2025-12-04T09:42:40.2179317Z Current runner version: '2.330.0' 2025-12-04T09:42:40.2185408Z Runner name: 'i-02fefd73b1a8f16c5' 2025-12-04T09:42:40.2186133Z Runner group name: 'default' 2025-12-04T09:42:40.2186999Z Machine name: 'ip-10-0-57-72' 2025-12-04T09:42:40.2189769Z ##[group]GITHUB_TOKEN Permissions 2025-12-04T09:42:40.2191973Z Contents: read 2025-12-04T09:42:40.2192599Z Metadata: read 2025-12-04T09:42:40.2193108Z ##[endgroup] 2025-12-04T09:42:40.2194979Z Secret source: Actions 2025-12-04T09:42:40.2195965Z Prepare workflow directory 2025-12-04T09:42:40.2673046Z Prepare all required actions 2025-12-04T09:42:40.2707650Z Getting action download info 2025-12-04T09:42:40.6578171Z Download action repository 'pytorch/test-infra@main' (SHA:39aa74d619174326f4e2fb0e216151c2f29d9ffd) 2025-12-04T09:42:43.1749246Z Download action repository 'pytorch/pytorch@main' (SHA:7716da9fb23f27a65b41f9f016a2afadf281c18f) 2025-12-04T09:43:00.0586466Z Download action repository 'actions/setup-python@a26af69be951a213d495a4c3e4e4022e16d87065' (SHA:a26af69be951a213d495a4c3e4e4022e16d87065) 2025-12-04T09:43:00.4132826Z Download action repository 'aws-actions/configure-aws-credentials@ececac1a45f3b08a01d2dd070d28d111c5fe6722' (SHA:ececac1a45f3b08a01d2dd070d28d111c5fe6722) 2025-12-04T09:43:00.6848306Z Download action repository 'aws-actions/amazon-ecr-login@062b18b96a7aff071d4dc91bc00c4c1a7945b076' (SHA:062b18b96a7aff071d4dc91bc00c4c1a7945b076) 2025-12-04T09:43:00.8687443Z Download action repository 'seemethere/download-artifact-s3@1da556a7aa0a088e3153970611f6c432d58e80e6' (SHA:1da556a7aa0a088e3153970611f6c432d58e80e6) 2025-12-04T09:43:01.1022498Z Download action repository 'seemethere/upload-artifact-s3@baba72d0712b404f646cebe0730933554ebce96a' (SHA:baba72d0712b404f646cebe0730933554ebce96a) 2025-12-04T09:43:01.3744551Z Getting action download info 2025-12-04T09:43:01.5754866Z Download action repository 'actions/checkout@v4' (SHA:34e114876b0b11c390a56381ad16ebd13914f8d5) 2025-12-04T09:43:01.8925191Z Getting action download info 2025-12-04T09:43:02.0364698Z Download action repository 'nick-fields/retry@v3.0.0' (SHA:7152eba30c6575329ac0576536151aca5a72780e) 2025-12-04T09:43:02.2605083Z Getting action download info 2025-12-04T09:43:02.3779278Z Download action repository 'nick-fields/retry@3e91a01664abd3c5cd539100d10d33b9c5b68482' (SHA:3e91a01664abd3c5cd539100d10d33b9c5b68482) 2025-12-04T09:43:02.5530527Z Getting action download info 2025-12-04T09:43:02.7301024Z Uses: pytorch/pytorch/.github/workflows/_linux-test.yml@refs/heads/main (ffd9b0fb4355e97af82fc42cf185c3ffa0fc0a32) 2025-12-04T09:43:02.7304757Z ##[group] Inputs 2025-12-04T09:43:02.7305092Z build-environment: linux-jammy-cuda12.8-py3.10-gcc11-debug 2025-12-04T09:43:02.7311294Z test-matrix: {"include": [{"config": "default", "shard": 1, "num_shards": 7, "runner": "linux.g6.4xlarge.experimental.nvidia.gpu", "owners": ["oncall:debug-build"], "mem_leak_check": "mem_leak_check"}, {"config": "default", "shard": 1, "num_shards": 7, "runner": "linux.g6.4xlarge.experimental.nvidia.gpu", "owners": ["oncall:debug-build"], "rerun_disabled_tests": "rerun_disabled_tests"}, {"config": "default", "shard": 2, "num_shards": 7, "runner": "linux.g6.4xlarge.experimental.nvidia.gpu", "owners": ["oncall:debug-build"], "mem_leak_check": "mem_leak_check"}, {"config": "default", "shard": 2, "num_shards": 7, "runner": "linux.g6.4xlarge.experimental.nvidia.gpu", "owners": ["oncall:debug-build"], "rerun_disabled_tests": "rerun_disabled_tests"}, {"config": "default", "shard": 3, "num_shards": 7, "runner": "linux.g6.4xlarge.experimental.nvidia.gpu", "owners": ["oncall:debug-build"], "mem_leak_check": "mem_leak_check"}, {"config": "default", "shard": 3, "num_shards": 7, "runner": "linux.g6.4xlarge.experimental.nvidia.gpu", "owners": ["oncall:debug-build"], "rerun_disabled_tests": "rerun_disabled_tests"}, {"config": "default", "shard": 4, "num_shards": 7, "runner": "linux.g6.4xlarge.experimental.nvidia.gpu", "owners": ["oncall:debug-build"], "mem_leak_check": "mem_leak_check"}, {"config": "default", "shard": 4, "num_shards": 7, "runner": "linux.g6.4xlarge.experimental.nvidia.gpu", "owners": ["oncall:debug-build"], "rerun_disabled_tests": "rerun_disabled_tests"}, {"config": "default", "shard": 5, "num_shards": 7, "runner": "linux.g6.4xlarge.experimental.nvidia.gpu", "owners": ["oncall:debug-build"], "mem_leak_check": "mem_leak_check"}, {"config": "default", "shard": 5, "num_shards": 7, "runner": "linux.g6.4xlarge.experimental.nvidia.gpu", "owners": ["oncall:debug-build"], "rerun_disabled_tests": "rerun_disabled_tests"}, {"config": "default", "shard": 6, "num_shards": 7, "runner": "linux.g6.4xlarge.experimental.nvidia.gpu", "owners": ["oncall:debug-build"], "mem_leak_check": "mem_leak_check"}, {"config": "default", "shard": 6, "num_shards": 7, "runner": "linux.g6.4xlarge.experimental.nvidia.gpu", "owners": ["oncall:debug-build"], "rerun_disabled_tests": "rerun_disabled_tests"}, {"config": "default", "shard": 7, "num_shards": 7, "runner": "linux.g6.4xlarge.experimental.nvidia.gpu", "owners": ["oncall:debug-build"], "mem_leak_check": "mem_leak_check"}, {"config": "default", "shard": 7, "num_shards": 7, "runner": "linux.g6.4xlarge.experimental.nvidia.gpu", "owners": ["oncall:debug-build"], "rerun_disabled_tests": "rerun_disabled_tests"}]} 2025-12-04T09:43:02.7317905Z docker-image: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/ci-image:pytorch-linux-jammy-cuda12.8-cudnn9-py3-gcc11-f0cd68561080d537ef3d3d6f81b25a6416ad600a 2025-12-04T09:43:02.7318499Z sync-tag: 2025-12-04T09:43:02.7319178Z timeout-minutes: 240 2025-12-04T09:43:02.7319382Z use-gha: 2025-12-04T09:43:02.7319543Z dashboard-tag: 2025-12-04T09:43:02.7319725Z s3-bucket: gha-artifacts 2025-12-04T09:43:02.7319929Z aws-role-to-assume: 2025-12-04T09:43:02.7320433Z disable-monitor: false 2025-12-04T09:43:02.7320669Z monitor-log-interval: 5 2025-12-04T09:43:02.7320895Z monitor-data-collect-interval: 1 2025-12-04T09:43:02.7321147Z ##[endgroup] 2025-12-04T09:43:02.7321706Z Complete job name: linux-jammy-cuda12.8-py3.10-gcc11-debug / test (default, 6, 7, linux.g6.4xlarge.experimental.nvidia.gpu, oncall:debug-build, mem_leak_check) 2025-12-04T09:43:02.7975507Z A job started hook has been configured by the self-hosted runner administrator 2025-12-04T09:43:02.8070576Z ##[group]Run '/home/ec2-user/runner-scripts/before_job.sh' 2025-12-04T09:43:02.8081068Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-12-04T09:43:02.8081646Z ##[endgroup] 2025-12-04T09:43:04.1289292Z Runner Type: linux.g6.4xlarge.experimental.nvidia.gpu 2025-12-04T09:43:04.1289787Z Instance Type: g6.4xlarge 2025-12-04T09:43:04.1290009Z AMI Name: unknown 2025-12-04T09:43:04.1327383Z AMI ID: ami-08982f1c5bf93d976 2025-12-04T09:43:08.9652818Z ##[group]Run pytorch/test-infra/.github/actions/setup-ssh@main 2025-12-04T09:43:08.9653177Z with: 2025-12-04T09:43:08.9653679Z github-secret: *** 2025-12-04T09:43:08.9654216Z instructions: All testing is done inside the container, to start an interactive session run: docker exec -it $(docker container ps --format '{{.ID}}') bash 2025-12-04T09:43:08.9654767Z activate-with-label: false 2025-12-04T09:43:08.9654984Z label: with-ssh 2025-12-04T09:43:08.9655173Z remove-existing-keys: true 2025-12-04T09:43:08.9655390Z fail-silently: true 2025-12-04T09:43:08.9655558Z env: 2025-12-04T09:43:08.9655715Z GIT_DEFAULT_BRANCH: main 2025-12-04T09:43:08.9655910Z ##[endgroup] 2025-12-04T09:43:09.0990530Z Please see https://github.com/pytorch/pytorch/wiki/Debugging-using-with-ssh-for-Github-Actions for more info. 2025-12-04T09:43:09.0991600Z Not on pull request and ciflow reference could not be extracted, skipping adding ssh keys 2025-12-04T09:43:09.1134771Z ##[group]Run pytorch/pytorch/.github/actions/checkout-pytorch@main 2025-12-04T09:43:09.1135115Z with: 2025-12-04T09:43:09.1135287Z no-sudo: true 2025-12-04T09:43:09.1135467Z submodules: recursive 2025-12-04T09:43:09.1135655Z fetch-depth: 0 2025-12-04T09:43:09.1135840Z env: 2025-12-04T09:43:09.1135997Z GIT_DEFAULT_BRANCH: main 2025-12-04T09:43:09.1136189Z ##[endgroup] 2025-12-04T09:43:09.1199165Z ##[group]Run echo "IN_CONTAINER_RUNNER=$(if [ -f /.inarc ] || [ -f /.incontainer ]; then echo true ; else echo false; fi)" >> "$GITHUB_OUTPUT" 2025-12-04T09:43:09.1200078Z echo "IN_CONTAINER_RUNNER=$(if [ -f /.inarc ] || [ -f /.incontainer ]; then echo true ; else echo false; fi)" >> "$GITHUB_OUTPUT" 2025-12-04T09:43:09.1212532Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-12-04T09:43:09.1212823Z env: 2025-12-04T09:43:09.1213021Z GIT_DEFAULT_BRANCH: main 2025-12-04T09:43:09.1213254Z ##[endgroup] 2025-12-04T09:43:09.1291854Z ##[group]Run # Use all available CPUs for fetching 2025-12-04T09:43:09.1292198Z # Use all available CPUs for fetching 2025-12-04T09:43:09.1292459Z cd "${GITHUB_WORKSPACE}" 2025-12-04T09:43:09.1292701Z git config --global fetch.parallel 0 2025-12-04T09:43:09.1292983Z git config --global submodule.fetchJobs 0 2025-12-04T09:43:09.1293228Z  2025-12-04T09:43:09.1293551Z # Clean workspace. The default checkout action should also do this, but 2025-12-04T09:43:09.1293929Z # do it here as well just in case 2025-12-04T09:43:09.1294166Z if [[ -d .git ]]; then 2025-12-04T09:43:09.1294386Z  if [ -z "${NO_SUDO}" ]; then 2025-12-04T09:43:09.1294612Z  sudo git clean -ffdx 2025-12-04T09:43:09.1294833Z  else 2025-12-04T09:43:09.1295017Z  git clean -ffdx 2025-12-04T09:43:09.1295199Z  fi 2025-12-04T09:43:09.1295359Z fi 2025-12-04T09:43:09.1302419Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-12-04T09:43:09.1302700Z env: 2025-12-04T09:43:09.1302862Z GIT_DEFAULT_BRANCH: main 2025-12-04T09:43:09.1303059Z NO_SUDO: true 2025-12-04T09:43:09.1303217Z ##[endgroup] 2025-12-04T09:43:09.1440408Z ##[group]Run actions/checkout@v4 2025-12-04T09:43:09.1440643Z with: 2025-12-04T09:43:09.1440836Z ref: ffd9b0fb4355e97af82fc42cf185c3ffa0fc0a32 2025-12-04T09:43:09.1441085Z fetch-depth: 0 2025-12-04T09:43:09.1441264Z submodules: recursive 2025-12-04T09:43:09.1441461Z show-progress: false 2025-12-04T09:43:09.1441648Z repository: pytorch/pytorch 2025-12-04T09:43:09.1441980Z token: *** 2025-12-04T09:43:09.1442152Z ssh-strict: true 2025-12-04T09:43:09.1442322Z ssh-user: git 2025-12-04T09:43:09.1442505Z persist-credentials: true 2025-12-04T09:43:09.1442701Z clean: true 2025-12-04T09:43:09.1442884Z sparse-checkout-cone-mode: true 2025-12-04T09:43:09.1443098Z fetch-tags: false 2025-12-04T09:43:09.1443266Z lfs: false 2025-12-04T09:43:09.1443433Z set-safe-directory: true 2025-12-04T09:43:09.1443621Z env: 2025-12-04T09:43:09.1443775Z GIT_DEFAULT_BRANCH: main 2025-12-04T09:43:09.1443959Z ##[endgroup] 2025-12-04T09:43:09.2561986Z Syncing repository: pytorch/pytorch 2025-12-04T09:43:09.2563443Z ##[group]Getting Git version info 2025-12-04T09:43:09.2563841Z Working directory is '/home/ec2-user/actions-runner/_work/pytorch/pytorch' 2025-12-04T09:43:09.2564358Z [command]/usr/bin/git version 2025-12-04T09:43:09.2781156Z git version 2.50.1 2025-12-04T09:43:09.2815627Z ##[endgroup] 2025-12-04T09:43:09.2825415Z Copying '/home/ec2-user/.gitconfig' to '/home/ec2-user/actions-runner/_work/_temp/739be14a-0795-43d0-9700-53e33e806dd0/.gitconfig' 2025-12-04T09:43:09.2844963Z Temporarily overriding HOME='/home/ec2-user/actions-runner/_work/_temp/739be14a-0795-43d0-9700-53e33e806dd0' before making global git config changes 2025-12-04T09:43:09.2845888Z Adding repository directory to the temporary git global config as a safe directory 2025-12-04T09:43:09.2849668Z [command]/usr/bin/git config --global --add safe.directory /home/ec2-user/actions-runner/_work/pytorch/pytorch 2025-12-04T09:43:09.2898051Z Deleting the contents of '/home/ec2-user/actions-runner/_work/pytorch/pytorch' 2025-12-04T09:43:09.2901225Z ##[group]Initializing the repository 2025-12-04T09:43:09.2904957Z [command]/usr/bin/git init /home/ec2-user/actions-runner/_work/pytorch/pytorch 2025-12-04T09:43:09.2974007Z hint: Using 'master' as the name for the initial branch. This default branch name 2025-12-04T09:43:09.2974806Z hint: is subject to change. To configure the initial branch name to use in all 2025-12-04T09:43:09.2975386Z hint: of your new repositories, which will suppress this warning, call: 2025-12-04T09:43:09.2975972Z hint: 2025-12-04T09:43:09.2976460Z hint: git config --global init.defaultBranch 2025-12-04T09:43:09.2976969Z hint: 2025-12-04T09:43:09.2977410Z hint: Names commonly chosen instead of 'master' are 'main', 'trunk' and 2025-12-04T09:43:09.2978636Z hint: 'development'. The just-created branch can be renamed via this command: 2025-12-04T09:43:09.2979214Z hint: 2025-12-04T09:43:09.2979499Z hint: git branch -m 2025-12-04T09:43:09.2979847Z hint: 2025-12-04T09:43:09.2980329Z hint: Disable this message with "git config set advice.defaultBranchName false" 2025-12-04T09:43:09.2987465Z Initialized empty Git repository in /home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/ 2025-12-04T09:43:09.2996386Z [command]/usr/bin/git remote add origin https://github.com/pytorch/pytorch 2025-12-04T09:43:09.3034856Z ##[endgroup] 2025-12-04T09:43:09.3035279Z ##[group]Disabling automatic garbage collection 2025-12-04T09:43:09.3038506Z [command]/usr/bin/git config --local gc.auto 0 2025-12-04T09:43:09.3066203Z ##[endgroup] 2025-12-04T09:43:09.3066558Z ##[group]Setting up auth 2025-12-04T09:43:09.3072041Z [command]/usr/bin/git config --local --name-only --get-regexp core\.sshCommand 2025-12-04T09:43:09.3102311Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'core\.sshCommand' && git config --local --unset-all 'core.sshCommand' || :" 2025-12-04T09:43:09.3483115Z [command]/usr/bin/git config --local --name-only --get-regexp http\.https\:\/\/github\.com\/\.extraheader 2025-12-04T09:43:09.3511484Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'http\.https\:\/\/github\.com\/\.extraheader' && git config --local --unset-all 'http.https://github.com/.extraheader' || :" 2025-12-04T09:43:09.3856284Z [command]/usr/bin/git config --local --name-only --get-regexp ^includeIf\.gitdir: 2025-12-04T09:43:09.3885468Z [command]/usr/bin/git submodule foreach --recursive git config --local --show-origin --name-only --get-regexp remote.origin.url 2025-12-04T09:43:09.4224448Z [command]/usr/bin/git config --local http.https://github.com/.extraheader AUTHORIZATION: basic *** 2025-12-04T09:43:09.4279862Z ##[endgroup] 2025-12-04T09:43:09.4280220Z ##[group]Fetching the repository 2025-12-04T09:43:09.4287394Z [command]/usr/bin/git -c protocol.version=2 fetch --prune --no-recurse-submodules origin +refs/heads/*:refs/remotes/origin/* +refs/tags/*:refs/tags/* 2025-12-04T09:43:53.1520150Z From https://github.com/pytorch/pytorch 2025-12-04T09:43:53.1520794Z * [new branch] 2.6.0.dev20241004+ -> origin/2.6.0.dev20241004+ 2025-12-04T09:43:53.1523043Z * [new branch] 2.9.1 -> origin/2.9.1 2025-12-04T09:43:53.1523699Z * [new branch] AaronWang04_addmmfusion_perftest -> origin/AaronWang04_addmmfusion_perftest 2025-12-04T09:43:53.1524328Z * [new branch] Flamefire-patch-1 -> origin/Flamefire-patch-1 2025-12-04T09:43:53.1524897Z * [new branch] HDCharles-2.6.0-release-notes -> origin/HDCharles-2.6.0-release-notes 2025-12-04T09:43:53.1525439Z * [new branch] HOPrintFunc -> origin/HOPrintFunc 2025-12-04T09:43:53.1528730Z * [new branch] IvanKobzarev/stack/1 -> origin/IvanKobzarev/stack/1 2025-12-04T09:43:53.1531241Z * [new branch] NicoshevSVE128 -> origin/NicoshevSVE128 2025-12-04T09:43:53.1532958Z * [new branch] PR-AOTInductorNoneBug -> origin/PR-AOTInductorNoneBug 2025-12-04T09:43:53.1534778Z * [new branch] PR-AOTInductorNoneBugFix -> origin/PR-AOTInductorNoneBugFix 2025-12-04T09:43:53.1536487Z * [new branch] PR-FixConfigsIssue -> origin/PR-FixConfigsIssue 2025-12-04T09:43:53.1538291Z * [new branch] PR-NoneBugFix-viable -> origin/PR-NoneBugFix-viable 2025-12-04T09:43:53.1540051Z * [new branch] PR-ResetToZero -> origin/PR-ResetToZero 2025-12-04T09:43:53.1541905Z * [new branch] Update-Flash-Packaging -> origin/Update-Flash-Packaging 2025-12-04T09:43:53.1543764Z * [new branch] VLA_exp -> origin/VLA_exp 2025-12-04T09:43:53.1545879Z * [new branch] activation_bench -> origin/activation_bench 2025-12-04T09:43:53.1547857Z * [new branch] addmm-heuristic -> origin/addmm-heuristic 2025-12-04T09:43:53.1550280Z * [new branch] adi/onednn_aarch64 -> origin/adi/onednn_aarch64 2025-12-04T09:43:53.1552037Z * [new branch] adi/test -> origin/adi/test 2025-12-04T09:43:53.1553798Z * [new branch] adi/test_bgemm -> origin/adi/test_bgemm 2025-12-04T09:43:53.1555581Z * [new branch] adi/test_m8g -> origin/adi/test_m8g 2025-12-04T09:43:53.1557342Z * [new branch] adi/test_onednn -> origin/adi/test_onednn 2025-12-04T09:43:53.1559120Z * [new branch] adi/test_onednn_v3.9 -> origin/adi/test_onednn_v3.9 2025-12-04T09:43:53.1560850Z * [new branch] adi/test_presve_change -> origin/adi/test_presve_change 2025-12-04T09:43:53.1562522Z * [new branch] adi/test_timm -> origin/adi/test_timm 2025-12-04T09:43:53.1564625Z * [new branch] adi/testpresve_change -> origin/adi/testpresve_change 2025-12-04T09:43:53.1567630Z * [new branch] aditew01/test/vec_bf16 -> origin/aditew01/test/vec_bf16 2025-12-04T09:43:53.1569487Z * [new branch] ah-globalfeedback-hook -> origin/ah-globalfeedback-hook 2025-12-04T09:43:53.1571871Z * [new branch] albanD-patch-1 -> origin/albanD-patch-1 2025-12-04T09:43:53.1573211Z * [new branch] also-surround-shimh -> origin/also-surround-shimh 2025-12-04T09:43:53.1575775Z * [new branch] angelayi/aot_compile -> origin/angelayi/aot_compile 2025-12-04T09:43:53.1577566Z * [new branch] angelayi/aoti_additional_files -> origin/angelayi/aoti_additional_files 2025-12-04T09:43:53.1579696Z * [new branch] angelayi/benchmark -> origin/angelayi/benchmark 2025-12-04T09:43:53.1581445Z * [new branch] angelayi/change_pytree_serialization -> origin/angelayi/change_pytree_serialization 2025-12-04T09:43:53.1583101Z * [new branch] angelayi/cpp_loader -> origin/angelayi/cpp_loader 2025-12-04T09:43:53.1585065Z * [new branch] angelayi/inductor_const -> origin/angelayi/inductor_const 2025-12-04T09:43:53.1586708Z * [new branch] angelayi/lstm -> origin/angelayi/lstm 2025-12-04T09:43:53.1588926Z * [new branch] angelayi/no_so_weight -> origin/angelayi/no_so_weight 2025-12-04T09:43:53.1591223Z * [new branch] angelayi/scan_layers -> origin/angelayi/scan_layers 2025-12-04T09:43:53.1592999Z * [new branch] angelayi/side_eff -> origin/angelayi/side_eff 2025-12-04T09:43:53.1594842Z * [new branch] angelayi/state_dict -> origin/angelayi/state_dict 2025-12-04T09:43:53.1596710Z * [new branch] angelayi/symint_input -> origin/angelayi/symint_input 2025-12-04T09:43:53.1598781Z * [new branch] angelayi/symm_mem -> origin/angelayi/symm_mem 2025-12-04T09:43:53.1600479Z * [new branch] angelayi/test_cpp -> origin/angelayi/test_cpp 2025-12-04T09:43:53.1602278Z * [new branch] angelayi/torch_size -> origin/angelayi/torch_size 2025-12-04T09:43:53.1604095Z * [new branch] annotate_assert -> origin/annotate_assert 2025-12-04T09:43:53.1605962Z * [new branch] annotate_fallback_kernel -> origin/annotate_fallback_kernel 2025-12-04T09:43:53.1607845Z * [new branch] annotation_deepcopy -> origin/annotation_deepcopy 2025-12-04T09:43:53.1609645Z * [new branch] annotation_dynamo -> origin/annotation_dynamo 2025-12-04T09:43:53.1611455Z * [new branch] aot_eager_stack_trace -> origin/aot_eager_stack_trace 2025-12-04T09:43:53.1613289Z * [new branch] aoti-cuda-alloc -> origin/aoti-cuda-alloc 2025-12-04T09:43:53.1615145Z * [new branch] aoti_const_device -> origin/aoti_const_device 2025-12-04T09:43:53.1616964Z * [new branch] aoti_fqn_name_interface -> origin/aoti_fqn_name_interface 2025-12-04T09:43:53.1618757Z * [new branch] aoti_package_weights_binary -> origin/aoti_package_weights_binary 2025-12-04T09:43:53.1620572Z * [new branch] aoti_target_windows -> origin/aoti_target_windows 2025-12-04T09:43:53.1623767Z * [new branch] arsh/feat/inductor_check_profiling -> origin/arsh/feat/inductor_check_profiling 2025-12-04T09:43:53.1625637Z * [new branch] async_tp -> origin/async_tp 2025-12-04T09:43:53.1627599Z * [new branch] atalman-inductor-perf-cu124 -> origin/atalman-inductor-perf-cu124 2025-12-04T09:43:53.1629770Z * [new branch] atalman-inductor-perf-cu124.1 -> origin/atalman-inductor-perf-cu124.1 2025-12-04T09:43:53.1631915Z * [new branch] atalman-patch-2 -> origin/atalman-patch-2 2025-12-04T09:43:53.1634132Z * [new branch] atalman-patch-3 -> origin/atalman-patch-3 2025-12-04T09:43:53.1635789Z * [new branch] atalman-patch-4 -> origin/atalman-patch-4 2025-12-04T09:43:53.1637716Z * [new branch] atalman-patch-5 -> origin/atalman-patch-5 2025-12-04T09:43:53.1639571Z * [new branch] atalman-patch-6 -> origin/atalman-patch-6 2025-12-04T09:43:53.1641824Z * [new branch] atalman-patch-7 -> origin/atalman-patch-7 2025-12-04T09:43:53.1643713Z * [new branch] atalman-patch-8 -> origin/atalman-patch-8 2025-12-04T09:43:53.1645559Z * [new branch] atalman_inductor_2.3.1 -> origin/atalman_inductor_2.3.1 2025-12-04T09:43:53.1647354Z * [new branch] atalman_inductor_2.4.0 -> origin/atalman_inductor_2.4.0 2025-12-04T09:43:53.1649229Z * [new branch] atalman_inductor_2.4.x -> origin/atalman_inductor_2.4.x 2025-12-04T09:43:53.1651180Z * [new branch] attention_benchmarking_clean -> origin/attention_benchmarking_clean 2025-12-04T09:43:53.1653529Z * [new branch] bahuang/dt_fix_scalar_add -> origin/bahuang/dt_fix_scalar_add 2025-12-04T09:43:53.1655191Z * [new branch] bahuang/fix_debug_mode -> origin/bahuang/fix_debug_mode 2025-12-04T09:43:53.1656977Z * [new branch] bahuang/fix_expand -> origin/bahuang/fix_expand 2025-12-04T09:43:53.1658707Z * [new branch] bahuang/test -> origin/bahuang/test 2025-12-04T09:43:53.1661266Z * [new branch] base/1.5 -> origin/base/1.5 2025-12-04T09:43:53.1663262Z * [new branch] batching_sdpa_efficient_attention -> origin/batching_sdpa_efficient_attention 2025-12-04T09:43:53.1665183Z * [new branch] bench_scaled_mm_ops -> origin/bench_scaled_mm_ops 2025-12-04T09:43:53.1667126Z * [new branch] benchmark-updates -> origin/benchmark-updates 2025-12-04T09:43:53.1668879Z * [new branch] benchmarking-script -> origin/benchmarking-script 2025-12-04T09:43:53.1671335Z * [new branch] bertmaher/pinbump26 -> origin/bertmaher/pinbump26 2025-12-04T09:43:53.1673789Z * [new branch] bertrand/cutlass -> origin/bertrand/cutlass 2025-12-04T09:43:53.1676215Z * [new branch] bf/bug-static-input -> origin/bf/bug-static-input 2025-12-04T09:43:53.1678496Z * [new branch] bf/cg-backend -> origin/bf/cg-backend 2025-12-04T09:43:53.1680519Z * [new branch] bf/cg-nccl-test -> origin/bf/cg-nccl-test 2025-12-04T09:43:53.1682250Z * [new branch] bf/cg-remove-check -> origin/bf/cg-remove-check 2025-12-04T09:43:53.1684075Z * [new branch] bf/clean-torchbench-hf -> origin/bf/clean-torchbench-hf 2025-12-04T09:43:53.1685691Z * [new branch] bf/combo-debug-log -> origin/bf/combo-debug-log 2025-12-04T09:43:53.1687468Z * [new branch] bf/cudagraph -> origin/bf/cudagraph 2025-12-04T09:43:53.1689730Z * [new branch] bf/cudagraph-disable-input-mutation -> origin/bf/cudagraph-disable-input-mutation 2025-12-04T09:43:53.1691707Z * [new branch] bf/cudagraph-enable-input-mutation-support-benchmark -> origin/bf/cudagraph-enable-input-mutation-support-benchmark 2025-12-04T09:43:53.1693311Z * [new branch] bf/cudagraph-partition -> origin/bf/cudagraph-partition 2025-12-04T09:43:53.1695107Z * [new branch] bf/donated-buffer-bench -> origin/bf/donated-buffer-bench 2025-12-04T09:43:53.1696939Z * [new branch] bf/dynamo-partition -> origin/bf/dynamo-partition 2025-12-04T09:43:53.1698808Z * [new branch] bf/lite -> origin/bf/lite 2025-12-04T09:43:53.1700584Z * [new branch] bf/pa-non-divisible -> origin/bf/pa-non-divisible 2025-12-04T09:43:53.1702531Z * [new branch] bf/partition-cache-free-symbols -> origin/bf/partition-cache-free-symbols 2025-12-04T09:43:53.1704503Z * [new branch] bf/partition-memory-plan -> origin/bf/partition-memory-plan 2025-12-04T09:43:53.1706345Z * [new branch] bf/partition-move-cpu -> origin/bf/partition-move-cpu 2025-12-04T09:43:53.1708208Z * [new branch] bf/partition-view-fallback -> origin/bf/partition-view-fallback 2025-12-04T09:43:53.1709926Z * [new branch] bf/remove-check-55b0c39d -> origin/bf/remove-check-55b0c39d 2025-12-04T09:43:53.1711704Z * [new branch] bf/timm-nov-26-2025 -> origin/bf/timm-nov-26-2025 2025-12-04T09:43:53.1713485Z * [new branch] bf/transformer-pin-4-57-3 -> origin/bf/transformer-pin-4-57-3 2025-12-04T09:43:53.1715298Z * [new branch] bisect_perf_hf_T5_3acc6eac492 -> origin/bisect_perf_hf_T5_3acc6eac492 2025-12-04T09:43:53.1717061Z * [new branch] bisect_perf_hf_T5_3fcf66f61fb -> origin/bisect_perf_hf_T5_3fcf66f61fb 2025-12-04T09:43:53.1718826Z * [new branch] bisect_perf_hf_T5_4009d154129 -> origin/bisect_perf_hf_T5_4009d154129 2025-12-04T09:43:53.1720651Z * [new branch] bisect_perf_hf_T5_40d0740e73d -> origin/bisect_perf_hf_T5_40d0740e73d 2025-12-04T09:43:53.1722388Z * [new branch] bisect_perf_hf_T5_5268754e -> origin/bisect_perf_hf_T5_5268754e 2025-12-04T09:43:53.1724148Z * [new branch] bisect_perf_hf_T5_7d89a8d385c -> origin/bisect_perf_hf_T5_7d89a8d385c 2025-12-04T09:43:53.1725856Z * [new branch] bisect_perf_hf_T5_b7a25c1ee7c -> origin/bisect_perf_hf_T5_b7a25c1ee7c 2025-12-04T09:43:53.1727605Z * [new branch] bisect_perf_hf_T5_c25b201583f -> origin/bisect_perf_hf_T5_c25b201583f 2025-12-04T09:43:53.1729415Z * [new branch] bisect_perf_hf_T5_c93e57efac0 -> origin/bisect_perf_hf_T5_c93e57efac0 2025-12-04T09:43:53.1731379Z * [new branch] bisect_perf_hf_T5_ca9813ea149 -> origin/bisect_perf_hf_T5_ca9813ea149 2025-12-04T09:43:53.1733066Z * [new branch] bisect_perf_hf_T5_d65f194a -> origin/bisect_perf_hf_T5_d65f194a 2025-12-04T09:43:53.1734724Z * [new branch] bisect_perf_hf_T5_da94ab0b -> origin/bisect_perf_hf_T5_da94ab0b 2025-12-04T09:43:53.1736586Z * [new branch] bisect_perf_hf_T5_da94ab0b_new -> origin/bisect_perf_hf_T5_da94ab0b_new 2025-12-04T09:43:53.1738384Z * [new branch] bisect_perf_hf_T5_db4e8a1d8a8 -> origin/bisect_perf_hf_T5_db4e8a1d8a8 2025-12-04T09:43:53.1740095Z * [new branch] bisect_perf_hf_T5_e0d97e936a2 -> origin/bisect_perf_hf_T5_e0d97e936a2 2025-12-04T09:43:53.1741829Z * [new branch] bisect_perf_hf_T5_f23621ec563 -> origin/bisect_perf_hf_T5_f23621ec563 2025-12-04T09:43:53.1744766Z * [new branch] brister/fx_device_type -> origin/brister/fx_device_type 2025-12-04T09:43:53.1746575Z * [new branch] brister/test_inductor_all_fx -> origin/brister/test_inductor_all_fx 2025-12-04T09:43:53.1748332Z * [new branch] brister/tiled_reduction_no_numel_check -> origin/brister/tiled_reduction_no_numel_check 2025-12-04T09:43:53.1750074Z * [new branch] bwd-backup -> origin/bwd-backup 2025-12-04T09:43:53.1751922Z * [new branch] c57382a49 -> origin/c57382a49 2025-12-04T09:43:53.1753679Z * [new branch] ca_0431d47eaa -> origin/ca_0431d47eaa 2025-12-04T09:43:53.1755424Z * [new branch] ca_fix_0431d47eaa -> origin/ca_fix_0431d47eaa 2025-12-04T09:43:53.1757934Z * [new branch] camyllh/test_setup_hooks_push -> origin/camyllh/test_setup_hooks_push 2025-12-04T09:43:53.1759813Z * [new branch] cccclai-patch-1 -> origin/cccclai-patch-1 2025-12-04T09:43:53.1761739Z * [new branch] cherry-pick-159969-by-pytorch_bot_bot_ -> origin/cherry-pick-159969-by-pytorch_bot_bot_ 2025-12-04T09:43:53.1763563Z * [new branch] cherry-pick-160586-by-pytorch_bot_bot_ -> origin/cherry-pick-160586-by-pytorch_bot_bot_ 2025-12-04T09:43:53.1765422Z * [new branch] cherry-pick-162208-by-pytorch_bot_bot_ -> origin/cherry-pick-162208-by-pytorch_bot_bot_ 2025-12-04T09:43:53.1767224Z * [new branch] cherry-pick-163169-by-pytorch_bot_bot_ -> origin/cherry-pick-163169-by-pytorch_bot_bot_ 2025-12-04T09:43:53.1769077Z * [new branch] cherry-pick-165086-by-pytorch_bot_bot_ -> origin/cherry-pick-165086-by-pytorch_bot_bot_ 2025-12-04T09:43:53.1771016Z * [new branch] cherry-pick-165514-by-pytorch_bot_bot_ -> origin/cherry-pick-165514-by-pytorch_bot_bot_ 2025-12-04T09:43:53.1773185Z * [new branch] cherry-pick-165601-by-pytorch_bot_bot_ -> origin/cherry-pick-165601-by-pytorch_bot_bot_ 2025-12-04T09:43:53.1774615Z * [new branch] cherry-pick-165667-by-pytorch_bot_bot_ -> origin/cherry-pick-165667-by-pytorch_bot_bot_ 2025-12-04T09:43:53.1776565Z * [new branch] cherry-pick-165815-by-pytorch_bot_bot_ -> origin/cherry-pick-165815-by-pytorch_bot_bot_ 2025-12-04T09:43:53.1778709Z * [new branch] cherry-pick-165922-by-pytorch_bot_bot_ -> origin/cherry-pick-165922-by-pytorch_bot_bot_ 2025-12-04T09:43:53.1780535Z * [new branch] cherry-pick-166148-by-pytorch_bot_bot_ -> origin/cherry-pick-166148-by-pytorch_bot_bot_ 2025-12-04T09:43:53.1782308Z * [new branch] cherry-pick-166181-by-pytorch_bot_bot_ -> origin/cherry-pick-166181-by-pytorch_bot_bot_ 2025-12-04T09:43:53.1784156Z * [new branch] cherry-pick-166404-by-pytorch_bot_bot_ -> origin/cherry-pick-166404-by-pytorch_bot_bot_ 2025-12-04T09:43:53.1786031Z * [new branch] cherry-pick-166427-by-pytorch_bot_bot_ -> origin/cherry-pick-166427-by-pytorch_bot_bot_ 2025-12-04T09:43:53.1787984Z * [new branch] cherry-pick-166480-by-pytorch_bot_bot_ -> origin/cherry-pick-166480-by-pytorch_bot_bot_ 2025-12-04T09:43:53.1789795Z * [new branch] cherry-pick-166570-by-pytorch_bot_bot_ -> origin/cherry-pick-166570-by-pytorch_bot_bot_ 2025-12-04T09:43:53.1791553Z * [new branch] cherry-pick-166993-by-pytorch_bot_bot_ -> origin/cherry-pick-166993-by-pytorch_bot_bot_ 2025-12-04T09:43:53.1793367Z * [new branch] cherry-pick-167111-by-pytorch_bot_bot_ -> origin/cherry-pick-167111-by-pytorch_bot_bot_ 2025-12-04T09:43:53.1795325Z * [new branch] cherry-pick-167478-by-pytorch_bot_bot_ -> origin/cherry-pick-167478-by-pytorch_bot_bot_ 2025-12-04T09:43:53.1796981Z * [new branch] cherry_pick_166036_166040 -> origin/cherry_pick_166036_166040 2025-12-04T09:43:53.1798793Z * [new branch] cherry_pick_166457 -> origin/cherry_pick_166457 2025-12-04T09:43:53.1800654Z * [new branch] cherrypick_166338 -> origin/cherrypick_166338 2025-12-04T09:43:53.1802512Z * [new branch] cherrypick_166458 -> origin/cherrypick_166458 2025-12-04T09:43:53.1804199Z * [new branch] cherrypick_166586 -> origin/cherrypick_166586 2025-12-04T09:43:53.1806044Z * [new branch] cherrypick_166956 -> origin/cherrypick_166956 2025-12-04T09:43:53.1807855Z * [new branch] ci_attn -> origin/ci_attn 2025-12-04T09:43:53.1809604Z * [new branch] codex-testing -> origin/codex-testing 2025-12-04T09:43:53.1812302Z * [new branch] codex/add-check_memory_overlap-helper-functions -> origin/codex/add-check_memory_overlap-helper-functions 2025-12-04T09:43:53.1813974Z * [new branch] codex/fix-issue-121219-in-pytorch -> origin/codex/fix-issue-121219-in-pytorch 2025-12-04T09:43:53.1816215Z * [new branch] codex/investigate-segfaults-in-get_tensor_storage_id -> origin/codex/investigate-segfaults-in-get_tensor_storage_id 2025-12-04T09:43:53.1818290Z * [new branch] codex/refactor-lintrunner-config-to-use-uv-run -> origin/codex/refactor-lintrunner-config-to-use-uv-run 2025-12-04T09:43:53.1819990Z * [new branch] compatiblpy39util -> origin/compatiblpy39util 2025-12-04T09:43:53.1821837Z * [new branch] cond_hop_device -> origin/cond_hop_device 2025-12-04T09:43:53.1823738Z * [new branch] context_test -> origin/context_test 2025-12-04T09:43:53.1826411Z * [new branch] copilot/code-style-cleanup-python-pip -> origin/copilot/code-style-cleanup-python-pip 2025-12-04T09:43:53.1828672Z * [new branch] cpio/fix_new_ami_tests -> origin/cpio/fix_new_ami_tests 2025-12-04T09:43:53.1830548Z * [new branch] cpp-docs-dependency-upgrade -> origin/cpp-docs-dependency-upgrade 2025-12-04T09:43:53.1833034Z * [new branch] crpa/typo-in-inductor_comm_lowering -> origin/crpa/typo-in-inductor_comm_lowering 2025-12-04T09:43:53.1835312Z * [new branch] csl/always_produce_xml -> origin/csl/always_produce_xml 2025-12-04T09:43:53.1837028Z * [new branch] csl/build_test_more_procs -> origin/csl/build_test_more_procs 2025-12-04T09:43:53.1838745Z * [new branch] csl/build_test_more_procs2 -> origin/csl/build_test_more_procs2 2025-12-04T09:43:53.1840459Z * [new branch] csl/clean_up -> origin/csl/clean_up 2025-12-04T09:43:53.1842259Z * [new branch] csl/fix_retry_segfault_exit -> origin/csl/fix_retry_segfault_exit 2025-12-04T09:43:53.1843961Z * [new branch] csl/katex -> origin/csl/katex 2025-12-04T09:43:53.1846007Z * [new branch] csl/larger_runner -> origin/csl/larger_runner 2025-12-04T09:43:53.1848159Z * [new branch] csl/lint_testing -> origin/csl/lint_testing 2025-12-04T09:43:53.1850309Z * [new branch] csl/lint_thing -> origin/csl/lint_thing 2025-12-04T09:43:53.1852258Z * [new branch] csl/lintrunner_stuff -> origin/csl/lintrunner_stuff 2025-12-04T09:43:53.1854054Z * [new branch] csl/manually_gen_json -> origin/csl/manually_gen_json 2025-12-04T09:43:53.1855802Z * [new branch] csl/mps_sharding -> origin/csl/mps_sharding 2025-12-04T09:43:53.1857650Z * [new branch] csl/multistage_docker -> origin/csl/multistage_docker 2025-12-04T09:43:53.1859481Z * [new branch] csl/print_timing -> origin/csl/print_timing 2025-12-04T09:43:53.1861183Z * [new branch] csl/remove_experiment -> origin/csl/remove_experiment 2025-12-04T09:43:53.1863027Z * [new branch] csl/remove_maybe_unused_var -> origin/csl/remove_maybe_unused_var 2025-12-04T09:43:53.1865108Z * [new branch] csl/remove_repo_specific_autolabel -> origin/csl/remove_repo_specific_autolabel 2025-12-04T09:43:53.1866964Z * [new branch] csl/remove_run_parallel -> origin/csl/remove_run_parallel 2025-12-04T09:43:53.1868627Z * [new branch] csl/remove_unused_vars -> origin/csl/remove_unused_vars 2025-12-04T09:43:53.1870440Z * [new branch] csl/revert_open -> origin/csl/revert_open 2025-12-04T09:43:53.1872431Z * [new branch] csl/skip_build -> origin/csl/skip_build 2025-12-04T09:43:53.1874203Z * [new branch] csl/smaller_avx_amx_runenrs -> origin/csl/smaller_avx_amx_runenrs 2025-12-04T09:43:53.1875914Z * [new branch] csl/td_job_level -> origin/csl/td_job_level 2025-12-04T09:43:53.1878113Z * [new branch] csl/test_cuda_build_large_runner -> origin/csl/test_cuda_build_large_runner 2025-12-04T09:43:53.1880110Z * [new branch] csl/test_owners_autograd_dispatch_nn -> origin/csl/test_owners_autograd_dispatch_nn 2025-12-04T09:43:53.1882348Z * [new branch] csl/test_owners_higher_confidence -> origin/csl/test_owners_higher_confidence 2025-12-04T09:43:53.1884189Z * [new branch] csl/upload_json_running -> origin/csl/upload_json_running 2025-12-04T09:43:53.1885982Z * [new branch] csl/win_sccache -> origin/csl/win_sccache 2025-12-04T09:43:53.1887763Z * [new branch] csl/xml_stuff -> origin/csl/xml_stuff 2025-12-04T09:43:53.1889592Z * [new branch] cublasrelax2 -> origin/cublasrelax2 2025-12-04T09:43:53.1891377Z * [new branch] cuda_mempool -> origin/cuda_mempool 2025-12-04T09:43:53.1893365Z * [new branch] custom_lowering_dict -> origin/custom_lowering_dict 2025-12-04T09:43:53.1895656Z * [new branch] d4l3k/debug_plane_frtrace -> origin/d4l3k/debug_plane_frtrace 2025-12-04T09:43:53.1897964Z * [new branch] daxia6/2.8o3 -> origin/daxia6/2.8o3 2025-12-04T09:43:53.1899744Z * [new branch] debug-guard -> origin/debug-guard 2025-12-04T09:43:53.1901666Z * [new branch] delete-quant-docs -> origin/delete-quant-docs 2025-12-04T09:43:53.1907066Z * [new branch] dependabot/pip/dot-ci/docker/ci_commit_pins/main/transformers-4.57.0 -> origin/dependabot/pip/dot-ci/docker/ci_commit_pins/main/transformers-4.57.0 2025-12-04T09:43:53.1908879Z * [new branch] dependabot/pip/dot-ci/docker/ci_commit_pins/main/transformers-4.57.1 -> origin/dependabot/pip/dot-ci/docker/ci_commit_pins/main/transformers-4.57.1 2025-12-04T09:43:53.1911142Z * [new branch] desertfire/test_cpp_wrapper -> origin/desertfire/test_cpp_wrapper 2025-12-04T09:43:53.1912927Z * [new branch] desertfire/triton-cpu-for-aarch64 -> origin/desertfire/triton-cpu-for-aarch64 2025-12-04T09:43:53.1915674Z * [new branch] dev/dhruva/flex_attn_opt -> origin/dev/dhruva/flex_attn_opt 2025-12-04T09:43:53.1918342Z * [new branch] dev/joona/MPSNDArrayAdd -> origin/dev/joona/MPSNDArrayAdd 2025-12-04T09:43:53.1920380Z * [new branch] dev/joona/Unranked -> origin/dev/joona/Unranked 2025-12-04T09:43:53.1922222Z * [new branch] dev/joona/cat -> origin/dev/joona/cat 2025-12-04T09:43:53.1923962Z * [new branch] dev/joona/embeddingbag -> origin/dev/joona/embeddingbag 2025-12-04T09:43:53.1925675Z * [new branch] dev/joona/fix_sdpa_memtest -> origin/dev/joona/fix_sdpa_memtest 2025-12-04T09:43:53.1927666Z * [new branch] dev/joona/getTensorsString -> origin/dev/joona/getTensorsString 2025-12-04T09:43:53.1929617Z * [new branch] dev/joona/mps_linear_macos14 -> origin/dev/joona/mps_linear_macos14 2025-12-04T09:43:53.1931895Z * [new branch] dev/joona/scalar_clamp -> origin/dev/joona/scalar_clamp 2025-12-04T09:43:53.1934125Z * [new branch] dev/joona/sdpa -> origin/dev/joona/sdpa 2025-12-04T09:43:53.1936591Z * [new branch] dev/joona/sdpa_api -> origin/dev/joona/sdpa_api 2025-12-04T09:43:53.1938541Z * [new branch] dev/joona/type_inf -> origin/dev/joona/type_inf 2025-12-04T09:43:53.1940578Z * [new branch] dev/joona/ulpAssertClose -> origin/dev/joona/ulpAssertClose 2025-12-04T09:43:53.1942573Z * [new branch] dev/joona/upsize3d -> origin/dev/joona/upsize3d 2025-12-04T09:43:53.1944424Z * [new branch] disp_counter -> origin/disp_counter 2025-12-04T09:43:53.1946275Z * [new branch] divyanshk-patch-1 -> origin/divyanshk-patch-1 2025-12-04T09:43:53.1948135Z * [new branch] docs -> origin/docs 2025-12-04T09:43:53.1950096Z * [new branch] documentation -> origin/documentation 2025-12-04T09:43:53.1951871Z * [new branch] eager_model_benchmarks -> origin/eager_model_benchmarks 2025-12-04T09:43:53.1954325Z * [new branch] embg/test_inductor_ci_control -> origin/embg/test_inductor_ci_control 2025-12-04T09:43:53.1955984Z * [new branch] embg/triton_l2_prefetch_128B -> origin/embg/triton_l2_prefetch_128B 2025-12-04T09:43:53.1957641Z * [new branch] embg/triton_l2_prefetch_256B -> origin/embg/triton_l2_prefetch_256B 2025-12-04T09:43:53.1959447Z * [new branch] eqy-patch-1 -> origin/eqy-patch-1 2025-12-04T09:43:53.1961198Z * [new branch] eqy-patch-2 -> origin/eqy-patch-2 2025-12-04T09:43:53.1963016Z * [new branch] eqy-patch-3 -> origin/eqy-patch-3 2025-12-04T09:43:53.1964821Z * [new branch] eqy-patch-4 -> origin/eqy-patch-4 2025-12-04T09:43:53.1966619Z * [new branch] eqy-patch-5 -> origin/eqy-patch-5 2025-12-04T09:43:53.1968341Z * [new branch] eqy-patch-6 -> origin/eqy-patch-6 2025-12-04T09:43:53.1970808Z * [new branch] exclamaforte/amd-ma -> origin/exclamaforte/amd-ma 2025-12-04T09:43:53.1972695Z * [new branch] exclamaforte/combo-kernels-perf-run -> origin/exclamaforte/combo-kernels-perf-run 2025-12-04T09:43:53.1974547Z * [new branch] exclamaforte/do_bench_refactor -> origin/exclamaforte/do_bench_refactor 2025-12-04T09:43:53.1976206Z * [new branch] exclamaforte/enable-mem-dep-fusion -> origin/exclamaforte/enable-mem-dep-fusion 2025-12-04T09:43:53.1978239Z * [new branch] exclamaforte/fix-exhaustive-autotuning -> origin/exclamaforte/fix-exhaustive-autotuning 2025-12-04T09:43:53.1980376Z * [new branch] exclamaforte/fix-trace-parsing-fx-svg -> origin/exclamaforte/fix-trace-parsing-fx-svg 2025-12-04T09:43:53.1982567Z * [new branch] exclamaforte/force-pointwise-cat-perf-run -> origin/exclamaforte/force-pointwise-cat-perf-run 2025-12-04T09:43:53.1984392Z * [new branch] exclamaforte/fusion-data -> origin/exclamaforte/fusion-data 2025-12-04T09:43:53.1986460Z * [new branch] exclamaforte/gemm-benchmark-run -> origin/exclamaforte/gemm-benchmark-run 2025-12-04T09:43:53.1988073Z * [new branch] exclamaforte/gemm-export-model -> origin/exclamaforte/gemm-export-model 2025-12-04T09:43:53.1989855Z * [new branch] exclamaforte/gemm-model -> origin/exclamaforte/gemm-model 2025-12-04T09:43:53.1991893Z * [new branch] exclamaforte/gemm-model-all-data-collection -> origin/exclamaforte/gemm-model-all-data-collection 2025-12-04T09:43:53.1993520Z * [new branch] exclamaforte/gemm-to-amd -> origin/exclamaforte/gemm-to-amd 2025-12-04T09:43:53.1995332Z * [new branch] exclamaforte/just-gemm-model -> origin/exclamaforte/just-gemm-model 2025-12-04T09:43:53.1997232Z * [new branch] exclamaforte/just-gemm-model-no-refactor -> origin/exclamaforte/just-gemm-model-no-refactor 2025-12-04T09:43:53.1999052Z * [new branch] exclamaforte/profile-diff-algo -> origin/exclamaforte/profile-diff-algo 2025-12-04T09:43:53.2000833Z * [new branch] exclamaforte/profiler-visualization -> origin/exclamaforte/profiler-visualization 2025-12-04T09:43:53.2002704Z * [new branch] exclamaforte/test_cpp_wrapper_mode -> origin/exclamaforte/test_cpp_wrapper_mode 2025-12-04T09:43:53.2004563Z * [new branch] exclamaforte/update-autotune-configs -> origin/exclamaforte/update-autotune-configs 2025-12-04T09:43:53.2006466Z * [new branch] exclamaforte/update-autotune-configs-2 -> origin/exclamaforte/update-autotune-configs-2 2025-12-04T09:43:53.2008153Z * [new branch] exec -> origin/exec 2025-12-04T09:43:53.2010089Z * [new branch] experimental-mosaic -> origin/experimental-mosaic 2025-12-04T09:43:53.2011975Z * [new branch] export-D61047529 -> origin/export-D61047529 2025-12-04T09:43:53.2013741Z * [new branch] export-D71412006 -> origin/export-D71412006 2025-12-04T09:43:53.2015602Z * [new branch] export-D73042989 -> origin/export-D73042989 2025-12-04T09:43:53.2017376Z * [new branch] export-D78957093 -> origin/export-D78957093 2025-12-04T09:43:53.2019168Z * [new branch] export-D78996107 -> origin/export-D78996107 2025-12-04T09:43:53.2021453Z * [new branch] export-D80823877 -> origin/export-D80823877 2025-12-04T09:43:53.2023309Z * [new branch] export-D80958642 -> origin/export-D80958642 2025-12-04T09:43:53.2025267Z * [new branch] export-D81054193 -> origin/export-D81054193 2025-12-04T09:43:53.2026951Z * [new branch] export-D81204584 -> origin/export-D81204584 2025-12-04T09:43:53.2028728Z * [new branch] export-D81429090 -> origin/export-D81429090 2025-12-04T09:43:53.2030680Z * [new branch] export-D82250826 -> origin/export-D82250826 2025-12-04T09:43:53.2032484Z * [new branch] export-D82253817 -> origin/export-D82253817 2025-12-04T09:43:53.2034285Z * [new branch] export-D83541846 -> origin/export-D83541846 2025-12-04T09:43:53.2036055Z * [new branch] export-D83627170 -> origin/export-D83627170 2025-12-04T09:43:53.2037891Z * [new branch] export-D83766701 -> origin/export-D83766701 2025-12-04T09:43:53.2039655Z * [new branch] export-D83768878 -> origin/export-D83768878 2025-12-04T09:43:53.2041391Z * [new branch] export-D83769447 -> origin/export-D83769447 2025-12-04T09:43:53.2043160Z * [new branch] export-D84089824 -> origin/export-D84089824 2025-12-04T09:43:53.2044947Z * [new branch] export-D84213020 -> origin/export-D84213020 2025-12-04T09:43:53.2047147Z * [new branch] export-D84373821 -> origin/export-D84373821 2025-12-04T09:43:53.2049139Z * [new branch] export-D84612194 -> origin/export-D84612194 2025-12-04T09:43:53.2050795Z * [new branch] export-D84890985 -> origin/export-D84890985 2025-12-04T09:43:53.2052524Z * [new branch] export-D85122326 -> origin/export-D85122326 2025-12-04T09:43:53.2054448Z * [new branch] export-D86256198 -> origin/export-D86256198 2025-12-04T09:43:53.2056207Z * [new branch] export-D86460608 -> origin/export-D86460608 2025-12-04T09:43:53.2058140Z * [new branch] export-D86474796 -> origin/export-D86474796 2025-12-04T09:43:53.2060017Z * [new branch] export-D86712396 -> origin/export-D86712396 2025-12-04T09:43:53.2061747Z * [new branch] export-D87022129 -> origin/export-D87022129 2025-12-04T09:43:53.2064215Z * [new branch] export-D87838959 -> origin/export-D87838959 2025-12-04T09:43:53.2066065Z * [new branch] export-D88319437 -> origin/export-D88319437 2025-12-04T09:43:53.2068041Z * [new branch] exported-model-train-idempotent -> origin/exported-model-train-idempotent 2025-12-04T09:43:53.2069822Z * [new branch] ezyang-titan-october -> origin/ezyang-titan-october 2025-12-04T09:43:53.2071566Z * [new branch] ezyang-titan-october2 -> origin/ezyang-titan-october2 2025-12-04T09:43:53.2073284Z * [new branch] ezyang-war -> origin/ezyang-war 2025-12-04T09:43:53.2075686Z * [new branch] ezyang/wip-aot-descriptors -> origin/ezyang/wip-aot-descriptors 2025-12-04T09:43:53.2077392Z * [new branch] fa_u8_brgemm -> origin/fa_u8_brgemm 2025-12-04T09:43:53.2080217Z * [new branch] fadeputr/sequence_fbgemm -> origin/fadeputr/sequence_fbgemm 2025-12-04T09:43:53.2081958Z * [new branch] fastmath_baseline -> origin/fastmath_baseline 2025-12-04T09:43:53.2084437Z * [new branch] fbcode/warm -> origin/fbcode/warm 2025-12-04T09:43:53.2086282Z * [new branch] fca -> origin/fca 2025-12-04T09:43:53.2088068Z * [new branch] fca2_ca5984c -> origin/fca2_ca5984c 2025-12-04T09:43:53.2089958Z * [new branch] fca5 -> origin/fca5 2025-12-04T09:43:53.2092351Z * [new branch] feature/justknobs-cpp -> origin/feature/justknobs-cpp 2025-12-04T09:43:53.2094305Z * [new branch] feature/numa-forkserver -> origin/feature/numa-forkserver 2025-12-04T09:43:53.2096325Z * [new branch] ffast_math_baseline -> origin/ffast_math_baseline 2025-12-04T09:43:53.2098056Z * [new branch] ffast_math_target -> origin/ffast_math_target 2025-12-04T09:43:53.2100486Z * [new branch] findhao/base_commit -> origin/findhao/base_commit 2025-12-04T09:43:53.2102279Z * [new branch] findhao/base_commit1 -> origin/findhao/base_commit1 2025-12-04T09:43:53.2104292Z * [new branch] findhao/multistream2 -> origin/findhao/multistream2 2025-12-04T09:43:53.2107110Z * [new branch] findhao/multistream5 -> origin/findhao/multistream5 2025-12-04T09:43:53.2109270Z * [new branch] findhao/multistream6 -> origin/findhao/multistream6 2025-12-04T09:43:53.2111075Z * [new branch] findhao/operatorbench3 -> origin/findhao/operatorbench3 2025-12-04T09:43:53.2113202Z * [new branch] findhao/operatorbench5 -> origin/findhao/operatorbench5 2025-12-04T09:43:53.2115133Z * [new branch] findhao/tritonparse -> origin/findhao/tritonparse 2025-12-04T09:43:53.2117150Z * [new branch] fix-ck-gemm-template-format -> origin/fix-ck-gemm-template-format 2025-12-04T09:43:53.2118983Z * [new branch] fix-config-ignore -> origin/fix-config-ignore 2025-12-04T09:43:53.2120685Z * [new branch] fix-dict-guard -> origin/fix-dict-guard 2025-12-04T09:43:53.2122554Z * [new branch] fix_addmm_issue -> origin/fix_addmm_issue 2025-12-04T09:43:53.2124337Z * [new branch] fix_amd_missing_cluster_dims -> origin/fix_amd_missing_cluster_dims 2025-12-04T09:43:53.2126089Z * [new branch] fix_bench_bwd_pass -> origin/fix_bench_bwd_pass 2025-12-04T09:43:53.2127869Z * [new branch] fix_mem_profiler_config -> origin/fix_mem_profiler_config 2025-12-04T09:43:53.2129686Z * [new branch] fix_nvrtc_discovery -> origin/fix_nvrtc_discovery 2025-12-04T09:43:53.2131369Z * [new branch] fix_op_runner -> origin/fix_op_runner 2025-12-04T09:43:53.2133168Z * [new branch] fix_ubn_159469 -> origin/fix_ubn_159469 2025-12-04T09:43:53.2135018Z * [new branch] fixes-triage -> origin/fixes-triage 2025-12-04T09:43:53.2136799Z * [new branch] fixflashinfer -> origin/fixflashinfer 2025-12-04T09:43:53.2138541Z * [new branch] flash_decoding_cpu -> origin/flash_decoding_cpu 2025-12-04T09:43:53.2140334Z * [new branch] flex-flash -> origin/flex-flash 2025-12-04T09:43:53.2142181Z * [new branch] flex_attention_functorch_grad -> origin/flex_attention_functorch_grad 2025-12-04T09:43:53.2144287Z * [new branch] flex_flash -> origin/flex_flash 2025-12-04T09:43:53.2146788Z * [new branch] fmassa/fix_memeff_sharding_rule -> origin/fmassa/fix_memeff_sharding_rule 2025-12-04T09:43:53.2148564Z * [new branch] fmassa/tests_comm_compute_scheduler -> origin/fmassa/tests_comm_compute_scheduler 2025-12-04T09:43:53.2150212Z * [new branch] forkserver_fix -> origin/forkserver_fix 2025-12-04T09:43:53.2152010Z * [new branch] fsdp2_trace_rules -> origin/fsdp2_trace_rules 2025-12-04T09:43:53.2153918Z * [new branch] fx_cpp -> origin/fx_cpp 2025-12-04T09:43:53.2156296Z * [new branch] fy/fix-win -> origin/fy/fix-win 2025-12-04T09:43:53.2158205Z * [new branch] galv-patch-1 -> origin/galv-patch-1 2025-12-04T09:43:53.2160886Z * [new branch] galv/cudagraphs-conditional-nodes-4 -> origin/galv/cudagraphs-conditional-nodes-4 2025-12-04T09:43:53.2163277Z * [new branch] georgehong/cmakelists-patch -> origin/georgehong/cmakelists-patch 2025-12-04T09:43:53.2166707Z * [new branch] gh/AlnisM/1/base -> origin/gh/AlnisM/1/base 2025-12-04T09:43:53.2168463Z * [new branch] gh/AlnisM/1/head -> origin/gh/AlnisM/1/head 2025-12-04T09:43:53.2171410Z * [new branch] gh/EikanWang/67/base -> origin/gh/EikanWang/67/base 2025-12-04T09:43:53.2173217Z * [new branch] gh/EikanWang/67/head -> origin/gh/EikanWang/67/head 2025-12-04T09:43:53.2176362Z * [new branch] gh/Gasoonjia/1/base -> origin/gh/Gasoonjia/1/base 2025-12-04T09:43:53.2178371Z * [new branch] gh/Gasoonjia/1/head -> origin/gh/Gasoonjia/1/head 2025-12-04T09:43:53.2183064Z * [new branch] gh/H-Huang/131/base -> origin/gh/H-Huang/131/base 2025-12-04T09:43:53.2184971Z * [new branch] gh/H-Huang/131/head -> origin/gh/H-Huang/131/head 2025-12-04T09:43:53.2186739Z * [new branch] gh/H-Huang/131/orig -> origin/gh/H-Huang/131/orig 2025-12-04T09:43:53.2189151Z * [new branch] gh/H-Huang/132/base -> origin/gh/H-Huang/132/base 2025-12-04T09:43:53.2190850Z * [new branch] gh/H-Huang/132/head -> origin/gh/H-Huang/132/head 2025-12-04T09:43:53.2192585Z * [new branch] gh/H-Huang/132/orig -> origin/gh/H-Huang/132/orig 2025-12-04T09:43:53.2195255Z * [new branch] gh/H-Huang/180/base -> origin/gh/H-Huang/180/base 2025-12-04T09:43:53.2196860Z * [new branch] gh/H-Huang/180/head -> origin/gh/H-Huang/180/head 2025-12-04T09:43:53.2198621Z * [new branch] gh/H-Huang/180/orig -> origin/gh/H-Huang/180/orig 2025-12-04T09:43:53.2200886Z * [new branch] gh/H-Huang/182/base -> origin/gh/H-Huang/182/base 2025-12-04T09:43:53.2202699Z * [new branch] gh/H-Huang/182/head -> origin/gh/H-Huang/182/head 2025-12-04T09:43:53.2204419Z * [new branch] gh/H-Huang/182/orig -> origin/gh/H-Huang/182/orig 2025-12-04T09:43:53.2207048Z * [new branch] gh/H-Huang/226/base -> origin/gh/H-Huang/226/base 2025-12-04T09:43:53.2208775Z * [new branch] gh/H-Huang/226/head -> origin/gh/H-Huang/226/head 2025-12-04T09:43:53.2210507Z * [new branch] gh/H-Huang/226/orig -> origin/gh/H-Huang/226/orig 2025-12-04T09:43:53.2212913Z * [new branch] gh/H-Huang/228/base -> origin/gh/H-Huang/228/base 2025-12-04T09:43:53.2214617Z * [new branch] gh/H-Huang/228/head -> origin/gh/H-Huang/228/head 2025-12-04T09:43:53.2216367Z * [new branch] gh/H-Huang/228/orig -> origin/gh/H-Huang/228/orig 2025-12-04T09:43:53.2219225Z * [new branch] gh/IvanKobzarev/150/base -> origin/gh/IvanKobzarev/150/base 2025-12-04T09:43:53.2220980Z * [new branch] gh/IvanKobzarev/150/head -> origin/gh/IvanKobzarev/150/head 2025-12-04T09:43:53.2222626Z * [new branch] gh/IvanKobzarev/150/orig -> origin/gh/IvanKobzarev/150/orig 2025-12-04T09:43:53.2225170Z * [new branch] gh/IvanKobzarev/157/base -> origin/gh/IvanKobzarev/157/base 2025-12-04T09:43:53.2226953Z * [new branch] gh/IvanKobzarev/157/head -> origin/gh/IvanKobzarev/157/head 2025-12-04T09:43:53.2228715Z * [new branch] gh/IvanKobzarev/157/orig -> origin/gh/IvanKobzarev/157/orig 2025-12-04T09:43:53.2231184Z * [new branch] gh/IvanKobzarev/159/base -> origin/gh/IvanKobzarev/159/base 2025-12-04T09:43:53.2232898Z * [new branch] gh/IvanKobzarev/159/head -> origin/gh/IvanKobzarev/159/head 2025-12-04T09:43:53.2234706Z * [new branch] gh/IvanKobzarev/159/orig -> origin/gh/IvanKobzarev/159/orig 2025-12-04T09:43:53.2237054Z * [new branch] gh/IvanKobzarev/162/base -> origin/gh/IvanKobzarev/162/base 2025-12-04T09:43:53.2239043Z * [new branch] gh/IvanKobzarev/162/head -> origin/gh/IvanKobzarev/162/head 2025-12-04T09:43:53.2240746Z * [new branch] gh/IvanKobzarev/162/orig -> origin/gh/IvanKobzarev/162/orig 2025-12-04T09:43:53.2243148Z * [new branch] gh/IvanKobzarev/163/base -> origin/gh/IvanKobzarev/163/base 2025-12-04T09:43:53.2244847Z * [new branch] gh/IvanKobzarev/163/head -> origin/gh/IvanKobzarev/163/head 2025-12-04T09:43:53.2246583Z * [new branch] gh/IvanKobzarev/163/orig -> origin/gh/IvanKobzarev/163/orig 2025-12-04T09:43:53.2248987Z * [new branch] gh/IvanKobzarev/166/base -> origin/gh/IvanKobzarev/166/base 2025-12-04T09:43:53.2250761Z * [new branch] gh/IvanKobzarev/166/head -> origin/gh/IvanKobzarev/166/head 2025-12-04T09:43:53.2252517Z * [new branch] gh/IvanKobzarev/166/orig -> origin/gh/IvanKobzarev/166/orig 2025-12-04T09:43:53.2254980Z * [new branch] gh/IvanKobzarev/167/base -> origin/gh/IvanKobzarev/167/base 2025-12-04T09:43:53.2256686Z * [new branch] gh/IvanKobzarev/167/head -> origin/gh/IvanKobzarev/167/head 2025-12-04T09:43:53.2258465Z * [new branch] gh/IvanKobzarev/167/orig -> origin/gh/IvanKobzarev/167/orig 2025-12-04T09:43:53.2260780Z * [new branch] gh/IvanKobzarev/168/base -> origin/gh/IvanKobzarev/168/base 2025-12-04T09:43:53.2262664Z * [new branch] gh/IvanKobzarev/168/head -> origin/gh/IvanKobzarev/168/head 2025-12-04T09:43:53.2264447Z * [new branch] gh/IvanKobzarev/168/orig -> origin/gh/IvanKobzarev/168/orig 2025-12-04T09:43:53.2266793Z * [new branch] gh/IvanKobzarev/169/base -> origin/gh/IvanKobzarev/169/base 2025-12-04T09:43:53.2268506Z * [new branch] gh/IvanKobzarev/169/head -> origin/gh/IvanKobzarev/169/head 2025-12-04T09:43:53.2270232Z * [new branch] gh/IvanKobzarev/169/orig -> origin/gh/IvanKobzarev/169/orig 2025-12-04T09:43:53.2272537Z * [new branch] gh/IvanKobzarev/170/base -> origin/gh/IvanKobzarev/170/base 2025-12-04T09:43:53.2274244Z * [new branch] gh/IvanKobzarev/170/head -> origin/gh/IvanKobzarev/170/head 2025-12-04T09:43:53.2275982Z * [new branch] gh/IvanKobzarev/170/orig -> origin/gh/IvanKobzarev/170/orig 2025-12-04T09:43:53.2278644Z * [new branch] gh/IvanKobzarev/171/base -> origin/gh/IvanKobzarev/171/base 2025-12-04T09:43:53.2281264Z * [new branch] gh/IvanKobzarev/171/head -> origin/gh/IvanKobzarev/171/head 2025-12-04T09:43:53.2283047Z * [new branch] gh/IvanKobzarev/171/orig -> origin/gh/IvanKobzarev/171/orig 2025-12-04T09:43:53.2285481Z * [new branch] gh/IvanKobzarev/172/base -> origin/gh/IvanKobzarev/172/base 2025-12-04T09:43:53.2287338Z * [new branch] gh/IvanKobzarev/172/head -> origin/gh/IvanKobzarev/172/head 2025-12-04T09:43:53.2289082Z * [new branch] gh/IvanKobzarev/172/orig -> origin/gh/IvanKobzarev/172/orig 2025-12-04T09:43:53.2291545Z * [new branch] gh/IvanKobzarev/173/base -> origin/gh/IvanKobzarev/173/base 2025-12-04T09:43:53.2293300Z * [new branch] gh/IvanKobzarev/173/head -> origin/gh/IvanKobzarev/173/head 2025-12-04T09:43:53.2295019Z * [new branch] gh/IvanKobzarev/173/orig -> origin/gh/IvanKobzarev/173/orig 2025-12-04T09:43:53.2297483Z * [new branch] gh/IvanKobzarev/174/base -> origin/gh/IvanKobzarev/174/base 2025-12-04T09:43:53.2299294Z * [new branch] gh/IvanKobzarev/174/head -> origin/gh/IvanKobzarev/174/head 2025-12-04T09:43:53.2301096Z * [new branch] gh/IvanKobzarev/174/orig -> origin/gh/IvanKobzarev/174/orig 2025-12-04T09:43:53.2303525Z * [new branch] gh/IvanKobzarev/175/base -> origin/gh/IvanKobzarev/175/base 2025-12-04T09:43:53.2305473Z * [new branch] gh/IvanKobzarev/175/head -> origin/gh/IvanKobzarev/175/head 2025-12-04T09:43:53.2307293Z * [new branch] gh/IvanKobzarev/175/orig -> origin/gh/IvanKobzarev/175/orig 2025-12-04T09:43:53.2309797Z * [new branch] gh/IvanKobzarev/176/base -> origin/gh/IvanKobzarev/176/base 2025-12-04T09:43:53.2311568Z * [new branch] gh/IvanKobzarev/176/head -> origin/gh/IvanKobzarev/176/head 2025-12-04T09:43:53.2313266Z * [new branch] gh/IvanKobzarev/176/orig -> origin/gh/IvanKobzarev/176/orig 2025-12-04T09:43:53.2315938Z * [new branch] gh/IvanKobzarev/177/base -> origin/gh/IvanKobzarev/177/base 2025-12-04T09:43:53.2317744Z * [new branch] gh/IvanKobzarev/177/head -> origin/gh/IvanKobzarev/177/head 2025-12-04T09:43:53.2319504Z * [new branch] gh/IvanKobzarev/177/orig -> origin/gh/IvanKobzarev/177/orig 2025-12-04T09:43:53.2322397Z * [new branch] gh/IvanKobzarev/178/base -> origin/gh/IvanKobzarev/178/base 2025-12-04T09:43:53.2324256Z * [new branch] gh/IvanKobzarev/178/head -> origin/gh/IvanKobzarev/178/head 2025-12-04T09:43:53.2326064Z * [new branch] gh/IvanKobzarev/178/orig -> origin/gh/IvanKobzarev/178/orig 2025-12-04T09:43:53.2328524Z * [new branch] gh/IvanKobzarev/179/base -> origin/gh/IvanKobzarev/179/base 2025-12-04T09:43:53.2330288Z * [new branch] gh/IvanKobzarev/179/head -> origin/gh/IvanKobzarev/179/head 2025-12-04T09:43:53.2332199Z * [new branch] gh/IvanKobzarev/179/orig -> origin/gh/IvanKobzarev/179/orig 2025-12-04T09:43:53.2334486Z * [new branch] gh/IvanKobzarev/180/base -> origin/gh/IvanKobzarev/180/base 2025-12-04T09:43:53.2336266Z * [new branch] gh/IvanKobzarev/180/head -> origin/gh/IvanKobzarev/180/head 2025-12-04T09:43:53.2338027Z * [new branch] gh/IvanKobzarev/180/orig -> origin/gh/IvanKobzarev/180/orig 2025-12-04T09:43:53.2340661Z * [new branch] gh/IvanKobzarev/181/base -> origin/gh/IvanKobzarev/181/base 2025-12-04T09:43:53.2342458Z * [new branch] gh/IvanKobzarev/181/head -> origin/gh/IvanKobzarev/181/head 2025-12-04T09:43:53.2344265Z * [new branch] gh/IvanKobzarev/181/orig -> origin/gh/IvanKobzarev/181/orig 2025-12-04T09:43:53.2346835Z * [new branch] gh/IvanKobzarev/182/base -> origin/gh/IvanKobzarev/182/base 2025-12-04T09:43:53.2348557Z * [new branch] gh/IvanKobzarev/182/head -> origin/gh/IvanKobzarev/182/head 2025-12-04T09:43:53.2350103Z * [new branch] gh/IvanKobzarev/182/orig -> origin/gh/IvanKobzarev/182/orig 2025-12-04T09:43:53.2352788Z * [new branch] gh/IvanKobzarev/183/base -> origin/gh/IvanKobzarev/183/base 2025-12-04T09:43:53.2354576Z * [new branch] gh/IvanKobzarev/183/head -> origin/gh/IvanKobzarev/183/head 2025-12-04T09:43:53.2356469Z * [new branch] gh/IvanKobzarev/183/orig -> origin/gh/IvanKobzarev/183/orig 2025-12-04T09:43:53.2358935Z * [new branch] gh/IvanKobzarev/184/base -> origin/gh/IvanKobzarev/184/base 2025-12-04T09:43:53.2360799Z * [new branch] gh/IvanKobzarev/184/head -> origin/gh/IvanKobzarev/184/head 2025-12-04T09:43:53.2362566Z * [new branch] gh/IvanKobzarev/184/orig -> origin/gh/IvanKobzarev/184/orig 2025-12-04T09:43:53.2365441Z * [new branch] gh/NikhilAPatel/1/base -> origin/gh/NikhilAPatel/1/base 2025-12-04T09:43:53.2367385Z * [new branch] gh/NikhilAPatel/1/head -> origin/gh/NikhilAPatel/1/head 2025-12-04T09:43:53.2369591Z * [new branch] gh/NikhilAPatel/2/base -> origin/gh/NikhilAPatel/2/base 2025-12-04T09:43:53.2371291Z * [new branch] gh/NikhilAPatel/2/head -> origin/gh/NikhilAPatel/2/head 2025-12-04T09:43:53.2373815Z * [new branch] gh/NikhilAPatel/4/base -> origin/gh/NikhilAPatel/4/base 2025-12-04T09:43:53.2375647Z * [new branch] gh/NikhilAPatel/4/head -> origin/gh/NikhilAPatel/4/head 2025-12-04T09:43:53.2378107Z * [new branch] gh/NikhilAPatel/5/base -> origin/gh/NikhilAPatel/5/base 2025-12-04T09:43:53.2379996Z * [new branch] gh/NikhilAPatel/5/head -> origin/gh/NikhilAPatel/5/head 2025-12-04T09:43:53.2381760Z * [new branch] gh/NikhilAPatel/5/orig -> origin/gh/NikhilAPatel/5/orig 2025-12-04T09:43:53.2384679Z * [new branch] gh/PaliC/17/base -> origin/gh/PaliC/17/base 2025-12-04T09:43:53.2386398Z * [new branch] gh/PaliC/17/head -> origin/gh/PaliC/17/head 2025-12-04T09:43:53.2388126Z * [new branch] gh/PaliC/17/orig -> origin/gh/PaliC/17/orig 2025-12-04T09:43:53.2390514Z * [new branch] gh/PaliC/18/base -> origin/gh/PaliC/18/base 2025-12-04T09:43:53.2392227Z * [new branch] gh/PaliC/18/head -> origin/gh/PaliC/18/head 2025-12-04T09:43:53.2394069Z * [new branch] gh/PaliC/18/orig -> origin/gh/PaliC/18/orig 2025-12-04T09:43:53.2396441Z * [new branch] gh/PaliC/20/base -> origin/gh/PaliC/20/base 2025-12-04T09:43:53.2398152Z * [new branch] gh/PaliC/20/head -> origin/gh/PaliC/20/head 2025-12-04T09:43:53.2400016Z * [new branch] gh/PaliC/20/orig -> origin/gh/PaliC/20/orig 2025-12-04T09:43:53.2402269Z * [new branch] gh/PaliC/21/base -> origin/gh/PaliC/21/base 2025-12-04T09:43:53.2404188Z * [new branch] gh/PaliC/21/head -> origin/gh/PaliC/21/head 2025-12-04T09:43:53.2419808Z * [new branch] gh/PaliC/21/orig -> origin/gh/PaliC/21/orig 2025-12-04T09:43:53.2420570Z * [new branch] gh/PaliC/23/base -> origin/gh/PaliC/23/base 2025-12-04T09:43:53.2420970Z * [new branch] gh/PaliC/23/head -> origin/gh/PaliC/23/head 2025-12-04T09:43:53.2421576Z * [new branch] gh/PaliC/23/orig -> origin/gh/PaliC/23/orig 2025-12-04T09:43:53.2421938Z * [new branch] gh/PaliC/24/base -> origin/gh/PaliC/24/base 2025-12-04T09:43:53.2422294Z * [new branch] gh/PaliC/24/head -> origin/gh/PaliC/24/head 2025-12-04T09:43:53.2422635Z * [new branch] gh/PaliC/24/orig -> origin/gh/PaliC/24/orig 2025-12-04T09:43:53.2422975Z * [new branch] gh/PaliC/25/head -> origin/gh/PaliC/25/head 2025-12-04T09:43:53.2423327Z * [new branch] gh/PaliC/25/next -> origin/gh/PaliC/25/next 2025-12-04T09:43:53.2423779Z * [new branch] gh/PaliC/25/orig -> origin/gh/PaliC/25/orig 2025-12-04T09:43:53.2425757Z * [new branch] gh/PaliC/26/head -> origin/gh/PaliC/26/head 2025-12-04T09:43:53.2427338Z * [new branch] gh/PaliC/26/next -> origin/gh/PaliC/26/next 2025-12-04T09:43:53.2429215Z * [new branch] gh/PaliC/26/orig -> origin/gh/PaliC/26/orig 2025-12-04T09:43:53.2431601Z * [new branch] gh/PaliC/27/next -> origin/gh/PaliC/27/next 2025-12-04T09:43:53.2433929Z * [new branch] gh/PaliC/28/head -> origin/gh/PaliC/28/head 2025-12-04T09:43:53.2435552Z * [new branch] gh/PaliC/28/next -> origin/gh/PaliC/28/next 2025-12-04T09:43:53.2437297Z * [new branch] gh/PaliC/28/orig -> origin/gh/PaliC/28/orig 2025-12-04T09:43:53.2439654Z * [new branch] gh/PaliC/29/head -> origin/gh/PaliC/29/head 2025-12-04T09:43:53.2441293Z * [new branch] gh/PaliC/29/next -> origin/gh/PaliC/29/next 2025-12-04T09:43:53.2443049Z * [new branch] gh/PaliC/29/orig -> origin/gh/PaliC/29/orig 2025-12-04T09:43:53.2445467Z * [new branch] gh/PaliC/30/head -> origin/gh/PaliC/30/head 2025-12-04T09:43:53.2447068Z * [new branch] gh/PaliC/30/next -> origin/gh/PaliC/30/next 2025-12-04T09:43:53.2448828Z * [new branch] gh/PaliC/30/orig -> origin/gh/PaliC/30/orig 2025-12-04T09:43:53.2451120Z * [new branch] gh/PaliC/31/head -> origin/gh/PaliC/31/head 2025-12-04T09:43:53.2452755Z * [new branch] gh/PaliC/31/next -> origin/gh/PaliC/31/next 2025-12-04T09:43:53.2454487Z * [new branch] gh/PaliC/31/orig -> origin/gh/PaliC/31/orig 2025-12-04T09:43:53.2457288Z * [new branch] gh/PaulZhang12/25/base -> origin/gh/PaulZhang12/25/base 2025-12-04T09:43:53.2459211Z * [new branch] gh/PaulZhang12/25/head -> origin/gh/PaulZhang12/25/head 2025-12-04T09:43:53.2460907Z * [new branch] gh/PaulZhang12/25/orig -> origin/gh/PaulZhang12/25/orig 2025-12-04T09:43:53.2463275Z * [new branch] gh/PaulZhang12/28/base -> origin/gh/PaulZhang12/28/base 2025-12-04T09:43:53.2465323Z * [new branch] gh/PaulZhang12/28/head -> origin/gh/PaulZhang12/28/head 2025-12-04T09:43:53.2467050Z * [new branch] gh/PaulZhang12/28/orig -> origin/gh/PaulZhang12/28/orig 2025-12-04T09:43:53.2469614Z * [new branch] gh/PaulZhang12/31/base -> origin/gh/PaulZhang12/31/base 2025-12-04T09:43:53.2472383Z * [new branch] gh/PaulZhang12/31/head -> origin/gh/PaulZhang12/31/head 2025-12-04T09:43:53.2473422Z * [new branch] gh/PaulZhang12/31/orig -> origin/gh/PaulZhang12/31/orig 2025-12-04T09:43:53.2475364Z * [new branch] gh/PaulZhang12/37/base -> origin/gh/PaulZhang12/37/base 2025-12-04T09:43:53.2476995Z * [new branch] gh/PaulZhang12/37/head -> origin/gh/PaulZhang12/37/head 2025-12-04T09:43:53.2478992Z * [new branch] gh/PaulZhang12/37/orig -> origin/gh/PaulZhang12/37/orig 2025-12-04T09:43:53.2481304Z * [new branch] gh/PaulZhang12/40/base -> origin/gh/PaulZhang12/40/base 2025-12-04T09:43:53.2483099Z * [new branch] gh/PaulZhang12/40/head -> origin/gh/PaulZhang12/40/head 2025-12-04T09:43:53.2484803Z * [new branch] gh/PaulZhang12/40/orig -> origin/gh/PaulZhang12/40/orig 2025-12-04T09:43:53.2487250Z * [new branch] gh/PaulZhang12/42/base -> origin/gh/PaulZhang12/42/base 2025-12-04T09:43:53.2488985Z * [new branch] gh/PaulZhang12/42/head -> origin/gh/PaulZhang12/42/head 2025-12-04T09:43:53.2491404Z * [new branch] gh/PaulZhang12/43/base -> origin/gh/PaulZhang12/43/base 2025-12-04T09:43:53.2493177Z * [new branch] gh/PaulZhang12/43/head -> origin/gh/PaulZhang12/43/head 2025-12-04T09:43:53.2494905Z * [new branch] gh/PaulZhang12/43/orig -> origin/gh/PaulZhang12/43/orig 2025-12-04T09:43:53.2497202Z * [new branch] gh/PaulZhang12/44/base -> origin/gh/PaulZhang12/44/base 2025-12-04T09:43:53.2498943Z * [new branch] gh/PaulZhang12/44/head -> origin/gh/PaulZhang12/44/head 2025-12-04T09:43:53.2501341Z * [new branch] gh/PaulZhang12/45/base -> origin/gh/PaulZhang12/45/base 2025-12-04T09:43:53.2503016Z * [new branch] gh/PaulZhang12/45/head -> origin/gh/PaulZhang12/45/head 2025-12-04T09:43:53.2504863Z * [new branch] gh/PaulZhang12/45/orig -> origin/gh/PaulZhang12/45/orig 2025-12-04T09:43:53.2507259Z * [new branch] gh/PaulZhang12/46/base -> origin/gh/PaulZhang12/46/base 2025-12-04T09:43:53.2508998Z * [new branch] gh/PaulZhang12/46/head -> origin/gh/PaulZhang12/46/head 2025-12-04T09:43:53.2510857Z * [new branch] gh/PaulZhang12/46/orig -> origin/gh/PaulZhang12/46/orig 2025-12-04T09:43:53.2513254Z * [new branch] gh/PaulZhang12/47/base -> origin/gh/PaulZhang12/47/base 2025-12-04T09:43:53.2515068Z * [new branch] gh/PaulZhang12/47/head -> origin/gh/PaulZhang12/47/head 2025-12-04T09:43:53.2516900Z * [new branch] gh/PaulZhang12/47/orig -> origin/gh/PaulZhang12/47/orig 2025-12-04T09:43:53.2519079Z * [new branch] gh/PaulZhang12/48/base -> origin/gh/PaulZhang12/48/base 2025-12-04T09:43:53.2520807Z * [new branch] gh/PaulZhang12/48/head -> origin/gh/PaulZhang12/48/head 2025-12-04T09:43:53.2522543Z * [new branch] gh/PaulZhang12/48/orig -> origin/gh/PaulZhang12/48/orig 2025-12-04T09:43:53.2525427Z * [new branch] gh/SamGinzburg/11/base -> origin/gh/SamGinzburg/11/base 2025-12-04T09:43:53.2527115Z * [new branch] gh/SamGinzburg/11/head -> origin/gh/SamGinzburg/11/head 2025-12-04T09:43:53.2530036Z * [new branch] gh/SherlockNoMad/1/base -> origin/gh/SherlockNoMad/1/base 2025-12-04T09:43:53.2531875Z * [new branch] gh/SherlockNoMad/1/head -> origin/gh/SherlockNoMad/1/head 2025-12-04T09:43:53.2534206Z * [new branch] gh/SherlockNoMad/10/base -> origin/gh/SherlockNoMad/10/base 2025-12-04T09:43:53.2535963Z * [new branch] gh/SherlockNoMad/10/head -> origin/gh/SherlockNoMad/10/head 2025-12-04T09:43:53.2537719Z * [new branch] gh/SherlockNoMad/10/orig -> origin/gh/SherlockNoMad/10/orig 2025-12-04T09:43:53.2539950Z * [new branch] gh/SherlockNoMad/11/base -> origin/gh/SherlockNoMad/11/base 2025-12-04T09:43:53.2541679Z * [new branch] gh/SherlockNoMad/11/head -> origin/gh/SherlockNoMad/11/head 2025-12-04T09:43:53.2543646Z * [new branch] gh/SherlockNoMad/11/orig -> origin/gh/SherlockNoMad/11/orig 2025-12-04T09:43:53.2545797Z * [new branch] gh/SherlockNoMad/12/base -> origin/gh/SherlockNoMad/12/base 2025-12-04T09:43:53.2547510Z * [new branch] gh/SherlockNoMad/12/head -> origin/gh/SherlockNoMad/12/head 2025-12-04T09:43:53.2549229Z * [new branch] gh/SherlockNoMad/12/orig -> origin/gh/SherlockNoMad/12/orig 2025-12-04T09:43:53.2551719Z * [new branch] gh/SherlockNoMad/15/base -> origin/gh/SherlockNoMad/15/base 2025-12-04T09:43:53.2553558Z * [new branch] gh/SherlockNoMad/15/head -> origin/gh/SherlockNoMad/15/head 2025-12-04T09:43:53.2555274Z * [new branch] gh/SherlockNoMad/15/orig -> origin/gh/SherlockNoMad/15/orig 2025-12-04T09:43:53.2557600Z * [new branch] gh/SherlockNoMad/17/base -> origin/gh/SherlockNoMad/17/base 2025-12-04T09:43:53.2559350Z * [new branch] gh/SherlockNoMad/17/head -> origin/gh/SherlockNoMad/17/head 2025-12-04T09:43:53.2561045Z * [new branch] gh/SherlockNoMad/17/orig -> origin/gh/SherlockNoMad/17/orig 2025-12-04T09:43:53.2563746Z * [new branch] gh/SherlockNoMad/18/base -> origin/gh/SherlockNoMad/18/base 2025-12-04T09:43:53.2565494Z * [new branch] gh/SherlockNoMad/18/head -> origin/gh/SherlockNoMad/18/head 2025-12-04T09:43:53.2567259Z * [new branch] gh/SherlockNoMad/18/orig -> origin/gh/SherlockNoMad/18/orig 2025-12-04T09:43:53.2569484Z * [new branch] gh/SherlockNoMad/19/base -> origin/gh/SherlockNoMad/19/base 2025-12-04T09:43:53.2571267Z * [new branch] gh/SherlockNoMad/19/head -> origin/gh/SherlockNoMad/19/head 2025-12-04T09:43:53.2573040Z * [new branch] gh/SherlockNoMad/19/orig -> origin/gh/SherlockNoMad/19/orig 2025-12-04T09:43:53.2575262Z * [new branch] gh/SherlockNoMad/2/base -> origin/gh/SherlockNoMad/2/base 2025-12-04T09:43:53.2577004Z * [new branch] gh/SherlockNoMad/2/head -> origin/gh/SherlockNoMad/2/head 2025-12-04T09:43:53.2579547Z * [new branch] gh/SherlockNoMad/20/base -> origin/gh/SherlockNoMad/20/base 2025-12-04T09:43:53.2581403Z * [new branch] gh/SherlockNoMad/20/head -> origin/gh/SherlockNoMad/20/head 2025-12-04T09:43:53.2583074Z * [new branch] gh/SherlockNoMad/20/orig -> origin/gh/SherlockNoMad/20/orig 2025-12-04T09:43:53.2585728Z * [new branch] gh/SherlockNoMad/21/base -> origin/gh/SherlockNoMad/21/base 2025-12-04T09:43:53.2587475Z * [new branch] gh/SherlockNoMad/21/head -> origin/gh/SherlockNoMad/21/head 2025-12-04T09:43:53.2589101Z * [new branch] gh/SherlockNoMad/21/orig -> origin/gh/SherlockNoMad/21/orig 2025-12-04T09:43:53.2591410Z * [new branch] gh/SherlockNoMad/3/base -> origin/gh/SherlockNoMad/3/base 2025-12-04T09:43:53.2593091Z * [new branch] gh/SherlockNoMad/3/head -> origin/gh/SherlockNoMad/3/head 2025-12-04T09:43:53.2595333Z * [new branch] gh/SherlockNoMad/4/base -> origin/gh/SherlockNoMad/4/base 2025-12-04T09:43:53.2597065Z * [new branch] gh/SherlockNoMad/4/head -> origin/gh/SherlockNoMad/4/head 2025-12-04T09:43:53.2599242Z * [new branch] gh/SherlockNoMad/5/base -> origin/gh/SherlockNoMad/5/base 2025-12-04T09:43:53.2601034Z * [new branch] gh/SherlockNoMad/5/head -> origin/gh/SherlockNoMad/5/head 2025-12-04T09:43:53.2604389Z * [new branch] gh/Sidharth123-cpu/24/base -> origin/gh/Sidharth123-cpu/24/base 2025-12-04T09:43:53.2606640Z * [new branch] gh/Sidharth123-cpu/25/base -> origin/gh/Sidharth123-cpu/25/base 2025-12-04T09:43:53.2608815Z * [new branch] gh/Sidharth123-cpu/26/base -> origin/gh/Sidharth123-cpu/26/base 2025-12-04T09:43:53.2611202Z * [new branch] gh/Sidharth123-cpu/27/base -> origin/gh/Sidharth123-cpu/27/base 2025-12-04T09:43:53.2614215Z * [new branch] gh/StrongerXi/1/base -> origin/gh/StrongerXi/1/base 2025-12-04T09:43:53.2615849Z * [new branch] gh/StrongerXi/1/head -> origin/gh/StrongerXi/1/head 2025-12-04T09:43:53.2618254Z * [new branch] gh/StrongerXi/71/base -> origin/gh/StrongerXi/71/base 2025-12-04T09:43:53.2620038Z * [new branch] gh/StrongerXi/71/head -> origin/gh/StrongerXi/71/head 2025-12-04T09:43:53.2622340Z * [new branch] gh/StrongerXi/72/base -> origin/gh/StrongerXi/72/base 2025-12-04T09:43:53.2624172Z * [new branch] gh/StrongerXi/72/head -> origin/gh/StrongerXi/72/head 2025-12-04T09:43:53.2626531Z * [new branch] gh/StrongerXi/73/base -> origin/gh/StrongerXi/73/base 2025-12-04T09:43:53.2628214Z * [new branch] gh/StrongerXi/73/head -> origin/gh/StrongerXi/73/head 2025-12-04T09:43:53.2630164Z * [new branch] gh/StrongerXi/73/orig -> origin/gh/StrongerXi/73/orig 2025-12-04T09:43:53.2632964Z * [new branch] gh/XilunWu/160/base -> origin/gh/XilunWu/160/base 2025-12-04T09:43:53.2634608Z * [new branch] gh/XilunWu/160/head -> origin/gh/XilunWu/160/head 2025-12-04T09:43:53.2636364Z * [new branch] gh/XilunWu/160/orig -> origin/gh/XilunWu/160/orig 2025-12-04T09:43:53.2638722Z * [new branch] gh/XilunWu/163/base -> origin/gh/XilunWu/163/base 2025-12-04T09:43:53.2640534Z * [new branch] gh/XilunWu/163/head -> origin/gh/XilunWu/163/head 2025-12-04T09:43:53.2642272Z * [new branch] gh/XilunWu/163/orig -> origin/gh/XilunWu/163/orig 2025-12-04T09:43:53.2644647Z * [new branch] gh/XilunWu/168/base -> origin/gh/XilunWu/168/base 2025-12-04T09:43:53.2646383Z * [new branch] gh/XilunWu/168/head -> origin/gh/XilunWu/168/head 2025-12-04T09:43:53.2648096Z * [new branch] gh/XilunWu/168/orig -> origin/gh/XilunWu/168/orig 2025-12-04T09:43:53.2650367Z * [new branch] gh/XilunWu/169/base -> origin/gh/XilunWu/169/base 2025-12-04T09:43:53.2652114Z * [new branch] gh/XilunWu/169/head -> origin/gh/XilunWu/169/head 2025-12-04T09:43:53.2653857Z * [new branch] gh/XilunWu/169/orig -> origin/gh/XilunWu/169/orig 2025-12-04T09:43:53.2656099Z * [new branch] gh/XilunWu/170/base -> origin/gh/XilunWu/170/base 2025-12-04T09:43:53.2657815Z * [new branch] gh/XilunWu/170/head -> origin/gh/XilunWu/170/head 2025-12-04T09:43:53.2659669Z * [new branch] gh/XilunWu/170/orig -> origin/gh/XilunWu/170/orig 2025-12-04T09:43:53.2662109Z * [new branch] gh/XilunWu/171/base -> origin/gh/XilunWu/171/base 2025-12-04T09:43:53.2663940Z * [new branch] gh/XilunWu/171/head -> origin/gh/XilunWu/171/head 2025-12-04T09:43:53.2665762Z * [new branch] gh/XilunWu/171/orig -> origin/gh/XilunWu/171/orig 2025-12-04T09:43:53.2668033Z * [new branch] gh/XilunWu/173/base -> origin/gh/XilunWu/173/base 2025-12-04T09:43:53.2669851Z * [new branch] gh/XilunWu/173/head -> origin/gh/XilunWu/173/head 2025-12-04T09:43:53.2671578Z * [new branch] gh/XilunWu/173/orig -> origin/gh/XilunWu/173/orig 2025-12-04T09:43:53.2673889Z * [new branch] gh/XilunWu/175/base -> origin/gh/XilunWu/175/base 2025-12-04T09:43:53.2675687Z * [new branch] gh/XilunWu/175/head -> origin/gh/XilunWu/175/head 2025-12-04T09:43:53.2677432Z * [new branch] gh/XilunWu/175/orig -> origin/gh/XilunWu/175/orig 2025-12-04T09:43:53.2680198Z * [new branch] gh/XilunWu/176/base -> origin/gh/XilunWu/176/base 2025-12-04T09:43:53.2682047Z * [new branch] gh/XilunWu/176/head -> origin/gh/XilunWu/176/head 2025-12-04T09:43:53.2684047Z * [new branch] gh/XilunWu/176/orig -> origin/gh/XilunWu/176/orig 2025-12-04T09:43:53.2686766Z * [new branch] gh/XuehaiPan/14/base -> origin/gh/XuehaiPan/14/base 2025-12-04T09:43:53.2688477Z * [new branch] gh/XuehaiPan/14/head -> origin/gh/XuehaiPan/14/head 2025-12-04T09:43:53.2690204Z * [new branch] gh/XuehaiPan/14/orig -> origin/gh/XuehaiPan/14/orig 2025-12-04T09:43:53.2692646Z * [new branch] gh/XuehaiPan/179/base -> origin/gh/XuehaiPan/179/base 2025-12-04T09:43:53.2694411Z * [new branch] gh/XuehaiPan/179/head -> origin/gh/XuehaiPan/179/head 2025-12-04T09:43:53.2696215Z * [new branch] gh/XuehaiPan/179/orig -> origin/gh/XuehaiPan/179/orig 2025-12-04T09:43:53.2698516Z * [new branch] gh/XuehaiPan/249/base -> origin/gh/XuehaiPan/249/base 2025-12-04T09:43:53.2700213Z * [new branch] gh/XuehaiPan/249/head -> origin/gh/XuehaiPan/249/head 2025-12-04T09:43:53.2701979Z * [new branch] gh/XuehaiPan/249/orig -> origin/gh/XuehaiPan/249/orig 2025-12-04T09:43:53.2704546Z * [new branch] gh/XuehaiPan/253/base -> origin/gh/XuehaiPan/253/base 2025-12-04T09:43:53.2706267Z * [new branch] gh/XuehaiPan/253/head -> origin/gh/XuehaiPan/253/head 2025-12-04T09:43:53.2707993Z * [new branch] gh/XuehaiPan/253/orig -> origin/gh/XuehaiPan/253/orig 2025-12-04T09:43:53.2710333Z * [new branch] gh/XuehaiPan/254/base -> origin/gh/XuehaiPan/254/base 2025-12-04T09:43:53.2712003Z * [new branch] gh/XuehaiPan/254/head -> origin/gh/XuehaiPan/254/head 2025-12-04T09:43:53.2713773Z * [new branch] gh/XuehaiPan/254/orig -> origin/gh/XuehaiPan/254/orig 2025-12-04T09:43:53.2716019Z * [new branch] gh/XuehaiPan/255/base -> origin/gh/XuehaiPan/255/base 2025-12-04T09:43:53.2717806Z * [new branch] gh/XuehaiPan/255/head -> origin/gh/XuehaiPan/255/head 2025-12-04T09:43:53.2719566Z * [new branch] gh/XuehaiPan/255/orig -> origin/gh/XuehaiPan/255/orig 2025-12-04T09:43:53.2721933Z * [new branch] gh/XuehaiPan/271/base -> origin/gh/XuehaiPan/271/base 2025-12-04T09:43:53.2723617Z * [new branch] gh/XuehaiPan/271/head -> origin/gh/XuehaiPan/271/head 2025-12-04T09:43:53.2725356Z * [new branch] gh/XuehaiPan/271/orig -> origin/gh/XuehaiPan/271/orig 2025-12-04T09:43:53.2727730Z * [new branch] gh/XuehaiPan/343/base -> origin/gh/XuehaiPan/343/base 2025-12-04T09:43:53.2729496Z * [new branch] gh/XuehaiPan/343/head -> origin/gh/XuehaiPan/343/head 2025-12-04T09:43:53.2731226Z * [new branch] gh/XuehaiPan/343/orig -> origin/gh/XuehaiPan/343/orig 2025-12-04T09:43:53.2733627Z * [new branch] gh/XuehaiPan/347/base -> origin/gh/XuehaiPan/347/base 2025-12-04T09:43:53.2735407Z * [new branch] gh/XuehaiPan/347/head -> origin/gh/XuehaiPan/347/head 2025-12-04T09:43:53.2737118Z * [new branch] gh/XuehaiPan/347/orig -> origin/gh/XuehaiPan/347/orig 2025-12-04T09:43:53.2739432Z * [new branch] gh/XuehaiPan/348/base -> origin/gh/XuehaiPan/348/base 2025-12-04T09:43:53.2741314Z * [new branch] gh/XuehaiPan/348/head -> origin/gh/XuehaiPan/348/head 2025-12-04T09:43:53.2743032Z * [new branch] gh/XuehaiPan/348/orig -> origin/gh/XuehaiPan/348/orig 2025-12-04T09:43:53.2745546Z * [new branch] gh/XuehaiPan/350/base -> origin/gh/XuehaiPan/350/base 2025-12-04T09:43:53.2747239Z * [new branch] gh/XuehaiPan/350/head -> origin/gh/XuehaiPan/350/head 2025-12-04T09:43:53.2749054Z * [new branch] gh/XuehaiPan/350/orig -> origin/gh/XuehaiPan/350/orig 2025-12-04T09:43:53.2751463Z * [new branch] gh/XuehaiPan/365/base -> origin/gh/XuehaiPan/365/base 2025-12-04T09:43:53.2753187Z * [new branch] gh/XuehaiPan/365/head -> origin/gh/XuehaiPan/365/head 2025-12-04T09:43:53.2754908Z * [new branch] gh/XuehaiPan/365/orig -> origin/gh/XuehaiPan/365/orig 2025-12-04T09:43:53.2757306Z * [new branch] gh/XuehaiPan/366/base -> origin/gh/XuehaiPan/366/base 2025-12-04T09:43:53.2759082Z * [new branch] gh/XuehaiPan/366/head -> origin/gh/XuehaiPan/366/head 2025-12-04T09:43:53.2761381Z * [new branch] gh/XuehaiPan/370/base -> origin/gh/XuehaiPan/370/base 2025-12-04T09:43:53.2763114Z * [new branch] gh/XuehaiPan/370/head -> origin/gh/XuehaiPan/370/head 2025-12-04T09:43:53.2764919Z * [new branch] gh/XuehaiPan/370/orig -> origin/gh/XuehaiPan/370/orig 2025-12-04T09:43:53.2767330Z * [new branch] gh/XuehaiPan/390/base -> origin/gh/XuehaiPan/390/base 2025-12-04T09:43:53.2769267Z * [new branch] gh/XuehaiPan/390/head -> origin/gh/XuehaiPan/390/head 2025-12-04T09:43:53.2771016Z * [new branch] gh/XuehaiPan/390/orig -> origin/gh/XuehaiPan/390/orig 2025-12-04T09:43:53.2773385Z * [new branch] gh/XuehaiPan/391/base -> origin/gh/XuehaiPan/391/base 2025-12-04T09:43:53.2775135Z * [new branch] gh/XuehaiPan/391/head -> origin/gh/XuehaiPan/391/head 2025-12-04T09:43:53.2776894Z * [new branch] gh/XuehaiPan/391/orig -> origin/gh/XuehaiPan/391/orig 2025-12-04T09:43:53.2781139Z * [new branch] gh/XuehaiPan/392/base -> origin/gh/XuehaiPan/392/base 2025-12-04T09:43:53.2782877Z * [new branch] gh/XuehaiPan/392/head -> origin/gh/XuehaiPan/392/head 2025-12-04T09:43:53.2784730Z * [new branch] gh/XuehaiPan/392/orig -> origin/gh/XuehaiPan/392/orig 2025-12-04T09:43:53.2787625Z * [new branch] gh/XuehaiPan/394/base -> origin/gh/XuehaiPan/394/base 2025-12-04T09:43:53.2789411Z * [new branch] gh/XuehaiPan/394/head -> origin/gh/XuehaiPan/394/head 2025-12-04T09:43:53.2791052Z * [new branch] gh/XuehaiPan/394/orig -> origin/gh/XuehaiPan/394/orig 2025-12-04T09:43:53.2793489Z * [new branch] gh/XuehaiPan/397/base -> origin/gh/XuehaiPan/397/base 2025-12-04T09:43:53.2795211Z * [new branch] gh/XuehaiPan/397/head -> origin/gh/XuehaiPan/397/head 2025-12-04T09:43:53.2796975Z * [new branch] gh/XuehaiPan/397/orig -> origin/gh/XuehaiPan/397/orig 2025-12-04T09:43:53.2799404Z * [new branch] gh/XuehaiPan/398/base -> origin/gh/XuehaiPan/398/base 2025-12-04T09:43:53.2801245Z * [new branch] gh/XuehaiPan/398/head -> origin/gh/XuehaiPan/398/head 2025-12-04T09:43:53.2803013Z * [new branch] gh/XuehaiPan/398/orig -> origin/gh/XuehaiPan/398/orig 2025-12-04T09:43:53.2805417Z * [new branch] gh/XuehaiPan/399/base -> origin/gh/XuehaiPan/399/base 2025-12-04T09:43:53.2807187Z * [new branch] gh/XuehaiPan/399/head -> origin/gh/XuehaiPan/399/head 2025-12-04T09:43:53.2808961Z * [new branch] gh/XuehaiPan/399/orig -> origin/gh/XuehaiPan/399/orig 2025-12-04T09:43:53.2811352Z * [new branch] gh/XuehaiPan/400/base -> origin/gh/XuehaiPan/400/base 2025-12-04T09:43:53.2813080Z * [new branch] gh/XuehaiPan/400/head -> origin/gh/XuehaiPan/400/head 2025-12-04T09:43:53.2814805Z * [new branch] gh/XuehaiPan/400/orig -> origin/gh/XuehaiPan/400/orig 2025-12-04T09:43:53.2817649Z * [new branch] gh/ZhiweiYan-96/39/base -> origin/gh/ZhiweiYan-96/39/base 2025-12-04T09:43:53.2819339Z * [new branch] gh/ZhiweiYan-96/39/head -> origin/gh/ZhiweiYan-96/39/head 2025-12-04T09:43:53.2821083Z * [new branch] gh/ZhiweiYan-96/39/orig -> origin/gh/ZhiweiYan-96/39/orig 2025-12-04T09:43:53.2824093Z * [new branch] gh/ZhiweiYan-96/44/base -> origin/gh/ZhiweiYan-96/44/base 2025-12-04T09:43:53.2825756Z * [new branch] gh/ZhiweiYan-96/44/head -> origin/gh/ZhiweiYan-96/44/head 2025-12-04T09:43:53.2828046Z * [new branch] gh/ZhiweiYan-96/45/base -> origin/gh/ZhiweiYan-96/45/base 2025-12-04T09:43:53.2829770Z * [new branch] gh/ZhiweiYan-96/45/head -> origin/gh/ZhiweiYan-96/45/head 2025-12-04T09:43:53.2832510Z * [new branch] gh/ZhiweiYan-96/49/base -> origin/gh/ZhiweiYan-96/49/base 2025-12-04T09:43:53.2834273Z * [new branch] gh/ZhiweiYan-96/49/head -> origin/gh/ZhiweiYan-96/49/head 2025-12-04T09:43:53.2836564Z * [new branch] gh/ZhiweiYan-96/62/base -> origin/gh/ZhiweiYan-96/62/base 2025-12-04T09:43:53.2838304Z * [new branch] gh/ZhiweiYan-96/62/head -> origin/gh/ZhiweiYan-96/62/head 2025-12-04T09:43:53.2840736Z * [new branch] gh/ZhiweiYan-96/66/base -> origin/gh/ZhiweiYan-96/66/base 2025-12-04T09:43:53.2842552Z * [new branch] gh/ZhiweiYan-96/66/head -> origin/gh/ZhiweiYan-96/66/head 2025-12-04T09:43:53.2845140Z * [new branch] gh/ZhiweiYan-96/67/base -> origin/gh/ZhiweiYan-96/67/base 2025-12-04T09:43:53.2846937Z * [new branch] gh/ZhiweiYan-96/67/head -> origin/gh/ZhiweiYan-96/67/head 2025-12-04T09:43:53.2849169Z * [new branch] gh/ZhiweiYan-96/68/base -> origin/gh/ZhiweiYan-96/68/base 2025-12-04T09:43:53.2850846Z * [new branch] gh/ZhiweiYan-96/68/head -> origin/gh/ZhiweiYan-96/68/head 2025-12-04T09:43:53.2852577Z * [new branch] gh/ZhiweiYan-96/68/orig -> origin/gh/ZhiweiYan-96/68/orig 2025-12-04T09:43:53.2855797Z * [new branch] gh/aakhundov/1/base -> origin/gh/aakhundov/1/base 2025-12-04T09:43:53.2857637Z * [new branch] gh/aakhundov/1/head -> origin/gh/aakhundov/1/head 2025-12-04T09:43:53.2859870Z * [new branch] gh/aakhundov/2/base -> origin/gh/aakhundov/2/base 2025-12-04T09:43:53.2861566Z * [new branch] gh/aakhundov/2/head -> origin/gh/aakhundov/2/head 2025-12-04T09:43:53.2864422Z * [new branch] gh/aditew01/openblas -> origin/gh/aditew01/openblas 2025-12-04T09:43:53.2866127Z * [new branch] gh/aditew01/sbgemm -> origin/gh/aditew01/sbgemm 2025-12-04T09:43:53.2867916Z * [new branch] gh/aditew01/vecbf16 -> origin/gh/aditew01/vecbf16 2025-12-04T09:43:53.2870786Z * [new branch] gh/albanD/4/base -> origin/gh/albanD/4/base 2025-12-04T09:43:53.2873437Z * [new branch] gh/albanD/4/head -> origin/gh/albanD/4/head 2025-12-04T09:43:53.2875221Z * [new branch] gh/albanD/4/orig -> origin/gh/albanD/4/orig 2025-12-04T09:43:53.2878102Z * [new branch] gh/alexbrauckmann/paddedtensor_faketensor_init -> origin/gh/alexbrauckmann/paddedtensor_faketensor_init 2025-12-04T09:43:53.2880786Z * [new branch] gh/alexsamardzic/12/base -> origin/gh/alexsamardzic/12/base 2025-12-04T09:43:53.2882781Z * [new branch] gh/alexsamardzic/12/head -> origin/gh/alexsamardzic/12/head 2025-12-04T09:43:53.2884566Z * [new branch] gh/alexsamardzic/12/orig -> origin/gh/alexsamardzic/12/orig 2025-12-04T09:43:53.2886934Z * [new branch] gh/alexsamardzic/14/base -> origin/gh/alexsamardzic/14/base 2025-12-04T09:43:53.2888729Z * [new branch] gh/alexsamardzic/14/head -> origin/gh/alexsamardzic/14/head 2025-12-04T09:43:53.2890463Z * [new branch] gh/alexsamardzic/14/orig -> origin/gh/alexsamardzic/14/orig 2025-12-04T09:43:53.2893093Z * [new branch] gh/alexsamardzic/15/base -> origin/gh/alexsamardzic/15/base 2025-12-04T09:43:53.2894894Z * [new branch] gh/alexsamardzic/15/head -> origin/gh/alexsamardzic/15/head 2025-12-04T09:43:53.2896771Z * [new branch] gh/alexsamardzic/15/orig -> origin/gh/alexsamardzic/15/orig 2025-12-04T09:43:53.2899703Z * [new branch] gh/amjames/18/base -> origin/gh/amjames/18/base 2025-12-04T09:43:53.2901545Z * [new branch] gh/amjames/18/head -> origin/gh/amjames/18/head 2025-12-04T09:43:53.2903323Z * [new branch] gh/amjames/18/orig -> origin/gh/amjames/18/orig 2025-12-04T09:43:53.2906543Z * [new branch] gh/andrewor14/35/base -> origin/gh/andrewor14/35/base 2025-12-04T09:43:53.2908386Z * [new branch] gh/andrewor14/35/head -> origin/gh/andrewor14/35/head 2025-12-04T09:43:53.2910449Z * [new branch] gh/andrewor14/35/orig -> origin/gh/andrewor14/35/orig 2025-12-04T09:43:53.2913095Z * [new branch] gh/andrewor14/50/base -> origin/gh/andrewor14/50/base 2025-12-04T09:43:53.2914964Z * [new branch] gh/andrewor14/50/head -> origin/gh/andrewor14/50/head 2025-12-04T09:43:53.2916755Z * [new branch] gh/andrewor14/50/orig -> origin/gh/andrewor14/50/orig 2025-12-04T09:43:53.2919673Z * [new branch] gh/andyanwang/30/base -> origin/gh/andyanwang/30/base 2025-12-04T09:43:53.2921775Z * [new branch] gh/andyanwang/30/orig -> origin/gh/andyanwang/30/orig 2025-12-04T09:43:53.2924195Z * [new branch] gh/andyanwang/31/base -> origin/gh/andyanwang/31/base 2025-12-04T09:43:53.2926144Z * [new branch] gh/andyanwang/31/orig -> origin/gh/andyanwang/31/orig 2025-12-04T09:43:53.2928526Z * [new branch] gh/andyanwang/39/base -> origin/gh/andyanwang/39/base 2025-12-04T09:43:53.2930405Z * [new branch] gh/andyanwang/39/head -> origin/gh/andyanwang/39/head 2025-12-04T09:43:53.2932422Z * [new branch] gh/andyanwang/39/orig -> origin/gh/andyanwang/39/orig 2025-12-04T09:43:53.2934961Z * [new branch] gh/andyanwang/42/base -> origin/gh/andyanwang/42/base 2025-12-04T09:43:53.2936669Z * [new branch] gh/andyanwang/42/head -> origin/gh/andyanwang/42/head 2025-12-04T09:43:53.2938513Z * [new branch] gh/andyanwang/42/orig -> origin/gh/andyanwang/42/orig 2025-12-04T09:43:53.2941294Z * [new branch] gh/andyanwang/45/base -> origin/gh/andyanwang/45/base 2025-12-04T09:43:53.2943258Z * [new branch] gh/andyanwang/45/head -> origin/gh/andyanwang/45/head 2025-12-04T09:43:53.2945261Z * [new branch] gh/andyanwang/45/orig -> origin/gh/andyanwang/45/orig 2025-12-04T09:43:53.2948226Z * [new branch] gh/angelayi/107/base -> origin/gh/angelayi/107/base 2025-12-04T09:43:53.2950049Z * [new branch] gh/angelayi/107/head -> origin/gh/angelayi/107/head 2025-12-04T09:43:53.2952455Z * [new branch] gh/angelayi/114/base -> origin/gh/angelayi/114/base 2025-12-04T09:43:53.2954264Z * [new branch] gh/angelayi/114/head -> origin/gh/angelayi/114/head 2025-12-04T09:43:53.2956081Z * [new branch] gh/angelayi/114/orig -> origin/gh/angelayi/114/orig 2025-12-04T09:43:53.2958671Z * [new branch] gh/angelayi/116/base -> origin/gh/angelayi/116/base 2025-12-04T09:43:53.2960539Z * [new branch] gh/angelayi/116/head -> origin/gh/angelayi/116/head 2025-12-04T09:43:53.2962246Z * [new branch] gh/angelayi/116/orig -> origin/gh/angelayi/116/orig 2025-12-04T09:43:53.2964676Z * [new branch] gh/angelayi/122/base -> origin/gh/angelayi/122/base 2025-12-04T09:43:53.2966348Z * [new branch] gh/angelayi/122/head -> origin/gh/angelayi/122/head 2025-12-04T09:43:53.2968395Z * [new branch] gh/angelayi/122/orig -> origin/gh/angelayi/122/orig 2025-12-04T09:43:53.2970922Z * [new branch] gh/angelayi/124/base -> origin/gh/angelayi/124/base 2025-12-04T09:43:53.2972731Z * [new branch] gh/angelayi/124/head -> origin/gh/angelayi/124/head 2025-12-04T09:43:53.2974424Z * [new branch] gh/angelayi/124/orig -> origin/gh/angelayi/124/orig 2025-12-04T09:43:53.2976823Z * [new branch] gh/angelayi/128/base -> origin/gh/angelayi/128/base 2025-12-04T09:43:53.2979272Z * [new branch] gh/angelayi/128/head -> origin/gh/angelayi/128/head 2025-12-04T09:43:53.2981059Z * [new branch] gh/angelayi/128/orig -> origin/gh/angelayi/128/orig 2025-12-04T09:43:53.2983496Z * [new branch] gh/angelayi/131/base -> origin/gh/angelayi/131/base 2025-12-04T09:43:53.2985440Z * [new branch] gh/angelayi/131/head -> origin/gh/angelayi/131/head 2025-12-04T09:43:53.2987351Z * [new branch] gh/angelayi/131/orig -> origin/gh/angelayi/131/orig 2025-12-04T09:43:53.2990138Z * [new branch] gh/angelayi/132/base -> origin/gh/angelayi/132/base 2025-12-04T09:43:53.2992170Z * [new branch] gh/angelayi/132/head -> origin/gh/angelayi/132/head 2025-12-04T09:43:53.2994371Z * [new branch] gh/angelayi/132/orig -> origin/gh/angelayi/132/orig 2025-12-04T09:43:53.2997616Z * [new branch] gh/angelayi/133/base -> origin/gh/angelayi/133/base 2025-12-04T09:43:53.2999261Z * [new branch] gh/angelayi/133/head -> origin/gh/angelayi/133/head 2025-12-04T09:43:53.3001064Z * [new branch] gh/angelayi/133/orig -> origin/gh/angelayi/133/orig 2025-12-04T09:43:53.3003626Z * [new branch] gh/angelayi/134/base -> origin/gh/angelayi/134/base 2025-12-04T09:43:53.3005550Z * [new branch] gh/angelayi/134/head -> origin/gh/angelayi/134/head 2025-12-04T09:43:53.3007340Z * [new branch] gh/angelayi/134/orig -> origin/gh/angelayi/134/orig 2025-12-04T09:43:53.3009887Z * [new branch] gh/angelayi/135/base -> origin/gh/angelayi/135/base 2025-12-04T09:43:53.3011699Z * [new branch] gh/angelayi/135/head -> origin/gh/angelayi/135/head 2025-12-04T09:43:53.3013478Z * [new branch] gh/angelayi/135/orig -> origin/gh/angelayi/135/orig 2025-12-04T09:43:53.3015879Z * [new branch] gh/angelayi/136/base -> origin/gh/angelayi/136/base 2025-12-04T09:43:53.3017604Z * [new branch] gh/angelayi/136/head -> origin/gh/angelayi/136/head 2025-12-04T09:43:53.3019300Z * [new branch] gh/angelayi/136/orig -> origin/gh/angelayi/136/orig 2025-12-04T09:43:53.3021617Z * [new branch] gh/angelayi/137/base -> origin/gh/angelayi/137/base 2025-12-04T09:43:53.3023289Z * [new branch] gh/angelayi/137/head -> origin/gh/angelayi/137/head 2025-12-04T09:43:53.3025406Z * [new branch] gh/angelayi/137/orig -> origin/gh/angelayi/137/orig 2025-12-04T09:43:53.3027720Z * [new branch] gh/angelayi/138/base -> origin/gh/angelayi/138/base 2025-12-04T09:43:53.3029397Z * [new branch] gh/angelayi/138/head -> origin/gh/angelayi/138/head 2025-12-04T09:43:53.3031119Z * [new branch] gh/angelayi/138/orig -> origin/gh/angelayi/138/orig 2025-12-04T09:43:53.3033474Z * [new branch] gh/angelayi/139/base -> origin/gh/angelayi/139/base 2025-12-04T09:43:53.3035223Z * [new branch] gh/angelayi/139/head -> origin/gh/angelayi/139/head 2025-12-04T09:43:53.3036942Z * [new branch] gh/angelayi/139/orig -> origin/gh/angelayi/139/orig 2025-12-04T09:43:53.3039428Z * [new branch] gh/angelayi/140/base -> origin/gh/angelayi/140/base 2025-12-04T09:43:53.3041304Z * [new branch] gh/angelayi/140/head -> origin/gh/angelayi/140/head 2025-12-04T09:43:53.3043098Z * [new branch] gh/angelayi/140/orig -> origin/gh/angelayi/140/orig 2025-12-04T09:43:53.3046115Z * [new branch] gh/angelayi/141/base -> origin/gh/angelayi/141/base 2025-12-04T09:43:53.3047780Z * [new branch] gh/angelayi/141/head -> origin/gh/angelayi/141/head 2025-12-04T09:43:53.3049524Z * [new branch] gh/angelayi/141/orig -> origin/gh/angelayi/141/orig 2025-12-04T09:43:53.3051926Z * [new branch] gh/angelayi/142/base -> origin/gh/angelayi/142/base 2025-12-04T09:43:53.3053709Z * [new branch] gh/angelayi/142/head -> origin/gh/angelayi/142/head 2025-12-04T09:43:53.3055450Z * [new branch] gh/angelayi/142/orig -> origin/gh/angelayi/142/orig 2025-12-04T09:43:53.3057822Z * [new branch] gh/angelayi/143/base -> origin/gh/angelayi/143/base 2025-12-04T09:43:53.3059611Z * [new branch] gh/angelayi/143/head -> origin/gh/angelayi/143/head 2025-12-04T09:43:53.3061308Z * [new branch] gh/angelayi/143/orig -> origin/gh/angelayi/143/orig 2025-12-04T09:43:53.3063852Z * [new branch] gh/angelayi/144/base -> origin/gh/angelayi/144/base 2025-12-04T09:43:53.3065762Z * [new branch] gh/angelayi/144/head -> origin/gh/angelayi/144/head 2025-12-04T09:43:53.3067621Z * [new branch] gh/angelayi/144/orig -> origin/gh/angelayi/144/orig 2025-12-04T09:43:53.3070666Z * [new branch] gh/anijain2305/753/base -> origin/gh/anijain2305/753/base 2025-12-04T09:43:53.3072424Z * [new branch] gh/anijain2305/753/head -> origin/gh/anijain2305/753/head 2025-12-04T09:43:53.3074195Z * [new branch] gh/anijain2305/753/orig -> origin/gh/anijain2305/753/orig 2025-12-04T09:43:53.3076669Z * [new branch] gh/anijain2305/810/base -> origin/gh/anijain2305/810/base 2025-12-04T09:43:53.3078473Z * [new branch] gh/anijain2305/810/head -> origin/gh/anijain2305/810/head 2025-12-04T09:43:53.3080460Z * [new branch] gh/anijain2305/810/orig -> origin/gh/anijain2305/810/orig 2025-12-04T09:43:53.3082799Z * [new branch] gh/anijain2305/854/base -> origin/gh/anijain2305/854/base 2025-12-04T09:43:53.3084615Z * [new branch] gh/anijain2305/854/head -> origin/gh/anijain2305/854/head 2025-12-04T09:43:53.3086342Z * [new branch] gh/anijain2305/854/orig -> origin/gh/anijain2305/854/orig 2025-12-04T09:43:53.3088825Z * [new branch] gh/anijain2305/864/base -> origin/gh/anijain2305/864/base 2025-12-04T09:43:53.3090602Z * [new branch] gh/anijain2305/864/head -> origin/gh/anijain2305/864/head 2025-12-04T09:43:53.3092362Z * [new branch] gh/anijain2305/864/orig -> origin/gh/anijain2305/864/orig 2025-12-04T09:43:53.3094864Z * [new branch] gh/anijain2305/870/base -> origin/gh/anijain2305/870/base 2025-12-04T09:43:53.3096612Z * [new branch] gh/anijain2305/870/head -> origin/gh/anijain2305/870/head 2025-12-04T09:43:53.3098364Z * [new branch] gh/anijain2305/870/orig -> origin/gh/anijain2305/870/orig 2025-12-04T09:43:53.3100811Z * [new branch] gh/anijain2305/873/base -> origin/gh/anijain2305/873/base 2025-12-04T09:43:53.3102506Z * [new branch] gh/anijain2305/873/head -> origin/gh/anijain2305/873/head 2025-12-04T09:43:53.3104336Z * [new branch] gh/anijain2305/873/orig -> origin/gh/anijain2305/873/orig 2025-12-04T09:43:53.3106684Z * [new branch] gh/anijain2305/894/base -> origin/gh/anijain2305/894/base 2025-12-04T09:43:53.3108401Z * [new branch] gh/anijain2305/894/head -> origin/gh/anijain2305/894/head 2025-12-04T09:43:53.3110147Z * [new branch] gh/anijain2305/894/orig -> origin/gh/anijain2305/894/orig 2025-12-04T09:43:53.3112527Z * [new branch] gh/anijain2305/895/base -> origin/gh/anijain2305/895/base 2025-12-04T09:43:53.3114339Z * [new branch] gh/anijain2305/895/head -> origin/gh/anijain2305/895/head 2025-12-04T09:43:53.3116212Z * [new branch] gh/anijain2305/895/orig -> origin/gh/anijain2305/895/orig 2025-12-04T09:43:53.3118562Z * [new branch] gh/anijain2305/910/base -> origin/gh/anijain2305/910/base 2025-12-04T09:43:53.3120357Z * [new branch] gh/anijain2305/910/head -> origin/gh/anijain2305/910/head 2025-12-04T09:43:53.3122085Z * [new branch] gh/anijain2305/910/orig -> origin/gh/anijain2305/910/orig 2025-12-04T09:43:53.3124553Z * [new branch] gh/anijain2305/919/base -> origin/gh/anijain2305/919/base 2025-12-04T09:43:53.3126330Z * [new branch] gh/anijain2305/919/head -> origin/gh/anijain2305/919/head 2025-12-04T09:43:53.3128102Z * [new branch] gh/anijain2305/919/orig -> origin/gh/anijain2305/919/orig 2025-12-04T09:43:53.3130474Z * [new branch] gh/anijain2305/922/base -> origin/gh/anijain2305/922/base 2025-12-04T09:43:53.3132334Z * [new branch] gh/anijain2305/922/head -> origin/gh/anijain2305/922/head 2025-12-04T09:43:53.3134139Z * [new branch] gh/anijain2305/922/orig -> origin/gh/anijain2305/922/orig 2025-12-04T09:43:53.3136530Z * [new branch] gh/anijain2305/932/base -> origin/gh/anijain2305/932/base 2025-12-04T09:43:53.3138410Z * [new branch] gh/anijain2305/932/head -> origin/gh/anijain2305/932/head 2025-12-04T09:43:53.3140246Z * [new branch] gh/anijain2305/932/orig -> origin/gh/anijain2305/932/orig 2025-12-04T09:43:53.3142674Z * [new branch] gh/anijain2305/940/base -> origin/gh/anijain2305/940/base 2025-12-04T09:43:53.3144550Z * [new branch] gh/anijain2305/940/head -> origin/gh/anijain2305/940/head 2025-12-04T09:43:53.3146236Z * [new branch] gh/anijain2305/940/orig -> origin/gh/anijain2305/940/orig 2025-12-04T09:43:53.3148654Z * [new branch] gh/anijain2305/941/base -> origin/gh/anijain2305/941/base 2025-12-04T09:43:53.3150370Z * [new branch] gh/anijain2305/941/head -> origin/gh/anijain2305/941/head 2025-12-04T09:43:53.3152125Z * [new branch] gh/anijain2305/941/orig -> origin/gh/anijain2305/941/orig 2025-12-04T09:43:53.3154476Z * [new branch] gh/anijain2305/942/base -> origin/gh/anijain2305/942/base 2025-12-04T09:43:53.3156277Z * [new branch] gh/anijain2305/942/head -> origin/gh/anijain2305/942/head 2025-12-04T09:43:53.3158142Z * [new branch] gh/anijain2305/942/orig -> origin/gh/anijain2305/942/orig 2025-12-04T09:43:53.3160537Z * [new branch] gh/anijain2305/943/base -> origin/gh/anijain2305/943/base 2025-12-04T09:43:53.3162328Z * [new branch] gh/anijain2305/943/head -> origin/gh/anijain2305/943/head 2025-12-04T09:43:53.3164062Z * [new branch] gh/anijain2305/943/orig -> origin/gh/anijain2305/943/orig 2025-12-04T09:43:53.3167046Z * [new branch] gh/anijain2305/944/base -> origin/gh/anijain2305/944/base 2025-12-04T09:43:53.3168790Z * [new branch] gh/anijain2305/944/head -> origin/gh/anijain2305/944/head 2025-12-04T09:43:53.3171031Z * [new branch] gh/anijain2305/944/orig -> origin/gh/anijain2305/944/orig 2025-12-04T09:43:53.3173487Z * [new branch] gh/anijain2305/945/base -> origin/gh/anijain2305/945/base 2025-12-04T09:43:53.3175256Z * [new branch] gh/anijain2305/945/head -> origin/gh/anijain2305/945/head 2025-12-04T09:43:53.3176983Z * [new branch] gh/anijain2305/945/orig -> origin/gh/anijain2305/945/orig 2025-12-04T09:43:53.3179717Z * [new branch] gh/anijain2305/946/base -> origin/gh/anijain2305/946/base 2025-12-04T09:43:53.3181368Z * [new branch] gh/anijain2305/946/head -> origin/gh/anijain2305/946/head 2025-12-04T09:43:53.3183305Z * [new branch] gh/anijain2305/946/orig -> origin/gh/anijain2305/946/orig 2025-12-04T09:43:53.3185829Z * [new branch] gh/anijain2305/947/base -> origin/gh/anijain2305/947/base 2025-12-04T09:43:53.3187434Z * [new branch] gh/anijain2305/947/head -> origin/gh/anijain2305/947/head 2025-12-04T09:43:53.3189143Z * [new branch] gh/anijain2305/947/orig -> origin/gh/anijain2305/947/orig 2025-12-04T09:43:53.3191711Z * [new branch] gh/anijain2305/948/base -> origin/gh/anijain2305/948/base 2025-12-04T09:43:53.3193458Z * [new branch] gh/anijain2305/948/head -> origin/gh/anijain2305/948/head 2025-12-04T09:43:53.3195264Z * [new branch] gh/anijain2305/948/orig -> origin/gh/anijain2305/948/orig 2025-12-04T09:43:53.3197696Z * [new branch] gh/anijain2305/949/base -> origin/gh/anijain2305/949/base 2025-12-04T09:43:53.3199615Z * [new branch] gh/anijain2305/949/head -> origin/gh/anijain2305/949/head 2025-12-04T09:43:53.3201303Z * [new branch] gh/anijain2305/949/orig -> origin/gh/anijain2305/949/orig 2025-12-04T09:43:53.3203708Z * [new branch] gh/anijain2305/950/base -> origin/gh/anijain2305/950/base 2025-12-04T09:43:53.3205530Z * [new branch] gh/anijain2305/950/head -> origin/gh/anijain2305/950/head 2025-12-04T09:43:53.3207291Z * [new branch] gh/anijain2305/950/orig -> origin/gh/anijain2305/950/orig 2025-12-04T09:43:53.3209812Z * [new branch] gh/anijain2305/951/base -> origin/gh/anijain2305/951/base 2025-12-04T09:43:53.3211598Z * [new branch] gh/anijain2305/951/head -> origin/gh/anijain2305/951/head 2025-12-04T09:43:53.3213404Z * [new branch] gh/anijain2305/951/orig -> origin/gh/anijain2305/951/orig 2025-12-04T09:43:53.3215924Z * [new branch] gh/anijain2305/952/base -> origin/gh/anijain2305/952/base 2025-12-04T09:43:53.3217681Z * [new branch] gh/anijain2305/952/head -> origin/gh/anijain2305/952/head 2025-12-04T09:43:53.3219407Z * [new branch] gh/anijain2305/952/orig -> origin/gh/anijain2305/952/orig 2025-12-04T09:43:53.3221712Z * [new branch] gh/anijain2305/953/base -> origin/gh/anijain2305/953/base 2025-12-04T09:43:53.3223431Z * [new branch] gh/anijain2305/953/head -> origin/gh/anijain2305/953/head 2025-12-04T09:43:53.3225330Z * [new branch] gh/anijain2305/953/orig -> origin/gh/anijain2305/953/orig 2025-12-04T09:43:53.3227779Z * [new branch] gh/anijain2305/954/base -> origin/gh/anijain2305/954/base 2025-12-04T09:43:53.3229635Z * [new branch] gh/anijain2305/954/head -> origin/gh/anijain2305/954/head 2025-12-04T09:43:53.3231313Z * [new branch] gh/anijain2305/954/orig -> origin/gh/anijain2305/954/orig 2025-12-04T09:43:53.3233970Z * [new branch] gh/anijain2305/955/base -> origin/gh/anijain2305/955/base 2025-12-04T09:43:53.3235608Z * [new branch] gh/anijain2305/955/head -> origin/gh/anijain2305/955/head 2025-12-04T09:43:53.3237337Z * [new branch] gh/anijain2305/955/orig -> origin/gh/anijain2305/955/orig 2025-12-04T09:43:53.3239920Z * [new branch] gh/anijain2305/956/base -> origin/gh/anijain2305/956/base 2025-12-04T09:43:53.3241700Z * [new branch] gh/anijain2305/956/head -> origin/gh/anijain2305/956/head 2025-12-04T09:43:53.3243467Z * [new branch] gh/anijain2305/956/orig -> origin/gh/anijain2305/956/orig 2025-12-04T09:43:53.3245974Z * [new branch] gh/anijain2305/957/base -> origin/gh/anijain2305/957/base 2025-12-04T09:43:53.3247719Z * [new branch] gh/anijain2305/957/head -> origin/gh/anijain2305/957/head 2025-12-04T09:43:53.3249523Z * [new branch] gh/anijain2305/957/orig -> origin/gh/anijain2305/957/orig 2025-12-04T09:43:53.3251893Z * [new branch] gh/anijain2305/958/base -> origin/gh/anijain2305/958/base 2025-12-04T09:43:53.3253752Z * [new branch] gh/anijain2305/958/head -> origin/gh/anijain2305/958/head 2025-12-04T09:43:53.3255386Z * [new branch] gh/anijain2305/958/orig -> origin/gh/anijain2305/958/orig 2025-12-04T09:43:53.3257916Z * [new branch] gh/anijain2305/959/base -> origin/gh/anijain2305/959/base 2025-12-04T09:43:53.3259640Z * [new branch] gh/anijain2305/959/head -> origin/gh/anijain2305/959/head 2025-12-04T09:43:53.3261409Z * [new branch] gh/anijain2305/959/orig -> origin/gh/anijain2305/959/orig 2025-12-04T09:43:53.3264015Z * [new branch] gh/anijain2305/960/base -> origin/gh/anijain2305/960/base 2025-12-04T09:43:53.3265860Z * [new branch] gh/anijain2305/960/head -> origin/gh/anijain2305/960/head 2025-12-04T09:43:53.3267605Z * [new branch] gh/anijain2305/960/orig -> origin/gh/anijain2305/960/orig 2025-12-04T09:43:53.3270075Z * [new branch] gh/anijain2305/961/base -> origin/gh/anijain2305/961/base 2025-12-04T09:43:53.3271938Z * [new branch] gh/anijain2305/961/head -> origin/gh/anijain2305/961/head 2025-12-04T09:43:53.3273635Z * [new branch] gh/anijain2305/961/orig -> origin/gh/anijain2305/961/orig 2025-12-04T09:43:53.3276126Z * [new branch] gh/anijain2305/962/base -> origin/gh/anijain2305/962/base 2025-12-04T09:43:53.3278058Z * [new branch] gh/anijain2305/962/head -> origin/gh/anijain2305/962/head 2025-12-04T09:43:53.3282076Z * [new branch] gh/anijain2305/962/orig -> origin/gh/anijain2305/962/orig 2025-12-04T09:43:53.3284839Z * [new branch] gh/anijain2305/963/base -> origin/gh/anijain2305/963/base 2025-12-04T09:43:53.3286745Z * [new branch] gh/anijain2305/963/head -> origin/gh/anijain2305/963/head 2025-12-04T09:43:53.3288661Z * [new branch] gh/anijain2305/963/orig -> origin/gh/anijain2305/963/orig 2025-12-04T09:43:53.3291245Z * [new branch] gh/anijain2305/964/base -> origin/gh/anijain2305/964/base 2025-12-04T09:43:53.3292997Z * [new branch] gh/anijain2305/964/head -> origin/gh/anijain2305/964/head 2025-12-04T09:43:53.3294779Z * [new branch] gh/anijain2305/964/orig -> origin/gh/anijain2305/964/orig 2025-12-04T09:43:53.3297206Z * [new branch] gh/anijain2305/965/base -> origin/gh/anijain2305/965/base 2025-12-04T09:43:53.3299196Z * [new branch] gh/anijain2305/965/head -> origin/gh/anijain2305/965/head 2025-12-04T09:43:53.3300925Z * [new branch] gh/anijain2305/965/orig -> origin/gh/anijain2305/965/orig 2025-12-04T09:43:53.3303270Z * [new branch] gh/anijain2305/966/base -> origin/gh/anijain2305/966/base 2025-12-04T09:43:53.3305227Z * [new branch] gh/anijain2305/966/head -> origin/gh/anijain2305/966/head 2025-12-04T09:43:53.3306905Z * [new branch] gh/anijain2305/966/orig -> origin/gh/anijain2305/966/orig 2025-12-04T09:43:53.3309400Z * [new branch] gh/anijain2305/967/base -> origin/gh/anijain2305/967/base 2025-12-04T09:43:53.3311157Z * [new branch] gh/anijain2305/967/head -> origin/gh/anijain2305/967/head 2025-12-04T09:43:53.3313014Z * [new branch] gh/anijain2305/967/orig -> origin/gh/anijain2305/967/orig 2025-12-04T09:43:53.3315489Z * [new branch] gh/anijain2305/968/base -> origin/gh/anijain2305/968/base 2025-12-04T09:43:53.3317243Z * [new branch] gh/anijain2305/968/head -> origin/gh/anijain2305/968/head 2025-12-04T09:43:53.3319050Z * [new branch] gh/anijain2305/968/orig -> origin/gh/anijain2305/968/orig 2025-12-04T09:43:53.3321400Z * [new branch] gh/anijain2305/969/base -> origin/gh/anijain2305/969/base 2025-12-04T09:43:53.3323276Z * [new branch] gh/anijain2305/969/head -> origin/gh/anijain2305/969/head 2025-12-04T09:43:53.3325198Z * [new branch] gh/anijain2305/969/orig -> origin/gh/anijain2305/969/orig 2025-12-04T09:43:53.3327622Z * [new branch] gh/anijain2305/970/base -> origin/gh/anijain2305/970/base 2025-12-04T09:43:53.3329586Z * [new branch] gh/anijain2305/970/head -> origin/gh/anijain2305/970/head 2025-12-04T09:43:53.3331009Z * [new branch] gh/anijain2305/970/orig -> origin/gh/anijain2305/970/orig 2025-12-04T09:43:53.3333897Z * [new branch] gh/anjali411/216/base -> origin/gh/anjali411/216/base 2025-12-04T09:43:53.3335670Z * [new branch] gh/anjali411/216/head -> origin/gh/anjali411/216/head 2025-12-04T09:43:53.3337484Z * [new branch] gh/anjali411/216/orig -> origin/gh/anjali411/216/orig 2025-12-04T09:43:53.3340474Z * [new branch] gh/anshul-si/1/base -> origin/gh/anshul-si/1/base 2025-12-04T09:43:53.3342270Z * [new branch] gh/anshul-si/1/head -> origin/gh/anshul-si/1/head 2025-12-04T09:43:53.3344621Z * [new branch] gh/anshul-si/2/base -> origin/gh/anshul-si/2/base 2025-12-04T09:43:53.3346289Z * [new branch] gh/anshul-si/2/head -> origin/gh/anshul-si/2/head 2025-12-04T09:43:53.3348516Z * [new branch] gh/anshul-si/3/base -> origin/gh/anshul-si/3/base 2025-12-04T09:43:53.3350325Z * [new branch] gh/anshul-si/3/head -> origin/gh/anshul-si/3/head 2025-12-04T09:43:53.3352536Z * [new branch] gh/anshul-si/4/base -> origin/gh/anshul-si/4/base 2025-12-04T09:43:53.3354242Z * [new branch] gh/anshul-si/4/head -> origin/gh/anshul-si/4/head 2025-12-04T09:43:53.3356459Z * [new branch] gh/anshul-si/5/base -> origin/gh/anshul-si/5/base 2025-12-04T09:43:53.3358249Z * [new branch] gh/anshul-si/5/head -> origin/gh/anshul-si/5/head 2025-12-04T09:43:53.3360665Z * [new branch] gh/anshul-si/53/base -> origin/gh/anshul-si/53/base 2025-12-04T09:43:53.3362444Z * [new branch] gh/anshul-si/53/head -> origin/gh/anshul-si/53/head 2025-12-04T09:43:53.3364983Z * [new branch] gh/anshul-si/58/base -> origin/gh/anshul-si/58/base 2025-12-04T09:43:53.3366675Z * [new branch] gh/anshul-si/58/head -> origin/gh/anshul-si/58/head 2025-12-04T09:43:53.3368901Z * [new branch] gh/anshul-si/66/base -> origin/gh/anshul-si/66/base 2025-12-04T09:43:53.3370730Z * [new branch] gh/anshul-si/66/head -> origin/gh/anshul-si/66/head 2025-12-04T09:43:53.3372478Z * [new branch] gh/anshul-si/66/orig -> origin/gh/anshul-si/66/orig 2025-12-04T09:43:53.3374750Z * [new branch] gh/anshul-si/67/base -> origin/gh/anshul-si/67/base 2025-12-04T09:43:53.3376517Z * [new branch] gh/anshul-si/67/head -> origin/gh/anshul-si/67/head 2025-12-04T09:43:53.3378318Z * [new branch] gh/anshul-si/67/orig -> origin/gh/anshul-si/67/orig 2025-12-04T09:43:53.3381161Z * [new branch] gh/anshul-si/68/base -> origin/gh/anshul-si/68/base 2025-12-04T09:43:53.3382820Z * [new branch] gh/anshul-si/68/head -> origin/gh/anshul-si/68/head 2025-12-04T09:43:53.3384608Z * [new branch] gh/anshul-si/68/orig -> origin/gh/anshul-si/68/orig 2025-12-04T09:43:53.3387243Z * [new branch] gh/anshul-si/69/base -> origin/gh/anshul-si/69/base 2025-12-04T09:43:53.3388921Z * [new branch] gh/anshul-si/69/head -> origin/gh/anshul-si/69/head 2025-12-04T09:43:53.3390722Z * [new branch] gh/anshul-si/69/orig -> origin/gh/anshul-si/69/orig 2025-12-04T09:43:53.3393056Z * [new branch] gh/anshul-si/70/base -> origin/gh/anshul-si/70/base 2025-12-04T09:43:53.3394906Z * [new branch] gh/anshul-si/70/head -> origin/gh/anshul-si/70/head 2025-12-04T09:43:53.3396831Z * [new branch] gh/anshul-si/70/orig -> origin/gh/anshul-si/70/orig 2025-12-04T09:43:53.3399088Z * [new branch] gh/anshul-si/71/base -> origin/gh/anshul-si/71/base 2025-12-04T09:43:53.3400867Z * [new branch] gh/anshul-si/71/head -> origin/gh/anshul-si/71/head 2025-12-04T09:43:53.3402631Z * [new branch] gh/anshul-si/71/orig -> origin/gh/anshul-si/71/orig 2025-12-04T09:43:53.3405017Z * [new branch] gh/anshul-si/72/base -> origin/gh/anshul-si/72/base 2025-12-04T09:43:53.3406860Z * [new branch] gh/anshul-si/72/head -> origin/gh/anshul-si/72/head 2025-12-04T09:43:53.3408755Z * [new branch] gh/anshul-si/72/orig -> origin/gh/anshul-si/72/orig 2025-12-04T09:43:53.3411072Z * [new branch] gh/anshul-si/73/base -> origin/gh/anshul-si/73/base 2025-12-04T09:43:53.3412849Z * [new branch] gh/anshul-si/73/head -> origin/gh/anshul-si/73/head 2025-12-04T09:43:53.3414583Z * [new branch] gh/anshul-si/73/orig -> origin/gh/anshul-si/73/orig 2025-12-04T09:43:53.3417513Z * [new branch] gh/aorenste/132/base -> origin/gh/aorenste/132/base 2025-12-04T09:43:53.3419304Z * [new branch] gh/aorenste/132/head -> origin/gh/aorenste/132/head 2025-12-04T09:43:53.3421692Z * [new branch] gh/aorenste/134/base -> origin/gh/aorenste/134/base 2025-12-04T09:43:53.3423643Z * [new branch] gh/aorenste/134/head -> origin/gh/aorenste/134/head 2025-12-04T09:43:53.3425546Z * [new branch] gh/aorenste/134/orig -> origin/gh/aorenste/134/orig 2025-12-04T09:43:53.3428069Z * [new branch] gh/aorenste/139/base -> origin/gh/aorenste/139/base 2025-12-04T09:43:53.3429849Z * [new branch] gh/aorenste/139/head -> origin/gh/aorenste/139/head 2025-12-04T09:43:53.3431551Z * [new branch] gh/aorenste/139/orig -> origin/gh/aorenste/139/orig 2025-12-04T09:43:53.3433931Z * [new branch] gh/aorenste/141/base -> origin/gh/aorenste/141/base 2025-12-04T09:43:53.3435646Z * [new branch] gh/aorenste/141/head -> origin/gh/aorenste/141/head 2025-12-04T09:43:53.3438264Z * [new branch] gh/aorenste/145/base -> origin/gh/aorenste/145/base 2025-12-04T09:43:53.3440124Z * [new branch] gh/aorenste/145/head -> origin/gh/aorenste/145/head 2025-12-04T09:43:53.3442034Z * [new branch] gh/aorenste/145/orig -> origin/gh/aorenste/145/orig 2025-12-04T09:43:53.3444486Z * [new branch] gh/aorenste/146/base -> origin/gh/aorenste/146/base 2025-12-04T09:43:53.3446295Z * [new branch] gh/aorenste/146/head -> origin/gh/aorenste/146/head 2025-12-04T09:43:53.3448095Z * [new branch] gh/aorenste/146/orig -> origin/gh/aorenste/146/orig 2025-12-04T09:43:53.3450586Z * [new branch] gh/aorenste/147/base -> origin/gh/aorenste/147/base 2025-12-04T09:43:53.3452522Z * [new branch] gh/aorenste/147/head -> origin/gh/aorenste/147/head 2025-12-04T09:43:53.3454285Z * [new branch] gh/aorenste/147/orig -> origin/gh/aorenste/147/orig 2025-12-04T09:43:53.3456670Z * [new branch] gh/aorenste/148/base -> origin/gh/aorenste/148/base 2025-12-04T09:43:53.3458430Z * [new branch] gh/aorenste/148/head -> origin/gh/aorenste/148/head 2025-12-04T09:43:53.3460275Z * [new branch] gh/aorenste/148/orig -> origin/gh/aorenste/148/orig 2025-12-04T09:43:53.3462643Z * [new branch] gh/aorenste/149/base -> origin/gh/aorenste/149/base 2025-12-04T09:43:53.3464512Z * [new branch] gh/aorenste/149/head -> origin/gh/aorenste/149/head 2025-12-04T09:43:53.3466188Z * [new branch] gh/aorenste/149/orig -> origin/gh/aorenste/149/orig 2025-12-04T09:43:53.3468635Z * [new branch] gh/aorenste/150/base -> origin/gh/aorenste/150/base 2025-12-04T09:43:53.3470420Z * [new branch] gh/aorenste/150/head -> origin/gh/aorenste/150/head 2025-12-04T09:43:53.3472131Z * [new branch] gh/aorenste/150/orig -> origin/gh/aorenste/150/orig 2025-12-04T09:43:53.3474456Z * [new branch] gh/aorenste/151/base -> origin/gh/aorenste/151/base 2025-12-04T09:43:53.3476289Z * [new branch] gh/aorenste/151/head -> origin/gh/aorenste/151/head 2025-12-04T09:43:53.3478098Z * [new branch] gh/aorenste/151/orig -> origin/gh/aorenste/151/orig 2025-12-04T09:43:53.3480768Z * [new branch] gh/aorenste/152/base -> origin/gh/aorenste/152/base 2025-12-04T09:43:53.3482520Z * [new branch] gh/aorenste/152/head -> origin/gh/aorenste/152/head 2025-12-04T09:43:53.3484247Z * [new branch] gh/aorenste/152/orig -> origin/gh/aorenste/152/orig 2025-12-04T09:43:53.3486523Z * [new branch] gh/aorenste/153/base -> origin/gh/aorenste/153/base 2025-12-04T09:43:53.3488193Z * [new branch] gh/aorenste/153/head -> origin/gh/aorenste/153/head 2025-12-04T09:43:53.3489853Z * [new branch] gh/aorenste/153/orig -> origin/gh/aorenste/153/orig 2025-12-04T09:43:53.3492325Z * [new branch] gh/aorenste/154/base -> origin/gh/aorenste/154/base 2025-12-04T09:43:53.3494546Z * [new branch] gh/aorenste/154/head -> origin/gh/aorenste/154/head 2025-12-04T09:43:53.3496859Z * [new branch] gh/aorenste/154/orig -> origin/gh/aorenste/154/orig 2025-12-04T09:43:53.3499981Z * [new branch] gh/aorenste/155/base -> origin/gh/aorenste/155/base 2025-12-04T09:43:53.3501824Z * [new branch] gh/aorenste/155/head -> origin/gh/aorenste/155/head 2025-12-04T09:43:53.3503536Z * [new branch] gh/aorenste/155/orig -> origin/gh/aorenste/155/orig 2025-12-04T09:43:53.3505868Z * [new branch] gh/aorenste/156/base -> origin/gh/aorenste/156/base 2025-12-04T09:43:53.3507614Z * [new branch] gh/aorenste/156/head -> origin/gh/aorenste/156/head 2025-12-04T09:43:53.3509283Z * [new branch] gh/aorenste/156/orig -> origin/gh/aorenste/156/orig 2025-12-04T09:43:53.3512048Z * [new branch] gh/aorenste/157/base -> origin/gh/aorenste/157/base 2025-12-04T09:43:53.3513891Z * [new branch] gh/aorenste/157/head -> origin/gh/aorenste/157/head 2025-12-04T09:43:53.3515662Z * [new branch] gh/aorenste/157/orig -> origin/gh/aorenste/157/orig 2025-12-04T09:43:53.3518001Z * [new branch] gh/aorenste/158/base -> origin/gh/aorenste/158/base 2025-12-04T09:43:53.3519793Z * [new branch] gh/aorenste/158/head -> origin/gh/aorenste/158/head 2025-12-04T09:43:53.3521492Z * [new branch] gh/aorenste/158/orig -> origin/gh/aorenste/158/orig 2025-12-04T09:43:53.3523740Z * [new branch] gh/aorenste/159/base -> origin/gh/aorenste/159/base 2025-12-04T09:43:53.3525647Z * [new branch] gh/aorenste/159/head -> origin/gh/aorenste/159/head 2025-12-04T09:43:53.3527268Z * [new branch] gh/aorenste/159/orig -> origin/gh/aorenste/159/orig 2025-12-04T09:43:53.3530213Z * [new branch] gh/avikchaudhuri/1/base -> origin/gh/avikchaudhuri/1/base 2025-12-04T09:43:53.3532006Z * [new branch] gh/avikchaudhuri/1/head -> origin/gh/avikchaudhuri/1/head 2025-12-04T09:43:53.3534191Z * [new branch] gh/avikchaudhuri/2/base -> origin/gh/avikchaudhuri/2/base 2025-12-04T09:43:53.3535848Z * [new branch] gh/avikchaudhuri/2/head -> origin/gh/avikchaudhuri/2/head 2025-12-04T09:43:53.3537610Z * [new branch] gh/avikchaudhuri/2/orig -> origin/gh/avikchaudhuri/2/orig 2025-12-04T09:43:53.3540797Z * [new branch] gh/bdhirsh/666/base -> origin/gh/bdhirsh/666/base 2025-12-04T09:43:53.3542477Z * [new branch] gh/bdhirsh/666/head -> origin/gh/bdhirsh/666/head 2025-12-04T09:43:53.3544330Z * [new branch] gh/bdhirsh/666/orig -> origin/gh/bdhirsh/666/orig 2025-12-04T09:43:53.3546843Z * [new branch] gh/bdhirsh/668/base -> origin/gh/bdhirsh/668/base 2025-12-04T09:43:53.3548623Z * [new branch] gh/bdhirsh/668/head -> origin/gh/bdhirsh/668/head 2025-12-04T09:43:53.3550357Z * [new branch] gh/bdhirsh/668/orig -> origin/gh/bdhirsh/668/orig 2025-12-04T09:43:53.3552830Z * [new branch] gh/bdhirsh/669/base -> origin/gh/bdhirsh/669/base 2025-12-04T09:43:53.3554512Z * [new branch] gh/bdhirsh/669/head -> origin/gh/bdhirsh/669/head 2025-12-04T09:43:53.3556260Z * [new branch] gh/bdhirsh/669/orig -> origin/gh/bdhirsh/669/orig 2025-12-04T09:43:53.3558782Z * [new branch] gh/bdhirsh/670/base -> origin/gh/bdhirsh/670/base 2025-12-04T09:43:53.3560600Z * [new branch] gh/bdhirsh/670/head -> origin/gh/bdhirsh/670/head 2025-12-04T09:43:53.3562370Z * [new branch] gh/bdhirsh/670/orig -> origin/gh/bdhirsh/670/orig 2025-12-04T09:43:53.3564802Z * [new branch] gh/bdhirsh/672/base -> origin/gh/bdhirsh/672/base 2025-12-04T09:43:53.3566504Z * [new branch] gh/bdhirsh/672/head -> origin/gh/bdhirsh/672/head 2025-12-04T09:43:53.3568218Z * [new branch] gh/bdhirsh/672/orig -> origin/gh/bdhirsh/672/orig 2025-12-04T09:43:53.3570861Z * [new branch] gh/bdhirsh/675/base -> origin/gh/bdhirsh/675/base 2025-12-04T09:43:53.3572737Z * [new branch] gh/bdhirsh/675/head -> origin/gh/bdhirsh/675/head 2025-12-04T09:43:53.3574548Z * [new branch] gh/bdhirsh/675/orig -> origin/gh/bdhirsh/675/orig 2025-12-04T09:43:53.3577012Z * [new branch] gh/bdhirsh/676/base -> origin/gh/bdhirsh/676/base 2025-12-04T09:43:53.3579015Z * [new branch] gh/bdhirsh/676/head -> origin/gh/bdhirsh/676/head 2025-12-04T09:43:53.3580757Z * [new branch] gh/bdhirsh/676/orig -> origin/gh/bdhirsh/676/orig 2025-12-04T09:43:53.3583130Z * [new branch] gh/bdhirsh/677/base -> origin/gh/bdhirsh/677/base 2025-12-04T09:43:53.3585353Z * [new branch] gh/bdhirsh/677/head -> origin/gh/bdhirsh/677/head 2025-12-04T09:43:53.3587106Z * [new branch] gh/bdhirsh/677/orig -> origin/gh/bdhirsh/677/orig 2025-12-04T09:43:53.3589531Z * [new branch] gh/bdhirsh/678/base -> origin/gh/bdhirsh/678/base 2025-12-04T09:43:53.3591404Z * [new branch] gh/bdhirsh/678/head -> origin/gh/bdhirsh/678/head 2025-12-04T09:43:53.3593173Z * [new branch] gh/bdhirsh/678/orig -> origin/gh/bdhirsh/678/orig 2025-12-04T09:43:53.3595712Z * [new branch] gh/bdhirsh/679/base -> origin/gh/bdhirsh/679/base 2025-12-04T09:43:53.3597536Z * [new branch] gh/bdhirsh/679/head -> origin/gh/bdhirsh/679/head 2025-12-04T09:43:53.3599319Z * [new branch] gh/bdhirsh/679/orig -> origin/gh/bdhirsh/679/orig 2025-12-04T09:43:53.3601657Z * [new branch] gh/bdhirsh/680/base -> origin/gh/bdhirsh/680/base 2025-12-04T09:43:53.3603481Z * [new branch] gh/bdhirsh/680/head -> origin/gh/bdhirsh/680/head 2025-12-04T09:43:53.3605225Z * [new branch] gh/bdhirsh/680/orig -> origin/gh/bdhirsh/680/orig 2025-12-04T09:43:53.3607479Z * [new branch] gh/bdhirsh/681/base -> origin/gh/bdhirsh/681/base 2025-12-04T09:43:53.3609373Z * [new branch] gh/bdhirsh/681/head -> origin/gh/bdhirsh/681/head 2025-12-04T09:43:53.3611263Z * [new branch] gh/bdhirsh/681/orig -> origin/gh/bdhirsh/681/orig 2025-12-04T09:43:53.3614030Z * [new branch] gh/benjaminglass1/101/base -> origin/gh/benjaminglass1/101/base 2025-12-04T09:43:53.3615762Z * [new branch] gh/benjaminglass1/101/head -> origin/gh/benjaminglass1/101/head 2025-12-04T09:43:53.3617513Z * [new branch] gh/benjaminglass1/101/orig -> origin/gh/benjaminglass1/101/orig 2025-12-04T09:43:53.3620119Z * [new branch] gh/benjaminglass1/102/base -> origin/gh/benjaminglass1/102/base 2025-12-04T09:43:53.3621929Z * [new branch] gh/benjaminglass1/102/head -> origin/gh/benjaminglass1/102/head 2025-12-04T09:43:53.3623699Z * [new branch] gh/benjaminglass1/102/orig -> origin/gh/benjaminglass1/102/orig 2025-12-04T09:43:53.3626236Z * [new branch] gh/benjaminglass1/106/base -> origin/gh/benjaminglass1/106/base 2025-12-04T09:43:53.3627909Z * [new branch] gh/benjaminglass1/106/head -> origin/gh/benjaminglass1/106/head 2025-12-04T09:43:53.3629612Z * [new branch] gh/benjaminglass1/106/orig -> origin/gh/benjaminglass1/106/orig 2025-12-04T09:43:53.3631933Z * [new branch] gh/benjaminglass1/107/base -> origin/gh/benjaminglass1/107/base 2025-12-04T09:43:53.3633625Z * [new branch] gh/benjaminglass1/107/head -> origin/gh/benjaminglass1/107/head 2025-12-04T09:43:53.3635446Z * [new branch] gh/benjaminglass1/107/orig -> origin/gh/benjaminglass1/107/orig 2025-12-04T09:43:53.3637796Z * [new branch] gh/benjaminglass1/108/base -> origin/gh/benjaminglass1/108/base 2025-12-04T09:43:53.3639680Z * [new branch] gh/benjaminglass1/108/head -> origin/gh/benjaminglass1/108/head 2025-12-04T09:43:53.3641427Z * [new branch] gh/benjaminglass1/108/orig -> origin/gh/benjaminglass1/108/orig 2025-12-04T09:43:53.3643973Z * [new branch] gh/benjaminglass1/109/base -> origin/gh/benjaminglass1/109/base 2025-12-04T09:43:53.3645602Z * [new branch] gh/benjaminglass1/109/head -> origin/gh/benjaminglass1/109/head 2025-12-04T09:43:53.3647386Z * [new branch] gh/benjaminglass1/109/orig -> origin/gh/benjaminglass1/109/orig 2025-12-04T09:43:53.3649744Z * [new branch] gh/benjaminglass1/97/base -> origin/gh/benjaminglass1/97/base 2025-12-04T09:43:53.3651500Z * [new branch] gh/benjaminglass1/97/head -> origin/gh/benjaminglass1/97/head 2025-12-04T09:43:53.3653260Z * [new branch] gh/benjaminglass1/97/orig -> origin/gh/benjaminglass1/97/orig 2025-12-04T09:43:53.3656047Z * [new branch] gh/bobrenjc93/570/base -> origin/gh/bobrenjc93/570/base 2025-12-04T09:43:53.3657793Z * [new branch] gh/bobrenjc93/570/head -> origin/gh/bobrenjc93/570/head 2025-12-04T09:43:53.3659542Z * [new branch] gh/bobrenjc93/570/orig -> origin/gh/bobrenjc93/570/orig 2025-12-04T09:43:53.3661747Z * [new branch] gh/bobrenjc93/604/base -> origin/gh/bobrenjc93/604/base 2025-12-04T09:43:53.3663610Z * [new branch] gh/bobrenjc93/604/head -> origin/gh/bobrenjc93/604/head 2025-12-04T09:43:53.3665524Z * [new branch] gh/bobrenjc93/604/orig -> origin/gh/bobrenjc93/604/orig 2025-12-04T09:43:53.3667828Z * [new branch] gh/bobrenjc93/638/base -> origin/gh/bobrenjc93/638/base 2025-12-04T09:43:53.3669590Z * [new branch] gh/bobrenjc93/638/head -> origin/gh/bobrenjc93/638/head 2025-12-04T09:43:53.3671299Z * [new branch] gh/bobrenjc93/638/orig -> origin/gh/bobrenjc93/638/orig 2025-12-04T09:43:53.3673634Z * [new branch] gh/bobrenjc93/653/base -> origin/gh/bobrenjc93/653/base 2025-12-04T09:43:53.3675354Z * [new branch] gh/bobrenjc93/653/head -> origin/gh/bobrenjc93/653/head 2025-12-04T09:43:53.3677137Z * [new branch] gh/bobrenjc93/653/orig -> origin/gh/bobrenjc93/653/orig 2025-12-04T09:43:53.3679994Z * [new branch] gh/bobrenjc93/654/base -> origin/gh/bobrenjc93/654/base 2025-12-04T09:43:53.3681678Z * [new branch] gh/bobrenjc93/654/head -> origin/gh/bobrenjc93/654/head 2025-12-04T09:43:53.3683346Z * [new branch] gh/bobrenjc93/654/orig -> origin/gh/bobrenjc93/654/orig 2025-12-04T09:43:53.3685734Z * [new branch] gh/bobrenjc93/657/base -> origin/gh/bobrenjc93/657/base 2025-12-04T09:43:53.3687509Z * [new branch] gh/bobrenjc93/657/head -> origin/gh/bobrenjc93/657/head 2025-12-04T09:43:53.3689302Z * [new branch] gh/bobrenjc93/657/orig -> origin/gh/bobrenjc93/657/orig 2025-12-04T09:43:53.3691647Z * [new branch] gh/bobrenjc93/672/base -> origin/gh/bobrenjc93/672/base 2025-12-04T09:43:53.3693310Z * [new branch] gh/bobrenjc93/672/head -> origin/gh/bobrenjc93/672/head 2025-12-04T09:43:53.3695062Z * [new branch] gh/bobrenjc93/672/orig -> origin/gh/bobrenjc93/672/orig 2025-12-04T09:43:53.3697458Z * [new branch] gh/bobrenjc93/679/base -> origin/gh/bobrenjc93/679/base 2025-12-04T09:43:53.3699308Z * [new branch] gh/bobrenjc93/679/head -> origin/gh/bobrenjc93/679/head 2025-12-04T09:43:53.3701173Z * [new branch] gh/bobrenjc93/679/orig -> origin/gh/bobrenjc93/679/orig 2025-12-04T09:43:53.3703703Z * [new branch] gh/bobrenjc93/680/base -> origin/gh/bobrenjc93/680/base 2025-12-04T09:43:53.3705508Z * [new branch] gh/bobrenjc93/680/head -> origin/gh/bobrenjc93/680/head 2025-12-04T09:43:53.3707264Z * [new branch] gh/bobrenjc93/680/orig -> origin/gh/bobrenjc93/680/orig 2025-12-04T09:43:53.3709453Z * [new branch] gh/bobrenjc93/681/base -> origin/gh/bobrenjc93/681/base 2025-12-04T09:43:53.3711203Z * [new branch] gh/bobrenjc93/681/head -> origin/gh/bobrenjc93/681/head 2025-12-04T09:43:53.3713033Z * [new branch] gh/bobrenjc93/681/orig -> origin/gh/bobrenjc93/681/orig 2025-12-04T09:43:53.3715293Z * [new branch] gh/bobrenjc93/682/base -> origin/gh/bobrenjc93/682/base 2025-12-04T09:43:53.3717036Z * [new branch] gh/bobrenjc93/682/head -> origin/gh/bobrenjc93/682/head 2025-12-04T09:43:53.3718784Z * [new branch] gh/bobrenjc93/682/orig -> origin/gh/bobrenjc93/682/orig 2025-12-04T09:43:53.3721117Z * [new branch] gh/bobrenjc93/683/base -> origin/gh/bobrenjc93/683/base 2025-12-04T09:43:53.3722913Z * [new branch] gh/bobrenjc93/683/head -> origin/gh/bobrenjc93/683/head 2025-12-04T09:43:53.3724621Z * [new branch] gh/bobrenjc93/683/orig -> origin/gh/bobrenjc93/683/orig 2025-12-04T09:43:53.3727037Z * [new branch] gh/bobrenjc93/684/base -> origin/gh/bobrenjc93/684/base 2025-12-04T09:43:53.3728918Z * [new branch] gh/bobrenjc93/684/head -> origin/gh/bobrenjc93/684/head 2025-12-04T09:43:53.3730784Z * [new branch] gh/bobrenjc93/684/orig -> origin/gh/bobrenjc93/684/orig 2025-12-04T09:43:53.3733024Z * [new branch] gh/bobrenjc93/685/base -> origin/gh/bobrenjc93/685/base 2025-12-04T09:43:53.3734960Z * [new branch] gh/bobrenjc93/685/head -> origin/gh/bobrenjc93/685/head 2025-12-04T09:43:53.3737050Z * [new branch] gh/bobrenjc93/685/orig -> origin/gh/bobrenjc93/685/orig 2025-12-04T09:43:53.3739568Z * [new branch] gh/bobrenjc93/686/base -> origin/gh/bobrenjc93/686/base 2025-12-04T09:43:53.3743018Z * [new branch] gh/bobrenjc93/686/head -> origin/gh/bobrenjc93/686/head 2025-12-04T09:43:53.3743425Z * [new branch] gh/bobrenjc93/686/orig -> origin/gh/bobrenjc93/686/orig 2025-12-04T09:43:53.3746000Z * [new branch] gh/bobrenjc93/687/base -> origin/gh/bobrenjc93/687/base 2025-12-04T09:43:53.3747972Z * [new branch] gh/bobrenjc93/687/head -> origin/gh/bobrenjc93/687/head 2025-12-04T09:43:53.3749719Z * [new branch] gh/bobrenjc93/687/orig -> origin/gh/bobrenjc93/687/orig 2025-12-04T09:43:53.3752531Z * [new branch] gh/bobrenjc93/688/base -> origin/gh/bobrenjc93/688/base 2025-12-04T09:43:53.3754268Z * [new branch] gh/bobrenjc93/688/head -> origin/gh/bobrenjc93/688/head 2025-12-04T09:43:53.3756027Z * [new branch] gh/bobrenjc93/688/orig -> origin/gh/bobrenjc93/688/orig 2025-12-04T09:43:53.3758374Z * [new branch] gh/bobrenjc93/689/base -> origin/gh/bobrenjc93/689/base 2025-12-04T09:43:53.3760072Z * [new branch] gh/bobrenjc93/689/head -> origin/gh/bobrenjc93/689/head 2025-12-04T09:43:53.3761930Z * [new branch] gh/bobrenjc93/689/orig -> origin/gh/bobrenjc93/689/orig 2025-12-04T09:43:53.3764315Z * [new branch] gh/bobrenjc93/690/base -> origin/gh/bobrenjc93/690/base 2025-12-04T09:43:53.3766071Z * [new branch] gh/bobrenjc93/690/head -> origin/gh/bobrenjc93/690/head 2025-12-04T09:43:53.3767856Z * [new branch] gh/bobrenjc93/690/orig -> origin/gh/bobrenjc93/690/orig 2025-12-04T09:43:53.3770787Z * [new branch] gh/bobrenjc93/691/base -> origin/gh/bobrenjc93/691/base 2025-12-04T09:43:53.3772730Z * [new branch] gh/bobrenjc93/691/head -> origin/gh/bobrenjc93/691/head 2025-12-04T09:43:53.3774854Z * [new branch] gh/bobrenjc93/691/orig -> origin/gh/bobrenjc93/691/orig 2025-12-04T09:43:53.3777779Z * [new branch] gh/bobrenjc93/692/base -> origin/gh/bobrenjc93/692/base 2025-12-04T09:43:53.3781756Z * [new branch] gh/bobrenjc93/692/head -> origin/gh/bobrenjc93/692/head 2025-12-04T09:43:53.3783511Z * [new branch] gh/bobrenjc93/692/orig -> origin/gh/bobrenjc93/692/orig 2025-12-04T09:43:53.3785892Z * [new branch] gh/bobrenjc93/693/base -> origin/gh/bobrenjc93/693/base 2025-12-04T09:43:53.3787658Z * [new branch] gh/bobrenjc93/693/head -> origin/gh/bobrenjc93/693/head 2025-12-04T09:43:53.3789527Z * [new branch] gh/bobrenjc93/693/orig -> origin/gh/bobrenjc93/693/orig 2025-12-04T09:43:53.3791926Z * [new branch] gh/bobrenjc93/694/base -> origin/gh/bobrenjc93/694/base 2025-12-04T09:43:53.3793692Z * [new branch] gh/bobrenjc93/694/head -> origin/gh/bobrenjc93/694/head 2025-12-04T09:43:53.3795469Z * [new branch] gh/bobrenjc93/694/orig -> origin/gh/bobrenjc93/694/orig 2025-12-04T09:43:53.3797746Z * [new branch] gh/bobrenjc93/695/base -> origin/gh/bobrenjc93/695/base 2025-12-04T09:43:53.3799584Z * [new branch] gh/bobrenjc93/695/head -> origin/gh/bobrenjc93/695/head 2025-12-04T09:43:53.3801336Z * [new branch] gh/bobrenjc93/695/orig -> origin/gh/bobrenjc93/695/orig 2025-12-04T09:43:53.3804307Z * [new branch] gh/c00w/23/base -> origin/gh/c00w/23/base 2025-12-04T09:43:53.3806055Z * [new branch] gh/c00w/23/head -> origin/gh/c00w/23/head 2025-12-04T09:43:53.3808526Z * [new branch] gh/c00w/53/base -> origin/gh/c00w/53/base 2025-12-04T09:43:53.3810074Z * [new branch] gh/c00w/53/head -> origin/gh/c00w/53/head 2025-12-04T09:43:53.3811973Z * [new branch] gh/c00w/53/orig -> origin/gh/c00w/53/orig 2025-12-04T09:43:53.3814237Z * [new branch] gh/c00w/54/base -> origin/gh/c00w/54/base 2025-12-04T09:43:53.3816080Z * [new branch] gh/c00w/54/head -> origin/gh/c00w/54/head 2025-12-04T09:43:53.3817897Z * [new branch] gh/c00w/54/orig -> origin/gh/c00w/54/orig 2025-12-04T09:43:53.3820237Z * [new branch] gh/c00w/56/base -> origin/gh/c00w/56/base 2025-12-04T09:43:53.3822155Z * [new branch] gh/c00w/56/head -> origin/gh/c00w/56/head 2025-12-04T09:43:53.3823797Z * [new branch] gh/c00w/56/orig -> origin/gh/c00w/56/orig 2025-12-04T09:43:53.3826205Z * [new branch] gh/c00w/57/base -> origin/gh/c00w/57/base 2025-12-04T09:43:53.3828166Z * [new branch] gh/c00w/57/head -> origin/gh/c00w/57/head 2025-12-04T09:43:53.3830100Z * [new branch] gh/c00w/57/orig -> origin/gh/c00w/57/orig 2025-12-04T09:43:53.3832391Z * [new branch] gh/c00w/58/base -> origin/gh/c00w/58/base 2025-12-04T09:43:53.3834134Z * [new branch] gh/c00w/58/head -> origin/gh/c00w/58/head 2025-12-04T09:43:53.3835906Z * [new branch] gh/c00w/58/orig -> origin/gh/c00w/58/orig 2025-12-04T09:43:53.3838719Z * [new branch] gh/clee2000/1/base -> origin/gh/clee2000/1/base 2025-12-04T09:43:53.3840517Z * [new branch] gh/clee2000/1/head -> origin/gh/clee2000/1/head 2025-12-04T09:43:53.3842223Z * [new branch] gh/clee2000/1/orig -> origin/gh/clee2000/1/orig 2025-12-04T09:43:53.3845216Z * [new branch] gh/coconutruben/1/base -> origin/gh/coconutruben/1/base 2025-12-04T09:43:53.3847017Z * [new branch] gh/coconutruben/1/head -> origin/gh/coconutruben/1/head 2025-12-04T09:43:53.3849641Z * [new branch] gh/coconutruben/55/base -> origin/gh/coconutruben/55/base 2025-12-04T09:43:53.3851324Z * [new branch] gh/coconutruben/55/head -> origin/gh/coconutruben/55/head 2025-12-04T09:43:53.3853107Z * [new branch] gh/coconutruben/55/orig -> origin/gh/coconutruben/55/orig 2025-12-04T09:43:53.3855559Z * [new branch] gh/coconutruben/57/base -> origin/gh/coconutruben/57/base 2025-12-04T09:43:53.3857517Z * [new branch] gh/coconutruben/57/head -> origin/gh/coconutruben/57/head 2025-12-04T09:43:53.3859341Z * [new branch] gh/coconutruben/57/orig -> origin/gh/coconutruben/57/orig 2025-12-04T09:43:53.3861717Z * [new branch] gh/coconutruben/70/base -> origin/gh/coconutruben/70/base 2025-12-04T09:43:53.3863516Z * [new branch] gh/coconutruben/70/head -> origin/gh/coconutruben/70/head 2025-12-04T09:43:53.3865466Z * [new branch] gh/coconutruben/70/orig -> origin/gh/coconutruben/70/orig 2025-12-04T09:43:53.3867645Z * [new branch] gh/coconutruben/71/base -> origin/gh/coconutruben/71/base 2025-12-04T09:43:53.3869476Z * [new branch] gh/coconutruben/71/head -> origin/gh/coconutruben/71/head 2025-12-04T09:43:53.3871228Z * [new branch] gh/coconutruben/71/orig -> origin/gh/coconutruben/71/orig 2025-12-04T09:43:53.3873659Z * [new branch] gh/coconutruben/72/base -> origin/gh/coconutruben/72/base 2025-12-04T09:43:53.3875299Z * [new branch] gh/coconutruben/72/head -> origin/gh/coconutruben/72/head 2025-12-04T09:43:53.3877016Z * [new branch] gh/coconutruben/72/orig -> origin/gh/coconutruben/72/orig 2025-12-04T09:43:53.3879452Z * [new branch] gh/coconutruben/73/base -> origin/gh/coconutruben/73/base 2025-12-04T09:43:53.3881268Z * [new branch] gh/coconutruben/73/head -> origin/gh/coconutruben/73/head 2025-12-04T09:43:53.3883010Z * [new branch] gh/coconutruben/73/orig -> origin/gh/coconutruben/73/orig 2025-12-04T09:43:53.3885534Z * [new branch] gh/coconutruben/74/base -> origin/gh/coconutruben/74/base 2025-12-04T09:43:53.3887419Z * [new branch] gh/coconutruben/74/head -> origin/gh/coconutruben/74/head 2025-12-04T09:43:53.3889167Z * [new branch] gh/coconutruben/74/orig -> origin/gh/coconutruben/74/orig 2025-12-04T09:43:53.3891892Z * [new branch] gh/coconutruben/79/base -> origin/gh/coconutruben/79/base 2025-12-04T09:43:53.3893753Z * [new branch] gh/coconutruben/79/head -> origin/gh/coconutruben/79/head 2025-12-04T09:43:53.3921732Z * [new branch] gh/coconutruben/79/orig -> origin/gh/coconutruben/79/orig 2025-12-04T09:43:53.3922311Z * [new branch] gh/coconutruben/80/base -> origin/gh/coconutruben/80/base 2025-12-04T09:43:53.3922822Z * [new branch] gh/coconutruben/80/head -> origin/gh/coconutruben/80/head 2025-12-04T09:43:53.3923324Z * [new branch] gh/coconutruben/80/orig -> origin/gh/coconutruben/80/orig 2025-12-04T09:43:53.3924071Z * [new branch] gh/coconutruben/82/base -> origin/gh/coconutruben/82/base 2025-12-04T09:43:53.3924614Z * [new branch] gh/coconutruben/82/head -> origin/gh/coconutruben/82/head 2025-12-04T09:43:53.3925145Z * [new branch] gh/coconutruben/82/orig -> origin/gh/coconutruben/82/orig 2025-12-04T09:43:53.3925623Z * [new branch] gh/coconutruben/83/base -> origin/gh/coconutruben/83/base 2025-12-04T09:43:53.3926106Z * [new branch] gh/coconutruben/83/head -> origin/gh/coconutruben/83/head 2025-12-04T09:43:53.3926593Z * [new branch] gh/coconutruben/83/orig -> origin/gh/coconutruben/83/orig 2025-12-04T09:43:53.3927075Z * [new branch] gh/coconutruben/84/base -> origin/gh/coconutruben/84/base 2025-12-04T09:43:53.3927555Z * [new branch] gh/coconutruben/84/head -> origin/gh/coconutruben/84/head 2025-12-04T09:43:53.3928040Z * [new branch] gh/coconutruben/84/orig -> origin/gh/coconutruben/84/orig 2025-12-04T09:43:53.3928526Z * [new branch] gh/coconutruben/85/base -> origin/gh/coconutruben/85/base 2025-12-04T09:43:53.3929038Z * [new branch] gh/coconutruben/85/head -> origin/gh/coconutruben/85/head 2025-12-04T09:43:53.3929520Z * [new branch] gh/coconutruben/85/orig -> origin/gh/coconutruben/85/orig 2025-12-04T09:43:53.3929938Z * [new branch] gh/coconutruben/86/base -> origin/gh/coconutruben/86/base 2025-12-04T09:43:53.3930325Z * [new branch] gh/coconutruben/86/head -> origin/gh/coconutruben/86/head 2025-12-04T09:43:53.3931291Z * [new branch] gh/coconutruben/86/orig -> origin/gh/coconutruben/86/orig 2025-12-04T09:43:53.3934315Z * [new branch] gh/colinchan15/1/base -> origin/gh/colinchan15/1/base 2025-12-04T09:43:53.3936192Z * [new branch] gh/colinchan15/1/head -> origin/gh/colinchan15/1/head 2025-12-04T09:43:53.3938254Z * [new branch] gh/colinchan15/2/base -> origin/gh/colinchan15/2/base 2025-12-04T09:43:53.3940005Z * [new branch] gh/colinchan15/2/head -> origin/gh/colinchan15/2/head 2025-12-04T09:43:53.3942270Z * [new branch] gh/colinchan15/3/base -> origin/gh/colinchan15/3/base 2025-12-04T09:43:53.3944052Z * [new branch] gh/colinchan15/3/head -> origin/gh/colinchan15/3/head 2025-12-04T09:43:53.3946309Z * [new branch] gh/colinchan15/6/base -> origin/gh/colinchan15/6/base 2025-12-04T09:43:53.3947973Z * [new branch] gh/colinchan15/6/head -> origin/gh/colinchan15/6/head 2025-12-04T09:43:53.3950758Z * [new branch] gh/d4l3k/1/base -> origin/gh/d4l3k/1/base 2025-12-04T09:43:53.3952517Z * [new branch] gh/d4l3k/1/head -> origin/gh/d4l3k/1/head 2025-12-04T09:43:53.3954898Z * [new branch] gh/d4l3k/2/base -> origin/gh/d4l3k/2/base 2025-12-04T09:43:53.3956614Z * [new branch] gh/d4l3k/2/head -> origin/gh/d4l3k/2/head 2025-12-04T09:43:53.3958427Z * [new branch] gh/d4l3k/2/orig -> origin/gh/d4l3k/2/orig 2025-12-04T09:43:53.3960840Z * [new branch] gh/d4l3k/3/base -> origin/gh/d4l3k/3/base 2025-12-04T09:43:53.3962600Z * [new branch] gh/d4l3k/3/head -> origin/gh/d4l3k/3/head 2025-12-04T09:43:53.3964488Z * [new branch] gh/d4l3k/3/orig -> origin/gh/d4l3k/3/orig 2025-12-04T09:43:53.3966744Z * [new branch] gh/d4l3k/4/base -> origin/gh/d4l3k/4/base 2025-12-04T09:43:53.3968448Z * [new branch] gh/d4l3k/4/head -> origin/gh/d4l3k/4/head 2025-12-04T09:43:53.3970198Z * [new branch] gh/d4l3k/4/orig -> origin/gh/d4l3k/4/orig 2025-12-04T09:43:53.3973067Z * [new branch] gh/d4l3k/5/base -> origin/gh/d4l3k/5/base 2025-12-04T09:43:53.3974813Z * [new branch] gh/d4l3k/5/orig -> origin/gh/d4l3k/5/orig 2025-12-04T09:43:53.3977770Z * [new branch] gh/davidberard98/392/base -> origin/gh/davidberard98/392/base 2025-12-04T09:43:53.3979754Z * [new branch] gh/davidberard98/392/head -> origin/gh/davidberard98/392/head 2025-12-04T09:43:53.3981506Z * [new branch] gh/davidberard98/392/orig -> origin/gh/davidberard98/392/orig 2025-12-04T09:43:53.3984018Z * [new branch] gh/davidberard98/399/base -> origin/gh/davidberard98/399/base 2025-12-04T09:43:53.3985990Z * [new branch] gh/davidberard98/399/head -> origin/gh/davidberard98/399/head 2025-12-04T09:43:53.3987769Z * [new branch] gh/davidberard98/399/orig -> origin/gh/davidberard98/399/orig 2025-12-04T09:43:53.3990596Z * [new branch] gh/desertfire/605/base -> origin/gh/desertfire/605/base 2025-12-04T09:43:53.3992355Z * [new branch] gh/desertfire/605/head -> origin/gh/desertfire/605/head 2025-12-04T09:43:53.3994079Z * [new branch] gh/desertfire/605/orig -> origin/gh/desertfire/605/orig 2025-12-04T09:43:53.3996951Z * [new branch] gh/desertfire/606/base -> origin/gh/desertfire/606/base 2025-12-04T09:43:53.3998680Z * [new branch] gh/desertfire/606/head -> origin/gh/desertfire/606/head 2025-12-04T09:43:53.4000643Z * [new branch] gh/desertfire/606/orig -> origin/gh/desertfire/606/orig 2025-12-04T09:43:53.4002929Z * [new branch] gh/desertfire/607/base -> origin/gh/desertfire/607/base 2025-12-04T09:43:53.4004648Z * [new branch] gh/desertfire/607/head -> origin/gh/desertfire/607/head 2025-12-04T09:43:53.4006434Z * [new branch] gh/desertfire/607/orig -> origin/gh/desertfire/607/orig 2025-12-04T09:43:53.4008725Z * [new branch] gh/desertfire/608/base -> origin/gh/desertfire/608/base 2025-12-04T09:43:53.4010452Z * [new branch] gh/desertfire/608/head -> origin/gh/desertfire/608/head 2025-12-04T09:43:53.4012230Z * [new branch] gh/desertfire/608/orig -> origin/gh/desertfire/608/orig 2025-12-04T09:43:53.4014542Z * [new branch] gh/desertfire/609/base -> origin/gh/desertfire/609/base 2025-12-04T09:43:53.4016323Z * [new branch] gh/desertfire/609/head -> origin/gh/desertfire/609/head 2025-12-04T09:43:53.4018117Z * [new branch] gh/desertfire/609/orig -> origin/gh/desertfire/609/orig 2025-12-04T09:43:53.4020642Z * [new branch] gh/desertfire/610/base -> origin/gh/desertfire/610/base 2025-12-04T09:43:53.4022482Z * [new branch] gh/desertfire/610/head -> origin/gh/desertfire/610/head 2025-12-04T09:43:53.4024444Z * [new branch] gh/desertfire/610/orig -> origin/gh/desertfire/610/orig 2025-12-04T09:43:53.4026696Z * [new branch] gh/desertfire/611/base -> origin/gh/desertfire/611/base 2025-12-04T09:43:53.4028479Z * [new branch] gh/desertfire/611/head -> origin/gh/desertfire/611/head 2025-12-04T09:43:53.4030151Z * [new branch] gh/desertfire/611/orig -> origin/gh/desertfire/611/orig 2025-12-04T09:43:53.4032569Z * [new branch] gh/desertfire/612/base -> origin/gh/desertfire/612/base 2025-12-04T09:43:53.4034502Z * [new branch] gh/desertfire/612/head -> origin/gh/desertfire/612/head 2025-12-04T09:43:53.4036098Z * [new branch] gh/desertfire/612/orig -> origin/gh/desertfire/612/orig 2025-12-04T09:43:53.4038692Z * [new branch] gh/desertfire/613/base -> origin/gh/desertfire/613/base 2025-12-04T09:43:53.4040515Z * [new branch] gh/desertfire/613/head -> origin/gh/desertfire/613/head 2025-12-04T09:43:53.4042362Z * [new branch] gh/desertfire/613/orig -> origin/gh/desertfire/613/orig 2025-12-04T09:43:53.4044879Z * [new branch] gh/desertfire/614/base -> origin/gh/desertfire/614/base 2025-12-04T09:43:53.4046760Z * [new branch] gh/desertfire/614/head -> origin/gh/desertfire/614/head 2025-12-04T09:43:53.4048491Z * [new branch] gh/desertfire/614/orig -> origin/gh/desertfire/614/orig 2025-12-04T09:43:53.4050866Z * [new branch] gh/desertfire/615/base -> origin/gh/desertfire/615/base 2025-12-04T09:43:53.4052850Z * [new branch] gh/desertfire/615/head -> origin/gh/desertfire/615/head 2025-12-04T09:43:53.4054526Z * [new branch] gh/desertfire/615/orig -> origin/gh/desertfire/615/orig 2025-12-04T09:43:53.4056735Z * [new branch] gh/desertfire/616/base -> origin/gh/desertfire/616/base 2025-12-04T09:43:53.4058562Z * [new branch] gh/desertfire/616/head -> origin/gh/desertfire/616/head 2025-12-04T09:43:53.4060270Z * [new branch] gh/desertfire/616/orig -> origin/gh/desertfire/616/orig 2025-12-04T09:43:53.4062488Z * [new branch] gh/desertfire/617/base -> origin/gh/desertfire/617/base 2025-12-04T09:43:53.4064441Z * [new branch] gh/desertfire/617/head -> origin/gh/desertfire/617/head 2025-12-04T09:43:53.4066087Z * [new branch] gh/desertfire/617/orig -> origin/gh/desertfire/617/orig 2025-12-04T09:43:53.4069008Z * [new branch] gh/dharakk/1/base -> origin/gh/dharakk/1/base 2025-12-04T09:43:53.4070845Z * [new branch] gh/dharakk/1/head -> origin/gh/dharakk/1/head 2025-12-04T09:43:53.4073713Z * [new branch] gh/drisspg/170/base -> origin/gh/drisspg/170/base 2025-12-04T09:43:53.4075519Z * [new branch] gh/drisspg/170/head -> origin/gh/drisspg/170/head 2025-12-04T09:43:53.4077301Z * [new branch] gh/drisspg/170/orig -> origin/gh/drisspg/170/orig 2025-12-04T09:43:53.4079910Z * [new branch] gh/drisspg/182/base -> origin/gh/drisspg/182/base 2025-12-04T09:43:53.4081649Z * [new branch] gh/drisspg/182/head -> origin/gh/drisspg/182/head 2025-12-04T09:43:53.4083843Z * [new branch] gh/drisspg/183/base -> origin/gh/drisspg/183/base 2025-12-04T09:43:53.4085616Z * [new branch] gh/drisspg/183/head -> origin/gh/drisspg/183/head 2025-12-04T09:43:53.4087887Z * [new branch] gh/drisspg/184/base -> origin/gh/drisspg/184/base 2025-12-04T09:43:53.4089546Z * [new branch] gh/drisspg/184/head -> origin/gh/drisspg/184/head 2025-12-04T09:43:53.4091912Z * [new branch] gh/drisspg/185/base -> origin/gh/drisspg/185/base 2025-12-04T09:43:53.4093615Z * [new branch] gh/drisspg/185/head -> origin/gh/drisspg/185/head 2025-12-04T09:43:53.4095920Z * [new branch] gh/drisspg/194/base -> origin/gh/drisspg/194/base 2025-12-04T09:43:53.4097664Z * [new branch] gh/drisspg/194/head -> origin/gh/drisspg/194/head 2025-12-04T09:43:53.4099366Z * [new branch] gh/drisspg/194/orig -> origin/gh/drisspg/194/orig 2025-12-04T09:43:53.4101681Z * [new branch] gh/drisspg/200/base -> origin/gh/drisspg/200/base 2025-12-04T09:43:53.4103427Z * [new branch] gh/drisspg/200/head -> origin/gh/drisspg/200/head 2025-12-04T09:43:53.4105529Z * [new branch] gh/drisspg/200/orig -> origin/gh/drisspg/200/orig 2025-12-04T09:43:53.4107633Z * [new branch] gh/drisspg/218/base -> origin/gh/drisspg/218/base 2025-12-04T09:43:53.4109335Z * [new branch] gh/drisspg/218/head -> origin/gh/drisspg/218/head 2025-12-04T09:43:53.4111203Z * [new branch] gh/drisspg/218/orig -> origin/gh/drisspg/218/orig 2025-12-04T09:43:53.4113566Z * [new branch] gh/drisspg/219/base -> origin/gh/drisspg/219/base 2025-12-04T09:43:53.4115317Z * [new branch] gh/drisspg/219/head -> origin/gh/drisspg/219/head 2025-12-04T09:43:53.4117089Z * [new branch] gh/drisspg/219/orig -> origin/gh/drisspg/219/orig 2025-12-04T09:43:53.4119376Z * [new branch] gh/drisspg/220/base -> origin/gh/drisspg/220/base 2025-12-04T09:43:53.4121067Z * [new branch] gh/drisspg/220/head -> origin/gh/drisspg/220/head 2025-12-04T09:43:53.4122778Z * [new branch] gh/drisspg/220/orig -> origin/gh/drisspg/220/orig 2025-12-04T09:43:53.4125145Z * [new branch] gh/drisspg/221/base -> origin/gh/drisspg/221/base 2025-12-04T09:43:53.4126848Z * [new branch] gh/drisspg/221/head -> origin/gh/drisspg/221/head 2025-12-04T09:43:53.4128583Z * [new branch] gh/drisspg/221/orig -> origin/gh/drisspg/221/orig 2025-12-04T09:43:53.4130957Z * [new branch] gh/drisspg/222/base -> origin/gh/drisspg/222/base 2025-12-04T09:43:53.4132690Z * [new branch] gh/drisspg/222/head -> origin/gh/drisspg/222/head 2025-12-04T09:43:53.4134666Z * [new branch] gh/drisspg/222/orig -> origin/gh/drisspg/222/orig 2025-12-04T09:43:53.4136896Z * [new branch] gh/drisspg/223/base -> origin/gh/drisspg/223/base 2025-12-04T09:43:53.4138574Z * [new branch] gh/drisspg/223/head -> origin/gh/drisspg/223/head 2025-12-04T09:43:53.4140287Z * [new branch] gh/drisspg/223/orig -> origin/gh/drisspg/223/orig 2025-12-04T09:43:53.4142623Z * [new branch] gh/drisspg/224/base -> origin/gh/drisspg/224/base 2025-12-04T09:43:53.4144450Z * [new branch] gh/drisspg/224/head -> origin/gh/drisspg/224/head 2025-12-04T09:43:53.4146240Z * [new branch] gh/drisspg/224/orig -> origin/gh/drisspg/224/orig 2025-12-04T09:43:53.4148548Z * [new branch] gh/drisspg/225/base -> origin/gh/drisspg/225/base 2025-12-04T09:43:53.4150247Z * [new branch] gh/drisspg/225/head -> origin/gh/drisspg/225/head 2025-12-04T09:43:53.4152018Z * [new branch] gh/drisspg/225/orig -> origin/gh/drisspg/225/orig 2025-12-04T09:43:53.4154316Z * [new branch] gh/drisspg/226/base -> origin/gh/drisspg/226/base 2025-12-04T09:43:53.4156009Z * [new branch] gh/drisspg/226/head -> origin/gh/drisspg/226/head 2025-12-04T09:43:53.4157851Z * [new branch] gh/drisspg/226/orig -> origin/gh/drisspg/226/orig 2025-12-04T09:43:53.4160673Z * [new branch] gh/drisspg/227/base -> origin/gh/drisspg/227/base 2025-12-04T09:43:53.4162427Z * [new branch] gh/drisspg/227/head -> origin/gh/drisspg/227/head 2025-12-04T09:43:53.4164173Z * [new branch] gh/drisspg/227/orig -> origin/gh/drisspg/227/orig 2025-12-04T09:43:53.4166581Z * [new branch] gh/drisspg/228/base -> origin/gh/drisspg/228/base 2025-12-04T09:43:53.4168335Z * [new branch] gh/drisspg/228/head -> origin/gh/drisspg/228/head 2025-12-04T09:43:53.4170031Z * [new branch] gh/drisspg/228/orig -> origin/gh/drisspg/228/orig 2025-12-04T09:43:53.4172472Z * [new branch] gh/drisspg/229/base -> origin/gh/drisspg/229/base 2025-12-04T09:43:53.4174175Z * [new branch] gh/drisspg/229/head -> origin/gh/drisspg/229/head 2025-12-04T09:43:53.4176000Z * [new branch] gh/drisspg/229/orig -> origin/gh/drisspg/229/orig 2025-12-04T09:43:53.4178631Z * [new branch] gh/drisspg/230/base -> origin/gh/drisspg/230/base 2025-12-04T09:43:53.4182156Z * [new branch] gh/drisspg/230/head -> origin/gh/drisspg/230/head 2025-12-04T09:43:53.4184132Z * [new branch] gh/drisspg/230/orig -> origin/gh/drisspg/230/orig 2025-12-04T09:43:53.4187078Z * [new branch] gh/dsjohns2/1/base -> origin/gh/dsjohns2/1/base 2025-12-04T09:43:53.4188882Z * [new branch] gh/dsjohns2/1/head -> origin/gh/dsjohns2/1/head 2025-12-04T09:43:53.4191710Z * [new branch] gh/dzmitry-huba/1/base -> origin/gh/dzmitry-huba/1/base 2025-12-04T09:43:53.4193688Z * [new branch] gh/dzmitry-huba/1/head -> origin/gh/dzmitry-huba/1/head 2025-12-04T09:43:53.4196104Z * [new branch] gh/dzmitry-huba/12/base -> origin/gh/dzmitry-huba/12/base 2025-12-04T09:43:53.4197892Z * [new branch] gh/dzmitry-huba/12/head -> origin/gh/dzmitry-huba/12/head 2025-12-04T09:43:53.4199627Z * [new branch] gh/dzmitry-huba/12/orig -> origin/gh/dzmitry-huba/12/orig 2025-12-04T09:43:53.4201963Z * [new branch] gh/dzmitry-huba/13/base -> origin/gh/dzmitry-huba/13/base 2025-12-04T09:43:53.4203841Z * [new branch] gh/dzmitry-huba/13/head -> origin/gh/dzmitry-huba/13/head 2025-12-04T09:43:53.4205665Z * [new branch] gh/dzmitry-huba/13/orig -> origin/gh/dzmitry-huba/13/orig 2025-12-04T09:43:53.4207963Z * [new branch] gh/dzmitry-huba/14/base -> origin/gh/dzmitry-huba/14/base 2025-12-04T09:43:53.4209709Z * [new branch] gh/dzmitry-huba/14/head -> origin/gh/dzmitry-huba/14/head 2025-12-04T09:43:53.4211572Z * [new branch] gh/dzmitry-huba/14/orig -> origin/gh/dzmitry-huba/14/orig 2025-12-04T09:43:53.4214511Z * [new branch] gh/dzmitry-huba/15/base -> origin/gh/dzmitry-huba/15/base 2025-12-04T09:43:53.4216186Z * [new branch] gh/dzmitry-huba/15/head -> origin/gh/dzmitry-huba/15/head 2025-12-04T09:43:53.4218010Z * [new branch] gh/dzmitry-huba/15/orig -> origin/gh/dzmitry-huba/15/orig 2025-12-04T09:43:53.4220461Z * [new branch] gh/dzmitry-huba/16/base -> origin/gh/dzmitry-huba/16/base 2025-12-04T09:43:53.4222309Z * [new branch] gh/dzmitry-huba/16/head -> origin/gh/dzmitry-huba/16/head 2025-12-04T09:43:53.4224223Z * [new branch] gh/dzmitry-huba/16/orig -> origin/gh/dzmitry-huba/16/orig 2025-12-04T09:43:53.4226588Z * [new branch] gh/dzmitry-huba/17/base -> origin/gh/dzmitry-huba/17/base 2025-12-04T09:43:53.4228260Z * [new branch] gh/dzmitry-huba/17/head -> origin/gh/dzmitry-huba/17/head 2025-12-04T09:43:53.4230123Z * [new branch] gh/dzmitry-huba/17/orig -> origin/gh/dzmitry-huba/17/orig 2025-12-04T09:43:53.4232316Z * [new branch] gh/dzmitry-huba/2/base -> origin/gh/dzmitry-huba/2/base 2025-12-04T09:43:53.4234024Z * [new branch] gh/dzmitry-huba/2/head -> origin/gh/dzmitry-huba/2/head 2025-12-04T09:43:53.4236242Z * [new branch] gh/dzmitry-huba/3/base -> origin/gh/dzmitry-huba/3/base 2025-12-04T09:43:53.4237886Z * [new branch] gh/dzmitry-huba/3/head -> origin/gh/dzmitry-huba/3/head 2025-12-04T09:43:53.4240676Z * [new branch] gh/eellison/808/base -> origin/gh/eellison/808/base 2025-12-04T09:43:53.4242588Z * [new branch] gh/eellison/808/head -> origin/gh/eellison/808/head 2025-12-04T09:43:53.4244280Z * [new branch] gh/eellison/808/orig -> origin/gh/eellison/808/orig 2025-12-04T09:43:53.4246932Z * [new branch] gh/eellison/822/base -> origin/gh/eellison/822/base 2025-12-04T09:43:53.4248967Z * [new branch] gh/eellison/822/head -> origin/gh/eellison/822/head 2025-12-04T09:43:53.4250677Z * [new branch] gh/eellison/822/orig -> origin/gh/eellison/822/orig 2025-12-04T09:43:53.4253131Z * [new branch] gh/eellison/823/base -> origin/gh/eellison/823/base 2025-12-04T09:43:53.4254850Z * [new branch] gh/eellison/823/head -> origin/gh/eellison/823/head 2025-12-04T09:43:53.4256612Z * [new branch] gh/eellison/823/orig -> origin/gh/eellison/823/orig 2025-12-04T09:43:53.4259008Z * [new branch] gh/eellison/862/base -> origin/gh/eellison/862/base 2025-12-04T09:43:53.4260611Z * [new branch] gh/eellison/862/head -> origin/gh/eellison/862/head 2025-12-04T09:43:53.4262310Z * [new branch] gh/eellison/862/orig -> origin/gh/eellison/862/orig 2025-12-04T09:43:53.4264829Z * [new branch] gh/eellison/863/base -> origin/gh/eellison/863/base 2025-12-04T09:43:53.4266532Z * [new branch] gh/eellison/863/head -> origin/gh/eellison/863/head 2025-12-04T09:43:53.4268333Z * [new branch] gh/eellison/863/orig -> origin/gh/eellison/863/orig 2025-12-04T09:43:53.4270569Z * [new branch] gh/eellison/864/base -> origin/gh/eellison/864/base 2025-12-04T09:43:53.4272384Z * [new branch] gh/eellison/864/head -> origin/gh/eellison/864/head 2025-12-04T09:43:53.4274366Z * [new branch] gh/eellison/864/orig -> origin/gh/eellison/864/orig 2025-12-04T09:43:53.4276904Z * [new branch] gh/eellison/865/base -> origin/gh/eellison/865/base 2025-12-04T09:43:53.4278908Z * [new branch] gh/eellison/865/head -> origin/gh/eellison/865/head 2025-12-04T09:43:53.4280820Z * [new branch] gh/eellison/865/orig -> origin/gh/eellison/865/orig 2025-12-04T09:43:53.4283111Z * [new branch] gh/eellison/866/base -> origin/gh/eellison/866/base 2025-12-04T09:43:53.4284862Z * [new branch] gh/eellison/866/head -> origin/gh/eellison/866/head 2025-12-04T09:43:53.4286607Z * [new branch] gh/eellison/866/orig -> origin/gh/eellison/866/orig 2025-12-04T09:43:53.4289011Z * [new branch] gh/eellison/867/base -> origin/gh/eellison/867/base 2025-12-04T09:43:53.4290829Z * [new branch] gh/eellison/867/head -> origin/gh/eellison/867/head 2025-12-04T09:43:53.4292685Z * [new branch] gh/eellison/867/orig -> origin/gh/eellison/867/orig 2025-12-04T09:43:53.4295180Z * [new branch] gh/eellison/868/base -> origin/gh/eellison/868/base 2025-12-04T09:43:53.4297203Z * [new branch] gh/eellison/868/head -> origin/gh/eellison/868/head 2025-12-04T09:43:53.4298934Z * [new branch] gh/eellison/868/orig -> origin/gh/eellison/868/orig 2025-12-04T09:43:53.4301362Z * [new branch] gh/eellison/869/base -> origin/gh/eellison/869/base 2025-12-04T09:43:53.4303721Z * [new branch] gh/eellison/869/head -> origin/gh/eellison/869/head 2025-12-04T09:43:53.4305538Z * [new branch] gh/eellison/869/orig -> origin/gh/eellison/869/orig 2025-12-04T09:43:53.4307907Z * [new branch] gh/eellison/870/base -> origin/gh/eellison/870/base 2025-12-04T09:43:53.4309715Z * [new branch] gh/eellison/870/head -> origin/gh/eellison/870/head 2025-12-04T09:43:53.4311294Z * [new branch] gh/eellison/870/orig -> origin/gh/eellison/870/orig 2025-12-04T09:43:53.4314050Z * [new branch] gh/eellison/871/base -> origin/gh/eellison/871/base 2025-12-04T09:43:53.4315374Z * [new branch] gh/eellison/871/head -> origin/gh/eellison/871/head 2025-12-04T09:43:53.4317076Z * [new branch] gh/eellison/871/orig -> origin/gh/eellison/871/orig 2025-12-04T09:43:53.4319632Z * [new branch] gh/eellison/872/base -> origin/gh/eellison/872/base 2025-12-04T09:43:53.4321287Z * [new branch] gh/eellison/872/head -> origin/gh/eellison/872/head 2025-12-04T09:43:53.4322994Z * [new branch] gh/eellison/872/orig -> origin/gh/eellison/872/orig 2025-12-04T09:43:53.4325500Z * [new branch] gh/eellison/873/base -> origin/gh/eellison/873/base 2025-12-04T09:43:53.4327189Z * [new branch] gh/eellison/873/head -> origin/gh/eellison/873/head 2025-12-04T09:43:53.4328966Z * [new branch] gh/eellison/873/orig -> origin/gh/eellison/873/orig 2025-12-04T09:43:53.4331272Z * [new branch] gh/eellison/874/base -> origin/gh/eellison/874/base 2025-12-04T09:43:53.4332983Z * [new branch] gh/eellison/874/head -> origin/gh/eellison/874/head 2025-12-04T09:43:53.4334726Z * [new branch] gh/eellison/874/orig -> origin/gh/eellison/874/orig 2025-12-04T09:43:53.4337816Z * [new branch] gh/eellison/875/base -> origin/gh/eellison/875/base 2025-12-04T09:43:53.4339716Z * [new branch] gh/eellison/875/head -> origin/gh/eellison/875/head 2025-12-04T09:43:53.4341368Z * [new branch] gh/eellison/875/orig -> origin/gh/eellison/875/orig 2025-12-04T09:43:53.4343892Z * [new branch] gh/eellison/876/base -> origin/gh/eellison/876/base 2025-12-04T09:43:53.4345782Z * [new branch] gh/eellison/876/head -> origin/gh/eellison/876/head 2025-12-04T09:43:53.4347492Z * [new branch] gh/eellison/876/orig -> origin/gh/eellison/876/orig 2025-12-04T09:43:53.4350038Z * [new branch] gh/eellison/877/base -> origin/gh/eellison/877/base 2025-12-04T09:43:53.4351800Z * [new branch] gh/eellison/877/head -> origin/gh/eellison/877/head 2025-12-04T09:43:53.4353529Z * [new branch] gh/eellison/877/orig -> origin/gh/eellison/877/orig 2025-12-04T09:43:53.4356013Z * [new branch] gh/eellison/878/base -> origin/gh/eellison/878/base 2025-12-04T09:43:53.4357657Z * [new branch] gh/eellison/878/head -> origin/gh/eellison/878/head 2025-12-04T09:43:53.4359409Z * [new branch] gh/eellison/878/orig -> origin/gh/eellison/878/orig 2025-12-04T09:43:53.4361734Z * [new branch] gh/eellison/879/base -> origin/gh/eellison/879/base 2025-12-04T09:43:53.4363501Z * [new branch] gh/eellison/879/head -> origin/gh/eellison/879/head 2025-12-04T09:43:53.4365187Z * [new branch] gh/eellison/879/orig -> origin/gh/eellison/879/orig 2025-12-04T09:43:53.4367488Z * [new branch] gh/eellison/880/base -> origin/gh/eellison/880/base 2025-12-04T09:43:53.4369293Z * [new branch] gh/eellison/880/head -> origin/gh/eellison/880/head 2025-12-04T09:43:53.4371116Z * [new branch] gh/eellison/880/orig -> origin/gh/eellison/880/orig 2025-12-04T09:43:53.4373609Z * [new branch] gh/eellison/881/base -> origin/gh/eellison/881/base 2025-12-04T09:43:53.4375331Z * [new branch] gh/eellison/881/head -> origin/gh/eellison/881/head 2025-12-04T09:43:53.4377073Z * [new branch] gh/eellison/881/orig -> origin/gh/eellison/881/orig 2025-12-04T09:43:53.4379730Z * [new branch] gh/eellison/882/base -> origin/gh/eellison/882/base 2025-12-04T09:43:53.4381452Z * [new branch] gh/eellison/882/head -> origin/gh/eellison/882/head 2025-12-04T09:43:53.4383338Z * [new branch] gh/eellison/882/orig -> origin/gh/eellison/882/orig 2025-12-04T09:43:53.4385831Z * [new branch] gh/eellison/883/base -> origin/gh/eellison/883/base 2025-12-04T09:43:53.4387574Z * [new branch] gh/eellison/883/head -> origin/gh/eellison/883/head 2025-12-04T09:43:53.4389376Z * [new branch] gh/eellison/883/orig -> origin/gh/eellison/883/orig 2025-12-04T09:43:53.4391640Z * [new branch] gh/eellison/884/base -> origin/gh/eellison/884/base 2025-12-04T09:43:53.4393386Z * [new branch] gh/eellison/884/head -> origin/gh/eellison/884/head 2025-12-04T09:43:53.4395047Z * [new branch] gh/eellison/884/orig -> origin/gh/eellison/884/orig 2025-12-04T09:43:53.4397956Z * [new branch] gh/etaf/147/base -> origin/gh/etaf/147/base 2025-12-04T09:43:53.4399831Z * [new branch] gh/etaf/147/head -> origin/gh/etaf/147/head 2025-12-04T09:43:53.4402488Z * [new branch] gh/etaf/154/base -> origin/gh/etaf/154/base 2025-12-04T09:43:53.4404247Z * [new branch] gh/etaf/154/head -> origin/gh/etaf/154/head 2025-12-04T09:43:53.4405971Z * [new branch] gh/etaf/154/orig -> origin/gh/etaf/154/orig 2025-12-04T09:43:53.4408239Z * [new branch] gh/etaf/156/base -> origin/gh/etaf/156/base 2025-12-04T09:43:53.4409963Z * [new branch] gh/etaf/156/head -> origin/gh/etaf/156/head 2025-12-04T09:43:53.4411807Z * [new branch] gh/etaf/156/orig -> origin/gh/etaf/156/orig 2025-12-04T09:43:53.4414246Z * [new branch] gh/etaf/157/base -> origin/gh/etaf/157/base 2025-12-04T09:43:53.4416015Z * [new branch] gh/etaf/157/head -> origin/gh/etaf/157/head 2025-12-04T09:43:53.4417753Z * [new branch] gh/etaf/157/orig -> origin/gh/etaf/157/orig 2025-12-04T09:43:53.4420126Z * [new branch] gh/etaf/158/base -> origin/gh/etaf/158/base 2025-12-04T09:43:53.4422032Z * [new branch] gh/etaf/158/head -> origin/gh/etaf/158/head 2025-12-04T09:43:53.4423820Z * [new branch] gh/etaf/158/orig -> origin/gh/etaf/158/orig 2025-12-04T09:43:53.4426340Z * [new branch] gh/etaf/159/base -> origin/gh/etaf/159/base 2025-12-04T09:43:53.4428095Z * [new branch] gh/etaf/159/head -> origin/gh/etaf/159/head 2025-12-04T09:43:53.4429869Z * [new branch] gh/etaf/159/orig -> origin/gh/etaf/159/orig 2025-12-04T09:43:53.4432316Z * [new branch] gh/etaf/160/base -> origin/gh/etaf/160/base 2025-12-04T09:43:53.4434132Z * [new branch] gh/etaf/160/head -> origin/gh/etaf/160/head 2025-12-04T09:43:53.4435878Z * [new branch] gh/etaf/160/orig -> origin/gh/etaf/160/orig 2025-12-04T09:43:53.4438239Z * [new branch] gh/etaf/161/base -> origin/gh/etaf/161/base 2025-12-04T09:43:53.4440077Z * [new branch] gh/etaf/161/head -> origin/gh/etaf/161/head 2025-12-04T09:43:53.4441831Z * [new branch] gh/etaf/161/orig -> origin/gh/etaf/161/orig 2025-12-04T09:43:53.4444268Z * [new branch] gh/etaf/166/base -> origin/gh/etaf/166/base 2025-12-04T09:43:53.4446154Z * [new branch] gh/etaf/166/head -> origin/gh/etaf/166/head 2025-12-04T09:43:53.4447839Z * [new branch] gh/etaf/166/orig -> origin/gh/etaf/166/orig 2025-12-04T09:43:53.4450060Z * [new branch] gh/etaf/167/base -> origin/gh/etaf/167/base 2025-12-04T09:43:53.4451913Z * [new branch] gh/etaf/167/head -> origin/gh/etaf/167/head 2025-12-04T09:43:53.4453801Z * [new branch] gh/etaf/167/orig -> origin/gh/etaf/167/orig 2025-12-04T09:43:53.4456216Z * [new branch] gh/etaf/168/base -> origin/gh/etaf/168/base 2025-12-04T09:43:53.4458029Z * [new branch] gh/etaf/168/head -> origin/gh/etaf/168/head 2025-12-04T09:43:53.4459763Z * [new branch] gh/etaf/168/orig -> origin/gh/etaf/168/orig 2025-12-04T09:43:53.4462422Z * [new branch] gh/etaf/172/base -> origin/gh/etaf/172/base 2025-12-04T09:43:53.4464171Z * [new branch] gh/etaf/172/head -> origin/gh/etaf/172/head 2025-12-04T09:43:53.4465960Z * [new branch] gh/etaf/172/orig -> origin/gh/etaf/172/orig 2025-12-04T09:43:53.4468511Z * [new branch] gh/etaf/173/base -> origin/gh/etaf/173/base 2025-12-04T09:43:53.4470291Z * [new branch] gh/etaf/173/head -> origin/gh/etaf/173/head 2025-12-04T09:43:53.4472126Z * [new branch] gh/etaf/173/orig -> origin/gh/etaf/173/orig 2025-12-04T09:43:53.4474627Z * [new branch] gh/etaf/174/base -> origin/gh/etaf/174/base 2025-12-04T09:43:53.4476335Z * [new branch] gh/etaf/174/head -> origin/gh/etaf/174/head 2025-12-04T09:43:53.4479568Z * [new branch] gh/etaf/175/base -> origin/gh/etaf/175/base 2025-12-04T09:43:53.4481318Z * [new branch] gh/etaf/175/head -> origin/gh/etaf/175/head 2025-12-04T09:43:53.4482931Z * [new branch] gh/etaf/175/orig -> origin/gh/etaf/175/orig 2025-12-04T09:43:53.4485334Z * [new branch] gh/etaf/176/base -> origin/gh/etaf/176/base 2025-12-04T09:43:53.4487209Z * [new branch] gh/etaf/176/head -> origin/gh/etaf/176/head 2025-12-04T09:43:53.4488973Z * [new branch] gh/etaf/176/orig -> origin/gh/etaf/176/orig 2025-12-04T09:43:53.4491922Z * [new branch] gh/etaf/177/base -> origin/gh/etaf/177/base 2025-12-04T09:43:53.4493826Z * [new branch] gh/etaf/177/head -> origin/gh/etaf/177/head 2025-12-04T09:43:53.4495648Z * [new branch] gh/etaf/177/orig -> origin/gh/etaf/177/orig 2025-12-04T09:43:53.4498239Z * [new branch] gh/etaf/178/base -> origin/gh/etaf/178/base 2025-12-04T09:43:53.4500115Z * [new branch] gh/etaf/178/head -> origin/gh/etaf/178/head 2025-12-04T09:43:53.4501936Z * [new branch] gh/etaf/178/orig -> origin/gh/etaf/178/orig 2025-12-04T09:43:53.4504522Z * [new branch] gh/etaf/179/base -> origin/gh/etaf/179/base 2025-12-04T09:43:53.4506231Z * [new branch] gh/etaf/179/head -> origin/gh/etaf/179/head 2025-12-04T09:43:53.4507972Z * [new branch] gh/etaf/179/orig -> origin/gh/etaf/179/orig 2025-12-04T09:43:53.4510292Z * [new branch] gh/etaf/180/base -> origin/gh/etaf/180/base 2025-12-04T09:43:53.4512029Z * [new branch] gh/etaf/180/head -> origin/gh/etaf/180/head 2025-12-04T09:43:53.4514321Z * [new branch] gh/etaf/180/orig -> origin/gh/etaf/180/orig 2025-12-04T09:43:53.4518128Z * [new branch] gh/exclamaforte/1/base -> origin/gh/exclamaforte/1/base 2025-12-04T09:43:53.4519832Z * [new branch] gh/exclamaforte/1/head -> origin/gh/exclamaforte/1/head 2025-12-04T09:43:53.4522275Z * [new branch] gh/exclamaforte/2/base -> origin/gh/exclamaforte/2/base 2025-12-04T09:43:53.4523952Z * [new branch] gh/exclamaforte/2/head -> origin/gh/exclamaforte/2/head 2025-12-04T09:43:53.4526255Z * [new branch] gh/exclamaforte/3/base -> origin/gh/exclamaforte/3/base 2025-12-04T09:43:53.4528104Z * [new branch] gh/exclamaforte/3/head -> origin/gh/exclamaforte/3/head 2025-12-04T09:43:53.4530430Z * [new branch] gh/exclamaforte/4/base -> origin/gh/exclamaforte/4/base 2025-12-04T09:43:53.4532165Z * [new branch] gh/exclamaforte/4/head -> origin/gh/exclamaforte/4/head 2025-12-04T09:43:53.4535002Z * [new branch] gh/ezyang/2374/base -> origin/gh/ezyang/2374/base 2025-12-04T09:43:53.4536759Z * [new branch] gh/ezyang/2374/head -> origin/gh/ezyang/2374/head 2025-12-04T09:43:53.4538769Z * [new branch] gh/ezyang/2374/orig -> origin/gh/ezyang/2374/orig 2025-12-04T09:43:53.4540955Z * [new branch] gh/ezyang/2973/base -> origin/gh/ezyang/2973/base 2025-12-04T09:43:53.4542664Z * [new branch] gh/ezyang/2973/head -> origin/gh/ezyang/2973/head 2025-12-04T09:43:53.4544566Z * [new branch] gh/ezyang/2973/orig -> origin/gh/ezyang/2973/orig 2025-12-04T09:43:53.4546970Z * [new branch] gh/ezyang/2974/base -> origin/gh/ezyang/2974/base 2025-12-04T09:43:53.4548673Z * [new branch] gh/ezyang/2974/head -> origin/gh/ezyang/2974/head 2025-12-04T09:43:53.4550368Z * [new branch] gh/ezyang/2974/orig -> origin/gh/ezyang/2974/orig 2025-12-04T09:43:53.4552724Z * [new branch] gh/ezyang/3131/base -> origin/gh/ezyang/3131/base 2025-12-04T09:43:53.4554456Z * [new branch] gh/ezyang/3131/head -> origin/gh/ezyang/3131/head 2025-12-04T09:43:53.4556189Z * [new branch] gh/ezyang/3131/orig -> origin/gh/ezyang/3131/orig 2025-12-04T09:43:53.4558517Z * [new branch] gh/ezyang/3139/base -> origin/gh/ezyang/3139/base 2025-12-04T09:43:53.4560364Z * [new branch] gh/ezyang/3139/head -> origin/gh/ezyang/3139/head 2025-12-04T09:43:53.4562149Z * [new branch] gh/ezyang/3139/orig -> origin/gh/ezyang/3139/orig 2025-12-04T09:43:53.4564437Z * [new branch] gh/ezyang/3140/base -> origin/gh/ezyang/3140/base 2025-12-04T09:43:53.4566142Z * [new branch] gh/ezyang/3140/head -> origin/gh/ezyang/3140/head 2025-12-04T09:43:53.4567889Z * [new branch] gh/ezyang/3140/orig -> origin/gh/ezyang/3140/orig 2025-12-04T09:43:53.4570135Z * [new branch] gh/ezyang/3143/base -> origin/gh/ezyang/3143/base 2025-12-04T09:43:53.4571770Z * [new branch] gh/ezyang/3143/head -> origin/gh/ezyang/3143/head 2025-12-04T09:43:53.4573549Z * [new branch] gh/ezyang/3143/orig -> origin/gh/ezyang/3143/orig 2025-12-04T09:43:53.4575917Z * [new branch] gh/ezyang/3144/base -> origin/gh/ezyang/3144/base 2025-12-04T09:43:53.4577640Z * [new branch] gh/ezyang/3144/head -> origin/gh/ezyang/3144/head 2025-12-04T09:43:53.4579653Z * [new branch] gh/ezyang/3144/orig -> origin/gh/ezyang/3144/orig 2025-12-04T09:43:53.4582140Z * [new branch] gh/ezyang/3167/base -> origin/gh/ezyang/3167/base 2025-12-04T09:43:53.4583921Z * [new branch] gh/ezyang/3167/head -> origin/gh/ezyang/3167/head 2025-12-04T09:43:53.4585759Z * [new branch] gh/ezyang/3167/orig -> origin/gh/ezyang/3167/orig 2025-12-04T09:43:53.4588224Z * [new branch] gh/ezyang/3173/base -> origin/gh/ezyang/3173/base 2025-12-04T09:43:53.4590083Z * [new branch] gh/ezyang/3173/head -> origin/gh/ezyang/3173/head 2025-12-04T09:43:53.4591914Z * [new branch] gh/ezyang/3173/orig -> origin/gh/ezyang/3173/orig 2025-12-04T09:43:53.4594241Z * [new branch] gh/ezyang/3175/base -> origin/gh/ezyang/3175/base 2025-12-04T09:43:53.4595955Z * [new branch] gh/ezyang/3175/head -> origin/gh/ezyang/3175/head 2025-12-04T09:43:53.4597705Z * [new branch] gh/ezyang/3175/orig -> origin/gh/ezyang/3175/orig 2025-12-04T09:43:53.4600024Z * [new branch] gh/ezyang/3182/base -> origin/gh/ezyang/3182/base 2025-12-04T09:43:53.4601687Z * [new branch] gh/ezyang/3182/head -> origin/gh/ezyang/3182/head 2025-12-04T09:43:53.4603839Z * [new branch] gh/ezyang/3182/orig -> origin/gh/ezyang/3182/orig 2025-12-04T09:43:53.4606181Z * [new branch] gh/ezyang/3185/base -> origin/gh/ezyang/3185/base 2025-12-04T09:43:53.4608073Z * [new branch] gh/ezyang/3185/head -> origin/gh/ezyang/3185/head 2025-12-04T09:43:53.4609835Z * [new branch] gh/ezyang/3185/orig -> origin/gh/ezyang/3185/orig 2025-12-04T09:43:53.4612199Z * [new branch] gh/ezyang/3189/base -> origin/gh/ezyang/3189/base 2025-12-04T09:43:53.4613905Z * [new branch] gh/ezyang/3189/head -> origin/gh/ezyang/3189/head 2025-12-04T09:43:53.4615622Z * [new branch] gh/ezyang/3189/orig -> origin/gh/ezyang/3189/orig 2025-12-04T09:43:53.4618020Z * [new branch] gh/ezyang/3191/base -> origin/gh/ezyang/3191/base 2025-12-04T09:43:53.4619700Z * [new branch] gh/ezyang/3191/head -> origin/gh/ezyang/3191/head 2025-12-04T09:43:53.4621456Z * [new branch] gh/ezyang/3191/orig -> origin/gh/ezyang/3191/orig 2025-12-04T09:43:53.4624568Z * [new branch] gh/ezyang/3192/base -> origin/gh/ezyang/3192/base 2025-12-04T09:43:53.4626244Z * [new branch] gh/ezyang/3192/head -> origin/gh/ezyang/3192/head 2025-12-04T09:43:53.4627970Z * [new branch] gh/ezyang/3192/orig -> origin/gh/ezyang/3192/orig 2025-12-04T09:43:53.4630332Z * [new branch] gh/ezyang/3193/base -> origin/gh/ezyang/3193/base 2025-12-04T09:43:53.4632020Z * [new branch] gh/ezyang/3193/head -> origin/gh/ezyang/3193/head 2025-12-04T09:43:53.4633904Z * [new branch] gh/ezyang/3193/orig -> origin/gh/ezyang/3193/orig 2025-12-04T09:43:53.4636365Z * [new branch] gh/ezyang/3194/base -> origin/gh/ezyang/3194/base 2025-12-04T09:43:53.4638072Z * [new branch] gh/ezyang/3194/head -> origin/gh/ezyang/3194/head 2025-12-04T09:43:53.4639899Z * [new branch] gh/ezyang/3194/orig -> origin/gh/ezyang/3194/orig 2025-12-04T09:43:53.4642283Z * [new branch] gh/ezyang/3195/base -> origin/gh/ezyang/3195/base 2025-12-04T09:43:53.4644005Z * [new branch] gh/ezyang/3195/head -> origin/gh/ezyang/3195/head 2025-12-04T09:43:53.4645759Z * [new branch] gh/ezyang/3195/orig -> origin/gh/ezyang/3195/orig 2025-12-04T09:43:53.4648154Z * [new branch] gh/ezyang/3196/base -> origin/gh/ezyang/3196/base 2025-12-04T09:43:53.4649914Z * [new branch] gh/ezyang/3196/head -> origin/gh/ezyang/3196/head 2025-12-04T09:43:53.4651780Z * [new branch] gh/ezyang/3196/orig -> origin/gh/ezyang/3196/orig 2025-12-04T09:43:53.4654143Z * [new branch] gh/ezyang/3197/base -> origin/gh/ezyang/3197/base 2025-12-04T09:43:53.4655891Z * [new branch] gh/ezyang/3197/head -> origin/gh/ezyang/3197/head 2025-12-04T09:43:53.4657713Z * [new branch] gh/ezyang/3197/orig -> origin/gh/ezyang/3197/orig 2025-12-04T09:43:53.4660030Z * [new branch] gh/ezyang/3198/base -> origin/gh/ezyang/3198/base 2025-12-04T09:43:53.4661833Z * [new branch] gh/ezyang/3198/head -> origin/gh/ezyang/3198/head 2025-12-04T09:43:53.4663932Z * [new branch] gh/ezyang/3198/orig -> origin/gh/ezyang/3198/orig 2025-12-04T09:43:53.4666387Z * [new branch] gh/ezyang/3199/base -> origin/gh/ezyang/3199/base 2025-12-04T09:43:53.4668058Z * [new branch] gh/ezyang/3199/head -> origin/gh/ezyang/3199/head 2025-12-04T09:43:53.4669959Z * [new branch] gh/ezyang/3199/orig -> origin/gh/ezyang/3199/orig 2025-12-04T09:43:53.4672289Z * [new branch] gh/ezyang/3200/base -> origin/gh/ezyang/3200/base 2025-12-04T09:43:53.4673992Z * [new branch] gh/ezyang/3200/head -> origin/gh/ezyang/3200/head 2025-12-04T09:43:53.4675783Z * [new branch] gh/ezyang/3200/orig -> origin/gh/ezyang/3200/orig 2025-12-04T09:43:53.4678347Z * [new branch] gh/ezyang/3201/base -> origin/gh/ezyang/3201/base 2025-12-04T09:43:53.4680666Z * [new branch] gh/ezyang/3201/head -> origin/gh/ezyang/3201/head 2025-12-04T09:43:53.4682110Z * [new branch] gh/ezyang/3201/orig -> origin/gh/ezyang/3201/orig 2025-12-04T09:43:53.4684480Z * [new branch] gh/ezyang/3202/base -> origin/gh/ezyang/3202/base 2025-12-04T09:43:53.4686167Z * [new branch] gh/ezyang/3202/head -> origin/gh/ezyang/3202/head 2025-12-04T09:43:53.4687945Z * [new branch] gh/ezyang/3202/orig -> origin/gh/ezyang/3202/orig 2025-12-04T09:43:53.4690198Z * [new branch] gh/ezyang/3203/base -> origin/gh/ezyang/3203/base 2025-12-04T09:43:53.4691966Z * [new branch] gh/ezyang/3203/head -> origin/gh/ezyang/3203/head 2025-12-04T09:43:53.4693784Z * [new branch] gh/ezyang/3203/orig -> origin/gh/ezyang/3203/orig 2025-12-04T09:43:53.4696210Z * [new branch] gh/ezyang/3204/base -> origin/gh/ezyang/3204/base 2025-12-04T09:43:53.4697977Z * [new branch] gh/ezyang/3204/head -> origin/gh/ezyang/3204/head 2025-12-04T09:43:53.4699641Z * [new branch] gh/ezyang/3204/orig -> origin/gh/ezyang/3204/orig 2025-12-04T09:43:53.4702102Z * [new branch] gh/ezyang/3205/base -> origin/gh/ezyang/3205/base 2025-12-04T09:43:53.4703910Z * [new branch] gh/ezyang/3205/head -> origin/gh/ezyang/3205/head 2025-12-04T09:43:53.4705786Z * [new branch] gh/ezyang/3205/orig -> origin/gh/ezyang/3205/orig 2025-12-04T09:43:53.4708186Z * [new branch] gh/ezyang/3206/base -> origin/gh/ezyang/3206/base 2025-12-04T09:43:53.4709866Z * [new branch] gh/ezyang/3206/head -> origin/gh/ezyang/3206/head 2025-12-04T09:43:53.4711781Z * [new branch] gh/ezyang/3206/orig -> origin/gh/ezyang/3206/orig 2025-12-04T09:43:53.4714149Z * [new branch] gh/ezyang/3207/base -> origin/gh/ezyang/3207/base 2025-12-04T09:43:53.4715950Z * [new branch] gh/ezyang/3207/head -> origin/gh/ezyang/3207/head 2025-12-04T09:43:53.4717698Z * [new branch] gh/ezyang/3207/orig -> origin/gh/ezyang/3207/orig 2025-12-04T09:43:53.4720041Z * [new branch] gh/ezyang/3208/base -> origin/gh/ezyang/3208/base 2025-12-04T09:43:53.4721712Z * [new branch] gh/ezyang/3208/head -> origin/gh/ezyang/3208/head 2025-12-04T09:43:53.4723477Z * [new branch] gh/ezyang/3208/orig -> origin/gh/ezyang/3208/orig 2025-12-04T09:43:53.4725909Z * [new branch] gh/ezyang/3209/base -> origin/gh/ezyang/3209/base 2025-12-04T09:43:53.4727697Z * [new branch] gh/ezyang/3209/head -> origin/gh/ezyang/3209/head 2025-12-04T09:43:53.4729511Z * [new branch] gh/ezyang/3209/orig -> origin/gh/ezyang/3209/orig 2025-12-04T09:43:53.4732292Z * [new branch] gh/fadara01/3/base -> origin/gh/fadara01/3/base 2025-12-04T09:43:53.4734036Z * [new branch] gh/fadara01/3/head -> origin/gh/fadara01/3/head 2025-12-04T09:43:53.4735780Z * [new branch] gh/fadara01/3/orig -> origin/gh/fadara01/3/orig 2025-12-04T09:43:53.4738079Z * [new branch] gh/fadara01/5/base -> origin/gh/fadara01/5/base 2025-12-04T09:43:53.4739940Z * [new branch] gh/fadara01/5/head -> origin/gh/fadara01/5/head 2025-12-04T09:43:53.4741688Z * [new branch] gh/fadara01/5/orig -> origin/gh/fadara01/5/orig 2025-12-04T09:43:53.4744086Z * [new branch] gh/fadara01/6/base -> origin/gh/fadara01/6/base 2025-12-04T09:43:53.4745881Z * [new branch] gh/fadara01/6/head -> origin/gh/fadara01/6/head 2025-12-04T09:43:53.4747566Z * [new branch] gh/fadara01/6/orig -> origin/gh/fadara01/6/orig 2025-12-04T09:43:53.4750019Z * [new branch] gh/fadara01/7/base -> origin/gh/fadara01/7/base 2025-12-04T09:43:53.4751746Z * [new branch] gh/fadara01/7/head -> origin/gh/fadara01/7/head 2025-12-04T09:43:53.4753612Z * [new branch] gh/fadara01/7/orig -> origin/gh/fadara01/7/orig 2025-12-04T09:43:53.4755951Z * [new branch] gh/fadara01/8/base -> origin/gh/fadara01/8/base 2025-12-04T09:43:53.4758143Z * [new branch] gh/fadara01/8/head -> origin/gh/fadara01/8/head 2025-12-04T09:43:53.4760039Z * [new branch] gh/fadara01/8/orig -> origin/gh/fadara01/8/orig 2025-12-04T09:43:53.4762258Z * [new branch] gh/fadara01/9/base -> origin/gh/fadara01/9/base 2025-12-04T09:43:53.4764025Z * [new branch] gh/fadara01/9/head -> origin/gh/fadara01/9/head 2025-12-04T09:43:53.4765773Z * [new branch] gh/fadara01/9/orig -> origin/gh/fadara01/9/orig 2025-12-04T09:43:53.4769179Z * [new branch] gh/fduwjj/182/base -> origin/gh/fduwjj/182/base 2025-12-04T09:43:53.4770963Z * [new branch] gh/fduwjj/182/head -> origin/gh/fduwjj/182/head 2025-12-04T09:43:53.4772685Z * [new branch] gh/fduwjj/182/orig -> origin/gh/fduwjj/182/orig 2025-12-04T09:43:53.4775147Z * [new branch] gh/fduwjj/211/base -> origin/gh/fduwjj/211/base 2025-12-04T09:43:53.4776912Z * [new branch] gh/fduwjj/211/head -> origin/gh/fduwjj/211/head 2025-12-04T09:43:53.4778967Z * [new branch] gh/fduwjj/211/orig -> origin/gh/fduwjj/211/orig 2025-12-04T09:43:53.4781229Z * [new branch] gh/fduwjj/212/base -> origin/gh/fduwjj/212/base 2025-12-04T09:43:53.4783046Z * [new branch] gh/fduwjj/212/head -> origin/gh/fduwjj/212/head 2025-12-04T09:43:53.4784905Z * [new branch] gh/fduwjj/212/orig -> origin/gh/fduwjj/212/orig 2025-12-04T09:43:53.4787210Z * [new branch] gh/fduwjj/213/base -> origin/gh/fduwjj/213/base 2025-12-04T09:43:53.4789011Z * [new branch] gh/fduwjj/213/head -> origin/gh/fduwjj/213/head 2025-12-04T09:43:53.4790686Z * [new branch] gh/fduwjj/213/orig -> origin/gh/fduwjj/213/orig 2025-12-04T09:43:53.4793138Z * [new branch] gh/fduwjj/226/base -> origin/gh/fduwjj/226/base 2025-12-04T09:43:53.4794800Z * [new branch] gh/fduwjj/226/head -> origin/gh/fduwjj/226/head 2025-12-04T09:43:53.4796565Z * [new branch] gh/fduwjj/226/orig -> origin/gh/fduwjj/226/orig 2025-12-04T09:43:53.4799101Z * [new branch] gh/fduwjj/229/base -> origin/gh/fduwjj/229/base 2025-12-04T09:43:53.4800766Z * [new branch] gh/fduwjj/229/head -> origin/gh/fduwjj/229/head 2025-12-04T09:43:53.4802718Z * [new branch] gh/fduwjj/229/orig -> origin/gh/fduwjj/229/orig 2025-12-04T09:43:53.4805065Z * [new branch] gh/fduwjj/233/base -> origin/gh/fduwjj/233/base 2025-12-04T09:43:53.4806796Z * [new branch] gh/fduwjj/233/head -> origin/gh/fduwjj/233/head 2025-12-04T09:43:53.4808528Z * [new branch] gh/fduwjj/233/orig -> origin/gh/fduwjj/233/orig 2025-12-04T09:43:53.4810913Z * [new branch] gh/fduwjj/234/base -> origin/gh/fduwjj/234/base 2025-12-04T09:43:53.4812758Z * [new branch] gh/fduwjj/234/head -> origin/gh/fduwjj/234/head 2025-12-04T09:43:53.4814492Z * [new branch] gh/fduwjj/234/orig -> origin/gh/fduwjj/234/orig 2025-12-04T09:43:53.4816812Z * [new branch] gh/fduwjj/235/base -> origin/gh/fduwjj/235/base 2025-12-04T09:43:53.4818558Z * [new branch] gh/fduwjj/235/head -> origin/gh/fduwjj/235/head 2025-12-04T09:43:53.4820254Z * [new branch] gh/fduwjj/235/orig -> origin/gh/fduwjj/235/orig 2025-12-04T09:43:53.4822827Z * [new branch] gh/fduwjj/236/base -> origin/gh/fduwjj/236/base 2025-12-04T09:43:53.4824604Z * [new branch] gh/fduwjj/236/head -> origin/gh/fduwjj/236/head 2025-12-04T09:43:53.4826279Z * [new branch] gh/fduwjj/236/orig -> origin/gh/fduwjj/236/orig 2025-12-04T09:43:53.4828481Z * [new branch] gh/fduwjj/237/base -> origin/gh/fduwjj/237/base 2025-12-04T09:43:53.4830264Z * [new branch] gh/fduwjj/237/head -> origin/gh/fduwjj/237/head 2025-12-04T09:43:53.4831946Z * [new branch] gh/fduwjj/237/orig -> origin/gh/fduwjj/237/orig 2025-12-04T09:43:53.4834284Z * [new branch] gh/fduwjj/238/base -> origin/gh/fduwjj/238/base 2025-12-04T09:43:53.4836121Z * [new branch] gh/fduwjj/238/head -> origin/gh/fduwjj/238/head 2025-12-04T09:43:53.4837861Z * [new branch] gh/fduwjj/238/orig -> origin/gh/fduwjj/238/orig 2025-12-04T09:43:53.4840226Z * [new branch] gh/fduwjj/239/base -> origin/gh/fduwjj/239/base 2025-12-04T09:43:53.4842107Z * [new branch] gh/fduwjj/239/head -> origin/gh/fduwjj/239/head 2025-12-04T09:43:53.4843871Z * [new branch] gh/fduwjj/239/orig -> origin/gh/fduwjj/239/orig 2025-12-04T09:43:53.4846706Z * [new branch] gh/fegin/332/base -> origin/gh/fegin/332/base 2025-12-04T09:43:53.4848471Z * [new branch] gh/fegin/332/head -> origin/gh/fegin/332/head 2025-12-04T09:43:53.4850279Z * [new branch] gh/fegin/332/orig -> origin/gh/fegin/332/orig 2025-12-04T09:43:53.4852529Z * [new branch] gh/fegin/333/base -> origin/gh/fegin/333/base 2025-12-04T09:43:53.4854258Z * [new branch] gh/fegin/333/head -> origin/gh/fegin/333/head 2025-12-04T09:43:53.4856005Z * [new branch] gh/fegin/333/orig -> origin/gh/fegin/333/orig 2025-12-04T09:43:53.4858346Z * [new branch] gh/fegin/334/base -> origin/gh/fegin/334/base 2025-12-04T09:43:53.4860042Z * [new branch] gh/fegin/334/head -> origin/gh/fegin/334/head 2025-12-04T09:43:53.4861891Z * [new branch] gh/fegin/334/orig -> origin/gh/fegin/334/orig 2025-12-04T09:43:53.4864361Z * [new branch] gh/fegin/335/base -> origin/gh/fegin/335/base 2025-12-04T09:43:53.4866104Z * [new branch] gh/fegin/335/head -> origin/gh/fegin/335/head 2025-12-04T09:43:53.4867793Z * [new branch] gh/fegin/335/orig -> origin/gh/fegin/335/orig 2025-12-04T09:43:53.4870800Z * [new branch] gh/fffrog/160/base -> origin/gh/fffrog/160/base 2025-12-04T09:43:53.4872662Z * [new branch] gh/fffrog/160/head -> origin/gh/fffrog/160/head 2025-12-04T09:43:53.4874974Z * [new branch] gh/fffrog/177/base -> origin/gh/fffrog/177/base 2025-12-04T09:43:53.4876718Z * [new branch] gh/fffrog/177/head -> origin/gh/fffrog/177/head 2025-12-04T09:43:53.4878535Z * [new branch] gh/fffrog/177/orig -> origin/gh/fffrog/177/orig 2025-12-04T09:43:53.4881078Z * [new branch] gh/fffrog/178/base -> origin/gh/fffrog/178/base 2025-12-04T09:43:53.4882901Z * [new branch] gh/fffrog/178/head -> origin/gh/fffrog/178/head 2025-12-04T09:43:53.4884662Z * [new branch] gh/fffrog/178/orig -> origin/gh/fffrog/178/orig 2025-12-04T09:43:53.4886903Z * [new branch] gh/fffrog/181/base -> origin/gh/fffrog/181/base 2025-12-04T09:43:53.4888627Z * [new branch] gh/fffrog/181/head -> origin/gh/fffrog/181/head 2025-12-04T09:43:53.4890358Z * [new branch] gh/fffrog/181/orig -> origin/gh/fffrog/181/orig 2025-12-04T09:43:53.4892976Z * [new branch] gh/fffrog/183/base -> origin/gh/fffrog/183/base 2025-12-04T09:43:53.4894542Z * [new branch] gh/fffrog/183/head -> origin/gh/fffrog/183/head 2025-12-04T09:43:53.4896241Z * [new branch] gh/fffrog/183/orig -> origin/gh/fffrog/183/orig 2025-12-04T09:43:53.4899148Z * [new branch] gh/fxdawnn/10/base -> origin/gh/fxdawnn/10/base 2025-12-04T09:43:53.4901031Z * [new branch] gh/fxdawnn/10/head -> origin/gh/fxdawnn/10/head 2025-12-04T09:43:53.4902723Z * [new branch] gh/fxdawnn/10/orig -> origin/gh/fxdawnn/10/orig 2025-12-04T09:43:53.4905553Z * [new branch] gh/fxdawnn/11/base -> origin/gh/fxdawnn/11/base 2025-12-04T09:43:53.4907168Z * [new branch] gh/fxdawnn/11/head -> origin/gh/fxdawnn/11/head 2025-12-04T09:43:53.4909520Z * [new branch] gh/fxdawnn/11/orig -> origin/gh/fxdawnn/11/orig 2025-12-04T09:43:53.4911555Z * [new branch] gh/fxdawnn/12/base -> origin/gh/fxdawnn/12/base 2025-12-04T09:43:53.4913319Z * [new branch] gh/fxdawnn/12/head -> origin/gh/fxdawnn/12/head 2025-12-04T09:43:53.4915117Z * [new branch] gh/fxdawnn/12/orig -> origin/gh/fxdawnn/12/orig 2025-12-04T09:43:53.4917455Z * [new branch] gh/fxdawnn/13/base -> origin/gh/fxdawnn/13/base 2025-12-04T09:43:53.4919248Z * [new branch] gh/fxdawnn/13/head -> origin/gh/fxdawnn/13/head 2025-12-04T09:43:53.4921024Z * [new branch] gh/fxdawnn/13/orig -> origin/gh/fxdawnn/13/orig 2025-12-04T09:43:53.4923413Z * [new branch] gh/fxdawnn/14/base -> origin/gh/fxdawnn/14/base 2025-12-04T09:43:53.4925123Z * [new branch] gh/fxdawnn/14/head -> origin/gh/fxdawnn/14/head 2025-12-04T09:43:53.4926870Z * [new branch] gh/fxdawnn/14/orig -> origin/gh/fxdawnn/14/orig 2025-12-04T09:43:53.4929197Z * [new branch] gh/fxdawnn/15/base -> origin/gh/fxdawnn/15/base 2025-12-04T09:43:53.4930941Z * [new branch] gh/fxdawnn/15/head -> origin/gh/fxdawnn/15/head 2025-12-04T09:43:53.4932731Z * [new branch] gh/fxdawnn/15/orig -> origin/gh/fxdawnn/15/orig 2025-12-04T09:43:53.4935047Z * [new branch] gh/fxdawnn/6/base -> origin/gh/fxdawnn/6/base 2025-12-04T09:43:53.4936772Z * [new branch] gh/fxdawnn/6/head -> origin/gh/fxdawnn/6/head 2025-12-04T09:43:53.4938635Z * [new branch] gh/fxdawnn/6/orig -> origin/gh/fxdawnn/6/orig 2025-12-04T09:43:53.4940950Z * [new branch] gh/fxdawnn/7/base -> origin/gh/fxdawnn/7/base 2025-12-04T09:43:53.4942788Z * [new branch] gh/fxdawnn/7/head -> origin/gh/fxdawnn/7/head 2025-12-04T09:43:53.4945048Z * [new branch] gh/fxdawnn/7/orig -> origin/gh/fxdawnn/7/orig 2025-12-04T09:43:53.4946991Z * [new branch] gh/fxdawnn/9/base -> origin/gh/fxdawnn/9/base 2025-12-04T09:43:53.4948669Z * [new branch] gh/fxdawnn/9/head -> origin/gh/fxdawnn/9/head 2025-12-04T09:43:53.4950295Z * [new branch] gh/fxdawnn/9/orig -> origin/gh/fxdawnn/9/orig 2025-12-04T09:43:53.4953124Z * [new branch] gh/galv/1/base -> origin/gh/galv/1/base 2025-12-04T09:43:53.4954823Z * [new branch] gh/galv/1/head -> origin/gh/galv/1/head 2025-12-04T09:43:53.4956666Z * [new branch] gh/galv/1/orig -> origin/gh/galv/1/orig 2025-12-04T09:43:53.4959084Z * [new branch] gh/galv/2/base -> origin/gh/galv/2/base 2025-12-04T09:43:53.4960893Z * [new branch] gh/galv/2/head -> origin/gh/galv/2/head 2025-12-04T09:43:53.4962684Z * [new branch] gh/galv/2/orig -> origin/gh/galv/2/orig 2025-12-04T09:43:53.4965380Z * [new branch] gh/galv/3/base -> origin/gh/galv/3/base 2025-12-04T09:43:53.4966840Z * [new branch] gh/galv/3/head -> origin/gh/galv/3/head 2025-12-04T09:43:53.4968658Z * [new branch] gh/galv/3/orig -> origin/gh/galv/3/orig 2025-12-04T09:43:53.4971502Z * [new branch] gh/guangyey/134/base -> origin/gh/guangyey/134/base 2025-12-04T09:43:53.4973355Z * [new branch] gh/guangyey/134/head -> origin/gh/guangyey/134/head 2025-12-04T09:43:53.4975119Z * [new branch] gh/guangyey/134/orig -> origin/gh/guangyey/134/orig 2025-12-04T09:43:53.4977393Z * [new branch] gh/guangyey/163/base -> origin/gh/guangyey/163/base 2025-12-04T09:43:53.4980837Z * [new branch] gh/guangyey/163/head -> origin/gh/guangyey/163/head 2025-12-04T09:43:53.4982594Z * [new branch] gh/guangyey/163/orig -> origin/gh/guangyey/163/orig 2025-12-04T09:43:53.4985139Z * [new branch] gh/guangyey/168/base -> origin/gh/guangyey/168/base 2025-12-04T09:43:53.4986881Z * [new branch] gh/guangyey/168/head -> origin/gh/guangyey/168/head 2025-12-04T09:43:53.4988668Z * [new branch] gh/guangyey/168/orig -> origin/gh/guangyey/168/orig 2025-12-04T09:43:53.4990954Z * [new branch] gh/guangyey/169/base -> origin/gh/guangyey/169/base 2025-12-04T09:43:53.4992712Z * [new branch] gh/guangyey/169/head -> origin/gh/guangyey/169/head 2025-12-04T09:43:53.4994483Z * [new branch] gh/guangyey/169/orig -> origin/gh/guangyey/169/orig 2025-12-04T09:43:53.4996822Z * [new branch] gh/guangyey/170/base -> origin/gh/guangyey/170/base 2025-12-04T09:43:53.4998543Z * [new branch] gh/guangyey/170/head -> origin/gh/guangyey/170/head 2025-12-04T09:43:53.5000388Z * [new branch] gh/guangyey/170/orig -> origin/gh/guangyey/170/orig 2025-12-04T09:43:53.5002788Z * [new branch] gh/guangyey/171/base -> origin/gh/guangyey/171/base 2025-12-04T09:43:53.5004519Z * [new branch] gh/guangyey/171/head -> origin/gh/guangyey/171/head 2025-12-04T09:43:53.5006224Z * [new branch] gh/guangyey/171/orig -> origin/gh/guangyey/171/orig 2025-12-04T09:43:53.5008624Z * [new branch] gh/guangyey/178/base -> origin/gh/guangyey/178/base 2025-12-04T09:43:53.5010428Z * [new branch] gh/guangyey/178/head -> origin/gh/guangyey/178/head 2025-12-04T09:43:53.5012141Z * [new branch] gh/guangyey/178/orig -> origin/gh/guangyey/178/orig 2025-12-04T09:43:53.5014462Z * [new branch] gh/guangyey/182/base -> origin/gh/guangyey/182/base 2025-12-04T09:43:53.5016278Z * [new branch] gh/guangyey/182/head -> origin/gh/guangyey/182/head 2025-12-04T09:43:53.5017961Z * [new branch] gh/guangyey/182/orig -> origin/gh/guangyey/182/orig 2025-12-04T09:43:53.5020309Z * [new branch] gh/guangyey/183/base -> origin/gh/guangyey/183/base 2025-12-04T09:43:53.5022032Z * [new branch] gh/guangyey/183/head -> origin/gh/guangyey/183/head 2025-12-04T09:43:53.5023895Z * [new branch] gh/guangyey/183/orig -> origin/gh/guangyey/183/orig 2025-12-04T09:43:53.5026254Z * [new branch] gh/guangyey/185/base -> origin/gh/guangyey/185/base 2025-12-04T09:43:53.5028022Z * [new branch] gh/guangyey/185/head -> origin/gh/guangyey/185/head 2025-12-04T09:43:53.5029833Z * [new branch] gh/guangyey/185/orig -> origin/gh/guangyey/185/orig 2025-12-04T09:43:53.5032706Z * [new branch] gh/guangyey/186/base -> origin/gh/guangyey/186/base 2025-12-04T09:43:53.5034509Z * [new branch] gh/guangyey/186/head -> origin/gh/guangyey/186/head 2025-12-04T09:43:53.5036349Z * [new branch] gh/guangyey/186/orig -> origin/gh/guangyey/186/orig 2025-12-04T09:43:53.5038592Z * [new branch] gh/guangyey/187/base -> origin/gh/guangyey/187/base 2025-12-04T09:43:53.5040208Z * [new branch] gh/guangyey/187/head -> origin/gh/guangyey/187/head 2025-12-04T09:43:53.5042109Z * [new branch] gh/guangyey/187/orig -> origin/gh/guangyey/187/orig 2025-12-04T09:43:53.5044436Z * [new branch] gh/guangyey/188/base -> origin/gh/guangyey/188/base 2025-12-04T09:43:53.5046217Z * [new branch] gh/guangyey/188/head -> origin/gh/guangyey/188/head 2025-12-04T09:43:53.5047944Z * [new branch] gh/guangyey/188/orig -> origin/gh/guangyey/188/orig 2025-12-04T09:43:53.5050247Z * [new branch] gh/guangyey/190/base -> origin/gh/guangyey/190/base 2025-12-04T09:43:53.5051928Z * [new branch] gh/guangyey/190/head -> origin/gh/guangyey/190/head 2025-12-04T09:43:53.5053631Z * [new branch] gh/guangyey/190/orig -> origin/gh/guangyey/190/orig 2025-12-04T09:43:53.5056014Z * [new branch] gh/guangyey/208/base -> origin/gh/guangyey/208/base 2025-12-04T09:43:53.5057750Z * [new branch] gh/guangyey/208/head -> origin/gh/guangyey/208/head 2025-12-04T09:43:53.5059486Z * [new branch] gh/guangyey/208/orig -> origin/gh/guangyey/208/orig 2025-12-04T09:43:53.5061731Z * [new branch] gh/guangyey/228/base -> origin/gh/guangyey/228/base 2025-12-04T09:43:53.5063531Z * [new branch] gh/guangyey/228/head -> origin/gh/guangyey/228/head 2025-12-04T09:43:53.5065383Z * [new branch] gh/guangyey/228/orig -> origin/gh/guangyey/228/orig 2025-12-04T09:43:53.5068161Z * [new branch] gh/guangyey/230/base -> origin/gh/guangyey/230/base 2025-12-04T09:43:53.5069939Z * [new branch] gh/guangyey/230/head -> origin/gh/guangyey/230/head 2025-12-04T09:43:53.5071719Z * [new branch] gh/guangyey/230/orig -> origin/gh/guangyey/230/orig 2025-12-04T09:43:53.5074117Z * [new branch] gh/guangyey/231/base -> origin/gh/guangyey/231/base 2025-12-04T09:43:53.5075964Z * [new branch] gh/guangyey/231/head -> origin/gh/guangyey/231/head 2025-12-04T09:43:53.5077778Z * [new branch] gh/guangyey/231/orig -> origin/gh/guangyey/231/orig 2025-12-04T09:43:53.5080410Z * [new branch] gh/guangyey/232/base -> origin/gh/guangyey/232/base 2025-12-04T09:43:53.5082241Z * [new branch] gh/guangyey/232/head -> origin/gh/guangyey/232/head 2025-12-04T09:43:53.5083963Z * [new branch] gh/guangyey/232/orig -> origin/gh/guangyey/232/orig 2025-12-04T09:43:53.5086402Z * [new branch] gh/guangyey/233/base -> origin/gh/guangyey/233/base 2025-12-04T09:43:53.5088079Z * [new branch] gh/guangyey/233/head -> origin/gh/guangyey/233/head 2025-12-04T09:43:53.5089909Z * [new branch] gh/guangyey/233/orig -> origin/gh/guangyey/233/orig 2025-12-04T09:43:53.5092303Z * [new branch] gh/guangyey/234/base -> origin/gh/guangyey/234/base 2025-12-04T09:43:53.5094050Z * [new branch] gh/guangyey/234/head -> origin/gh/guangyey/234/head 2025-12-04T09:43:53.5095775Z * [new branch] gh/guangyey/234/orig -> origin/gh/guangyey/234/orig 2025-12-04T09:43:53.5098173Z * [new branch] gh/guangyey/235/base -> origin/gh/guangyey/235/base 2025-12-04T09:43:53.5099814Z * [new branch] gh/guangyey/235/head -> origin/gh/guangyey/235/head 2025-12-04T09:43:53.5101641Z * [new branch] gh/guangyey/235/orig -> origin/gh/guangyey/235/orig 2025-12-04T09:43:53.5104204Z * [new branch] gh/guangyey/236/base -> origin/gh/guangyey/236/base 2025-12-04T09:43:53.5106254Z * [new branch] gh/guangyey/236/head -> origin/gh/guangyey/236/head 2025-12-04T09:43:53.5107855Z * [new branch] gh/guangyey/236/orig -> origin/gh/guangyey/236/orig 2025-12-04T09:43:53.5110141Z * [new branch] gh/guangyey/237/base -> origin/gh/guangyey/237/base 2025-12-04T09:43:53.5111936Z * [new branch] gh/guangyey/237/head -> origin/gh/guangyey/237/head 2025-12-04T09:43:53.5113674Z * [new branch] gh/guangyey/237/orig -> origin/gh/guangyey/237/orig 2025-12-04T09:43:53.5116128Z * [new branch] gh/guangyey/238/base -> origin/gh/guangyey/238/base 2025-12-04T09:43:53.5117809Z * [new branch] gh/guangyey/238/head -> origin/gh/guangyey/238/head 2025-12-04T09:43:53.5120153Z * [new branch] gh/guangyey/239/base -> origin/gh/guangyey/239/base 2025-12-04T09:43:53.5121893Z * [new branch] gh/guangyey/239/head -> origin/gh/guangyey/239/head 2025-12-04T09:43:53.5123660Z * [new branch] gh/guangyey/239/orig -> origin/gh/guangyey/239/orig 2025-12-04T09:43:53.5126085Z * [new branch] gh/guangyey/240/base -> origin/gh/guangyey/240/base 2025-12-04T09:43:53.5127913Z * [new branch] gh/guangyey/240/head -> origin/gh/guangyey/240/head 2025-12-04T09:43:53.5129651Z * [new branch] gh/guangyey/240/orig -> origin/gh/guangyey/240/orig 2025-12-04T09:43:53.5132103Z * [new branch] gh/guangyey/241/base -> origin/gh/guangyey/241/base 2025-12-04T09:43:53.5133816Z * [new branch] gh/guangyey/241/head -> origin/gh/guangyey/241/head 2025-12-04T09:43:53.5135557Z * [new branch] gh/guangyey/241/orig -> origin/gh/guangyey/241/orig 2025-12-04T09:43:53.5137964Z * [new branch] gh/guangyey/242/base -> origin/gh/guangyey/242/base 2025-12-04T09:43:53.5139707Z * [new branch] gh/guangyey/242/head -> origin/gh/guangyey/242/head 2025-12-04T09:43:53.5141344Z * [new branch] gh/guangyey/242/orig -> origin/gh/guangyey/242/orig 2025-12-04T09:43:53.5143789Z * [new branch] gh/guangyey/243/base -> origin/gh/guangyey/243/base 2025-12-04T09:43:53.5145596Z * [new branch] gh/guangyey/243/head -> origin/gh/guangyey/243/head 2025-12-04T09:43:53.5147297Z * [new branch] gh/guangyey/243/orig -> origin/gh/guangyey/243/orig 2025-12-04T09:43:53.5149721Z * [new branch] gh/guangyey/244/base -> origin/gh/guangyey/244/base 2025-12-04T09:43:53.5151671Z * [new branch] gh/guangyey/244/head -> origin/gh/guangyey/244/head 2025-12-04T09:43:53.5153424Z * [new branch] gh/guangyey/244/orig -> origin/gh/guangyey/244/orig 2025-12-04T09:43:53.5155764Z * [new branch] gh/guangyey/245/base -> origin/gh/guangyey/245/base 2025-12-04T09:43:53.5157503Z * [new branch] gh/guangyey/245/head -> origin/gh/guangyey/245/head 2025-12-04T09:43:53.5159213Z * [new branch] gh/guangyey/245/orig -> origin/gh/guangyey/245/orig 2025-12-04T09:43:53.5161610Z * [new branch] gh/guangyey/246/base -> origin/gh/guangyey/246/base 2025-12-04T09:43:53.5163302Z * [new branch] gh/guangyey/246/head -> origin/gh/guangyey/246/head 2025-12-04T09:43:53.5165043Z * [new branch] gh/guangyey/246/orig -> origin/gh/guangyey/246/orig 2025-12-04T09:43:53.5167542Z * [new branch] gh/guangyey/247/base -> origin/gh/guangyey/247/base 2025-12-04T09:43:53.5169320Z * [new branch] gh/guangyey/247/head -> origin/gh/guangyey/247/head 2025-12-04T09:43:53.5171056Z * [new branch] gh/guangyey/247/orig -> origin/gh/guangyey/247/orig 2025-12-04T09:43:53.5173453Z * [new branch] gh/guangyey/248/base -> origin/gh/guangyey/248/base 2025-12-04T09:43:53.5175340Z * [new branch] gh/guangyey/248/head -> origin/gh/guangyey/248/head 2025-12-04T09:43:53.5176964Z * [new branch] gh/guangyey/248/orig -> origin/gh/guangyey/248/orig 2025-12-04T09:43:53.5179667Z * [new branch] gh/guangyey/249/base -> origin/gh/guangyey/249/base 2025-12-04T09:43:53.5181524Z * [new branch] gh/guangyey/249/head -> origin/gh/guangyey/249/head 2025-12-04T09:43:53.5183225Z * [new branch] gh/guangyey/249/orig -> origin/gh/guangyey/249/orig 2025-12-04T09:43:53.5185794Z * [new branch] gh/guangyey/250/base -> origin/gh/guangyey/250/base 2025-12-04T09:43:53.5187495Z * [new branch] gh/guangyey/250/head -> origin/gh/guangyey/250/head 2025-12-04T09:43:53.5189236Z * [new branch] gh/guangyey/250/orig -> origin/gh/guangyey/250/orig 2025-12-04T09:43:53.5191582Z * [new branch] gh/guangyey/251/base -> origin/gh/guangyey/251/base 2025-12-04T09:43:53.5193317Z * [new branch] gh/guangyey/251/head -> origin/gh/guangyey/251/head 2025-12-04T09:43:53.5195139Z * [new branch] gh/guangyey/251/orig -> origin/gh/guangyey/251/orig 2025-12-04T09:43:53.5197509Z * [new branch] gh/guangyey/252/base -> origin/gh/guangyey/252/base 2025-12-04T09:43:53.5199321Z * [new branch] gh/guangyey/252/head -> origin/gh/guangyey/252/head 2025-12-04T09:43:53.5201035Z * [new branch] gh/guangyey/252/orig -> origin/gh/guangyey/252/orig 2025-12-04T09:43:53.5203412Z * [new branch] gh/guangyey/253/base -> origin/gh/guangyey/253/base 2025-12-04T09:43:53.5205604Z * [new branch] gh/guangyey/253/head -> origin/gh/guangyey/253/head 2025-12-04T09:43:53.5207370Z * [new branch] gh/guangyey/253/orig -> origin/gh/guangyey/253/orig 2025-12-04T09:43:53.5209828Z * [new branch] gh/guangyey/254/base -> origin/gh/guangyey/254/base 2025-12-04T09:43:53.5211546Z * [new branch] gh/guangyey/254/head -> origin/gh/guangyey/254/head 2025-12-04T09:43:53.5213364Z * [new branch] gh/guangyey/254/orig -> origin/gh/guangyey/254/orig 2025-12-04T09:43:53.5215766Z * [new branch] gh/guangyey/255/base -> origin/gh/guangyey/255/base 2025-12-04T09:43:53.5218003Z * [new branch] gh/guangyey/255/head -> origin/gh/guangyey/255/head 2025-12-04T09:43:53.5219774Z * [new branch] gh/guangyey/255/orig -> origin/gh/guangyey/255/orig 2025-12-04T09:43:53.5222807Z * [new branch] gh/guilhermeleobas/107/base -> origin/gh/guilhermeleobas/107/base 2025-12-04T09:43:53.5224871Z * [new branch] gh/guilhermeleobas/107/head -> origin/gh/guilhermeleobas/107/head 2025-12-04T09:43:53.5227119Z * [new branch] gh/guilhermeleobas/107/orig -> origin/gh/guilhermeleobas/107/orig 2025-12-04T09:43:53.5230368Z * [new branch] gh/guilhermeleobas/108/base -> origin/gh/guilhermeleobas/108/base 2025-12-04T09:43:53.5232767Z * [new branch] gh/guilhermeleobas/108/head -> origin/gh/guilhermeleobas/108/head 2025-12-04T09:43:53.5235117Z * [new branch] gh/guilhermeleobas/108/orig -> origin/gh/guilhermeleobas/108/orig 2025-12-04T09:43:53.5238494Z * [new branch] gh/guilhermeleobas/150/base -> origin/gh/guilhermeleobas/150/base 2025-12-04T09:43:53.5242196Z * [new branch] gh/guilhermeleobas/150/head -> origin/gh/guilhermeleobas/150/head 2025-12-04T09:43:53.5245486Z * [new branch] gh/guilhermeleobas/150/orig -> origin/gh/guilhermeleobas/150/orig 2025-12-04T09:43:53.5247093Z * [new branch] gh/guilhermeleobas/168/base -> origin/gh/guilhermeleobas/168/base 2025-12-04T09:43:53.5248869Z * [new branch] gh/guilhermeleobas/168/head -> origin/gh/guilhermeleobas/168/head 2025-12-04T09:43:53.5250471Z * [new branch] gh/guilhermeleobas/168/orig -> origin/gh/guilhermeleobas/168/orig 2025-12-04T09:43:53.5253048Z * [new branch] gh/guilhermeleobas/169/base -> origin/gh/guilhermeleobas/169/base 2025-12-04T09:43:53.5254693Z * [new branch] gh/guilhermeleobas/169/head -> origin/gh/guilhermeleobas/169/head 2025-12-04T09:43:53.5256481Z * [new branch] gh/guilhermeleobas/169/orig -> origin/gh/guilhermeleobas/169/orig 2025-12-04T09:43:53.5258790Z * [new branch] gh/guilhermeleobas/170/base -> origin/gh/guilhermeleobas/170/base 2025-12-04T09:43:53.5260719Z * [new branch] gh/guilhermeleobas/170/head -> origin/gh/guilhermeleobas/170/head 2025-12-04T09:43:53.5262461Z * [new branch] gh/guilhermeleobas/170/orig -> origin/gh/guilhermeleobas/170/orig 2025-12-04T09:43:53.5265011Z * [new branch] gh/guilhermeleobas/171/base -> origin/gh/guilhermeleobas/171/base 2025-12-04T09:43:53.5266739Z * [new branch] gh/guilhermeleobas/171/head -> origin/gh/guilhermeleobas/171/head 2025-12-04T09:43:53.5268547Z * [new branch] gh/guilhermeleobas/171/orig -> origin/gh/guilhermeleobas/171/orig 2025-12-04T09:43:53.5270802Z * [new branch] gh/guilhermeleobas/173/base -> origin/gh/guilhermeleobas/173/base 2025-12-04T09:43:53.5272511Z * [new branch] gh/guilhermeleobas/173/head -> origin/gh/guilhermeleobas/173/head 2025-12-04T09:43:53.5274254Z * [new branch] gh/guilhermeleobas/173/orig -> origin/gh/guilhermeleobas/173/orig 2025-12-04T09:43:53.5276672Z * [new branch] gh/guilhermeleobas/193/base -> origin/gh/guilhermeleobas/193/base 2025-12-04T09:43:53.5278785Z * [new branch] gh/guilhermeleobas/193/head -> origin/gh/guilhermeleobas/193/head 2025-12-04T09:43:53.5280542Z * [new branch] gh/guilhermeleobas/193/orig -> origin/gh/guilhermeleobas/193/orig 2025-12-04T09:43:53.5282909Z * [new branch] gh/guilhermeleobas/204/base -> origin/gh/guilhermeleobas/204/base 2025-12-04T09:43:53.5284766Z * [new branch] gh/guilhermeleobas/204/head -> origin/gh/guilhermeleobas/204/head 2025-12-04T09:43:53.5286420Z * [new branch] gh/guilhermeleobas/204/orig -> origin/gh/guilhermeleobas/204/orig 2025-12-04T09:43:53.5288947Z * [new branch] gh/guilhermeleobas/211/base -> origin/gh/guilhermeleobas/211/base 2025-12-04T09:43:53.5290701Z * [new branch] gh/guilhermeleobas/211/head -> origin/gh/guilhermeleobas/211/head 2025-12-04T09:43:53.5292451Z * [new branch] gh/guilhermeleobas/211/orig -> origin/gh/guilhermeleobas/211/orig 2025-12-04T09:43:53.5294794Z * [new branch] gh/guilhermeleobas/226/base -> origin/gh/guilhermeleobas/226/base 2025-12-04T09:43:53.5296500Z * [new branch] gh/guilhermeleobas/226/head -> origin/gh/guilhermeleobas/226/head 2025-12-04T09:43:53.5298292Z * [new branch] gh/guilhermeleobas/226/orig -> origin/gh/guilhermeleobas/226/orig 2025-12-04T09:43:53.5300577Z * [new branch] gh/guilhermeleobas/236/base -> origin/gh/guilhermeleobas/236/base 2025-12-04T09:43:53.5302418Z * [new branch] gh/guilhermeleobas/236/head -> origin/gh/guilhermeleobas/236/head 2025-12-04T09:43:53.5304296Z * [new branch] gh/guilhermeleobas/236/orig -> origin/gh/guilhermeleobas/236/orig 2025-12-04T09:43:53.5306630Z * [new branch] gh/guilhermeleobas/247/base -> origin/gh/guilhermeleobas/247/base 2025-12-04T09:43:53.5308300Z * [new branch] gh/guilhermeleobas/247/head -> origin/gh/guilhermeleobas/247/head 2025-12-04T09:43:53.5310107Z * [new branch] gh/guilhermeleobas/247/orig -> origin/gh/guilhermeleobas/247/orig 2025-12-04T09:43:53.5312486Z * [new branch] gh/guilhermeleobas/248/base -> origin/gh/guilhermeleobas/248/base 2025-12-04T09:43:53.5314228Z * [new branch] gh/guilhermeleobas/248/head -> origin/gh/guilhermeleobas/248/head 2025-12-04T09:43:53.5316027Z * [new branch] gh/guilhermeleobas/248/orig -> origin/gh/guilhermeleobas/248/orig 2025-12-04T09:43:53.5320313Z * [new branch] gh/guilhermeleobas/250/base -> origin/gh/guilhermeleobas/250/base 2025-12-04T09:43:53.5320960Z * [new branch] gh/guilhermeleobas/250/head -> origin/gh/guilhermeleobas/250/head 2025-12-04T09:43:53.5322500Z * [new branch] gh/guilhermeleobas/250/orig -> origin/gh/guilhermeleobas/250/orig 2025-12-04T09:43:53.5325457Z * [new branch] gh/guilhermeleobas/253/base -> origin/gh/guilhermeleobas/253/base 2025-12-04T09:43:53.5327215Z * [new branch] gh/guilhermeleobas/253/head -> origin/gh/guilhermeleobas/253/head 2025-12-04T09:43:53.5328953Z * [new branch] gh/guilhermeleobas/253/orig -> origin/gh/guilhermeleobas/253/orig 2025-12-04T09:43:53.5331416Z * [new branch] gh/guilhermeleobas/254/base -> origin/gh/guilhermeleobas/254/base 2025-12-04T09:43:53.5333151Z * [new branch] gh/guilhermeleobas/254/head -> origin/gh/guilhermeleobas/254/head 2025-12-04T09:43:53.5334877Z * [new branch] gh/guilhermeleobas/254/orig -> origin/gh/guilhermeleobas/254/orig 2025-12-04T09:43:53.5337540Z * [new branch] gh/guilhermeleobas/255/base -> origin/gh/guilhermeleobas/255/base 2025-12-04T09:43:53.5339167Z * [new branch] gh/guilhermeleobas/255/head -> origin/gh/guilhermeleobas/255/head 2025-12-04T09:43:53.5341625Z * [new branch] gh/guilhermeleobas/255/orig -> origin/gh/guilhermeleobas/255/orig 2025-12-04T09:43:53.5344059Z * [new branch] gh/guilhermeleobas/256/base -> origin/gh/guilhermeleobas/256/base 2025-12-04T09:43:53.5346110Z * [new branch] gh/guilhermeleobas/256/head -> origin/gh/guilhermeleobas/256/head 2025-12-04T09:43:53.5347601Z * [new branch] gh/guilhermeleobas/256/orig -> origin/gh/guilhermeleobas/256/orig 2025-12-04T09:43:53.5350042Z * [new branch] gh/guilhermeleobas/257/base -> origin/gh/guilhermeleobas/257/base 2025-12-04T09:43:53.5351884Z * [new branch] gh/guilhermeleobas/257/head -> origin/gh/guilhermeleobas/257/head 2025-12-04T09:43:53.5353698Z * [new branch] gh/guilhermeleobas/257/orig -> origin/gh/guilhermeleobas/257/orig 2025-12-04T09:43:53.5356069Z * [new branch] gh/guilhermeleobas/258/base -> origin/gh/guilhermeleobas/258/base 2025-12-04T09:43:53.5357824Z * [new branch] gh/guilhermeleobas/258/head -> origin/gh/guilhermeleobas/258/head 2025-12-04T09:43:53.5359588Z * [new branch] gh/guilhermeleobas/258/orig -> origin/gh/guilhermeleobas/258/orig 2025-12-04T09:43:53.5362084Z * [new branch] gh/guilhermeleobas/259/base -> origin/gh/guilhermeleobas/259/base 2025-12-04T09:43:53.5363857Z * [new branch] gh/guilhermeleobas/259/head -> origin/gh/guilhermeleobas/259/head 2025-12-04T09:43:53.5365620Z * [new branch] gh/guilhermeleobas/259/orig -> origin/gh/guilhermeleobas/259/orig 2025-12-04T09:43:53.5368041Z * [new branch] gh/guilhermeleobas/260/base -> origin/gh/guilhermeleobas/260/base 2025-12-04T09:43:53.5369706Z * [new branch] gh/guilhermeleobas/260/head -> origin/gh/guilhermeleobas/260/head 2025-12-04T09:43:53.5371490Z * [new branch] gh/guilhermeleobas/260/orig -> origin/gh/guilhermeleobas/260/orig 2025-12-04T09:43:53.5373929Z * [new branch] gh/guilhermeleobas/261/base -> origin/gh/guilhermeleobas/261/base 2025-12-04T09:43:53.5375643Z * [new branch] gh/guilhermeleobas/261/head -> origin/gh/guilhermeleobas/261/head 2025-12-04T09:43:53.5377376Z * [new branch] gh/guilhermeleobas/261/orig -> origin/gh/guilhermeleobas/261/orig 2025-12-04T09:43:53.5381717Z * [new branch] gh/guilhermeleobas/262/base -> origin/gh/guilhermeleobas/262/base 2025-12-04T09:43:53.5384011Z * [new branch] gh/guilhermeleobas/262/head -> origin/gh/guilhermeleobas/262/head 2025-12-04T09:43:53.5385512Z * [new branch] gh/guilhermeleobas/262/orig -> origin/gh/guilhermeleobas/262/orig 2025-12-04T09:43:53.5387986Z * [new branch] gh/guilhermeleobas/263/base -> origin/gh/guilhermeleobas/263/base 2025-12-04T09:43:53.5389618Z * [new branch] gh/guilhermeleobas/263/head -> origin/gh/guilhermeleobas/263/head 2025-12-04T09:43:53.5391383Z * [new branch] gh/guilhermeleobas/263/orig -> origin/gh/guilhermeleobas/263/orig 2025-12-04T09:43:53.5393891Z * [new branch] gh/guilhermeleobas/264/base -> origin/gh/guilhermeleobas/264/base 2025-12-04T09:43:53.5395642Z * [new branch] gh/guilhermeleobas/264/head -> origin/gh/guilhermeleobas/264/head 2025-12-04T09:43:53.5397311Z * [new branch] gh/guilhermeleobas/264/orig -> origin/gh/guilhermeleobas/264/orig 2025-12-04T09:43:53.5400257Z * [new branch] gh/guilhermeleobas/265/base -> origin/gh/guilhermeleobas/265/base 2025-12-04T09:43:53.5402023Z * [new branch] gh/guilhermeleobas/265/head -> origin/gh/guilhermeleobas/265/head 2025-12-04T09:43:53.5403847Z * [new branch] gh/guilhermeleobas/265/orig -> origin/gh/guilhermeleobas/265/orig 2025-12-04T09:43:53.5406248Z * [new branch] gh/guilhermeleobas/266/base -> origin/gh/guilhermeleobas/266/base 2025-12-04T09:43:53.5408012Z * [new branch] gh/guilhermeleobas/266/head -> origin/gh/guilhermeleobas/266/head 2025-12-04T09:43:53.5409784Z * [new branch] gh/guilhermeleobas/266/orig -> origin/gh/guilhermeleobas/266/orig 2025-12-04T09:43:53.5412817Z * [new branch] gh/guilhermeleobas/267/base -> origin/gh/guilhermeleobas/267/base 2025-12-04T09:43:53.5414627Z * [new branch] gh/guilhermeleobas/267/head -> origin/gh/guilhermeleobas/267/head 2025-12-04T09:43:53.5416366Z * [new branch] gh/guilhermeleobas/267/orig -> origin/gh/guilhermeleobas/267/orig 2025-12-04T09:43:53.5419221Z * [new branch] gh/hameerabbasi/1/base -> origin/gh/hameerabbasi/1/base 2025-12-04T09:43:53.5421492Z * [new branch] gh/hameerabbasi/1/head -> origin/gh/hameerabbasi/1/head 2025-12-04T09:43:53.5423891Z * [new branch] gh/hameerabbasi/2/base -> origin/gh/hameerabbasi/2/base 2025-12-04T09:43:53.5425705Z * [new branch] gh/hameerabbasi/2/head -> origin/gh/hameerabbasi/2/head 2025-12-04T09:43:53.5427504Z * [new branch] gh/hameerabbasi/2/orig -> origin/gh/hameerabbasi/2/orig 2025-12-04T09:43:53.5429740Z * [new branch] gh/hameerabbasi/3/base -> origin/gh/hameerabbasi/3/base 2025-12-04T09:43:53.5431486Z * [new branch] gh/hameerabbasi/3/head -> origin/gh/hameerabbasi/3/head 2025-12-04T09:43:53.5433315Z * [new branch] gh/hameerabbasi/3/orig -> origin/gh/hameerabbasi/3/orig 2025-12-04T09:43:53.5435596Z * [new branch] gh/hameerabbasi/4/base -> origin/gh/hameerabbasi/4/base 2025-12-04T09:43:53.5437358Z * [new branch] gh/hameerabbasi/4/head -> origin/gh/hameerabbasi/4/head 2025-12-04T09:43:53.5439017Z * [new branch] gh/hameerabbasi/4/orig -> origin/gh/hameerabbasi/4/orig 2025-12-04T09:43:53.5441817Z * [new branch] gh/huydhn/1/next -> origin/gh/huydhn/1/next 2025-12-04T09:43:53.5444598Z * [new branch] gh/huydhn/2/next -> origin/gh/huydhn/2/next 2025-12-04T09:43:53.5447057Z * [new branch] gh/huydhn/3/next -> origin/gh/huydhn/3/next 2025-12-04T09:43:53.5449339Z * [new branch] gh/huydhn/4/next -> origin/gh/huydhn/4/next 2025-12-04T09:43:53.5451725Z * [new branch] gh/huydhn/5/next -> origin/gh/huydhn/5/next 2025-12-04T09:43:53.5454040Z * [new branch] gh/huydhn/6/next -> origin/gh/huydhn/6/next 2025-12-04T09:43:53.5456823Z * [new branch] gh/int3/97/base -> origin/gh/int3/97/base 2025-12-04T09:43:53.5458522Z * [new branch] gh/int3/97/head -> origin/gh/int3/97/head 2025-12-04T09:43:53.5461529Z * [new branch] gh/isuruf/101/base -> origin/gh/isuruf/101/base 2025-12-04T09:43:53.5463174Z * [new branch] gh/isuruf/101/head -> origin/gh/isuruf/101/head 2025-12-04T09:43:53.5465765Z * [new branch] gh/isuruf/146/base -> origin/gh/isuruf/146/base 2025-12-04T09:43:53.5467438Z * [new branch] gh/isuruf/146/head -> origin/gh/isuruf/146/head 2025-12-04T09:43:53.5469156Z * [new branch] gh/isuruf/146/orig -> origin/gh/isuruf/146/orig 2025-12-04T09:43:53.5471466Z * [new branch] gh/isuruf/158/base -> origin/gh/isuruf/158/base 2025-12-04T09:43:53.5473114Z * [new branch] gh/isuruf/158/head -> origin/gh/isuruf/158/head 2025-12-04T09:43:53.5475365Z * [new branch] gh/isuruf/159/base -> origin/gh/isuruf/159/base 2025-12-04T09:43:53.5477072Z * [new branch] gh/isuruf/159/head -> origin/gh/isuruf/159/head 2025-12-04T09:43:53.5479872Z * [new branch] gh/isuruf/160/base -> origin/gh/isuruf/160/base 2025-12-04T09:43:53.5481465Z * [new branch] gh/isuruf/160/head -> origin/gh/isuruf/160/head 2025-12-04T09:43:53.5483252Z * [new branch] gh/isuruf/160/orig -> origin/gh/isuruf/160/orig 2025-12-04T09:43:53.5485555Z * [new branch] gh/isuruf/81/base -> origin/gh/isuruf/81/base 2025-12-04T09:43:53.5487359Z * [new branch] gh/isuruf/81/head -> origin/gh/isuruf/81/head 2025-12-04T09:43:53.5489106Z * [new branch] gh/isuruf/81/orig -> origin/gh/isuruf/81/orig 2025-12-04T09:43:53.5491869Z * [new branch] gh/jamesjwu/176/base -> origin/gh/jamesjwu/176/base 2025-12-04T09:43:53.5493692Z * [new branch] gh/jamesjwu/176/head -> origin/gh/jamesjwu/176/head 2025-12-04T09:43:53.5495393Z * [new branch] gh/jamesjwu/176/orig -> origin/gh/jamesjwu/176/orig 2025-12-04T09:43:53.5497682Z * [new branch] gh/jamesjwu/187/base -> origin/gh/jamesjwu/187/base 2025-12-04T09:43:53.5499427Z * [new branch] gh/jamesjwu/187/head -> origin/gh/jamesjwu/187/head 2025-12-04T09:43:53.5501436Z * [new branch] gh/jamesjwu/187/orig -> origin/gh/jamesjwu/187/orig 2025-12-04T09:43:53.5503795Z * [new branch] gh/jamesjwu/196/base -> origin/gh/jamesjwu/196/base 2025-12-04T09:43:53.5505562Z * [new branch] gh/jamesjwu/196/head -> origin/gh/jamesjwu/196/head 2025-12-04T09:43:53.5507274Z * [new branch] gh/jamesjwu/196/orig -> origin/gh/jamesjwu/196/orig 2025-12-04T09:43:53.5509666Z * [new branch] gh/jamesjwu/198/base -> origin/gh/jamesjwu/198/base 2025-12-04T09:43:53.5511398Z * [new branch] gh/jamesjwu/198/head -> origin/gh/jamesjwu/198/head 2025-12-04T09:43:53.5513152Z * [new branch] gh/jamesjwu/198/orig -> origin/gh/jamesjwu/198/orig 2025-12-04T09:43:53.5515537Z * [new branch] gh/jamesjwu/207/base -> origin/gh/jamesjwu/207/base 2025-12-04T09:43:53.5517439Z * [new branch] gh/jamesjwu/207/head -> origin/gh/jamesjwu/207/head 2025-12-04T09:43:53.5519174Z * [new branch] gh/jamesjwu/207/orig -> origin/gh/jamesjwu/207/orig 2025-12-04T09:43:53.5521614Z * [new branch] gh/jamesjwu/208/base -> origin/gh/jamesjwu/208/base 2025-12-04T09:43:53.5523458Z * [new branch] gh/jamesjwu/208/head -> origin/gh/jamesjwu/208/head 2025-12-04T09:43:53.5525164Z * [new branch] gh/jamesjwu/208/orig -> origin/gh/jamesjwu/208/orig 2025-12-04T09:43:53.5527593Z * [new branch] gh/jamesjwu/52/base -> origin/gh/jamesjwu/52/base 2025-12-04T09:43:53.5529358Z * [new branch] gh/jamesjwu/52/head -> origin/gh/jamesjwu/52/head 2025-12-04T09:43:53.5531875Z * [new branch] gh/jamesjwu/53/base -> origin/gh/jamesjwu/53/base 2025-12-04T09:43:53.5533433Z * [new branch] gh/jamesjwu/53/head -> origin/gh/jamesjwu/53/head 2025-12-04T09:43:53.5535641Z * [new branch] gh/jamesjwu/54/base -> origin/gh/jamesjwu/54/base 2025-12-04T09:43:53.5537328Z * [new branch] gh/jamesjwu/54/head -> origin/gh/jamesjwu/54/head 2025-12-04T09:43:53.5539551Z * [new branch] gh/jamesjwu/55/base -> origin/gh/jamesjwu/55/base 2025-12-04T09:43:53.5541380Z * [new branch] gh/jamesjwu/55/head -> origin/gh/jamesjwu/55/head 2025-12-04T09:43:53.5543606Z * [new branch] gh/jamesjwu/56/base -> origin/gh/jamesjwu/56/base 2025-12-04T09:43:53.5545442Z * [new branch] gh/jamesjwu/56/head -> origin/gh/jamesjwu/56/head 2025-12-04T09:43:53.5547605Z * [new branch] gh/jamesjwu/57/base -> origin/gh/jamesjwu/57/base 2025-12-04T09:43:53.5549291Z * [new branch] gh/jamesjwu/57/head -> origin/gh/jamesjwu/57/head 2025-12-04T09:43:53.5551438Z * [new branch] gh/jamesjwu/58/base -> origin/gh/jamesjwu/58/base 2025-12-04T09:43:53.5553160Z * [new branch] gh/jamesjwu/58/head -> origin/gh/jamesjwu/58/head 2025-12-04T09:43:53.5555424Z * [new branch] gh/jamesjwu/59/base -> origin/gh/jamesjwu/59/base 2025-12-04T09:43:53.5557207Z * [new branch] gh/jamesjwu/59/head -> origin/gh/jamesjwu/59/head 2025-12-04T09:43:53.5559421Z * [new branch] gh/jamesjwu/60/base -> origin/gh/jamesjwu/60/base 2025-12-04T09:43:53.5561235Z * [new branch] gh/jamesjwu/60/head -> origin/gh/jamesjwu/60/head 2025-12-04T09:43:53.5563369Z * [new branch] gh/jamesjwu/61/base -> origin/gh/jamesjwu/61/base 2025-12-04T09:43:53.5565056Z * [new branch] gh/jamesjwu/61/head -> origin/gh/jamesjwu/61/head 2025-12-04T09:43:53.5567266Z * [new branch] gh/jamesjwu/62/base -> origin/gh/jamesjwu/62/base 2025-12-04T09:43:53.5568971Z * [new branch] gh/jamesjwu/62/head -> origin/gh/jamesjwu/62/head 2025-12-04T09:43:53.5571138Z * [new branch] gh/jamesjwu/63/base -> origin/gh/jamesjwu/63/base 2025-12-04T09:43:53.5572884Z * [new branch] gh/jamesjwu/63/head -> origin/gh/jamesjwu/63/head 2025-12-04T09:43:53.5575839Z * [new branch] gh/jamesjwu/64/base -> origin/gh/jamesjwu/64/base 2025-12-04T09:43:53.5577618Z * [new branch] gh/jamesjwu/64/head -> origin/gh/jamesjwu/64/head 2025-12-04T09:43:53.5581238Z * [new branch] gh/jamesjwu/65/base -> origin/gh/jamesjwu/65/base 2025-12-04T09:43:53.5582884Z * [new branch] gh/jamesjwu/65/head -> origin/gh/jamesjwu/65/head 2025-12-04T09:43:53.5586004Z * [new branch] gh/janeyx99/165/base -> origin/gh/janeyx99/165/base 2025-12-04T09:43:53.5587719Z * [new branch] gh/janeyx99/165/head -> origin/gh/janeyx99/165/head 2025-12-04T09:43:53.5589450Z * [new branch] gh/janeyx99/165/orig -> origin/gh/janeyx99/165/orig 2025-12-04T09:43:53.5591795Z * [new branch] gh/janeyx99/201/base -> origin/gh/janeyx99/201/base 2025-12-04T09:43:53.5593480Z * [new branch] gh/janeyx99/201/head -> origin/gh/janeyx99/201/head 2025-12-04T09:43:53.5595165Z * [new branch] gh/janeyx99/201/orig -> origin/gh/janeyx99/201/orig 2025-12-04T09:43:53.5597755Z * [new branch] gh/janeyx99/225/base -> origin/gh/janeyx99/225/base 2025-12-04T09:43:53.5599570Z * [new branch] gh/janeyx99/225/head -> origin/gh/janeyx99/225/head 2025-12-04T09:43:53.5601420Z * [new branch] gh/janeyx99/225/orig -> origin/gh/janeyx99/225/orig 2025-12-04T09:43:53.5603748Z * [new branch] gh/janeyx99/299/base -> origin/gh/janeyx99/299/base 2025-12-04T09:43:53.5605716Z * [new branch] gh/janeyx99/299/head -> origin/gh/janeyx99/299/head 2025-12-04T09:43:53.5607333Z * [new branch] gh/janeyx99/299/orig -> origin/gh/janeyx99/299/orig 2025-12-04T09:43:53.5609894Z * [new branch] gh/janeyx99/302/base -> origin/gh/janeyx99/302/base 2025-12-04T09:43:53.5611671Z * [new branch] gh/janeyx99/302/head -> origin/gh/janeyx99/302/head 2025-12-04T09:43:53.5613928Z * [new branch] gh/janeyx99/303/base -> origin/gh/janeyx99/303/base 2025-12-04T09:43:53.5615619Z * [new branch] gh/janeyx99/303/head -> origin/gh/janeyx99/303/head 2025-12-04T09:43:53.5617916Z * [new branch] gh/janeyx99/305/base -> origin/gh/janeyx99/305/base 2025-12-04T09:43:53.5619699Z * [new branch] gh/janeyx99/305/head -> origin/gh/janeyx99/305/head 2025-12-04T09:43:53.5622005Z * [new branch] gh/janeyx99/306/base -> origin/gh/janeyx99/306/base 2025-12-04T09:43:53.5623762Z * [new branch] gh/janeyx99/306/head -> origin/gh/janeyx99/306/head 2025-12-04T09:43:53.5626209Z * [new branch] gh/janeyx99/314/base -> origin/gh/janeyx99/314/base 2025-12-04T09:43:53.5627925Z * [new branch] gh/janeyx99/314/head -> origin/gh/janeyx99/314/head 2025-12-04T09:43:53.5629658Z * [new branch] gh/janeyx99/314/orig -> origin/gh/janeyx99/314/orig 2025-12-04T09:43:53.5631983Z * [new branch] gh/janeyx99/315/base -> origin/gh/janeyx99/315/base 2025-12-04T09:43:53.5633697Z * [new branch] gh/janeyx99/315/head -> origin/gh/janeyx99/315/head 2025-12-04T09:43:53.5635525Z * [new branch] gh/janeyx99/315/orig -> origin/gh/janeyx99/315/orig 2025-12-04T09:43:53.5637866Z * [new branch] gh/janeyx99/316/base -> origin/gh/janeyx99/316/base 2025-12-04T09:43:53.5639661Z * [new branch] gh/janeyx99/316/head -> origin/gh/janeyx99/316/head 2025-12-04T09:43:53.5641388Z * [new branch] gh/janeyx99/316/orig -> origin/gh/janeyx99/316/orig 2025-12-04T09:43:53.5644436Z * [new branch] gh/janeyx99/317/base -> origin/gh/janeyx99/317/base 2025-12-04T09:43:53.5646230Z * [new branch] gh/janeyx99/317/head -> origin/gh/janeyx99/317/head 2025-12-04T09:43:53.5647887Z * [new branch] gh/janeyx99/317/orig -> origin/gh/janeyx99/317/orig 2025-12-04T09:43:53.5650268Z * [new branch] gh/janeyx99/325/base -> origin/gh/janeyx99/325/base 2025-12-04T09:43:53.5652034Z * [new branch] gh/janeyx99/325/head -> origin/gh/janeyx99/325/head 2025-12-04T09:43:53.5653978Z * [new branch] gh/janeyx99/325/orig -> origin/gh/janeyx99/325/orig 2025-12-04T09:43:53.5656666Z * [new branch] gh/janeyx99/327/base -> origin/gh/janeyx99/327/base 2025-12-04T09:43:53.5658386Z * [new branch] gh/janeyx99/327/head -> origin/gh/janeyx99/327/head 2025-12-04T09:43:53.5660009Z * [new branch] gh/janeyx99/327/orig -> origin/gh/janeyx99/327/orig 2025-12-04T09:43:53.5662382Z * [new branch] gh/janeyx99/328/base -> origin/gh/janeyx99/328/base 2025-12-04T09:43:53.5664252Z * [new branch] gh/janeyx99/328/head -> origin/gh/janeyx99/328/head 2025-12-04T09:43:53.5666028Z * [new branch] gh/janeyx99/328/orig -> origin/gh/janeyx99/328/orig 2025-12-04T09:43:53.5668352Z * [new branch] gh/janeyx99/329/base -> origin/gh/janeyx99/329/base 2025-12-04T09:43:53.5670211Z * [new branch] gh/janeyx99/329/head -> origin/gh/janeyx99/329/head 2025-12-04T09:43:53.5672045Z * [new branch] gh/janeyx99/329/orig -> origin/gh/janeyx99/329/orig 2025-12-04T09:43:53.5674895Z * [new branch] gh/janeyx99/330/base -> origin/gh/janeyx99/330/base 2025-12-04T09:43:53.5677055Z * [new branch] gh/janeyx99/330/head -> origin/gh/janeyx99/330/head 2025-12-04T09:43:53.5679141Z * [new branch] gh/janeyx99/330/orig -> origin/gh/janeyx99/330/orig 2025-12-04T09:43:53.5681408Z * [new branch] gh/janeyx99/331/base -> origin/gh/janeyx99/331/base 2025-12-04T09:43:53.5683165Z * [new branch] gh/janeyx99/331/head -> origin/gh/janeyx99/331/head 2025-12-04T09:43:53.5684939Z * [new branch] gh/janeyx99/331/orig -> origin/gh/janeyx99/331/orig 2025-12-04T09:43:53.5687297Z * [new branch] gh/janeyx99/332/base -> origin/gh/janeyx99/332/base 2025-12-04T09:43:53.5689292Z * [new branch] gh/janeyx99/332/head -> origin/gh/janeyx99/332/head 2025-12-04T09:43:53.5690785Z * [new branch] gh/janeyx99/332/orig -> origin/gh/janeyx99/332/orig 2025-12-04T09:43:53.5693411Z * [new branch] gh/janeyx99/333/base -> origin/gh/janeyx99/333/base 2025-12-04T09:43:53.5695245Z * [new branch] gh/janeyx99/333/head -> origin/gh/janeyx99/333/head 2025-12-04T09:43:53.5696978Z * [new branch] gh/janeyx99/333/orig -> origin/gh/janeyx99/333/orig 2025-12-04T09:43:53.5699528Z * [new branch] gh/janeyx99/88/base -> origin/gh/janeyx99/88/base 2025-12-04T09:43:53.5701267Z * [new branch] gh/janeyx99/88/head -> origin/gh/janeyx99/88/head 2025-12-04T09:43:53.5703021Z * [new branch] gh/janeyx99/88/orig -> origin/gh/janeyx99/88/orig 2025-12-04T09:43:53.5706166Z * [new branch] gh/jansel/360/base -> origin/gh/jansel/360/base 2025-12-04T09:43:53.5707888Z * [new branch] gh/jansel/360/head -> origin/gh/jansel/360/head 2025-12-04T09:43:53.5710162Z * [new branch] gh/jansel/451/base -> origin/gh/jansel/451/base 2025-12-04T09:43:53.5712121Z * [new branch] gh/jansel/451/head -> origin/gh/jansel/451/head 2025-12-04T09:43:53.5713898Z * [new branch] gh/jansel/451/orig -> origin/gh/jansel/451/orig 2025-12-04T09:43:53.5716255Z * [new branch] gh/jansel/462/base -> origin/gh/jansel/462/base 2025-12-04T09:43:53.5718095Z * [new branch] gh/jansel/462/head -> origin/gh/jansel/462/head 2025-12-04T09:43:53.5719851Z * [new branch] gh/jansel/462/orig -> origin/gh/jansel/462/orig 2025-12-04T09:43:53.5722128Z * [new branch] gh/jansel/533/base -> origin/gh/jansel/533/base 2025-12-04T09:43:53.5724292Z * [new branch] gh/jansel/533/head -> origin/gh/jansel/533/head 2025-12-04T09:43:53.5726063Z * [new branch] gh/jansel/533/orig -> origin/gh/jansel/533/orig 2025-12-04T09:43:53.5728374Z * [new branch] gh/jansel/552/base -> origin/gh/jansel/552/base 2025-12-04T09:43:53.5730049Z * [new branch] gh/jansel/552/head -> origin/gh/jansel/552/head 2025-12-04T09:43:53.5731902Z * [new branch] gh/jansel/552/orig -> origin/gh/jansel/552/orig 2025-12-04T09:43:53.5734702Z * [new branch] gh/jansel/553/base -> origin/gh/jansel/553/base 2025-12-04T09:43:53.5736496Z * [new branch] gh/jansel/553/head -> origin/gh/jansel/553/head 2025-12-04T09:43:53.5738262Z * [new branch] gh/jansel/553/orig -> origin/gh/jansel/553/orig 2025-12-04T09:43:53.5740557Z * [new branch] gh/jansel/554/base -> origin/gh/jansel/554/base 2025-12-04T09:43:53.5742451Z * [new branch] gh/jansel/554/head -> origin/gh/jansel/554/head 2025-12-04T09:43:53.5744281Z * [new branch] gh/jansel/554/orig -> origin/gh/jansel/554/orig 2025-12-04T09:43:53.5746618Z * [new branch] gh/jansel/555/base -> origin/gh/jansel/555/base 2025-12-04T09:43:53.5748533Z * [new branch] gh/jansel/555/head -> origin/gh/jansel/555/head 2025-12-04T09:43:53.5750087Z * [new branch] gh/jansel/555/orig -> origin/gh/jansel/555/orig 2025-12-04T09:43:53.5752360Z * [new branch] gh/jansel/556/base -> origin/gh/jansel/556/base 2025-12-04T09:43:53.5754263Z * [new branch] gh/jansel/556/head -> origin/gh/jansel/556/head 2025-12-04T09:43:53.5756413Z * [new branch] gh/jansel/556/orig -> origin/gh/jansel/556/orig 2025-12-04T09:43:53.5759867Z * [new branch] gh/jansel/557/base -> origin/gh/jansel/557/base 2025-12-04T09:43:53.5762277Z * [new branch] gh/jansel/557/head -> origin/gh/jansel/557/head 2025-12-04T09:43:53.5764656Z * [new branch] gh/jansel/557/orig -> origin/gh/jansel/557/orig 2025-12-04T09:43:53.5767822Z * [new branch] gh/jansel/558/base -> origin/gh/jansel/558/base 2025-12-04T09:43:53.5770157Z * [new branch] gh/jansel/558/head -> origin/gh/jansel/558/head 2025-12-04T09:43:53.5772405Z * [new branch] gh/jansel/558/orig -> origin/gh/jansel/558/orig 2025-12-04T09:43:53.5775605Z * [new branch] gh/jansel/559/base -> origin/gh/jansel/559/base 2025-12-04T09:43:53.5778056Z * [new branch] gh/jansel/559/head -> origin/gh/jansel/559/head 2025-12-04T09:43:53.5780485Z * [new branch] gh/jansel/559/orig -> origin/gh/jansel/559/orig 2025-12-04T09:43:53.5783825Z * [new branch] gh/jansel/560/base -> origin/gh/jansel/560/base 2025-12-04T09:43:53.5786383Z * [new branch] gh/jansel/560/head -> origin/gh/jansel/560/head 2025-12-04T09:43:53.5788023Z * [new branch] gh/jansel/560/orig -> origin/gh/jansel/560/orig 2025-12-04T09:43:53.5790359Z * [new branch] gh/jansel/561/base -> origin/gh/jansel/561/base 2025-12-04T09:43:53.5792180Z * [new branch] gh/jansel/561/head -> origin/gh/jansel/561/head 2025-12-04T09:43:53.5793931Z * [new branch] gh/jansel/561/orig -> origin/gh/jansel/561/orig 2025-12-04T09:43:53.5796364Z * [new branch] gh/jansel/562/base -> origin/gh/jansel/562/base 2025-12-04T09:43:53.5798206Z * [new branch] gh/jansel/562/head -> origin/gh/jansel/562/head 2025-12-04T09:43:53.5800028Z * [new branch] gh/jansel/562/orig -> origin/gh/jansel/562/orig 2025-12-04T09:43:53.5802445Z * [new branch] gh/jansel/563/base -> origin/gh/jansel/563/base 2025-12-04T09:43:53.5804186Z * [new branch] gh/jansel/563/head -> origin/gh/jansel/563/head 2025-12-04T09:43:53.5805912Z * [new branch] gh/jansel/563/orig -> origin/gh/jansel/563/orig 2025-12-04T09:43:53.5808928Z * [new branch] gh/jansel/564/base -> origin/gh/jansel/564/base 2025-12-04T09:43:53.5810591Z * [new branch] gh/jansel/564/head -> origin/gh/jansel/564/head 2025-12-04T09:43:53.5812315Z * [new branch] gh/jansel/564/orig -> origin/gh/jansel/564/orig 2025-12-04T09:43:53.5814706Z * [new branch] gh/jansel/565/base -> origin/gh/jansel/565/base 2025-12-04T09:43:53.5816497Z * [new branch] gh/jansel/565/head -> origin/gh/jansel/565/head 2025-12-04T09:43:53.5818713Z * [new branch] gh/jansel/565/orig -> origin/gh/jansel/565/orig 2025-12-04T09:43:53.5821241Z * [new branch] gh/jansel/566/base -> origin/gh/jansel/566/base 2025-12-04T09:43:53.5823100Z * [new branch] gh/jansel/566/head -> origin/gh/jansel/566/head 2025-12-04T09:43:53.5825006Z * [new branch] gh/jansel/566/orig -> origin/gh/jansel/566/orig 2025-12-04T09:43:53.5827393Z * [new branch] gh/jansel/567/base -> origin/gh/jansel/567/base 2025-12-04T09:43:53.5829307Z * [new branch] gh/jansel/567/head -> origin/gh/jansel/567/head 2025-12-04T09:43:53.5830858Z * [new branch] gh/jansel/567/orig -> origin/gh/jansel/567/orig 2025-12-04T09:43:53.5833327Z * [new branch] gh/jansel/568/base -> origin/gh/jansel/568/base 2025-12-04T09:43:53.5835023Z * [new branch] gh/jansel/568/head -> origin/gh/jansel/568/head 2025-12-04T09:43:53.5836806Z * [new branch] gh/jansel/568/orig -> origin/gh/jansel/568/orig 2025-12-04T09:43:53.5839198Z * [new branch] gh/jansel/569/base -> origin/gh/jansel/569/base 2025-12-04T09:43:53.5840934Z * [new branch] gh/jansel/569/head -> origin/gh/jansel/569/head 2025-12-04T09:43:53.5842665Z * [new branch] gh/jansel/569/orig -> origin/gh/jansel/569/orig 2025-12-04T09:43:53.5845471Z * [new branch] gh/jansel/570/base -> origin/gh/jansel/570/base 2025-12-04T09:43:53.5847282Z * [new branch] gh/jansel/570/head -> origin/gh/jansel/570/head 2025-12-04T09:43:53.5849014Z * [new branch] gh/jansel/570/orig -> origin/gh/jansel/570/orig 2025-12-04T09:43:53.5851425Z * [new branch] gh/jansel/571/base -> origin/gh/jansel/571/base 2025-12-04T09:43:53.5853171Z * [new branch] gh/jansel/571/head -> origin/gh/jansel/571/head 2025-12-04T09:43:53.5854913Z * [new branch] gh/jansel/571/orig -> origin/gh/jansel/571/orig 2025-12-04T09:43:53.5857219Z * [new branch] gh/jansel/572/base -> origin/gh/jansel/572/base 2025-12-04T09:43:53.5858928Z * [new branch] gh/jansel/572/head -> origin/gh/jansel/572/head 2025-12-04T09:43:53.5860682Z * [new branch] gh/jansel/572/orig -> origin/gh/jansel/572/orig 2025-12-04T09:43:53.5863221Z * [new branch] gh/jansel/573/base -> origin/gh/jansel/573/base 2025-12-04T09:43:53.5865198Z * [new branch] gh/jansel/573/head -> origin/gh/jansel/573/head 2025-12-04T09:43:53.5866862Z * [new branch] gh/jansel/573/orig -> origin/gh/jansel/573/orig 2025-12-04T09:43:53.5869274Z * [new branch] gh/jansel/574/base -> origin/gh/jansel/574/base 2025-12-04T09:43:53.5871097Z * [new branch] gh/jansel/574/head -> origin/gh/jansel/574/head 2025-12-04T09:43:53.5872886Z * [new branch] gh/jansel/574/orig -> origin/gh/jansel/574/orig 2025-12-04T09:43:53.5875237Z * [new branch] gh/jansel/575/base -> origin/gh/jansel/575/base 2025-12-04T09:43:53.5876994Z * [new branch] gh/jansel/575/head -> origin/gh/jansel/575/head 2025-12-04T09:43:53.5879015Z * [new branch] gh/jansel/575/orig -> origin/gh/jansel/575/orig 2025-12-04T09:43:53.5881405Z * [new branch] gh/jansel/576/base -> origin/gh/jansel/576/base 2025-12-04T09:43:53.5883103Z * [new branch] gh/jansel/576/head -> origin/gh/jansel/576/head 2025-12-04T09:43:53.5884854Z * [new branch] gh/jansel/576/orig -> origin/gh/jansel/576/orig 2025-12-04T09:43:53.5887758Z * [new branch] gh/jbschlosser/247/base -> origin/gh/jbschlosser/247/base 2025-12-04T09:43:53.5889467Z * [new branch] gh/jbschlosser/247/head -> origin/gh/jbschlosser/247/head 2025-12-04T09:43:53.5891177Z * [new branch] gh/jbschlosser/247/orig -> origin/gh/jbschlosser/247/orig 2025-12-04T09:43:53.5893588Z * [new branch] gh/jbschlosser/250/base -> origin/gh/jbschlosser/250/base 2025-12-04T09:43:53.5895334Z * [new branch] gh/jbschlosser/250/head -> origin/gh/jbschlosser/250/head 2025-12-04T09:43:53.5897039Z * [new branch] gh/jbschlosser/250/orig -> origin/gh/jbschlosser/250/orig 2025-12-04T09:43:53.5900178Z * [new branch] gh/jerryzh168/1/base -> origin/gh/jerryzh168/1/base 2025-12-04T09:43:53.5902251Z * [new branch] gh/jerryzh168/1/head -> origin/gh/jerryzh168/1/head 2025-12-04T09:43:53.5904045Z * [new branch] gh/jerryzh168/1/orig -> origin/gh/jerryzh168/1/orig 2025-12-04T09:43:53.5906905Z * [new branch] gh/jiayisunx/59/base -> origin/gh/jiayisunx/59/base 2025-12-04T09:43:53.5908613Z * [new branch] gh/jiayisunx/59/head -> origin/gh/jiayisunx/59/head 2025-12-04T09:43:53.5910369Z * [new branch] gh/jiayisunx/59/orig -> origin/gh/jiayisunx/59/orig 2025-12-04T09:43:53.5912823Z * [new branch] gh/jiayisunx/61/base -> origin/gh/jiayisunx/61/base 2025-12-04T09:43:53.5914517Z * [new branch] gh/jiayisunx/61/head -> origin/gh/jiayisunx/61/head 2025-12-04T09:43:53.5916327Z * [new branch] gh/jiayisunx/61/orig -> origin/gh/jiayisunx/61/orig 2025-12-04T09:43:53.5919179Z * [new branch] gh/jiayisunx/68/base -> origin/gh/jiayisunx/68/base 2025-12-04T09:43:53.5920822Z * [new branch] gh/jiayisunx/68/head -> origin/gh/jiayisunx/68/head 2025-12-04T09:43:53.5922565Z * [new branch] gh/jiayisunx/68/orig -> origin/gh/jiayisunx/68/orig 2025-12-04T09:43:53.5925005Z * [new branch] gh/jiayisunx/77/base -> origin/gh/jiayisunx/77/base 2025-12-04T09:43:53.5926657Z * [new branch] gh/jiayisunx/77/head -> origin/gh/jiayisunx/77/head 2025-12-04T09:43:53.5928683Z * [new branch] gh/jiayisunx/77/orig -> origin/gh/jiayisunx/77/orig 2025-12-04T09:43:53.5931097Z * [new branch] gh/jiayisunx/78/base -> origin/gh/jiayisunx/78/base 2025-12-04T09:43:53.5932855Z * [new branch] gh/jiayisunx/78/head -> origin/gh/jiayisunx/78/head 2025-12-04T09:43:53.5934622Z * [new branch] gh/jiayisunx/78/orig -> origin/gh/jiayisunx/78/orig 2025-12-04T09:43:53.5936955Z * [new branch] gh/jiayisunx/79/base -> origin/gh/jiayisunx/79/base 2025-12-04T09:43:53.5938743Z * [new branch] gh/jiayisunx/79/head -> origin/gh/jiayisunx/79/head 2025-12-04T09:43:53.5940507Z * [new branch] gh/jiayisunx/79/orig -> origin/gh/jiayisunx/79/orig 2025-12-04T09:43:53.5942874Z * [new branch] gh/jiayisunx/82/base -> origin/gh/jiayisunx/82/base 2025-12-04T09:43:53.5944751Z * [new branch] gh/jiayisunx/82/head -> origin/gh/jiayisunx/82/head 2025-12-04T09:43:53.5946467Z * [new branch] gh/jiayisunx/82/orig -> origin/gh/jiayisunx/82/orig 2025-12-04T09:43:53.5948824Z * [new branch] gh/jiayisunx/83/base -> origin/gh/jiayisunx/83/base 2025-12-04T09:43:53.5950561Z * [new branch] gh/jiayisunx/83/head -> origin/gh/jiayisunx/83/head 2025-12-04T09:43:53.5952411Z * [new branch] gh/jiayisunx/83/orig -> origin/gh/jiayisunx/83/orig 2025-12-04T09:43:53.5954629Z * [new branch] gh/jiayisunx/84/base -> origin/gh/jiayisunx/84/base 2025-12-04T09:43:53.5956337Z * [new branch] gh/jiayisunx/84/head -> origin/gh/jiayisunx/84/head 2025-12-04T09:43:53.5958045Z * [new branch] gh/jiayisunx/84/orig -> origin/gh/jiayisunx/84/orig 2025-12-04T09:43:53.5960322Z * [new branch] gh/jiayisunx/85/base -> origin/gh/jiayisunx/85/base 2025-12-04T09:43:53.5962491Z * [new branch] gh/jiayisunx/85/head -> origin/gh/jiayisunx/85/head 2025-12-04T09:43:53.5964530Z * [new branch] gh/jiayisunx/85/orig -> origin/gh/jiayisunx/85/orig 2025-12-04T09:43:53.5967097Z * [new branch] gh/jiayisunx/86/base -> origin/gh/jiayisunx/86/base 2025-12-04T09:43:53.5968814Z * [new branch] gh/jiayisunx/86/head -> origin/gh/jiayisunx/86/head 2025-12-04T09:43:53.5970771Z * [new branch] gh/jiayisunx/86/orig -> origin/gh/jiayisunx/86/orig 2025-12-04T09:43:53.5972969Z * [new branch] gh/jiayisunx/87/base -> origin/gh/jiayisunx/87/base 2025-12-04T09:43:53.5974810Z * [new branch] gh/jiayisunx/87/head -> origin/gh/jiayisunx/87/head 2025-12-04T09:43:53.5976517Z * [new branch] gh/jiayisunx/87/orig -> origin/gh/jiayisunx/87/orig 2025-12-04T09:43:53.5980499Z * [new branch] gh/jiayisunx/88/base -> origin/gh/jiayisunx/88/base 2025-12-04T09:43:53.5982240Z * [new branch] gh/jiayisunx/88/head -> origin/gh/jiayisunx/88/head 2025-12-04T09:43:53.5984094Z * [new branch] gh/jiayisunx/88/orig -> origin/gh/jiayisunx/88/orig 2025-12-04T09:43:53.5986432Z * [new branch] gh/jiayisunx/89/base -> origin/gh/jiayisunx/89/base 2025-12-04T09:43:53.5988124Z * [new branch] gh/jiayisunx/89/head -> origin/gh/jiayisunx/89/head 2025-12-04T09:43:53.5989923Z * [new branch] gh/jiayisunx/89/orig -> origin/gh/jiayisunx/89/orig 2025-12-04T09:43:53.5992273Z * [new branch] gh/jiayisunx/90/base -> origin/gh/jiayisunx/90/base 2025-12-04T09:43:53.5993981Z * [new branch] gh/jiayisunx/90/head -> origin/gh/jiayisunx/90/head 2025-12-04T09:43:53.5995701Z * [new branch] gh/jiayisunx/90/orig -> origin/gh/jiayisunx/90/orig 2025-12-04T09:43:53.5998405Z * [new branch] gh/jjwu@meta.com/1/base -> origin/gh/jjwu@meta.com/1/base 2025-12-04T09:43:53.6000114Z * [new branch] gh/jjwu@meta.com/1/head -> origin/gh/jjwu@meta.com/1/head 2025-12-04T09:43:53.6003009Z * [new branch] gh/jturney/1/base -> origin/gh/jturney/1/base 2025-12-04T09:43:53.6004712Z * [new branch] gh/jturney/1/head -> origin/gh/jturney/1/head 2025-12-04T09:43:53.6006430Z * [new branch] gh/jturney/1/orig -> origin/gh/jturney/1/orig 2025-12-04T09:43:53.6008781Z * [new branch] gh/jturney/2/base -> origin/gh/jturney/2/base 2025-12-04T09:43:53.6010494Z * [new branch] gh/jturney/2/head -> origin/gh/jturney/2/head 2025-12-04T09:43:53.6012235Z * [new branch] gh/jturney/2/orig -> origin/gh/jturney/2/orig 2025-12-04T09:43:53.6015110Z * [new branch] gh/karthickai/10/base -> origin/gh/karthickai/10/base 2025-12-04T09:43:53.6017000Z * [new branch] gh/karthickai/10/head -> origin/gh/karthickai/10/head 2025-12-04T09:43:53.6018724Z * [new branch] gh/karthickai/10/orig -> origin/gh/karthickai/10/orig 2025-12-04T09:43:53.6021004Z * [new branch] gh/karthickai/11/base -> origin/gh/karthickai/11/base 2025-12-04T09:43:53.6022803Z * [new branch] gh/karthickai/11/head -> origin/gh/karthickai/11/head 2025-12-04T09:43:53.6024808Z * [new branch] gh/karthickai/11/orig -> origin/gh/karthickai/11/orig 2025-12-04T09:43:53.6027400Z * [new branch] gh/karthickai/12/base -> origin/gh/karthickai/12/base 2025-12-04T09:43:53.6029219Z * [new branch] gh/karthickai/12/head -> origin/gh/karthickai/12/head 2025-12-04T09:43:53.6030996Z * [new branch] gh/karthickai/12/orig -> origin/gh/karthickai/12/orig 2025-12-04T09:43:53.6033476Z * [new branch] gh/karthickai/13/base -> origin/gh/karthickai/13/base 2025-12-04T09:43:53.6035259Z * [new branch] gh/karthickai/13/head -> origin/gh/karthickai/13/head 2025-12-04T09:43:53.6037004Z * [new branch] gh/karthickai/13/orig -> origin/gh/karthickai/13/orig 2025-12-04T09:43:53.6039545Z * [new branch] gh/karthickai/14/base -> origin/gh/karthickai/14/base 2025-12-04T09:43:53.6041395Z * [new branch] gh/karthickai/14/head -> origin/gh/karthickai/14/head 2025-12-04T09:43:53.6043311Z * [new branch] gh/karthickai/14/orig -> origin/gh/karthickai/14/orig 2025-12-04T09:43:53.6045804Z * [new branch] gh/karthickai/15/base -> origin/gh/karthickai/15/base 2025-12-04T09:43:53.6047502Z * [new branch] gh/karthickai/15/head -> origin/gh/karthickai/15/head 2025-12-04T09:43:53.6049287Z * [new branch] gh/karthickai/15/orig -> origin/gh/karthickai/15/orig 2025-12-04T09:43:53.6051512Z * [new branch] gh/karthickai/16/base -> origin/gh/karthickai/16/base 2025-12-04T09:43:53.6053312Z * [new branch] gh/karthickai/16/head -> origin/gh/karthickai/16/head 2025-12-04T09:43:53.6055121Z * [new branch] gh/karthickai/16/orig -> origin/gh/karthickai/16/orig 2025-12-04T09:43:53.6057468Z * [new branch] gh/karthickai/17/base -> origin/gh/karthickai/17/base 2025-12-04T09:43:53.6059172Z * [new branch] gh/karthickai/17/head -> origin/gh/karthickai/17/head 2025-12-04T09:43:53.6060871Z * [new branch] gh/karthickai/17/orig -> origin/gh/karthickai/17/orig 2025-12-04T09:43:53.6063260Z * [new branch] gh/karthickai/18/base -> origin/gh/karthickai/18/base 2025-12-04T09:43:53.6065314Z * [new branch] gh/karthickai/18/head -> origin/gh/karthickai/18/head 2025-12-04T09:43:53.6067257Z * [new branch] gh/karthickai/18/orig -> origin/gh/karthickai/18/orig 2025-12-04T09:43:53.6069615Z * [new branch] gh/karthickai/19/base -> origin/gh/karthickai/19/base 2025-12-04T09:43:53.6071354Z * [new branch] gh/karthickai/19/head -> origin/gh/karthickai/19/head 2025-12-04T09:43:53.6073120Z * [new branch] gh/karthickai/19/orig -> origin/gh/karthickai/19/orig 2025-12-04T09:43:53.6076054Z * [new branch] gh/karthickai/20/base -> origin/gh/karthickai/20/base 2025-12-04T09:43:53.6078366Z * [new branch] gh/karthickai/20/head -> origin/gh/karthickai/20/head 2025-12-04T09:43:53.6080297Z * [new branch] gh/karthickai/20/orig -> origin/gh/karthickai/20/orig 2025-12-04T09:43:53.6082669Z * [new branch] gh/karthickai/21/base -> origin/gh/karthickai/21/base 2025-12-04T09:43:53.6084565Z * [new branch] gh/karthickai/21/head -> origin/gh/karthickai/21/head 2025-12-04T09:43:53.6086342Z * [new branch] gh/karthickai/21/orig -> origin/gh/karthickai/21/orig 2025-12-04T09:43:53.6088871Z * [new branch] gh/karthickai/22/base -> origin/gh/karthickai/22/base 2025-12-04T09:43:53.6090551Z * [new branch] gh/karthickai/22/head -> origin/gh/karthickai/22/head 2025-12-04T09:43:53.6092441Z * [new branch] gh/karthickai/22/orig -> origin/gh/karthickai/22/orig 2025-12-04T09:43:53.6094875Z * [new branch] gh/karthickai/23/base -> origin/gh/karthickai/23/base 2025-12-04T09:43:53.6096752Z * [new branch] gh/karthickai/23/head -> origin/gh/karthickai/23/head 2025-12-04T09:43:53.6098454Z * [new branch] gh/karthickai/23/orig -> origin/gh/karthickai/23/orig 2025-12-04T09:43:53.6100768Z * [new branch] gh/karthickai/24/base -> origin/gh/karthickai/24/base 2025-12-04T09:43:53.6102613Z * [new branch] gh/karthickai/24/head -> origin/gh/karthickai/24/head 2025-12-04T09:43:53.6104518Z * [new branch] gh/karthickai/24/orig -> origin/gh/karthickai/24/orig 2025-12-04T09:43:53.6107311Z * [new branch] gh/karthickai/25/base -> origin/gh/karthickai/25/base 2025-12-04T09:43:53.6109122Z * [new branch] gh/karthickai/25/head -> origin/gh/karthickai/25/head 2025-12-04T09:43:53.6110797Z * [new branch] gh/karthickai/25/orig -> origin/gh/karthickai/25/orig 2025-12-04T09:43:53.6113271Z * [new branch] gh/karthickai/26/base -> origin/gh/karthickai/26/base 2025-12-04T09:43:53.6115195Z * [new branch] gh/karthickai/26/head -> origin/gh/karthickai/26/head 2025-12-04T09:43:53.6116833Z * [new branch] gh/karthickai/26/orig -> origin/gh/karthickai/26/orig 2025-12-04T09:43:53.6120229Z * [new branch] gh/karthickai/6/base -> origin/gh/karthickai/6/base 2025-12-04T09:43:53.6122407Z * [new branch] gh/karthickai/6/head -> origin/gh/karthickai/6/head 2025-12-04T09:43:53.6124205Z * [new branch] gh/karthickai/6/orig -> origin/gh/karthickai/6/orig 2025-12-04T09:43:53.6127010Z * [new branch] gh/krocki/1/base -> origin/gh/krocki/1/base 2025-12-04T09:43:53.6128761Z * [new branch] gh/krocki/1/head -> origin/gh/krocki/1/head 2025-12-04T09:43:53.6130546Z * [new branch] gh/krocki/1/orig -> origin/gh/krocki/1/orig 2025-12-04T09:43:53.6133041Z * [new branch] gh/krocki/2/base -> origin/gh/krocki/2/base 2025-12-04T09:43:53.6134783Z * [new branch] gh/krocki/2/head -> origin/gh/krocki/2/head 2025-12-04T09:43:53.6136489Z * [new branch] gh/krocki/2/orig -> origin/gh/krocki/2/orig 2025-12-04T09:43:53.6139542Z * [new branch] gh/kurtamohler/60/base -> origin/gh/kurtamohler/60/base 2025-12-04T09:43:53.6141364Z * [new branch] gh/kurtamohler/60/head -> origin/gh/kurtamohler/60/head 2025-12-04T09:43:53.6143127Z * [new branch] gh/kurtamohler/60/orig -> origin/gh/kurtamohler/60/orig 2025-12-04T09:43:53.6145575Z * [new branch] gh/kurtamohler/61/base -> origin/gh/kurtamohler/61/base 2025-12-04T09:43:53.6147285Z * [new branch] gh/kurtamohler/61/head -> origin/gh/kurtamohler/61/head 2025-12-04T09:43:53.6149005Z * [new branch] gh/kurtamohler/61/orig -> origin/gh/kurtamohler/61/orig 2025-12-04T09:43:53.6151293Z * [new branch] gh/kurtamohler/62/base -> origin/gh/kurtamohler/62/base 2025-12-04T09:43:53.6152997Z * [new branch] gh/kurtamohler/62/head -> origin/gh/kurtamohler/62/head 2025-12-04T09:43:53.6154826Z * [new branch] gh/kurtamohler/62/orig -> origin/gh/kurtamohler/62/orig 2025-12-04T09:43:53.6157143Z * [new branch] gh/kurtamohler/63/base -> origin/gh/kurtamohler/63/base 2025-12-04T09:43:53.6158879Z * [new branch] gh/kurtamohler/63/head -> origin/gh/kurtamohler/63/head 2025-12-04T09:43:53.6160592Z * [new branch] gh/kurtamohler/63/orig -> origin/gh/kurtamohler/63/orig 2025-12-04T09:43:53.6163010Z * [new branch] gh/kurtamohler/64/base -> origin/gh/kurtamohler/64/base 2025-12-04T09:43:53.6164738Z * [new branch] gh/kurtamohler/64/head -> origin/gh/kurtamohler/64/head 2025-12-04T09:43:53.6166475Z * [new branch] gh/kurtamohler/64/orig -> origin/gh/kurtamohler/64/orig 2025-12-04T09:43:53.6169386Z * [new branch] gh/kurtamohler/65/base -> origin/gh/kurtamohler/65/base 2025-12-04T09:43:53.6171170Z * [new branch] gh/kurtamohler/65/head -> origin/gh/kurtamohler/65/head 2025-12-04T09:43:53.6172891Z * [new branch] gh/kurtamohler/65/orig -> origin/gh/kurtamohler/65/orig 2025-12-04T09:43:53.6175187Z * [new branch] gh/kurtamohler/66/base -> origin/gh/kurtamohler/66/base 2025-12-04T09:43:53.6177018Z * [new branch] gh/kurtamohler/66/head -> origin/gh/kurtamohler/66/head 2025-12-04T09:43:53.6179009Z * [new branch] gh/kurtamohler/66/orig -> origin/gh/kurtamohler/66/orig 2025-12-04T09:43:53.6181326Z * [new branch] gh/kurtamohler/67/base -> origin/gh/kurtamohler/67/base 2025-12-04T09:43:53.6183041Z * [new branch] gh/kurtamohler/67/head -> origin/gh/kurtamohler/67/head 2025-12-04T09:43:53.6185127Z * [new branch] gh/kurtamohler/67/orig -> origin/gh/kurtamohler/67/orig 2025-12-04T09:43:53.6187886Z * [new branch] gh/kwen2501/130/base -> origin/gh/kwen2501/130/base 2025-12-04T09:43:53.6189646Z * [new branch] gh/kwen2501/130/head -> origin/gh/kwen2501/130/head 2025-12-04T09:43:53.6191573Z * [new branch] gh/kwen2501/130/orig -> origin/gh/kwen2501/130/orig 2025-12-04T09:43:53.6193977Z * [new branch] gh/kwen2501/170/base -> origin/gh/kwen2501/170/base 2025-12-04T09:43:53.6195709Z * [new branch] gh/kwen2501/170/head -> origin/gh/kwen2501/170/head 2025-12-04T09:43:53.6198059Z * [new branch] gh/kwen2501/187/base -> origin/gh/kwen2501/187/base 2025-12-04T09:43:53.6199945Z * [new branch] gh/kwen2501/187/head -> origin/gh/kwen2501/187/head 2025-12-04T09:43:53.6201655Z * [new branch] gh/kwen2501/187/orig -> origin/gh/kwen2501/187/orig 2025-12-04T09:43:53.6204007Z * [new branch] gh/kwen2501/188/base -> origin/gh/kwen2501/188/base 2025-12-04T09:43:53.6205709Z * [new branch] gh/kwen2501/188/head -> origin/gh/kwen2501/188/head 2025-12-04T09:43:53.6207481Z * [new branch] gh/kwen2501/188/orig -> origin/gh/kwen2501/188/orig 2025-12-04T09:43:53.6209839Z * [new branch] gh/kwen2501/211/base -> origin/gh/kwen2501/211/base 2025-12-04T09:43:53.6211579Z * [new branch] gh/kwen2501/211/head -> origin/gh/kwen2501/211/head 2025-12-04T09:43:53.6213856Z * [new branch] gh/kwen2501/224/base -> origin/gh/kwen2501/224/base 2025-12-04T09:43:53.6215568Z * [new branch] gh/kwen2501/224/head -> origin/gh/kwen2501/224/head 2025-12-04T09:43:53.6217329Z * [new branch] gh/kwen2501/224/orig -> origin/gh/kwen2501/224/orig 2025-12-04T09:43:53.6219634Z * [new branch] gh/kwen2501/228/base -> origin/gh/kwen2501/228/base 2025-12-04T09:43:53.6221407Z * [new branch] gh/kwen2501/228/head -> origin/gh/kwen2501/228/head 2025-12-04T09:43:53.6223109Z * [new branch] gh/kwen2501/228/orig -> origin/gh/kwen2501/228/orig 2025-12-04T09:43:53.6225806Z * [new branch] gh/kwen2501/234/base -> origin/gh/kwen2501/234/base 2025-12-04T09:43:53.6227508Z * [new branch] gh/kwen2501/234/head -> origin/gh/kwen2501/234/head 2025-12-04T09:43:53.6229257Z * [new branch] gh/kwen2501/234/orig -> origin/gh/kwen2501/234/orig 2025-12-04T09:43:53.6231617Z * [new branch] gh/kwen2501/235/base -> origin/gh/kwen2501/235/base 2025-12-04T09:43:53.6233364Z * [new branch] gh/kwen2501/235/head -> origin/gh/kwen2501/235/head 2025-12-04T09:43:53.6235104Z * [new branch] gh/kwen2501/235/orig -> origin/gh/kwen2501/235/orig 2025-12-04T09:43:53.6237493Z * [new branch] gh/kwen2501/236/base -> origin/gh/kwen2501/236/base 2025-12-04T09:43:53.6239220Z * [new branch] gh/kwen2501/236/head -> origin/gh/kwen2501/236/head 2025-12-04T09:43:53.6241143Z * [new branch] gh/kwen2501/236/orig -> origin/gh/kwen2501/236/orig 2025-12-04T09:43:53.6243456Z * [new branch] gh/kwen2501/237/base -> origin/gh/kwen2501/237/base 2025-12-04T09:43:53.6245089Z * [new branch] gh/kwen2501/237/head -> origin/gh/kwen2501/237/head 2025-12-04T09:43:53.6246844Z * [new branch] gh/kwen2501/237/orig -> origin/gh/kwen2501/237/orig 2025-12-04T09:43:53.6249247Z * [new branch] gh/kwen2501/238/base -> origin/gh/kwen2501/238/base 2025-12-04T09:43:53.6250936Z * [new branch] gh/kwen2501/238/head -> origin/gh/kwen2501/238/head 2025-12-04T09:43:53.6252879Z * [new branch] gh/kwen2501/238/orig -> origin/gh/kwen2501/238/orig 2025-12-04T09:43:53.6256031Z * [new branch] gh/kwen2501/240/base -> origin/gh/kwen2501/240/base 2025-12-04T09:43:53.6257487Z * [new branch] gh/kwen2501/240/head -> origin/gh/kwen2501/240/head 2025-12-04T09:43:53.6259136Z * [new branch] gh/kwen2501/240/orig -> origin/gh/kwen2501/240/orig 2025-12-04T09:43:53.6261408Z * [new branch] gh/kwen2501/241/base -> origin/gh/kwen2501/241/base 2025-12-04T09:43:53.6263175Z * [new branch] gh/kwen2501/241/head -> origin/gh/kwen2501/241/head 2025-12-04T09:43:53.6265090Z * [new branch] gh/kwen2501/241/orig -> origin/gh/kwen2501/241/orig 2025-12-04T09:43:53.6267369Z * [new branch] gh/kwen2501/247/base -> origin/gh/kwen2501/247/base 2025-12-04T09:43:53.6269078Z * [new branch] gh/kwen2501/247/head -> origin/gh/kwen2501/247/head 2025-12-04T09:43:53.6270857Z * [new branch] gh/kwen2501/247/orig -> origin/gh/kwen2501/247/orig 2025-12-04T09:43:53.6273222Z * [new branch] gh/kwen2501/252/base -> origin/gh/kwen2501/252/base 2025-12-04T09:43:53.6274967Z * [new branch] gh/kwen2501/252/head -> origin/gh/kwen2501/252/head 2025-12-04T09:43:53.6276704Z * [new branch] gh/kwen2501/252/orig -> origin/gh/kwen2501/252/orig 2025-12-04T09:43:53.6279873Z * [new branch] gh/kwen2501/259/base -> origin/gh/kwen2501/259/base 2025-12-04T09:43:53.6281620Z * [new branch] gh/kwen2501/259/head -> origin/gh/kwen2501/259/head 2025-12-04T09:43:53.6283400Z * [new branch] gh/kwen2501/259/orig -> origin/gh/kwen2501/259/orig 2025-12-04T09:43:53.6285866Z * [new branch] gh/kwen2501/260/base -> origin/gh/kwen2501/260/base 2025-12-04T09:43:53.6287676Z * [new branch] gh/kwen2501/260/head -> origin/gh/kwen2501/260/head 2025-12-04T09:43:53.6289414Z * [new branch] gh/kwen2501/260/orig -> origin/gh/kwen2501/260/orig 2025-12-04T09:43:53.6291788Z * [new branch] gh/kwen2501/268/base -> origin/gh/kwen2501/268/base 2025-12-04T09:43:53.6293538Z * [new branch] gh/kwen2501/268/head -> origin/gh/kwen2501/268/head 2025-12-04T09:43:53.6295261Z * [new branch] gh/kwen2501/268/orig -> origin/gh/kwen2501/268/orig 2025-12-04T09:43:53.6297748Z * [new branch] gh/kwen2501/269/base -> origin/gh/kwen2501/269/base 2025-12-04T09:43:53.6299593Z * [new branch] gh/kwen2501/269/head -> origin/gh/kwen2501/269/head 2025-12-04T09:43:53.6301329Z * [new branch] gh/kwen2501/269/orig -> origin/gh/kwen2501/269/orig 2025-12-04T09:43:53.6303911Z * [new branch] gh/kwen2501/270/base -> origin/gh/kwen2501/270/base 2025-12-04T09:43:53.6305787Z * [new branch] gh/kwen2501/270/head -> origin/gh/kwen2501/270/head 2025-12-04T09:43:53.6307468Z * [new branch] gh/kwen2501/270/orig -> origin/gh/kwen2501/270/orig 2025-12-04T09:43:53.6309887Z * [new branch] gh/kwen2501/271/base -> origin/gh/kwen2501/271/base 2025-12-04T09:43:53.6311748Z * [new branch] gh/kwen2501/271/head -> origin/gh/kwen2501/271/head 2025-12-04T09:43:53.6313505Z * [new branch] gh/kwen2501/271/orig -> origin/gh/kwen2501/271/orig 2025-12-04T09:43:53.6316002Z * [new branch] gh/kwen2501/274/base -> origin/gh/kwen2501/274/base 2025-12-04T09:43:53.6317833Z * [new branch] gh/kwen2501/274/head -> origin/gh/kwen2501/274/head 2025-12-04T09:43:53.6319574Z * [new branch] gh/kwen2501/274/orig -> origin/gh/kwen2501/274/orig 2025-12-04T09:43:53.6322287Z * [new branch] gh/kwen2501/275/base -> origin/gh/kwen2501/275/base 2025-12-04T09:43:53.6324101Z * [new branch] gh/kwen2501/275/head -> origin/gh/kwen2501/275/head 2025-12-04T09:43:53.6326096Z * [new branch] gh/kwen2501/275/orig -> origin/gh/kwen2501/275/orig 2025-12-04T09:43:53.6328397Z * [new branch] gh/kwen2501/276/base -> origin/gh/kwen2501/276/base 2025-12-04T09:43:53.6330069Z * [new branch] gh/kwen2501/276/head -> origin/gh/kwen2501/276/head 2025-12-04T09:43:53.6331825Z * [new branch] gh/kwen2501/276/orig -> origin/gh/kwen2501/276/orig 2025-12-04T09:43:53.6334233Z * [new branch] gh/kwen2501/277/base -> origin/gh/kwen2501/277/base 2025-12-04T09:43:53.6335991Z * [new branch] gh/kwen2501/277/head -> origin/gh/kwen2501/277/head 2025-12-04T09:43:53.6337715Z * [new branch] gh/kwen2501/277/orig -> origin/gh/kwen2501/277/orig 2025-12-04T09:43:53.6340059Z * [new branch] gh/kwen2501/278/base -> origin/gh/kwen2501/278/base 2025-12-04T09:43:53.6341834Z * [new branch] gh/kwen2501/278/head -> origin/gh/kwen2501/278/head 2025-12-04T09:43:53.6343690Z * [new branch] gh/kwen2501/278/orig -> origin/gh/kwen2501/278/orig 2025-12-04T09:43:53.6346400Z * [new branch] gh/kwen2501/279/base -> origin/gh/kwen2501/279/base 2025-12-04T09:43:53.6348203Z * [new branch] gh/kwen2501/279/head -> origin/gh/kwen2501/279/head 2025-12-04T09:43:53.6350056Z * [new branch] gh/kwen2501/279/orig -> origin/gh/kwen2501/279/orig 2025-12-04T09:43:53.6352480Z * [new branch] gh/kwen2501/280/base -> origin/gh/kwen2501/280/base 2025-12-04T09:43:53.6354282Z * [new branch] gh/kwen2501/280/head -> origin/gh/kwen2501/280/head 2025-12-04T09:43:53.6356442Z * [new branch] gh/kwen2501/280/orig -> origin/gh/kwen2501/280/orig 2025-12-04T09:43:53.6358935Z * [new branch] gh/kwen2501/281/base -> origin/gh/kwen2501/281/base 2025-12-04T09:43:53.6360598Z * [new branch] gh/kwen2501/281/head -> origin/gh/kwen2501/281/head 2025-12-04T09:43:53.6362430Z * [new branch] gh/kwen2501/281/orig -> origin/gh/kwen2501/281/orig 2025-12-04T09:43:53.6364889Z * [new branch] gh/kwen2501/282/base -> origin/gh/kwen2501/282/base 2025-12-04T09:43:53.6366673Z * [new branch] gh/kwen2501/282/head -> origin/gh/kwen2501/282/head 2025-12-04T09:43:53.6368405Z * [new branch] gh/kwen2501/282/orig -> origin/gh/kwen2501/282/orig 2025-12-04T09:43:53.6370813Z * [new branch] gh/kwen2501/283/base -> origin/gh/kwen2501/283/base 2025-12-04T09:43:53.6372583Z * [new branch] gh/kwen2501/283/head -> origin/gh/kwen2501/283/head 2025-12-04T09:43:53.6374471Z * [new branch] gh/kwen2501/283/orig -> origin/gh/kwen2501/283/orig 2025-12-04T09:43:53.6376876Z * [new branch] gh/kwen2501/284/base -> origin/gh/kwen2501/284/base 2025-12-04T09:43:53.6379692Z * [new branch] gh/kwen2501/284/head -> origin/gh/kwen2501/284/head 2025-12-04T09:43:53.6381476Z * [new branch] gh/kwen2501/284/orig -> origin/gh/kwen2501/284/orig 2025-12-04T09:43:53.6383971Z * [new branch] gh/kwen2501/285/base -> origin/gh/kwen2501/285/base 2025-12-04T09:43:53.6385714Z * [new branch] gh/kwen2501/285/head -> origin/gh/kwen2501/285/head 2025-12-04T09:43:53.6387504Z * [new branch] gh/kwen2501/285/orig -> origin/gh/kwen2501/285/orig 2025-12-04T09:43:53.6389935Z * [new branch] gh/kwen2501/286/base -> origin/gh/kwen2501/286/base 2025-12-04T09:43:53.6391746Z * [new branch] gh/kwen2501/286/head -> origin/gh/kwen2501/286/head 2025-12-04T09:43:53.6393425Z * [new branch] gh/kwen2501/286/orig -> origin/gh/kwen2501/286/orig 2025-12-04T09:43:53.6395755Z * [new branch] gh/kwen2501/287/base -> origin/gh/kwen2501/287/base 2025-12-04T09:43:53.6397638Z * [new branch] gh/kwen2501/287/head -> origin/gh/kwen2501/287/head 2025-12-04T09:43:53.6399281Z * [new branch] gh/kwen2501/287/orig -> origin/gh/kwen2501/287/orig 2025-12-04T09:43:53.6402122Z * [new branch] gh/kwen2501/288/base -> origin/gh/kwen2501/288/base 2025-12-04T09:43:53.6404001Z * [new branch] gh/kwen2501/288/head -> origin/gh/kwen2501/288/head 2025-12-04T09:43:53.6405756Z * [new branch] gh/kwen2501/288/orig -> origin/gh/kwen2501/288/orig 2025-12-04T09:43:53.6408637Z * [new branch] gh/laithsakka/251/base -> origin/gh/laithsakka/251/base 2025-12-04T09:43:53.6410321Z * [new branch] gh/laithsakka/251/head -> origin/gh/laithsakka/251/head 2025-12-04T09:43:53.6412210Z * [new branch] gh/laithsakka/251/orig -> origin/gh/laithsakka/251/orig 2025-12-04T09:43:53.6414480Z * [new branch] gh/laithsakka/276/base -> origin/gh/laithsakka/276/base 2025-12-04T09:43:53.6416254Z * [new branch] gh/laithsakka/276/head -> origin/gh/laithsakka/276/head 2025-12-04T09:43:53.6418003Z * [new branch] gh/laithsakka/276/orig -> origin/gh/laithsakka/276/orig 2025-12-04T09:43:53.6420882Z * [new branch] gh/laithsakka/28/base -> origin/gh/laithsakka/28/base 2025-12-04T09:43:53.6423198Z * [new branch] gh/laithsakka/29/base -> origin/gh/laithsakka/29/base 2025-12-04T09:43:53.6426067Z * [new branch] gh/laithsakka/30/base -> origin/gh/laithsakka/30/base 2025-12-04T09:43:53.6427795Z * [new branch] gh/laithsakka/30/head -> origin/gh/laithsakka/30/head 2025-12-04T09:43:53.6429877Z * [new branch] gh/laithsakka/31/base -> origin/gh/laithsakka/31/base 2025-12-04T09:43:53.6431666Z * [new branch] gh/laithsakka/31/head -> origin/gh/laithsakka/31/head 2025-12-04T09:43:53.6434138Z * [new branch] gh/laithsakka/313/base -> origin/gh/laithsakka/313/base 2025-12-04T09:43:53.6435911Z * [new branch] gh/laithsakka/313/head -> origin/gh/laithsakka/313/head 2025-12-04T09:43:53.6437641Z * [new branch] gh/laithsakka/313/orig -> origin/gh/laithsakka/313/orig 2025-12-04T09:43:53.6440204Z * [new branch] gh/laithsakka/316/base -> origin/gh/laithsakka/316/base 2025-12-04T09:43:53.6441908Z * [new branch] gh/laithsakka/316/head -> origin/gh/laithsakka/316/head 2025-12-04T09:43:53.6443616Z * [new branch] gh/laithsakka/316/orig -> origin/gh/laithsakka/316/orig 2025-12-04T09:43:53.6446090Z * [new branch] gh/laithsakka/317/base -> origin/gh/laithsakka/317/base 2025-12-04T09:43:53.6447795Z * [new branch] gh/laithsakka/317/head -> origin/gh/laithsakka/317/head 2025-12-04T09:43:53.6449477Z * [new branch] gh/laithsakka/317/orig -> origin/gh/laithsakka/317/orig 2025-12-04T09:43:53.6452383Z * [new branch] gh/laithsakka/319/base -> origin/gh/laithsakka/319/base 2025-12-04T09:43:53.6454199Z * [new branch] gh/laithsakka/319/head -> origin/gh/laithsakka/319/head 2025-12-04T09:43:53.6455967Z * [new branch] gh/laithsakka/319/orig -> origin/gh/laithsakka/319/orig 2025-12-04T09:43:53.6458164Z * [new branch] gh/laithsakka/32/base -> origin/gh/laithsakka/32/base 2025-12-04T09:43:53.6459889Z * [new branch] gh/laithsakka/32/head -> origin/gh/laithsakka/32/head 2025-12-04T09:43:53.6462536Z * [new branch] gh/laithsakka/320/base -> origin/gh/laithsakka/320/base 2025-12-04T09:43:53.6464412Z * [new branch] gh/laithsakka/320/head -> origin/gh/laithsakka/320/head 2025-12-04T09:43:53.6466109Z * [new branch] gh/laithsakka/320/orig -> origin/gh/laithsakka/320/orig 2025-12-04T09:43:53.6468419Z * [new branch] gh/laithsakka/321/base -> origin/gh/laithsakka/321/base 2025-12-04T09:43:53.6470299Z * [new branch] gh/laithsakka/321/head -> origin/gh/laithsakka/321/head 2025-12-04T09:43:53.6471967Z * [new branch] gh/laithsakka/321/orig -> origin/gh/laithsakka/321/orig 2025-12-04T09:43:53.6482870Z * [new branch] gh/laithsakka/322/base -> origin/gh/laithsakka/322/base 2025-12-04T09:43:53.6483163Z * [new branch] gh/laithsakka/322/head -> origin/gh/laithsakka/322/head 2025-12-04T09:43:53.6483355Z * [new branch] gh/laithsakka/322/orig -> origin/gh/laithsakka/322/orig 2025-12-04T09:43:53.6483694Z * [new branch] gh/laithsakka/323/base -> origin/gh/laithsakka/323/base 2025-12-04T09:43:53.6483872Z * [new branch] gh/laithsakka/323/head -> origin/gh/laithsakka/323/head 2025-12-04T09:43:53.6484030Z * [new branch] gh/laithsakka/323/orig -> origin/gh/laithsakka/323/orig 2025-12-04T09:43:53.6486663Z * [new branch] gh/laithsakka/324/base -> origin/gh/laithsakka/324/base 2025-12-04T09:43:53.6488515Z * [new branch] gh/laithsakka/324/head -> origin/gh/laithsakka/324/head 2025-12-04T09:43:53.6490171Z * [new branch] gh/laithsakka/324/orig -> origin/gh/laithsakka/324/orig 2025-12-04T09:43:53.6492627Z * [new branch] gh/laithsakka/325/base -> origin/gh/laithsakka/325/base 2025-12-04T09:43:53.6494344Z * [new branch] gh/laithsakka/325/head -> origin/gh/laithsakka/325/head 2025-12-04T09:43:53.6496135Z * [new branch] gh/laithsakka/325/orig -> origin/gh/laithsakka/325/orig 2025-12-04T09:43:53.6498735Z * [new branch] gh/laithsakka/326/base -> origin/gh/laithsakka/326/base 2025-12-04T09:43:53.6500531Z * [new branch] gh/laithsakka/326/head -> origin/gh/laithsakka/326/head 2025-12-04T09:43:53.6502706Z * [new branch] gh/laithsakka/326/orig -> origin/gh/laithsakka/326/orig 2025-12-04T09:43:53.6505438Z * [new branch] gh/laithsakka/327/base -> origin/gh/laithsakka/327/base 2025-12-04T09:43:53.6507114Z * [new branch] gh/laithsakka/327/head -> origin/gh/laithsakka/327/head 2025-12-04T09:43:53.6508868Z * [new branch] gh/laithsakka/327/orig -> origin/gh/laithsakka/327/orig 2025-12-04T09:43:53.6511237Z * [new branch] gh/laithsakka/328/base -> origin/gh/laithsakka/328/base 2025-12-04T09:43:53.6513064Z * [new branch] gh/laithsakka/328/head -> origin/gh/laithsakka/328/head 2025-12-04T09:43:53.6514812Z * [new branch] gh/laithsakka/328/orig -> origin/gh/laithsakka/328/orig 2025-12-04T09:43:53.6518052Z * [new branch] gh/liangel/4/base -> origin/gh/liangel/4/base 2025-12-04T09:43:53.6520252Z * [new branch] gh/liangel/4/head -> origin/gh/liangel/4/head 2025-12-04T09:43:53.6521998Z * [new branch] gh/liangel/4/orig -> origin/gh/liangel/4/orig 2025-12-04T09:43:53.6526198Z * [new branch] gh/lucaskabela/1/base -> origin/gh/lucaskabela/1/base 2025-12-04T09:43:53.6527948Z * [new branch] gh/lucaskabela/1/head -> origin/gh/lucaskabela/1/head 2025-12-04T09:43:53.6530632Z * [new branch] gh/lw/4/base -> origin/gh/lw/4/base 2025-12-04T09:43:53.6532403Z * [new branch] gh/lw/4/head -> origin/gh/lw/4/head 2025-12-04T09:43:53.6534100Z * [new branch] gh/lw/4/orig -> origin/gh/lw/4/orig 2025-12-04T09:43:53.6536503Z * [new branch] gh/lw/5/base -> origin/gh/lw/5/base 2025-12-04T09:43:53.6538275Z * [new branch] gh/lw/5/head -> origin/gh/lw/5/head 2025-12-04T09:43:53.6540019Z * [new branch] gh/lw/5/orig -> origin/gh/lw/5/orig 2025-12-04T09:43:53.6542378Z * [new branch] gh/lw/6/base -> origin/gh/lw/6/base 2025-12-04T09:43:53.6544439Z * [new branch] gh/lw/6/head -> origin/gh/lw/6/head 2025-12-04T09:43:53.6546064Z * [new branch] gh/lw/6/orig -> origin/gh/lw/6/orig 2025-12-04T09:43:53.6548879Z * [new branch] gh/malfet/14/base -> origin/gh/malfet/14/base 2025-12-04T09:43:53.6551135Z * [new branch] gh/malfet/417/base -> origin/gh/malfet/417/base 2025-12-04T09:43:53.6552973Z * [new branch] gh/malfet/417/head -> origin/gh/malfet/417/head 2025-12-04T09:43:53.6554779Z * [new branch] gh/malfet/417/orig -> origin/gh/malfet/417/orig 2025-12-04T09:43:53.6557102Z * [new branch] gh/malfet/506/base -> origin/gh/malfet/506/base 2025-12-04T09:43:53.6559052Z * [new branch] gh/malfet/506/head -> origin/gh/malfet/506/head 2025-12-04T09:43:53.6560707Z * [new branch] gh/malfet/506/orig -> origin/gh/malfet/506/orig 2025-12-04T09:43:53.6563136Z * [new branch] gh/malfet/517/base -> origin/gh/malfet/517/base 2025-12-04T09:43:53.6565348Z * [new branch] gh/malfet/517/head -> origin/gh/malfet/517/head 2025-12-04T09:43:53.6567660Z * [new branch] gh/malfet/528/base -> origin/gh/malfet/528/base 2025-12-04T09:43:53.6569392Z * [new branch] gh/malfet/528/head -> origin/gh/malfet/528/head 2025-12-04T09:43:53.6571186Z * [new branch] gh/malfet/528/orig -> origin/gh/malfet/528/orig 2025-12-04T09:43:53.6573465Z * [new branch] gh/malfet/537/base -> origin/gh/malfet/537/base 2025-12-04T09:43:53.6575182Z * [new branch] gh/malfet/537/head -> origin/gh/malfet/537/head 2025-12-04T09:43:53.6576961Z * [new branch] gh/malfet/537/orig -> origin/gh/malfet/537/orig 2025-12-04T09:43:53.6579694Z * [new branch] gh/malfet/546/base -> origin/gh/malfet/546/base 2025-12-04T09:43:53.6581339Z * [new branch] gh/malfet/546/head -> origin/gh/malfet/546/head 2025-12-04T09:43:53.6583105Z * [new branch] gh/malfet/546/orig -> origin/gh/malfet/546/orig 2025-12-04T09:43:53.6585541Z * [new branch] gh/malfet/565/base -> origin/gh/malfet/565/base 2025-12-04T09:43:53.6587282Z * [new branch] gh/malfet/565/head -> origin/gh/malfet/565/head 2025-12-04T09:43:53.6588987Z * [new branch] gh/malfet/565/orig -> origin/gh/malfet/565/orig 2025-12-04T09:43:53.6591302Z * [new branch] gh/malfet/575/base -> origin/gh/malfet/575/base 2025-12-04T09:43:53.6593043Z * [new branch] gh/malfet/575/head -> origin/gh/malfet/575/head 2025-12-04T09:43:53.6594812Z * [new branch] gh/malfet/575/orig -> origin/gh/malfet/575/orig 2025-12-04T09:43:53.6597147Z * [new branch] gh/malfet/580/base -> origin/gh/malfet/580/base 2025-12-04T09:43:53.6598878Z * [new branch] gh/malfet/580/head -> origin/gh/malfet/580/head 2025-12-04T09:43:53.6600637Z * [new branch] gh/malfet/580/orig -> origin/gh/malfet/580/orig 2025-12-04T09:43:53.6603023Z * [new branch] gh/malfet/581/base -> origin/gh/malfet/581/base 2025-12-04T09:43:53.6604779Z * [new branch] gh/malfet/581/head -> origin/gh/malfet/581/head 2025-12-04T09:43:53.6606513Z * [new branch] gh/malfet/581/orig -> origin/gh/malfet/581/orig 2025-12-04T09:43:53.6608839Z * [new branch] gh/malfet/583/base -> origin/gh/malfet/583/base 2025-12-04T09:43:53.6610604Z * [new branch] gh/malfet/583/head -> origin/gh/malfet/583/head 2025-12-04T09:43:53.6612370Z * [new branch] gh/malfet/583/orig -> origin/gh/malfet/583/orig 2025-12-04T09:43:53.6614617Z * [new branch] gh/malfet/586/base -> origin/gh/malfet/586/base 2025-12-04T09:43:53.6616562Z * [new branch] gh/malfet/586/head -> origin/gh/malfet/586/head 2025-12-04T09:43:53.6618089Z * [new branch] gh/malfet/586/orig -> origin/gh/malfet/586/orig 2025-12-04T09:43:53.6620420Z * [new branch] gh/malfet/587/base -> origin/gh/malfet/587/base 2025-12-04T09:43:53.6622130Z * [new branch] gh/malfet/587/head -> origin/gh/malfet/587/head 2025-12-04T09:43:53.6624008Z * [new branch] gh/malfet/587/orig -> origin/gh/malfet/587/orig 2025-12-04T09:43:53.6626423Z * [new branch] gh/malfet/588/base -> origin/gh/malfet/588/base 2025-12-04T09:43:53.6628165Z * [new branch] gh/malfet/588/head -> origin/gh/malfet/588/head 2025-12-04T09:43:53.6629971Z * [new branch] gh/malfet/588/orig -> origin/gh/malfet/588/orig 2025-12-04T09:43:53.6632258Z * [new branch] gh/malfet/589/base -> origin/gh/malfet/589/base 2025-12-04T09:43:53.6634004Z * [new branch] gh/malfet/589/head -> origin/gh/malfet/589/head 2025-12-04T09:43:53.6635775Z * [new branch] gh/malfet/589/orig -> origin/gh/malfet/589/orig 2025-12-04T09:43:53.6638099Z * [new branch] gh/malfet/590/base -> origin/gh/malfet/590/base 2025-12-04T09:43:53.6639844Z * [new branch] gh/malfet/590/head -> origin/gh/malfet/590/head 2025-12-04T09:43:53.6641903Z * [new branch] gh/malfet/590/orig -> origin/gh/malfet/590/orig 2025-12-04T09:43:53.6644651Z * [new branch] gh/malfet/591/base -> origin/gh/malfet/591/base 2025-12-04T09:43:53.6646827Z * [new branch] gh/malfet/591/head -> origin/gh/malfet/591/head 2025-12-04T09:43:53.6648636Z * [new branch] gh/malfet/591/orig -> origin/gh/malfet/591/orig 2025-12-04T09:43:53.6650942Z * [new branch] gh/malfet/592/base -> origin/gh/malfet/592/base 2025-12-04T09:43:53.6652829Z * [new branch] gh/malfet/592/head -> origin/gh/malfet/592/head 2025-12-04T09:43:53.6654536Z * [new branch] gh/malfet/592/orig -> origin/gh/malfet/592/orig 2025-12-04T09:43:53.6656976Z * [new branch] gh/malfet/593/base -> origin/gh/malfet/593/base 2025-12-04T09:43:53.6658684Z * [new branch] gh/malfet/593/head -> origin/gh/malfet/593/head 2025-12-04T09:43:53.6660420Z * [new branch] gh/malfet/593/orig -> origin/gh/malfet/593/orig 2025-12-04T09:43:53.6662948Z * [new branch] gh/malfet/594/base -> origin/gh/malfet/594/base 2025-12-04T09:43:53.6664814Z * [new branch] gh/malfet/594/head -> origin/gh/malfet/594/head 2025-12-04T09:43:53.6666588Z * [new branch] gh/malfet/594/orig -> origin/gh/malfet/594/orig 2025-12-04T09:43:53.6668913Z * [new branch] gh/malfet/595/base -> origin/gh/malfet/595/base 2025-12-04T09:43:53.6670626Z * [new branch] gh/malfet/595/head -> origin/gh/malfet/595/head 2025-12-04T09:43:53.6672500Z * [new branch] gh/malfet/595/orig -> origin/gh/malfet/595/orig 2025-12-04T09:43:53.6674879Z * [new branch] gh/malfet/596/base -> origin/gh/malfet/596/base 2025-12-04T09:43:53.6676666Z * [new branch] gh/malfet/596/head -> origin/gh/malfet/596/head 2025-12-04T09:43:53.6678655Z * [new branch] gh/malfet/596/orig -> origin/gh/malfet/596/orig 2025-12-04T09:43:53.6682071Z * [new branch] gh/malfet/597/base -> origin/gh/malfet/597/base 2025-12-04T09:43:53.6683853Z * [new branch] gh/malfet/597/head -> origin/gh/malfet/597/head 2025-12-04T09:43:53.6685621Z * [new branch] gh/malfet/597/orig -> origin/gh/malfet/597/orig 2025-12-04T09:43:53.6687993Z * [new branch] gh/malfet/598/base -> origin/gh/malfet/598/base 2025-12-04T09:43:53.6689924Z * [new branch] gh/malfet/598/head -> origin/gh/malfet/598/head 2025-12-04T09:43:53.6691515Z * [new branch] gh/malfet/598/orig -> origin/gh/malfet/598/orig 2025-12-04T09:43:53.6693907Z * [new branch] gh/malfet/599/base -> origin/gh/malfet/599/base 2025-12-04T09:43:53.6695613Z * [new branch] gh/malfet/599/head -> origin/gh/malfet/599/head 2025-12-04T09:43:53.6697417Z * [new branch] gh/malfet/599/orig -> origin/gh/malfet/599/orig 2025-12-04T09:43:53.6699705Z * [new branch] gh/malfet/600/base -> origin/gh/malfet/600/base 2025-12-04T09:43:53.6701423Z * [new branch] gh/malfet/600/head -> origin/gh/malfet/600/head 2025-12-04T09:43:53.6703118Z * [new branch] gh/malfet/600/orig -> origin/gh/malfet/600/orig 2025-12-04T09:43:53.6705811Z * [new branch] gh/malfet/601/base -> origin/gh/malfet/601/base 2025-12-04T09:43:53.6707521Z * [new branch] gh/malfet/601/head -> origin/gh/malfet/601/head 2025-12-04T09:43:53.6709261Z * [new branch] gh/malfet/601/orig -> origin/gh/malfet/601/orig 2025-12-04T09:43:53.6712217Z * [new branch] gh/malfet/602/base -> origin/gh/malfet/602/base 2025-12-04T09:43:53.6713931Z * [new branch] gh/malfet/602/head -> origin/gh/malfet/602/head 2025-12-04T09:43:53.6715682Z * [new branch] gh/malfet/602/orig -> origin/gh/malfet/602/orig 2025-12-04T09:43:53.6717996Z * [new branch] gh/malfet/603/base -> origin/gh/malfet/603/base 2025-12-04T09:43:53.6719673Z * [new branch] gh/malfet/603/head -> origin/gh/malfet/603/head 2025-12-04T09:43:53.6721372Z * [new branch] gh/malfet/603/orig -> origin/gh/malfet/603/orig 2025-12-04T09:43:53.6723820Z * [new branch] gh/malfet/604/base -> origin/gh/malfet/604/base 2025-12-04T09:43:53.6725608Z * [new branch] gh/malfet/604/head -> origin/gh/malfet/604/head 2025-12-04T09:43:53.6727340Z * [new branch] gh/malfet/604/orig -> origin/gh/malfet/604/orig 2025-12-04T09:43:53.6729834Z * [new branch] gh/malfet/605/base -> origin/gh/malfet/605/base 2025-12-04T09:43:53.6731555Z * [new branch] gh/malfet/605/head -> origin/gh/malfet/605/head 2025-12-04T09:43:53.6733346Z * [new branch] gh/malfet/605/orig -> origin/gh/malfet/605/orig 2025-12-04T09:43:53.6736227Z * [new branch] gh/malfet/606/base -> origin/gh/malfet/606/base 2025-12-04T09:43:53.6738021Z * [new branch] gh/malfet/606/head -> origin/gh/malfet/606/head 2025-12-04T09:43:53.6739799Z * [new branch] gh/malfet/606/orig -> origin/gh/malfet/606/orig 2025-12-04T09:43:53.6742268Z * [new branch] gh/malfet/607/base -> origin/gh/malfet/607/base 2025-12-04T09:43:53.6744074Z * [new branch] gh/malfet/607/head -> origin/gh/malfet/607/head 2025-12-04T09:43:53.6745914Z * [new branch] gh/malfet/607/orig -> origin/gh/malfet/607/orig 2025-12-04T09:43:53.6748299Z * [new branch] gh/malfet/608/base -> origin/gh/malfet/608/base 2025-12-04T09:43:53.6750122Z * [new branch] gh/malfet/608/head -> origin/gh/malfet/608/head 2025-12-04T09:43:53.6751917Z * [new branch] gh/malfet/608/orig -> origin/gh/malfet/608/orig 2025-12-04T09:43:53.6754302Z * [new branch] gh/malfet/609/base -> origin/gh/malfet/609/base 2025-12-04T09:43:53.6756101Z * [new branch] gh/malfet/609/head -> origin/gh/malfet/609/head 2025-12-04T09:43:53.6757854Z * [new branch] gh/malfet/609/orig -> origin/gh/malfet/609/orig 2025-12-04T09:43:53.6760424Z * [new branch] gh/malfet/610/base -> origin/gh/malfet/610/base 2025-12-04T09:43:53.6762621Z * [new branch] gh/malfet/610/head -> origin/gh/malfet/610/head 2025-12-04T09:43:53.6764389Z * [new branch] gh/malfet/610/orig -> origin/gh/malfet/610/orig 2025-12-04T09:43:53.6766941Z * [new branch] gh/malfet/611/base -> origin/gh/malfet/611/base 2025-12-04T09:43:53.6768677Z * [new branch] gh/malfet/611/head -> origin/gh/malfet/611/head 2025-12-04T09:43:53.6770405Z * [new branch] gh/malfet/611/orig -> origin/gh/malfet/611/orig 2025-12-04T09:43:53.6772647Z * [new branch] gh/malfet/612/base -> origin/gh/malfet/612/base 2025-12-04T09:43:53.6774400Z * [new branch] gh/malfet/612/head -> origin/gh/malfet/612/head 2025-12-04T09:43:53.6776793Z * [new branch] gh/malfet/612/orig -> origin/gh/malfet/612/orig 2025-12-04T09:43:53.6781043Z * [new branch] gh/malfet/64/base -> origin/gh/malfet/64/base 2025-12-04T09:43:53.6782800Z * [new branch] gh/malfet/64/head -> origin/gh/malfet/64/head 2025-12-04T09:43:53.6786218Z * [new branch] gh/manuelcandales/11/base -> origin/gh/manuelcandales/11/base 2025-12-04T09:43:53.6788096Z * [new branch] gh/manuelcandales/11/head -> origin/gh/manuelcandales/11/head 2025-12-04T09:43:53.6789771Z * [new branch] gh/manuelcandales/11/orig -> origin/gh/manuelcandales/11/orig 2025-12-04T09:43:53.6792713Z * [new branch] gh/markkm/1/base -> origin/gh/markkm/1/base 2025-12-04T09:43:53.6795581Z * [new branch] gh/masnesral/1/base -> origin/gh/masnesral/1/base 2025-12-04T09:43:53.6797286Z * [new branch] gh/masnesral/1/head -> origin/gh/masnesral/1/head 2025-12-04T09:43:53.6799085Z * [new branch] gh/masnesral/1/orig -> origin/gh/masnesral/1/orig 2025-12-04T09:43:53.6801835Z * [new branch] gh/mhorowitz/0/base -> origin/gh/mhorowitz/0/base 2025-12-04T09:43:53.6803610Z * [new branch] gh/mhorowitz/0/head -> origin/gh/mhorowitz/0/head 2025-12-04T09:43:53.6805802Z * [new branch] gh/mhorowitz/1/base -> origin/gh/mhorowitz/1/base 2025-12-04T09:43:53.6807518Z * [new branch] gh/mhorowitz/1/head -> origin/gh/mhorowitz/1/head 2025-12-04T09:43:53.6809788Z * [new branch] gh/mhorowitz/2/base -> origin/gh/mhorowitz/2/base 2025-12-04T09:43:53.6811527Z * [new branch] gh/mhorowitz/2/head -> origin/gh/mhorowitz/2/head 2025-12-04T09:43:53.6813756Z * [new branch] gh/mhorowitz/3/base -> origin/gh/mhorowitz/3/base 2025-12-04T09:43:53.6815433Z * [new branch] gh/mhorowitz/3/head -> origin/gh/mhorowitz/3/head 2025-12-04T09:43:53.6817633Z * [new branch] gh/mhorowitz/4/base -> origin/gh/mhorowitz/4/base 2025-12-04T09:43:53.6819391Z * [new branch] gh/mhorowitz/4/head -> origin/gh/mhorowitz/4/head 2025-12-04T09:43:53.6821574Z * [new branch] gh/mhorowitz/5/base -> origin/gh/mhorowitz/5/base 2025-12-04T09:43:53.6823233Z * [new branch] gh/mhorowitz/5/head -> origin/gh/mhorowitz/5/head 2025-12-04T09:43:53.6825621Z * [new branch] gh/mhorowitz/6/base -> origin/gh/mhorowitz/6/base 2025-12-04T09:43:53.6827251Z * [new branch] gh/mhorowitz/6/head -> origin/gh/mhorowitz/6/head 2025-12-04T09:43:53.6830283Z * [new branch] gh/mikaylagawarecki/234/base -> origin/gh/mikaylagawarecki/234/base 2025-12-04T09:43:53.6832029Z * [new branch] gh/mikaylagawarecki/234/head -> origin/gh/mikaylagawarecki/234/head 2025-12-04T09:43:53.6834303Z * [new branch] gh/mikaylagawarecki/235/base -> origin/gh/mikaylagawarecki/235/base 2025-12-04T09:43:53.6836140Z * [new branch] gh/mikaylagawarecki/235/head -> origin/gh/mikaylagawarecki/235/head 2025-12-04T09:43:53.6838362Z * [new branch] gh/mikaylagawarecki/236/base -> origin/gh/mikaylagawarecki/236/base 2025-12-04T09:43:53.6840182Z * [new branch] gh/mikaylagawarecki/236/head -> origin/gh/mikaylagawarecki/236/head 2025-12-04T09:43:53.6842432Z * [new branch] gh/mikaylagawarecki/237/base -> origin/gh/mikaylagawarecki/237/base 2025-12-04T09:43:53.6844096Z * [new branch] gh/mikaylagawarecki/237/head -> origin/gh/mikaylagawarecki/237/head 2025-12-04T09:43:53.6846539Z * [new branch] gh/mikaylagawarecki/238/base -> origin/gh/mikaylagawarecki/238/base 2025-12-04T09:43:53.6848277Z * [new branch] gh/mikaylagawarecki/238/head -> origin/gh/mikaylagawarecki/238/head 2025-12-04T09:43:53.6850501Z * [new branch] gh/mikaylagawarecki/336/base -> origin/gh/mikaylagawarecki/336/base 2025-12-04T09:43:53.6852340Z * [new branch] gh/mikaylagawarecki/336/head -> origin/gh/mikaylagawarecki/336/head 2025-12-04T09:43:53.6854061Z * [new branch] gh/mikaylagawarecki/336/orig -> origin/gh/mikaylagawarecki/336/orig 2025-12-04T09:43:53.6856480Z * [new branch] gh/mikaylagawarecki/341/base -> origin/gh/mikaylagawarecki/341/base 2025-12-04T09:43:53.6858165Z * [new branch] gh/mikaylagawarecki/341/head -> origin/gh/mikaylagawarecki/341/head 2025-12-04T09:43:53.6859920Z * [new branch] gh/mikaylagawarecki/341/orig -> origin/gh/mikaylagawarecki/341/orig 2025-12-04T09:43:53.6862481Z * [new branch] gh/mikaylagawarecki/342/base -> origin/gh/mikaylagawarecki/342/base 2025-12-04T09:43:53.6864288Z * [new branch] gh/mikaylagawarecki/342/head -> origin/gh/mikaylagawarecki/342/head 2025-12-04T09:43:53.6866128Z * [new branch] gh/mikaylagawarecki/342/orig -> origin/gh/mikaylagawarecki/342/orig 2025-12-04T09:43:53.6868538Z * [new branch] gh/mikaylagawarecki/345/base -> origin/gh/mikaylagawarecki/345/base 2025-12-04T09:43:53.6870245Z * [new branch] gh/mikaylagawarecki/345/head -> origin/gh/mikaylagawarecki/345/head 2025-12-04T09:43:53.6871953Z * [new branch] gh/mikaylagawarecki/345/orig -> origin/gh/mikaylagawarecki/345/orig 2025-12-04T09:43:53.6874369Z * [new branch] gh/mikaylagawarecki/346/base -> origin/gh/mikaylagawarecki/346/base 2025-12-04T09:43:53.6876088Z * [new branch] gh/mikaylagawarecki/346/head -> origin/gh/mikaylagawarecki/346/head 2025-12-04T09:43:53.6878052Z * [new branch] gh/mikaylagawarecki/346/orig -> origin/gh/mikaylagawarecki/346/orig 2025-12-04T09:43:53.6880414Z * [new branch] gh/mikaylagawarecki/347/base -> origin/gh/mikaylagawarecki/347/base 2025-12-04T09:43:53.6881983Z * [new branch] gh/mikaylagawarecki/347/head -> origin/gh/mikaylagawarecki/347/head 2025-12-04T09:43:53.6883698Z * [new branch] gh/mikaylagawarecki/347/orig -> origin/gh/mikaylagawarecki/347/orig 2025-12-04T09:43:53.6886212Z * [new branch] gh/mikaylagawarecki/350/base -> origin/gh/mikaylagawarecki/350/base 2025-12-04T09:43:53.6887970Z * [new branch] gh/mikaylagawarecki/350/head -> origin/gh/mikaylagawarecki/350/head 2025-12-04T09:43:53.6889670Z * [new branch] gh/mikaylagawarecki/350/orig -> origin/gh/mikaylagawarecki/350/orig 2025-12-04T09:43:53.6892398Z * [new branch] gh/mikaylagawarecki/351/base -> origin/gh/mikaylagawarecki/351/base 2025-12-04T09:43:53.6894171Z * [new branch] gh/mikaylagawarecki/351/head -> origin/gh/mikaylagawarecki/351/head 2025-12-04T09:43:53.6895890Z * [new branch] gh/mikaylagawarecki/351/orig -> origin/gh/mikaylagawarecki/351/orig 2025-12-04T09:43:53.6898486Z * [new branch] gh/mikaylagawarecki/352/base -> origin/gh/mikaylagawarecki/352/base 2025-12-04T09:43:53.6900379Z * [new branch] gh/mikaylagawarecki/352/head -> origin/gh/mikaylagawarecki/352/head 2025-12-04T09:43:53.6902345Z * [new branch] gh/mikaylagawarecki/352/orig -> origin/gh/mikaylagawarecki/352/orig 2025-12-04T09:43:53.6904856Z * [new branch] gh/mikaylagawarecki/353/base -> origin/gh/mikaylagawarecki/353/base 2025-12-04T09:43:53.6906754Z * [new branch] gh/mikaylagawarecki/353/head -> origin/gh/mikaylagawarecki/353/head 2025-12-04T09:43:53.6908553Z * [new branch] gh/mikaylagawarecki/353/orig -> origin/gh/mikaylagawarecki/353/orig 2025-12-04T09:43:53.6910876Z * [new branch] gh/mikaylagawarecki/354/base -> origin/gh/mikaylagawarecki/354/base 2025-12-04T09:43:53.6912801Z * [new branch] gh/mikaylagawarecki/354/head -> origin/gh/mikaylagawarecki/354/head 2025-12-04T09:43:53.6914484Z * [new branch] gh/mikaylagawarecki/354/orig -> origin/gh/mikaylagawarecki/354/orig 2025-12-04T09:43:53.6917449Z * [new branch] gh/mikaylagawarecki/356/base -> origin/gh/mikaylagawarecki/356/base 2025-12-04T09:43:53.6919228Z * [new branch] gh/mikaylagawarecki/356/head -> origin/gh/mikaylagawarecki/356/head 2025-12-04T09:43:53.6920888Z * [new branch] gh/mikaylagawarecki/356/orig -> origin/gh/mikaylagawarecki/356/orig 2025-12-04T09:43:53.6923228Z * [new branch] gh/mikaylagawarecki/357/base -> origin/gh/mikaylagawarecki/357/base 2025-12-04T09:43:53.6924974Z * [new branch] gh/mikaylagawarecki/357/head -> origin/gh/mikaylagawarecki/357/head 2025-12-04T09:43:53.6926720Z * [new branch] gh/mikaylagawarecki/357/orig -> origin/gh/mikaylagawarecki/357/orig 2025-12-04T09:43:53.6929235Z * [new branch] gh/mikaylagawarecki/359/base -> origin/gh/mikaylagawarecki/359/base 2025-12-04T09:43:53.6931069Z * [new branch] gh/mikaylagawarecki/359/head -> origin/gh/mikaylagawarecki/359/head 2025-12-04T09:43:53.6932938Z * [new branch] gh/mikaylagawarecki/359/orig -> origin/gh/mikaylagawarecki/359/orig 2025-12-04T09:43:53.6935406Z * [new branch] gh/mikaylagawarecki/360/base -> origin/gh/mikaylagawarecki/360/base 2025-12-04T09:43:53.6937225Z * [new branch] gh/mikaylagawarecki/360/head -> origin/gh/mikaylagawarecki/360/head 2025-12-04T09:43:53.6939015Z * [new branch] gh/mikaylagawarecki/360/orig -> origin/gh/mikaylagawarecki/360/orig 2025-12-04T09:43:53.6941358Z * [new branch] gh/mikaylagawarecki/361/base -> origin/gh/mikaylagawarecki/361/base 2025-12-04T09:43:53.6943101Z * [new branch] gh/mikaylagawarecki/361/head -> origin/gh/mikaylagawarecki/361/head 2025-12-04T09:43:53.6945031Z * [new branch] gh/mikaylagawarecki/361/orig -> origin/gh/mikaylagawarecki/361/orig 2025-12-04T09:43:53.6947480Z * [new branch] gh/mikaylagawarecki/362/base -> origin/gh/mikaylagawarecki/362/base 2025-12-04T09:43:53.6949391Z * [new branch] gh/mikaylagawarecki/362/head -> origin/gh/mikaylagawarecki/362/head 2025-12-04T09:43:53.6951158Z * [new branch] gh/mikaylagawarecki/362/orig -> origin/gh/mikaylagawarecki/362/orig 2025-12-04T09:43:53.6953846Z * [new branch] gh/mikaylagawarecki/363/base -> origin/gh/mikaylagawarecki/363/base 2025-12-04T09:43:53.6955757Z * [new branch] gh/mikaylagawarecki/363/head -> origin/gh/mikaylagawarecki/363/head 2025-12-04T09:43:53.6957501Z * [new branch] gh/mikaylagawarecki/363/orig -> origin/gh/mikaylagawarecki/363/orig 2025-12-04T09:43:53.6960396Z * [new branch] gh/mikaylagawarecki/364/base -> origin/gh/mikaylagawarecki/364/base 2025-12-04T09:43:53.6962104Z * [new branch] gh/mikaylagawarecki/364/head -> origin/gh/mikaylagawarecki/364/head 2025-12-04T09:43:53.6963897Z * [new branch] gh/mikaylagawarecki/364/orig -> origin/gh/mikaylagawarecki/364/orig 2025-12-04T09:43:53.6966408Z * [new branch] gh/mikaylagawarecki/365/base -> origin/gh/mikaylagawarecki/365/base 2025-12-04T09:43:53.6968233Z * [new branch] gh/mikaylagawarecki/365/head -> origin/gh/mikaylagawarecki/365/head 2025-12-04T09:43:53.6970005Z * [new branch] gh/mikaylagawarecki/365/orig -> origin/gh/mikaylagawarecki/365/orig 2025-12-04T09:43:53.6972533Z * [new branch] gh/mikaylagawarecki/366/base -> origin/gh/mikaylagawarecki/366/base 2025-12-04T09:43:53.6974180Z * [new branch] gh/mikaylagawarecki/366/head -> origin/gh/mikaylagawarecki/366/head 2025-12-04T09:43:53.6975926Z * [new branch] gh/mikaylagawarecki/366/orig -> origin/gh/mikaylagawarecki/366/orig 2025-12-04T09:43:53.6978391Z * [new branch] gh/mikaylagawarecki/367/base -> origin/gh/mikaylagawarecki/367/base 2025-12-04T09:43:53.6980393Z * [new branch] gh/mikaylagawarecki/367/head -> origin/gh/mikaylagawarecki/367/head 2025-12-04T09:43:53.6982089Z * [new branch] gh/mikaylagawarecki/367/orig -> origin/gh/mikaylagawarecki/367/orig 2025-12-04T09:43:53.6985234Z * [new branch] gh/mikaylagawarecki/368/base -> origin/gh/mikaylagawarecki/368/base 2025-12-04T09:43:53.6986975Z * [new branch] gh/mikaylagawarecki/368/head -> origin/gh/mikaylagawarecki/368/head 2025-12-04T09:43:53.6988674Z * [new branch] gh/mikaylagawarecki/368/orig -> origin/gh/mikaylagawarecki/368/orig 2025-12-04T09:43:53.6991022Z * [new branch] gh/mikaylagawarecki/369/base -> origin/gh/mikaylagawarecki/369/base 2025-12-04T09:43:53.6992900Z * [new branch] gh/mikaylagawarecki/369/head -> origin/gh/mikaylagawarecki/369/head 2025-12-04T09:43:53.6994633Z * [new branch] gh/mikaylagawarecki/369/orig -> origin/gh/mikaylagawarecki/369/orig 2025-12-04T09:43:53.6997114Z * [new branch] gh/mikaylagawarecki/370/base -> origin/gh/mikaylagawarecki/370/base 2025-12-04T09:43:53.6998894Z * [new branch] gh/mikaylagawarecki/370/head -> origin/gh/mikaylagawarecki/370/head 2025-12-04T09:43:53.7000511Z * [new branch] gh/mikaylagawarecki/370/orig -> origin/gh/mikaylagawarecki/370/orig 2025-12-04T09:43:53.7003132Z * [new branch] gh/mikaylagawarecki/371/base -> origin/gh/mikaylagawarecki/371/base 2025-12-04T09:43:53.7004774Z * [new branch] gh/mikaylagawarecki/371/head -> origin/gh/mikaylagawarecki/371/head 2025-12-04T09:43:53.7006604Z * [new branch] gh/mikaylagawarecki/371/orig -> origin/gh/mikaylagawarecki/371/orig 2025-12-04T09:43:53.7009181Z * [new branch] gh/mikaylagawarecki/372/base -> origin/gh/mikaylagawarecki/372/base 2025-12-04T09:43:53.7010853Z * [new branch] gh/mikaylagawarecki/372/head -> origin/gh/mikaylagawarecki/372/head 2025-12-04T09:43:53.7012658Z * [new branch] gh/mikaylagawarecki/372/orig -> origin/gh/mikaylagawarecki/372/orig 2025-12-04T09:43:53.7014948Z * [new branch] gh/mikaylagawarecki/373/base -> origin/gh/mikaylagawarecki/373/base 2025-12-04T09:43:53.7016700Z * [new branch] gh/mikaylagawarecki/373/head -> origin/gh/mikaylagawarecki/373/head 2025-12-04T09:43:53.7018462Z * [new branch] gh/mikaylagawarecki/373/orig -> origin/gh/mikaylagawarecki/373/orig 2025-12-04T09:43:53.7020773Z * [new branch] gh/mikaylagawarecki/374/base -> origin/gh/mikaylagawarecki/374/base 2025-12-04T09:43:53.7022619Z * [new branch] gh/mikaylagawarecki/374/head -> origin/gh/mikaylagawarecki/374/head 2025-12-04T09:43:53.7024512Z * [new branch] gh/mikaylagawarecki/374/orig -> origin/gh/mikaylagawarecki/374/orig 2025-12-04T09:43:53.7026913Z * [new branch] gh/mikaylagawarecki/375/base -> origin/gh/mikaylagawarecki/375/base 2025-12-04T09:43:53.7028699Z * [new branch] gh/mikaylagawarecki/375/head -> origin/gh/mikaylagawarecki/375/head 2025-12-04T09:43:53.7030382Z * [new branch] gh/mikaylagawarecki/375/orig -> origin/gh/mikaylagawarecki/375/orig 2025-12-04T09:43:53.7032880Z * [new branch] gh/mikaylagawarecki/376/base -> origin/gh/mikaylagawarecki/376/base 2025-12-04T09:43:53.7034840Z * [new branch] gh/mikaylagawarecki/376/head -> origin/gh/mikaylagawarecki/376/head 2025-12-04T09:43:53.7036450Z * [new branch] gh/mikaylagawarecki/376/orig -> origin/gh/mikaylagawarecki/376/orig 2025-12-04T09:43:53.7038823Z * [new branch] gh/mikaylagawarecki/377/base -> origin/gh/mikaylagawarecki/377/base 2025-12-04T09:43:53.7041066Z * [new branch] gh/mikaylagawarecki/377/head -> origin/gh/mikaylagawarecki/377/head 2025-12-04T09:43:53.7042871Z * [new branch] gh/mikaylagawarecki/377/orig -> origin/gh/mikaylagawarecki/377/orig 2025-12-04T09:43:53.7045334Z * [new branch] gh/mikaylagawarecki/378/base -> origin/gh/mikaylagawarecki/378/base 2025-12-04T09:43:53.7047187Z * [new branch] gh/mikaylagawarecki/378/head -> origin/gh/mikaylagawarecki/378/head 2025-12-04T09:43:53.7048931Z * [new branch] gh/mikaylagawarecki/378/orig -> origin/gh/mikaylagawarecki/378/orig 2025-12-04T09:43:53.7051304Z * [new branch] gh/mikaylagawarecki/379/base -> origin/gh/mikaylagawarecki/379/base 2025-12-04T09:43:53.7053102Z * [new branch] gh/mikaylagawarecki/379/head -> origin/gh/mikaylagawarecki/379/head 2025-12-04T09:43:53.7054818Z * [new branch] gh/mikaylagawarecki/379/orig -> origin/gh/mikaylagawarecki/379/orig 2025-12-04T09:43:53.7057291Z * [new branch] gh/mikaylagawarecki/380/base -> origin/gh/mikaylagawarecki/380/base 2025-12-04T09:43:53.7058948Z * [new branch] gh/mikaylagawarecki/380/head -> origin/gh/mikaylagawarecki/380/head 2025-12-04T09:43:53.7060683Z * [new branch] gh/mikaylagawarecki/380/orig -> origin/gh/mikaylagawarecki/380/orig 2025-12-04T09:43:53.7062992Z * [new branch] gh/mikaylagawarecki/381/base -> origin/gh/mikaylagawarecki/381/base 2025-12-04T09:43:53.7064934Z * [new branch] gh/mikaylagawarecki/381/head -> origin/gh/mikaylagawarecki/381/head 2025-12-04T09:43:53.7066591Z * [new branch] gh/mikaylagawarecki/381/orig -> origin/gh/mikaylagawarecki/381/orig 2025-12-04T09:43:53.7068881Z * [new branch] gh/mikaylagawarecki/382/base -> origin/gh/mikaylagawarecki/382/base 2025-12-04T09:43:53.7070657Z * [new branch] gh/mikaylagawarecki/382/head -> origin/gh/mikaylagawarecki/382/head 2025-12-04T09:43:53.7072449Z * [new branch] gh/mikaylagawarecki/382/orig -> origin/gh/mikaylagawarecki/382/orig 2025-12-04T09:43:53.7074884Z * [new branch] gh/mikaylagawarecki/383/base -> origin/gh/mikaylagawarecki/383/base 2025-12-04T09:43:53.7076707Z * [new branch] gh/mikaylagawarecki/383/head -> origin/gh/mikaylagawarecki/383/head 2025-12-04T09:43:53.7078503Z * [new branch] gh/mikaylagawarecki/383/orig -> origin/gh/mikaylagawarecki/383/orig 2025-12-04T09:43:53.7081249Z * [new branch] gh/mikaylagawarecki/384/base -> origin/gh/mikaylagawarecki/384/base 2025-12-04T09:43:53.7082991Z * [new branch] gh/mikaylagawarecki/384/head -> origin/gh/mikaylagawarecki/384/head 2025-12-04T09:43:53.7084747Z * [new branch] gh/mikaylagawarecki/384/orig -> origin/gh/mikaylagawarecki/384/orig 2025-12-04T09:43:53.7087153Z * [new branch] gh/mikaylagawarecki/385/base -> origin/gh/mikaylagawarecki/385/base 2025-12-04T09:43:53.7088943Z * [new branch] gh/mikaylagawarecki/385/head -> origin/gh/mikaylagawarecki/385/head 2025-12-04T09:43:53.7090620Z * [new branch] gh/mikaylagawarecki/385/orig -> origin/gh/mikaylagawarecki/385/orig 2025-12-04T09:43:53.7093101Z * [new branch] gh/mikaylagawarecki/386/base -> origin/gh/mikaylagawarecki/386/base 2025-12-04T09:43:53.7094816Z * [new branch] gh/mikaylagawarecki/386/head -> origin/gh/mikaylagawarecki/386/head 2025-12-04T09:43:53.7096616Z * [new branch] gh/mikaylagawarecki/386/orig -> origin/gh/mikaylagawarecki/386/orig 2025-12-04T09:43:53.7099190Z * [new branch] gh/mikaylagawarecki/387/base -> origin/gh/mikaylagawarecki/387/base 2025-12-04T09:43:53.7100816Z * [new branch] gh/mikaylagawarecki/387/head -> origin/gh/mikaylagawarecki/387/head 2025-12-04T09:43:53.7102509Z * [new branch] gh/mikaylagawarecki/387/orig -> origin/gh/mikaylagawarecki/387/orig 2025-12-04T09:43:53.7105003Z * [new branch] gh/mikaylagawarecki/388/base -> origin/gh/mikaylagawarecki/388/base 2025-12-04T09:43:53.7106867Z * [new branch] gh/mikaylagawarecki/388/head -> origin/gh/mikaylagawarecki/388/head 2025-12-04T09:43:53.7108780Z * [new branch] gh/mikaylagawarecki/388/orig -> origin/gh/mikaylagawarecki/388/orig 2025-12-04T09:43:53.7111719Z * [new branch] gh/mikaylagawarecki/389/base -> origin/gh/mikaylagawarecki/389/base 2025-12-04T09:43:53.7113450Z * [new branch] gh/mikaylagawarecki/389/head -> origin/gh/mikaylagawarecki/389/head 2025-12-04T09:43:53.7115117Z * [new branch] gh/mikaylagawarecki/389/orig -> origin/gh/mikaylagawarecki/389/orig 2025-12-04T09:43:53.7119529Z * [new branch] gh/mikaylagawarecki/390/base -> origin/gh/mikaylagawarecki/390/base 2025-12-04T09:43:53.7121176Z * [new branch] gh/mikaylagawarecki/390/head -> origin/gh/mikaylagawarecki/390/head 2025-12-04T09:43:53.7122918Z * [new branch] gh/mikaylagawarecki/390/orig -> origin/gh/mikaylagawarecki/390/orig 2025-12-04T09:43:53.7125437Z * [new branch] gh/mikaylagawarecki/391/base -> origin/gh/mikaylagawarecki/391/base 2025-12-04T09:43:53.7127854Z * [new branch] gh/mikaylagawarecki/391/head -> origin/gh/mikaylagawarecki/391/head 2025-12-04T09:43:53.7129688Z * [new branch] gh/mikaylagawarecki/391/orig -> origin/gh/mikaylagawarecki/391/orig 2025-12-04T09:43:53.7132235Z * [new branch] gh/mikaylagawarecki/392/base -> origin/gh/mikaylagawarecki/392/base 2025-12-04T09:43:53.7133974Z * [new branch] gh/mikaylagawarecki/392/head -> origin/gh/mikaylagawarecki/392/head 2025-12-04T09:43:53.7136090Z * [new branch] gh/mikaylagawarecki/392/orig -> origin/gh/mikaylagawarecki/392/orig 2025-12-04T09:43:53.7139033Z * [new branch] gh/mlazos/41/base -> origin/gh/mlazos/41/base 2025-12-04T09:43:53.7140753Z * [new branch] gh/mlazos/41/head -> origin/gh/mlazos/41/head 2025-12-04T09:43:53.7142464Z * [new branch] gh/mlazos/41/orig -> origin/gh/mlazos/41/orig 2025-12-04T09:43:53.7145060Z * [new branch] gh/mlazos/42/base -> origin/gh/mlazos/42/base 2025-12-04T09:43:53.7146696Z * [new branch] gh/mlazos/42/head -> origin/gh/mlazos/42/head 2025-12-04T09:43:53.7148396Z * [new branch] gh/mlazos/42/orig -> origin/gh/mlazos/42/orig 2025-12-04T09:43:53.7150566Z * [new branch] gh/mlazos/43/base -> origin/gh/mlazos/43/base 2025-12-04T09:43:53.7152354Z * [new branch] gh/mlazos/43/head -> origin/gh/mlazos/43/head 2025-12-04T09:43:53.7154062Z * [new branch] gh/mlazos/43/orig -> origin/gh/mlazos/43/orig 2025-12-04T09:43:53.7157000Z * [new branch] gh/mlazos/44/base -> origin/gh/mlazos/44/base 2025-12-04T09:43:53.7158738Z * [new branch] gh/mlazos/44/head -> origin/gh/mlazos/44/head 2025-12-04T09:43:53.7160550Z * [new branch] gh/mlazos/44/orig -> origin/gh/mlazos/44/orig 2025-12-04T09:43:53.7162977Z * [new branch] gh/mlazos/47/base -> origin/gh/mlazos/47/base 2025-12-04T09:43:53.7164644Z * [new branch] gh/mlazos/47/head -> origin/gh/mlazos/47/head 2025-12-04T09:43:53.7166458Z * [new branch] gh/mlazos/47/orig -> origin/gh/mlazos/47/orig 2025-12-04T09:43:53.7168722Z * [new branch] gh/mlazos/48/base -> origin/gh/mlazos/48/base 2025-12-04T09:43:53.7170583Z * [new branch] gh/mlazos/48/head -> origin/gh/mlazos/48/head 2025-12-04T09:43:53.7172450Z * [new branch] gh/mlazos/48/orig -> origin/gh/mlazos/48/orig 2025-12-04T09:43:53.7175106Z * [new branch] gh/mlazos/49/base -> origin/gh/mlazos/49/base 2025-12-04T09:43:53.7176834Z * [new branch] gh/mlazos/49/head -> origin/gh/mlazos/49/head 2025-12-04T09:43:53.7178950Z * [new branch] gh/mlazos/49/orig -> origin/gh/mlazos/49/orig 2025-12-04T09:43:53.7182850Z * [new branch] gh/mlazos/50/base -> origin/gh/mlazos/50/base 2025-12-04T09:43:53.7184648Z * [new branch] gh/mlazos/50/head -> origin/gh/mlazos/50/head 2025-12-04T09:43:53.7186349Z * [new branch] gh/mlazos/50/orig -> origin/gh/mlazos/50/orig 2025-12-04T09:43:53.7188627Z * [new branch] gh/mlazos/51/base -> origin/gh/mlazos/51/base 2025-12-04T09:43:53.7190525Z * [new branch] gh/mlazos/51/head -> origin/gh/mlazos/51/head 2025-12-04T09:43:53.7192204Z * [new branch] gh/mlazos/51/orig -> origin/gh/mlazos/51/orig 2025-12-04T09:43:53.7194563Z * [new branch] gh/mlazos/52/base -> origin/gh/mlazos/52/base 2025-12-04T09:43:53.7196287Z * [new branch] gh/mlazos/52/head -> origin/gh/mlazos/52/head 2025-12-04T09:43:53.7197995Z * [new branch] gh/mlazos/52/orig -> origin/gh/mlazos/52/orig 2025-12-04T09:43:53.7200351Z * [new branch] gh/mlazos/53/base -> origin/gh/mlazos/53/base 2025-12-04T09:43:53.7202110Z * [new branch] gh/mlazos/53/head -> origin/gh/mlazos/53/head 2025-12-04T09:43:53.7203810Z * [new branch] gh/mlazos/53/orig -> origin/gh/mlazos/53/orig 2025-12-04T09:43:53.7206213Z * [new branch] gh/mlazos/54/base -> origin/gh/mlazos/54/base 2025-12-04T09:43:53.7207935Z * [new branch] gh/mlazos/54/head -> origin/gh/mlazos/54/head 2025-12-04T09:43:53.7209665Z * [new branch] gh/mlazos/54/orig -> origin/gh/mlazos/54/orig 2025-12-04T09:43:53.7212061Z * [new branch] gh/mlazos/55/base -> origin/gh/mlazos/55/base 2025-12-04T09:43:53.7213802Z * [new branch] gh/mlazos/55/head -> origin/gh/mlazos/55/head 2025-12-04T09:43:53.7215496Z * [new branch] gh/mlazos/55/orig -> origin/gh/mlazos/55/orig 2025-12-04T09:43:53.7217954Z * [new branch] gh/mlazos/56/base -> origin/gh/mlazos/56/base 2025-12-04T09:43:53.7219601Z * [new branch] gh/mlazos/56/head -> origin/gh/mlazos/56/head 2025-12-04T09:43:53.7221294Z * [new branch] gh/mlazos/56/orig -> origin/gh/mlazos/56/orig 2025-12-04T09:43:53.7223734Z * [new branch] gh/mlazos/57/base -> origin/gh/mlazos/57/base 2025-12-04T09:43:53.7225549Z * [new branch] gh/mlazos/57/head -> origin/gh/mlazos/57/head 2025-12-04T09:43:53.7227277Z * [new branch] gh/mlazos/57/orig -> origin/gh/mlazos/57/orig 2025-12-04T09:43:53.7229710Z * [new branch] gh/mlazos/58/base -> origin/gh/mlazos/58/base 2025-12-04T09:43:53.7231412Z * [new branch] gh/mlazos/58/head -> origin/gh/mlazos/58/head 2025-12-04T09:43:53.7233283Z * [new branch] gh/mlazos/58/orig -> origin/gh/mlazos/58/orig 2025-12-04T09:43:53.7235567Z * [new branch] gh/mlazos/59/base -> origin/gh/mlazos/59/base 2025-12-04T09:43:53.7237330Z * [new branch] gh/mlazos/59/head -> origin/gh/mlazos/59/head 2025-12-04T09:43:53.7239048Z * [new branch] gh/mlazos/59/orig -> origin/gh/mlazos/59/orig 2025-12-04T09:43:53.7241472Z * [new branch] gh/mlazos/60/base -> origin/gh/mlazos/60/base 2025-12-04T09:43:53.7243281Z * [new branch] gh/mlazos/60/head -> origin/gh/mlazos/60/head 2025-12-04T09:43:53.7244933Z * [new branch] gh/mlazos/60/orig -> origin/gh/mlazos/60/orig 2025-12-04T09:43:53.7247690Z * [new branch] gh/mlazos/61/base -> origin/gh/mlazos/61/base 2025-12-04T09:43:53.7252016Z * [new branch] gh/mlazos/61/head -> origin/gh/mlazos/61/head 2025-12-04T09:43:53.7253017Z * [new branch] gh/mlazos/61/orig -> origin/gh/mlazos/61/orig 2025-12-04T09:43:53.7254125Z * [new branch] gh/mlazos/62/base -> origin/gh/mlazos/62/base 2025-12-04T09:43:53.7255829Z * [new branch] gh/mlazos/62/head -> origin/gh/mlazos/62/head 2025-12-04T09:43:53.7257563Z * [new branch] gh/mlazos/62/orig -> origin/gh/mlazos/62/orig 2025-12-04T09:43:53.7260411Z * [new branch] gh/mlazos/63/base -> origin/gh/mlazos/63/base 2025-12-04T09:43:53.7262127Z * [new branch] gh/mlazos/63/head -> origin/gh/mlazos/63/head 2025-12-04T09:43:53.7263979Z * [new branch] gh/mlazos/63/orig -> origin/gh/mlazos/63/orig 2025-12-04T09:43:53.7266388Z * [new branch] gh/mlazos/64/base -> origin/gh/mlazos/64/base 2025-12-04T09:43:53.7268157Z * [new branch] gh/mlazos/64/head -> origin/gh/mlazos/64/head 2025-12-04T09:43:53.7269876Z * [new branch] gh/mlazos/64/orig -> origin/gh/mlazos/64/orig 2025-12-04T09:43:53.7272307Z * [new branch] gh/mlazos/65/base -> origin/gh/mlazos/65/base 2025-12-04T09:43:53.7274011Z * [new branch] gh/mlazos/65/head -> origin/gh/mlazos/65/head 2025-12-04T09:43:53.7275785Z * [new branch] gh/mlazos/65/orig -> origin/gh/mlazos/65/orig 2025-12-04T09:43:53.7278346Z * [new branch] gh/mlazos/66/base -> origin/gh/mlazos/66/base 2025-12-04T09:43:53.7280291Z * [new branch] gh/mlazos/66/head -> origin/gh/mlazos/66/head 2025-12-04T09:43:53.7282030Z * [new branch] gh/mlazos/66/orig -> origin/gh/mlazos/66/orig 2025-12-04T09:43:53.7284420Z * [new branch] gh/mlazos/67/base -> origin/gh/mlazos/67/base 2025-12-04T09:43:53.7286091Z * [new branch] gh/mlazos/67/head -> origin/gh/mlazos/67/head 2025-12-04T09:43:53.7287735Z * [new branch] gh/mlazos/67/orig -> origin/gh/mlazos/67/orig 2025-12-04T09:43:53.7290201Z * [new branch] gh/mlazos/68/base -> origin/gh/mlazos/68/base 2025-12-04T09:43:53.7291895Z * [new branch] gh/mlazos/68/head -> origin/gh/mlazos/68/head 2025-12-04T09:43:53.7293615Z * [new branch] gh/mlazos/68/orig -> origin/gh/mlazos/68/orig 2025-12-04T09:43:53.7296090Z * [new branch] gh/mlazos/69/base -> origin/gh/mlazos/69/base 2025-12-04T09:43:53.7297813Z * [new branch] gh/mlazos/69/head -> origin/gh/mlazos/69/head 2025-12-04T09:43:53.7299612Z * [new branch] gh/mlazos/69/orig -> origin/gh/mlazos/69/orig 2025-12-04T09:43:53.7302039Z * [new branch] gh/mlazos/70/base -> origin/gh/mlazos/70/base 2025-12-04T09:43:53.7303881Z * [new branch] gh/mlazos/70/head -> origin/gh/mlazos/70/head 2025-12-04T09:43:53.7305758Z * [new branch] gh/mlazos/70/orig -> origin/gh/mlazos/70/orig 2025-12-04T09:43:53.7308134Z * [new branch] gh/mlazos/71/base -> origin/gh/mlazos/71/base 2025-12-04T09:43:53.7309835Z * [new branch] gh/mlazos/71/head -> origin/gh/mlazos/71/head 2025-12-04T09:43:53.7311608Z * [new branch] gh/mlazos/71/orig -> origin/gh/mlazos/71/orig 2025-12-04T09:43:53.7313922Z * [new branch] gh/mlazos/72/base -> origin/gh/mlazos/72/base 2025-12-04T09:43:53.7315798Z * [new branch] gh/mlazos/72/head -> origin/gh/mlazos/72/head 2025-12-04T09:43:53.7317381Z * [new branch] gh/mlazos/72/orig -> origin/gh/mlazos/72/orig 2025-12-04T09:43:53.7319922Z * [new branch] gh/mlazos/73/base -> origin/gh/mlazos/73/base 2025-12-04T09:43:53.7321706Z * [new branch] gh/mlazos/73/head -> origin/gh/mlazos/73/head 2025-12-04T09:43:53.7323477Z * [new branch] gh/mlazos/73/orig -> origin/gh/mlazos/73/orig 2025-12-04T09:43:53.7326358Z * [new branch] gh/mrmiywj/1/base -> origin/gh/mrmiywj/1/base 2025-12-04T09:43:53.7328199Z * [new branch] gh/mrmiywj/1/head -> origin/gh/mrmiywj/1/head 2025-12-04T09:43:53.7330910Z * [new branch] gh/muchulee8/73/base -> origin/gh/muchulee8/73/base 2025-12-04T09:43:53.7332790Z * [new branch] gh/muchulee8/73/head -> origin/gh/muchulee8/73/head 2025-12-04T09:43:53.7334556Z * [new branch] gh/muchulee8/73/orig -> origin/gh/muchulee8/73/orig 2025-12-04T09:43:53.7337607Z * [new branch] gh/naveenthangudu/1/base -> origin/gh/naveenthangudu/1/base 2025-12-04T09:43:53.7339305Z * [new branch] gh/naveenthangudu/1/head -> origin/gh/naveenthangudu/1/head 2025-12-04T09:43:53.7341125Z * [new branch] gh/naveenthangudu/1/orig -> origin/gh/naveenthangudu/1/orig 2025-12-04T09:43:53.7344029Z * [new branch] gh/naveenthangudu/2/base -> origin/gh/naveenthangudu/2/base 2025-12-04T09:43:53.7345938Z * [new branch] gh/naveenthangudu/2/head -> origin/gh/naveenthangudu/2/head 2025-12-04T09:43:53.7347731Z * [new branch] gh/naveenthangudu/2/orig -> origin/gh/naveenthangudu/2/orig 2025-12-04T09:43:53.7349974Z * [new branch] gh/naveenthangudu/3/base -> origin/gh/naveenthangudu/3/base 2025-12-04T09:43:53.7351723Z * [new branch] gh/naveenthangudu/3/head -> origin/gh/naveenthangudu/3/head 2025-12-04T09:43:53.7353462Z * [new branch] gh/naveenthangudu/3/orig -> origin/gh/naveenthangudu/3/orig 2025-12-04T09:43:53.7355756Z * [new branch] gh/naveenthangudu/4/base -> origin/gh/naveenthangudu/4/base 2025-12-04T09:43:53.7357538Z * [new branch] gh/naveenthangudu/4/head -> origin/gh/naveenthangudu/4/head 2025-12-04T09:43:53.7359858Z * [new branch] gh/naveenthangudu/4/orig -> origin/gh/naveenthangudu/4/orig 2025-12-04T09:43:53.7362189Z * [new branch] gh/naveenthangudu/5/base -> origin/gh/naveenthangudu/5/base 2025-12-04T09:43:53.7363905Z * [new branch] gh/naveenthangudu/5/head -> origin/gh/naveenthangudu/5/head 2025-12-04T09:43:53.7365865Z * [new branch] gh/naveenthangudu/5/orig -> origin/gh/naveenthangudu/5/orig 2025-12-04T09:43:53.7368254Z * [new branch] gh/naveenthangudu/6/base -> origin/gh/naveenthangudu/6/base 2025-12-04T09:43:53.7370123Z * [new branch] gh/naveenthangudu/6/head -> origin/gh/naveenthangudu/6/head 2025-12-04T09:43:53.7371723Z * [new branch] gh/naveenthangudu/6/orig -> origin/gh/naveenthangudu/6/orig 2025-12-04T09:43:53.7374182Z * [new branch] gh/naveenthangudu/7/base -> origin/gh/naveenthangudu/7/base 2025-12-04T09:43:53.7375922Z * [new branch] gh/naveenthangudu/7/head -> origin/gh/naveenthangudu/7/head 2025-12-04T09:43:53.7377672Z * [new branch] gh/naveenthangudu/7/orig -> origin/gh/naveenthangudu/7/orig 2025-12-04T09:43:53.7380270Z * [new branch] gh/naveenthangudu/8/base -> origin/gh/naveenthangudu/8/base 2025-12-04T09:43:53.7382094Z * [new branch] gh/naveenthangudu/8/head -> origin/gh/naveenthangudu/8/head 2025-12-04T09:43:53.7383949Z * [new branch] gh/naveenthangudu/8/orig -> origin/gh/naveenthangudu/8/orig 2025-12-04T09:43:53.7386448Z * [new branch] gh/naveenthangudu/9/base -> origin/gh/naveenthangudu/9/base 2025-12-04T09:43:53.7388056Z * [new branch] gh/naveenthangudu/9/head -> origin/gh/naveenthangudu/9/head 2025-12-04T09:43:53.7389790Z * [new branch] gh/naveenthangudu/9/orig -> origin/gh/naveenthangudu/9/orig 2025-12-04T09:43:53.7392592Z * [new branch] gh/nikitaved/1/base -> origin/gh/nikitaved/1/base 2025-12-04T09:43:53.7394386Z * [new branch] gh/nikitaved/1/head -> origin/gh/nikitaved/1/head 2025-12-04T09:43:53.7396143Z * [new branch] gh/nikitaved/1/orig -> origin/gh/nikitaved/1/orig 2025-12-04T09:43:53.7398735Z * [new branch] gh/nikitaved/10/base -> origin/gh/nikitaved/10/base 2025-12-04T09:43:53.7400917Z * [new branch] gh/nikitaved/10/head -> origin/gh/nikitaved/10/head 2025-12-04T09:43:53.7402628Z * [new branch] gh/nikitaved/10/orig -> origin/gh/nikitaved/10/orig 2025-12-04T09:43:53.7404861Z * [new branch] gh/nikitaved/11/base -> origin/gh/nikitaved/11/base 2025-12-04T09:43:53.7406762Z * [new branch] gh/nikitaved/11/head -> origin/gh/nikitaved/11/head 2025-12-04T09:43:53.7408512Z * [new branch] gh/nikitaved/11/orig -> origin/gh/nikitaved/11/orig 2025-12-04T09:43:53.7410764Z * [new branch] gh/nikitaved/12/base -> origin/gh/nikitaved/12/base 2025-12-04T09:43:53.7412508Z * [new branch] gh/nikitaved/12/head -> origin/gh/nikitaved/12/head 2025-12-04T09:43:53.7414316Z * [new branch] gh/nikitaved/12/orig -> origin/gh/nikitaved/12/orig 2025-12-04T09:43:53.7416797Z * [new branch] gh/nikitaved/13/base -> origin/gh/nikitaved/13/base 2025-12-04T09:43:53.7418946Z * [new branch] gh/nikitaved/13/head -> origin/gh/nikitaved/13/head 2025-12-04T09:43:53.7420689Z * [new branch] gh/nikitaved/13/orig -> origin/gh/nikitaved/13/orig 2025-12-04T09:43:53.7423057Z * [new branch] gh/nikitaved/14/base -> origin/gh/nikitaved/14/base 2025-12-04T09:43:53.7424983Z * [new branch] gh/nikitaved/14/head -> origin/gh/nikitaved/14/head 2025-12-04T09:43:53.7426637Z * [new branch] gh/nikitaved/14/orig -> origin/gh/nikitaved/14/orig 2025-12-04T09:43:53.7428931Z * [new branch] gh/nikitaved/15/base -> origin/gh/nikitaved/15/base 2025-12-04T09:43:53.7430657Z * [new branch] gh/nikitaved/15/head -> origin/gh/nikitaved/15/head 2025-12-04T09:43:53.7432482Z * [new branch] gh/nikitaved/15/orig -> origin/gh/nikitaved/15/orig 2025-12-04T09:43:53.7434783Z * [new branch] gh/nikitaved/16/base -> origin/gh/nikitaved/16/base 2025-12-04T09:43:53.7436493Z * [new branch] gh/nikitaved/16/head -> origin/gh/nikitaved/16/head 2025-12-04T09:43:53.7438231Z * [new branch] gh/nikitaved/16/orig -> origin/gh/nikitaved/16/orig 2025-12-04T09:43:53.7440681Z * [new branch] gh/nikitaved/2/base -> origin/gh/nikitaved/2/base 2025-12-04T09:43:53.7442377Z * [new branch] gh/nikitaved/2/head -> origin/gh/nikitaved/2/head 2025-12-04T09:43:53.7444087Z * [new branch] gh/nikitaved/2/orig -> origin/gh/nikitaved/2/orig 2025-12-04T09:43:53.7446399Z * [new branch] gh/nikitaved/4/base -> origin/gh/nikitaved/4/base 2025-12-04T09:43:53.7448150Z * [new branch] gh/nikitaved/4/head -> origin/gh/nikitaved/4/head 2025-12-04T09:43:53.7449858Z * [new branch] gh/nikitaved/4/orig -> origin/gh/nikitaved/4/orig 2025-12-04T09:43:53.7452201Z * [new branch] gh/nikitaved/5/base -> origin/gh/nikitaved/5/base 2025-12-04T09:43:53.7453924Z * [new branch] gh/nikitaved/5/head -> origin/gh/nikitaved/5/head 2025-12-04T09:43:53.7455869Z * [new branch] gh/nikitaved/5/orig -> origin/gh/nikitaved/5/orig 2025-12-04T09:43:53.7458042Z * [new branch] gh/nikitaved/6/base -> origin/gh/nikitaved/6/base 2025-12-04T09:43:53.7459829Z * [new branch] gh/nikitaved/6/head -> origin/gh/nikitaved/6/head 2025-12-04T09:43:53.7461526Z * [new branch] gh/nikitaved/6/orig -> origin/gh/nikitaved/6/orig 2025-12-04T09:43:53.7463974Z * [new branch] gh/nikitaved/8/base -> origin/gh/nikitaved/8/base 2025-12-04T09:43:53.7465731Z * [new branch] gh/nikitaved/8/head -> origin/gh/nikitaved/8/head 2025-12-04T09:43:53.7467426Z * [new branch] gh/nikitaved/8/orig -> origin/gh/nikitaved/8/orig 2025-12-04T09:43:53.7469745Z * [new branch] gh/nikitaved/9/base -> origin/gh/nikitaved/9/base 2025-12-04T09:43:53.7471654Z * [new branch] gh/nikitaved/9/head -> origin/gh/nikitaved/9/head 2025-12-04T09:43:53.7473369Z * [new branch] gh/nikitaved/9/orig -> origin/gh/nikitaved/9/orig 2025-12-04T09:43:53.7476178Z * [new branch] gh/oulgen/10/base -> origin/gh/oulgen/10/base 2025-12-04T09:43:53.7478188Z * [new branch] gh/oulgen/10/head -> origin/gh/oulgen/10/head 2025-12-04T09:43:53.7480040Z * [new branch] gh/oulgen/10/orig -> origin/gh/oulgen/10/orig 2025-12-04T09:43:53.7482323Z * [new branch] gh/oulgen/11/base -> origin/gh/oulgen/11/base 2025-12-04T09:43:53.7484040Z * [new branch] gh/oulgen/11/head -> origin/gh/oulgen/11/head 2025-12-04T09:43:53.7485775Z * [new branch] gh/oulgen/11/orig -> origin/gh/oulgen/11/orig 2025-12-04T09:43:53.7488182Z * [new branch] gh/oulgen/12/base -> origin/gh/oulgen/12/base 2025-12-04T09:43:53.7489907Z * [new branch] gh/oulgen/12/head -> origin/gh/oulgen/12/head 2025-12-04T09:43:53.7491625Z * [new branch] gh/oulgen/12/orig -> origin/gh/oulgen/12/orig 2025-12-04T09:43:53.7493899Z * [new branch] gh/oulgen/13/base -> origin/gh/oulgen/13/base 2025-12-04T09:43:53.7496122Z * [new branch] gh/oulgen/13/head -> origin/gh/oulgen/13/head 2025-12-04T09:43:53.7498244Z * [new branch] gh/oulgen/13/orig -> origin/gh/oulgen/13/orig 2025-12-04T09:43:53.7500719Z * [new branch] gh/oulgen/14/base -> origin/gh/oulgen/14/base 2025-12-04T09:43:53.7502443Z * [new branch] gh/oulgen/14/head -> origin/gh/oulgen/14/head 2025-12-04T09:43:53.7504566Z * [new branch] gh/oulgen/14/orig -> origin/gh/oulgen/14/orig 2025-12-04T09:43:53.7506694Z * [new branch] gh/oulgen/15/base -> origin/gh/oulgen/15/base 2025-12-04T09:43:53.7508385Z * [new branch] gh/oulgen/15/head -> origin/gh/oulgen/15/head 2025-12-04T09:43:53.7510067Z * [new branch] gh/oulgen/15/orig -> origin/gh/oulgen/15/orig 2025-12-04T09:43:53.7512385Z * [new branch] gh/oulgen/16/base -> origin/gh/oulgen/16/base 2025-12-04T09:43:53.7514068Z * [new branch] gh/oulgen/16/head -> origin/gh/oulgen/16/head 2025-12-04T09:43:53.7515770Z * [new branch] gh/oulgen/16/orig -> origin/gh/oulgen/16/orig 2025-12-04T09:43:53.7518101Z * [new branch] gh/oulgen/17/base -> origin/gh/oulgen/17/base 2025-12-04T09:43:53.7519826Z * [new branch] gh/oulgen/17/head -> origin/gh/oulgen/17/head 2025-12-04T09:43:53.7521487Z * [new branch] gh/oulgen/17/orig -> origin/gh/oulgen/17/orig 2025-12-04T09:43:53.7524118Z * [new branch] gh/oulgen/18/base -> origin/gh/oulgen/18/base 2025-12-04T09:43:53.7525581Z * [new branch] gh/oulgen/18/head -> origin/gh/oulgen/18/head 2025-12-04T09:43:53.7527464Z * [new branch] gh/oulgen/18/orig -> origin/gh/oulgen/18/orig 2025-12-04T09:43:53.7529558Z * [new branch] gh/oulgen/19/base -> origin/gh/oulgen/19/base 2025-12-04T09:43:53.7531488Z * [new branch] gh/oulgen/19/head -> origin/gh/oulgen/19/head 2025-12-04T09:43:53.7533156Z * [new branch] gh/oulgen/19/orig -> origin/gh/oulgen/19/orig 2025-12-04T09:43:53.7535541Z * [new branch] gh/oulgen/20/base -> origin/gh/oulgen/20/base 2025-12-04T09:43:53.7537280Z * [new branch] gh/oulgen/20/head -> origin/gh/oulgen/20/head 2025-12-04T09:43:53.7539010Z * [new branch] gh/oulgen/20/orig -> origin/gh/oulgen/20/orig 2025-12-04T09:43:53.7541264Z * [new branch] gh/oulgen/21/base -> origin/gh/oulgen/21/base 2025-12-04T09:43:53.7542972Z * [new branch] gh/oulgen/21/head -> origin/gh/oulgen/21/head 2025-12-04T09:43:53.7544940Z * [new branch] gh/oulgen/21/orig -> origin/gh/oulgen/21/orig 2025-12-04T09:43:53.7547227Z * [new branch] gh/oulgen/22/base -> origin/gh/oulgen/22/base 2025-12-04T09:43:53.7548949Z * [new branch] gh/oulgen/22/head -> origin/gh/oulgen/22/head 2025-12-04T09:43:53.7550657Z * [new branch] gh/oulgen/22/orig -> origin/gh/oulgen/22/orig 2025-12-04T09:43:53.7552937Z * [new branch] gh/oulgen/23/base -> origin/gh/oulgen/23/base 2025-12-04T09:43:53.7554685Z * [new branch] gh/oulgen/23/head -> origin/gh/oulgen/23/head 2025-12-04T09:43:53.7556358Z * [new branch] gh/oulgen/23/orig -> origin/gh/oulgen/23/orig 2025-12-04T09:43:53.7558703Z * [new branch] gh/oulgen/24/base -> origin/gh/oulgen/24/base 2025-12-04T09:43:53.7560422Z * [new branch] gh/oulgen/24/head -> origin/gh/oulgen/24/head 2025-12-04T09:43:53.7562129Z * [new branch] gh/oulgen/24/orig -> origin/gh/oulgen/24/orig 2025-12-04T09:43:53.7564383Z * [new branch] gh/oulgen/25/base -> origin/gh/oulgen/25/base 2025-12-04T09:43:53.7566071Z * [new branch] gh/oulgen/25/head -> origin/gh/oulgen/25/head 2025-12-04T09:43:53.7567905Z * [new branch] gh/oulgen/25/orig -> origin/gh/oulgen/25/orig 2025-12-04T09:43:53.7570175Z * [new branch] gh/oulgen/26/base -> origin/gh/oulgen/26/base 2025-12-04T09:43:53.7571855Z * [new branch] gh/oulgen/26/head -> origin/gh/oulgen/26/head 2025-12-04T09:43:53.7573666Z * [new branch] gh/oulgen/26/orig -> origin/gh/oulgen/26/orig 2025-12-04T09:43:53.7576004Z * [new branch] gh/oulgen/4/base -> origin/gh/oulgen/4/base 2025-12-04T09:43:53.7577712Z * [new branch] gh/oulgen/4/head -> origin/gh/oulgen/4/head 2025-12-04T09:43:53.7581559Z * [new branch] gh/oulgen/4/orig -> origin/gh/oulgen/4/orig 2025-12-04T09:43:53.7584725Z * [new branch] gh/oulgen/7/base -> origin/gh/oulgen/7/base 2025-12-04T09:43:53.7586402Z * [new branch] gh/oulgen/7/head -> origin/gh/oulgen/7/head 2025-12-04T09:43:53.7588175Z * [new branch] gh/oulgen/7/orig -> origin/gh/oulgen/7/orig 2025-12-04T09:43:53.7590682Z * [new branch] gh/oulgen/8/base -> origin/gh/oulgen/8/base 2025-12-04T09:43:53.7592652Z * [new branch] gh/oulgen/8/head -> origin/gh/oulgen/8/head 2025-12-04T09:43:53.7594319Z * [new branch] gh/oulgen/8/orig -> origin/gh/oulgen/8/orig 2025-12-04T09:43:53.7596618Z * [new branch] gh/oulgen/9/base -> origin/gh/oulgen/9/base 2025-12-04T09:43:53.7598269Z * [new branch] gh/oulgen/9/head -> origin/gh/oulgen/9/head 2025-12-04T09:43:53.7600060Z * [new branch] gh/oulgen/9/orig -> origin/gh/oulgen/9/orig 2025-12-04T09:43:53.7602408Z * [new branch] gh/patvig/mtia-serialization -> origin/gh/patvig/mtia-serialization 2025-12-04T09:43:53.7605546Z * [new branch] gh/pearu/108/base -> origin/gh/pearu/108/base 2025-12-04T09:43:53.7607299Z * [new branch] gh/pearu/108/head -> origin/gh/pearu/108/head 2025-12-04T09:43:53.7609151Z * [new branch] gh/pearu/108/orig -> origin/gh/pearu/108/orig 2025-12-04T09:43:53.7611538Z * [new branch] gh/pearu/109/base -> origin/gh/pearu/109/base 2025-12-04T09:43:53.7613188Z * [new branch] gh/pearu/109/head -> origin/gh/pearu/109/head 2025-12-04T09:43:53.7614989Z * [new branch] gh/pearu/109/orig -> origin/gh/pearu/109/orig 2025-12-04T09:43:53.7617366Z * [new branch] gh/pearu/110/base -> origin/gh/pearu/110/base 2025-12-04T09:43:53.7619117Z * [new branch] gh/pearu/110/head -> origin/gh/pearu/110/head 2025-12-04T09:43:53.7620887Z * [new branch] gh/pearu/110/orig -> origin/gh/pearu/110/orig 2025-12-04T09:43:53.7623198Z * [new branch] gh/pearu/111/base -> origin/gh/pearu/111/base 2025-12-04T09:43:53.7625584Z * [new branch] gh/pearu/111/head -> origin/gh/pearu/111/head 2025-12-04T09:43:53.7627376Z * [new branch] gh/pearu/111/orig -> origin/gh/pearu/111/orig 2025-12-04T09:43:53.7629770Z * [new branch] gh/pearu/112/base -> origin/gh/pearu/112/base 2025-12-04T09:43:53.7631490Z * [new branch] gh/pearu/112/head -> origin/gh/pearu/112/head 2025-12-04T09:43:53.7633200Z * [new branch] gh/pearu/112/orig -> origin/gh/pearu/112/orig 2025-12-04T09:43:53.7635493Z * [new branch] gh/pearu/115/base -> origin/gh/pearu/115/base 2025-12-04T09:43:53.7637174Z * [new branch] gh/pearu/115/head -> origin/gh/pearu/115/head 2025-12-04T09:43:53.7638892Z * [new branch] gh/pearu/115/orig -> origin/gh/pearu/115/orig 2025-12-04T09:43:53.7641132Z * [new branch] gh/pearu/116/base -> origin/gh/pearu/116/base 2025-12-04T09:43:53.7642956Z * [new branch] gh/pearu/116/head -> origin/gh/pearu/116/head 2025-12-04T09:43:53.7645085Z * [new branch] gh/pearu/116/orig -> origin/gh/pearu/116/orig 2025-12-04T09:43:53.7647524Z * [new branch] gh/pearu/117/base -> origin/gh/pearu/117/base 2025-12-04T09:43:53.7649217Z * [new branch] gh/pearu/117/head -> origin/gh/pearu/117/head 2025-12-04T09:43:53.7651005Z * [new branch] gh/pearu/117/orig -> origin/gh/pearu/117/orig 2025-12-04T09:43:53.7653328Z * [new branch] gh/pearu/118/base -> origin/gh/pearu/118/base 2025-12-04T09:43:53.7655025Z * [new branch] gh/pearu/118/head -> origin/gh/pearu/118/head 2025-12-04T09:43:53.7656757Z * [new branch] gh/pearu/118/orig -> origin/gh/pearu/118/orig 2025-12-04T09:43:53.7659050Z * [new branch] gh/pearu/119/base -> origin/gh/pearu/119/base 2025-12-04T09:43:53.7660776Z * [new branch] gh/pearu/119/head -> origin/gh/pearu/119/head 2025-12-04T09:43:53.7662494Z * [new branch] gh/pearu/119/orig -> origin/gh/pearu/119/orig 2025-12-04T09:43:53.7665017Z * [new branch] gh/pearu/139/base -> origin/gh/pearu/139/base 2025-12-04T09:43:53.7667213Z * [new branch] gh/pearu/139/head -> origin/gh/pearu/139/head 2025-12-04T09:43:53.7668928Z * [new branch] gh/pearu/139/orig -> origin/gh/pearu/139/orig 2025-12-04T09:43:53.7671281Z * [new branch] gh/pearu/140/base -> origin/gh/pearu/140/base 2025-12-04T09:43:53.7673113Z * [new branch] gh/pearu/140/head -> origin/gh/pearu/140/head 2025-12-04T09:43:53.7674785Z * [new branch] gh/pearu/140/orig -> origin/gh/pearu/140/orig 2025-12-04T09:43:53.7677191Z * [new branch] gh/pearu/142/base -> origin/gh/pearu/142/base 2025-12-04T09:43:53.7679302Z * [new branch] gh/pearu/142/head -> origin/gh/pearu/142/head 2025-12-04T09:43:53.7680999Z * [new branch] gh/pearu/142/orig -> origin/gh/pearu/142/orig 2025-12-04T09:43:53.7683337Z * [new branch] gh/pearu/143/base -> origin/gh/pearu/143/base 2025-12-04T09:43:53.7685035Z * [new branch] gh/pearu/143/head -> origin/gh/pearu/143/head 2025-12-04T09:43:53.7686819Z * [new branch] gh/pearu/143/orig -> origin/gh/pearu/143/orig 2025-12-04T09:43:53.7689203Z * [new branch] gh/pearu/147/base -> origin/gh/pearu/147/base 2025-12-04T09:43:53.7690912Z * [new branch] gh/pearu/147/head -> origin/gh/pearu/147/head 2025-12-04T09:43:53.7692646Z * [new branch] gh/pearu/147/orig -> origin/gh/pearu/147/orig 2025-12-04T09:43:53.7695017Z * [new branch] gh/pearu/149/base -> origin/gh/pearu/149/base 2025-12-04T09:43:53.7696819Z * [new branch] gh/pearu/149/head -> origin/gh/pearu/149/head 2025-12-04T09:43:53.7698531Z * [new branch] gh/pearu/149/orig -> origin/gh/pearu/149/orig 2025-12-04T09:43:53.7701376Z * [new branch] gh/pearu/150/base -> origin/gh/pearu/150/base 2025-12-04T09:43:53.7703118Z * [new branch] gh/pearu/150/head -> origin/gh/pearu/150/head 2025-12-04T09:43:53.7705070Z * [new branch] gh/pearu/150/orig -> origin/gh/pearu/150/orig 2025-12-04T09:43:53.7708817Z * [new branch] gh/pearu/151/base -> origin/gh/pearu/151/base 2025-12-04T09:43:53.7710881Z * [new branch] gh/pearu/151/head -> origin/gh/pearu/151/head 2025-12-04T09:43:53.7712605Z * [new branch] gh/pearu/151/orig -> origin/gh/pearu/151/orig 2025-12-04T09:43:53.7715059Z * [new branch] gh/pearu/152/base -> origin/gh/pearu/152/base 2025-12-04T09:43:53.7716775Z * [new branch] gh/pearu/152/head -> origin/gh/pearu/152/head 2025-12-04T09:43:53.7718493Z * [new branch] gh/pearu/152/orig -> origin/gh/pearu/152/orig 2025-12-04T09:43:53.7720976Z * [new branch] gh/pearu/153/base -> origin/gh/pearu/153/base 2025-12-04T09:43:53.7722768Z * [new branch] gh/pearu/153/head -> origin/gh/pearu/153/head 2025-12-04T09:43:53.7724461Z * [new branch] gh/pearu/153/orig -> origin/gh/pearu/153/orig 2025-12-04T09:43:53.7726995Z * [new branch] gh/pearu/154/base -> origin/gh/pearu/154/base 2025-12-04T09:43:53.7728721Z * [new branch] gh/pearu/154/head -> origin/gh/pearu/154/head 2025-12-04T09:43:53.7730470Z * [new branch] gh/pearu/154/orig -> origin/gh/pearu/154/orig 2025-12-04T09:43:53.7733399Z * [new branch] gh/pearu/155/base -> origin/gh/pearu/155/base 2025-12-04T09:43:53.7735127Z * [new branch] gh/pearu/155/head -> origin/gh/pearu/155/head 2025-12-04T09:43:53.7736864Z * [new branch] gh/pearu/155/orig -> origin/gh/pearu/155/orig 2025-12-04T09:43:53.7739238Z * [new branch] gh/pearu/156/base -> origin/gh/pearu/156/base 2025-12-04T09:43:53.7740972Z * [new branch] gh/pearu/156/head -> origin/gh/pearu/156/head 2025-12-04T09:43:53.7742702Z * [new branch] gh/pearu/156/orig -> origin/gh/pearu/156/orig 2025-12-04T09:43:53.7745595Z * [new branch] gh/pearu/56/base -> origin/gh/pearu/56/base 2025-12-04T09:43:53.7747657Z * [new branch] gh/pearu/56/head -> origin/gh/pearu/56/head 2025-12-04T09:43:53.7749259Z * [new branch] gh/pearu/56/orig -> origin/gh/pearu/56/orig 2025-12-04T09:43:53.7751887Z * [new branch] gh/pearu/97/base -> origin/gh/pearu/97/base 2025-12-04T09:43:53.7753648Z * [new branch] gh/pearu/97/head -> origin/gh/pearu/97/head 2025-12-04T09:43:53.7755431Z * [new branch] gh/pearu/97/orig -> origin/gh/pearu/97/orig 2025-12-04T09:43:53.7758271Z * [new branch] gh/pianpwk/21/base -> origin/gh/pianpwk/21/base 2025-12-04T09:43:53.7759970Z * [new branch] gh/pianpwk/21/head -> origin/gh/pianpwk/21/head 2025-12-04T09:43:53.7762371Z * [new branch] gh/pianpwk/28/base -> origin/gh/pianpwk/28/base 2025-12-04T09:43:53.7764181Z * [new branch] gh/pianpwk/28/head -> origin/gh/pianpwk/28/head 2025-12-04T09:43:53.7765930Z * [new branch] gh/pianpwk/28/orig -> origin/gh/pianpwk/28/orig 2025-12-04T09:43:53.7768323Z * [new branch] gh/pianpwk/29/base -> origin/gh/pianpwk/29/base 2025-12-04T09:43:53.7770585Z * [new branch] gh/pianpwk/29/head -> origin/gh/pianpwk/29/head 2025-12-04T09:43:53.7772350Z * [new branch] gh/pianpwk/29/orig -> origin/gh/pianpwk/29/orig 2025-12-04T09:43:53.7774904Z * [new branch] gh/pianpwk/30/base -> origin/gh/pianpwk/30/base 2025-12-04T09:43:53.7776599Z * [new branch] gh/pianpwk/30/head -> origin/gh/pianpwk/30/head 2025-12-04T09:43:53.7778457Z * [new branch] gh/pianpwk/30/orig -> origin/gh/pianpwk/30/orig 2025-12-04T09:43:53.7781138Z * [new branch] gh/pianpwk/31/base -> origin/gh/pianpwk/31/base 2025-12-04T09:43:53.7782845Z * [new branch] gh/pianpwk/31/head -> origin/gh/pianpwk/31/head 2025-12-04T09:43:53.7784710Z * [new branch] gh/pianpwk/31/orig -> origin/gh/pianpwk/31/orig 2025-12-04T09:43:53.7786914Z * [new branch] gh/pianpwk/32/base -> origin/gh/pianpwk/32/base 2025-12-04T09:43:53.7788661Z * [new branch] gh/pianpwk/32/head -> origin/gh/pianpwk/32/head 2025-12-04T09:43:53.7790333Z * [new branch] gh/pianpwk/32/orig -> origin/gh/pianpwk/32/orig 2025-12-04T09:43:53.7793015Z * [new branch] gh/pianpwk/33/base -> origin/gh/pianpwk/33/base 2025-12-04T09:43:53.7794751Z * [new branch] gh/pianpwk/33/head -> origin/gh/pianpwk/33/head 2025-12-04T09:43:53.7796472Z * [new branch] gh/pianpwk/33/orig -> origin/gh/pianpwk/33/orig 2025-12-04T09:43:53.7799121Z * [new branch] gh/pianpwk/34/base -> origin/gh/pianpwk/34/base 2025-12-04T09:43:53.7801066Z * [new branch] gh/pianpwk/34/head -> origin/gh/pianpwk/34/head 2025-12-04T09:43:53.7802935Z * [new branch] gh/pianpwk/34/orig -> origin/gh/pianpwk/34/orig 2025-12-04T09:43:53.7805296Z * [new branch] gh/pianpwk/35/base -> origin/gh/pianpwk/35/base 2025-12-04T09:43:53.7807130Z * [new branch] gh/pianpwk/35/head -> origin/gh/pianpwk/35/head 2025-12-04T09:43:53.7808982Z * [new branch] gh/pianpwk/35/orig -> origin/gh/pianpwk/35/orig 2025-12-04T09:43:53.7811725Z * [new branch] gh/rec/141/base -> origin/gh/rec/141/base 2025-12-04T09:43:53.7813664Z * [new branch] gh/rec/141/head -> origin/gh/rec/141/head 2025-12-04T09:43:53.7815980Z * [new branch] gh/rec/153/base -> origin/gh/rec/153/base 2025-12-04T09:43:53.7817717Z * [new branch] gh/rec/153/head -> origin/gh/rec/153/head 2025-12-04T09:43:53.7819563Z * [new branch] gh/rec/153/orig -> origin/gh/rec/153/orig 2025-12-04T09:43:53.7822044Z * [new branch] gh/rec/154/base -> origin/gh/rec/154/base 2025-12-04T09:43:53.7823722Z * [new branch] gh/rec/154/head -> origin/gh/rec/154/head 2025-12-04T09:43:53.7825467Z * [new branch] gh/rec/154/orig -> origin/gh/rec/154/orig 2025-12-04T09:43:53.7827861Z * [new branch] gh/rec/164/base -> origin/gh/rec/164/base 2025-12-04T09:43:53.7829535Z * [new branch] gh/rec/164/head -> origin/gh/rec/164/head 2025-12-04T09:43:53.7831273Z * [new branch] gh/rec/164/orig -> origin/gh/rec/164/orig 2025-12-04T09:43:53.7833562Z * [new branch] gh/rec/166/base -> origin/gh/rec/166/base 2025-12-04T09:43:53.7835334Z * [new branch] gh/rec/166/head -> origin/gh/rec/166/head 2025-12-04T09:43:53.7837050Z * [new branch] gh/rec/166/orig -> origin/gh/rec/166/orig 2025-12-04T09:43:53.7839373Z * [new branch] gh/rec/167/base -> origin/gh/rec/167/base 2025-12-04T09:43:53.7841120Z * [new branch] gh/rec/167/head -> origin/gh/rec/167/head 2025-12-04T09:43:53.7843381Z * [new branch] gh/rec/167/orig -> origin/gh/rec/167/orig 2025-12-04T09:43:53.7845769Z * [new branch] gh/rec/168/base -> origin/gh/rec/168/base 2025-12-04T09:43:53.7847511Z * [new branch] gh/rec/168/head -> origin/gh/rec/168/head 2025-12-04T09:43:53.7849243Z * [new branch] gh/rec/168/orig -> origin/gh/rec/168/orig 2025-12-04T09:43:53.7851623Z * [new branch] gh/rec/169/base -> origin/gh/rec/169/base 2025-12-04T09:43:53.7853355Z * [new branch] gh/rec/169/head -> origin/gh/rec/169/head 2025-12-04T09:43:53.7855063Z * [new branch] gh/rec/169/orig -> origin/gh/rec/169/orig 2025-12-04T09:43:53.7857370Z * [new branch] gh/rec/170/base -> origin/gh/rec/170/base 2025-12-04T09:43:53.7859182Z * [new branch] gh/rec/170/head -> origin/gh/rec/170/head 2025-12-04T09:43:53.7860851Z * [new branch] gh/rec/170/orig -> origin/gh/rec/170/orig 2025-12-04T09:43:53.7863194Z * [new branch] gh/rec/171/base -> origin/gh/rec/171/base 2025-12-04T09:43:53.7865060Z * [new branch] gh/rec/171/head -> origin/gh/rec/171/head 2025-12-04T09:43:53.7866797Z * [new branch] gh/rec/171/orig -> origin/gh/rec/171/orig 2025-12-04T09:43:53.7869096Z * [new branch] gh/rec/172/base -> origin/gh/rec/172/base 2025-12-04T09:43:53.7870795Z * [new branch] gh/rec/172/head -> origin/gh/rec/172/head 2025-12-04T09:43:53.7872556Z * [new branch] gh/rec/172/orig -> origin/gh/rec/172/orig 2025-12-04T09:43:53.7874952Z * [new branch] gh/rec/173/base -> origin/gh/rec/173/base 2025-12-04T09:43:53.7876777Z * [new branch] gh/rec/173/head -> origin/gh/rec/173/head 2025-12-04T09:43:53.7878761Z * [new branch] gh/rec/173/orig -> origin/gh/rec/173/orig 2025-12-04T09:43:53.7881144Z * [new branch] gh/rec/174/base -> origin/gh/rec/174/base 2025-12-04T09:43:53.7882881Z * [new branch] gh/rec/174/head -> origin/gh/rec/174/head 2025-12-04T09:43:53.7884619Z * [new branch] gh/rec/174/orig -> origin/gh/rec/174/orig 2025-12-04T09:43:53.7886898Z * [new branch] gh/rec/175/base -> origin/gh/rec/175/base 2025-12-04T09:43:53.7888603Z * [new branch] gh/rec/175/head -> origin/gh/rec/175/head 2025-12-04T09:43:53.7890348Z * [new branch] gh/rec/175/orig -> origin/gh/rec/175/orig 2025-12-04T09:43:53.7892813Z * [new branch] gh/rec/176/base -> origin/gh/rec/176/base 2025-12-04T09:43:53.7894442Z * [new branch] gh/rec/176/head -> origin/gh/rec/176/head 2025-12-04T09:43:53.7896132Z * [new branch] gh/rec/176/orig -> origin/gh/rec/176/orig 2025-12-04T09:43:53.7898426Z * [new branch] gh/rec/177/base -> origin/gh/rec/177/base 2025-12-04T09:43:53.7900158Z * [new branch] gh/rec/177/head -> origin/gh/rec/177/head 2025-12-04T09:43:53.7901928Z * [new branch] gh/rec/177/orig -> origin/gh/rec/177/orig 2025-12-04T09:43:53.7905600Z * [new branch] gh/robert-hardwick/3/base -> origin/gh/robert-hardwick/3/base 2025-12-04T09:43:53.7907552Z * [new branch] gh/robert-hardwick/3/head -> origin/gh/robert-hardwick/3/head 2025-12-04T09:43:53.7909251Z * [new branch] gh/robert-hardwick/3/orig -> origin/gh/robert-hardwick/3/orig 2025-12-04T09:43:53.7911573Z * [new branch] gh/robert-hardwick/4/base -> origin/gh/robert-hardwick/4/base 2025-12-04T09:43:53.7913284Z * [new branch] gh/robert-hardwick/4/head -> origin/gh/robert-hardwick/4/head 2025-12-04T09:43:53.7915001Z * [new branch] gh/robert-hardwick/4/orig -> origin/gh/robert-hardwick/4/orig 2025-12-04T09:43:53.7917478Z * [new branch] gh/robert-hardwick/5/base -> origin/gh/robert-hardwick/5/base 2025-12-04T09:43:53.7919181Z * [new branch] gh/robert-hardwick/5/head -> origin/gh/robert-hardwick/5/head 2025-12-04T09:43:53.7921052Z * [new branch] gh/robert-hardwick/5/orig -> origin/gh/robert-hardwick/5/orig 2025-12-04T09:43:53.7923385Z * [new branch] gh/robert-hardwick/6/base -> origin/gh/robert-hardwick/6/base 2025-12-04T09:43:53.7925099Z * [new branch] gh/robert-hardwick/6/head -> origin/gh/robert-hardwick/6/head 2025-12-04T09:43:53.7926837Z * [new branch] gh/robert-hardwick/6/orig -> origin/gh/robert-hardwick/6/orig 2025-12-04T09:43:53.7929229Z * [new branch] gh/robert-hardwick/7/base -> origin/gh/robert-hardwick/7/base 2025-12-04T09:43:53.7930925Z * [new branch] gh/robert-hardwick/7/head -> origin/gh/robert-hardwick/7/head 2025-12-04T09:43:53.7932729Z * [new branch] gh/robert-hardwick/7/orig -> origin/gh/robert-hardwick/7/orig 2025-12-04T09:43:53.7935067Z * [new branch] gh/robert-hardwick/8/base -> origin/gh/robert-hardwick/8/base 2025-12-04T09:43:53.7936852Z * [new branch] gh/robert-hardwick/8/head -> origin/gh/robert-hardwick/8/head 2025-12-04T09:43:53.7938543Z * [new branch] gh/robert-hardwick/8/orig -> origin/gh/robert-hardwick/8/orig 2025-12-04T09:43:53.7940940Z * [new branch] gh/robert-hardwick/9/base -> origin/gh/robert-hardwick/9/base 2025-12-04T09:43:53.7942704Z * [new branch] gh/robert-hardwick/9/head -> origin/gh/robert-hardwick/9/head 2025-12-04T09:43:53.7944637Z * [new branch] gh/robert-hardwick/9/orig -> origin/gh/robert-hardwick/9/orig 2025-12-04T09:43:53.7947329Z * [new branch] gh/rtimpe/1/base -> origin/gh/rtimpe/1/base 2025-12-04T09:43:53.7949023Z * [new branch] gh/rtimpe/1/head -> origin/gh/rtimpe/1/head 2025-12-04T09:43:53.7951274Z * [new branch] gh/rtimpe/2/base -> origin/gh/rtimpe/2/base 2025-12-04T09:43:53.7953004Z * [new branch] gh/rtimpe/2/head -> origin/gh/rtimpe/2/head 2025-12-04T09:43:53.7955338Z * [new branch] gh/rtimpe/22/base -> origin/gh/rtimpe/22/base 2025-12-04T09:43:53.7957096Z * [new branch] gh/rtimpe/22/head -> origin/gh/rtimpe/22/head 2025-12-04T09:43:53.7958829Z * [new branch] gh/rtimpe/22/orig -> origin/gh/rtimpe/22/orig 2025-12-04T09:43:53.7961194Z * [new branch] gh/rtimpe/23/base -> origin/gh/rtimpe/23/base 2025-12-04T09:43:53.7962968Z * [new branch] gh/rtimpe/23/head -> origin/gh/rtimpe/23/head 2025-12-04T09:43:53.7964593Z * [new branch] gh/rtimpe/23/orig -> origin/gh/rtimpe/23/orig 2025-12-04T09:43:53.7966839Z * [new branch] gh/rtimpe/24/base -> origin/gh/rtimpe/24/base 2025-12-04T09:43:53.7968646Z * [new branch] gh/rtimpe/24/head -> origin/gh/rtimpe/24/head 2025-12-04T09:43:53.7970385Z * [new branch] gh/rtimpe/24/orig -> origin/gh/rtimpe/24/orig 2025-12-04T09:43:53.7972675Z * [new branch] gh/rtimpe/25/base -> origin/gh/rtimpe/25/base 2025-12-04T09:43:53.7974376Z * [new branch] gh/rtimpe/25/head -> origin/gh/rtimpe/25/head 2025-12-04T09:43:53.7976118Z * [new branch] gh/rtimpe/25/orig -> origin/gh/rtimpe/25/orig 2025-12-04T09:43:53.7978486Z * [new branch] gh/rtimpe/26/base -> origin/gh/rtimpe/26/base 2025-12-04T09:43:53.7982227Z * [new branch] gh/rtimpe/26/head -> origin/gh/rtimpe/26/head 2025-12-04T09:43:53.7984108Z * [new branch] gh/rtimpe/26/orig -> origin/gh/rtimpe/26/orig 2025-12-04T09:43:53.7986592Z * [new branch] gh/rtimpe/27/base -> origin/gh/rtimpe/27/base 2025-12-04T09:43:53.7988229Z * [new branch] gh/rtimpe/27/head -> origin/gh/rtimpe/27/head 2025-12-04T09:43:53.7990015Z * [new branch] gh/rtimpe/27/orig -> origin/gh/rtimpe/27/orig 2025-12-04T09:43:53.7992318Z * [new branch] gh/rtimpe/28/base -> origin/gh/rtimpe/28/base 2025-12-04T09:43:53.7994040Z * [new branch] gh/rtimpe/28/head -> origin/gh/rtimpe/28/head 2025-12-04T09:43:53.7995900Z * [new branch] gh/rtimpe/28/orig -> origin/gh/rtimpe/28/orig 2025-12-04T09:43:53.7998231Z * [new branch] gh/rtimpe/29/base -> origin/gh/rtimpe/29/base 2025-12-04T09:43:53.7999969Z * [new branch] gh/rtimpe/29/head -> origin/gh/rtimpe/29/head 2025-12-04T09:43:53.8001658Z * [new branch] gh/rtimpe/29/orig -> origin/gh/rtimpe/29/orig 2025-12-04T09:43:53.8003914Z * [new branch] gh/rtimpe/3/base -> origin/gh/rtimpe/3/base 2025-12-04T09:43:53.8005598Z * [new branch] gh/rtimpe/3/head -> origin/gh/rtimpe/3/head 2025-12-04T09:43:53.8007938Z * [new branch] gh/rtimpe/30/base -> origin/gh/rtimpe/30/base 2025-12-04T09:43:53.8009753Z * [new branch] gh/rtimpe/30/head -> origin/gh/rtimpe/30/head 2025-12-04T09:43:53.8011464Z * [new branch] gh/rtimpe/30/orig -> origin/gh/rtimpe/30/orig 2025-12-04T09:43:53.8013826Z * [new branch] gh/rtimpe/31/base -> origin/gh/rtimpe/31/base 2025-12-04T09:43:53.8015538Z * [new branch] gh/rtimpe/31/head -> origin/gh/rtimpe/31/head 2025-12-04T09:43:53.8017366Z * [new branch] gh/rtimpe/31/orig -> origin/gh/rtimpe/31/orig 2025-12-04T09:43:53.8019821Z * [new branch] gh/rtimpe/32/base -> origin/gh/rtimpe/32/base 2025-12-04T09:43:53.8021628Z * [new branch] gh/rtimpe/32/head -> origin/gh/rtimpe/32/head 2025-12-04T09:43:53.8023398Z * [new branch] gh/rtimpe/32/orig -> origin/gh/rtimpe/32/orig 2025-12-04T09:43:53.8025893Z * [new branch] gh/rtimpe/33/base -> origin/gh/rtimpe/33/base 2025-12-04T09:43:53.8027627Z * [new branch] gh/rtimpe/33/head -> origin/gh/rtimpe/33/head 2025-12-04T09:43:53.8029359Z * [new branch] gh/rtimpe/33/orig -> origin/gh/rtimpe/33/orig 2025-12-04T09:43:53.8031605Z * [new branch] gh/rtimpe/34/base -> origin/gh/rtimpe/34/base 2025-12-04T09:43:53.8033384Z * [new branch] gh/rtimpe/34/head -> origin/gh/rtimpe/34/head 2025-12-04T09:43:53.8035219Z * [new branch] gh/rtimpe/34/orig -> origin/gh/rtimpe/34/orig 2025-12-04T09:43:53.8037527Z * [new branch] gh/rtimpe/35/base -> origin/gh/rtimpe/35/base 2025-12-04T09:43:53.8039315Z * [new branch] gh/rtimpe/35/head -> origin/gh/rtimpe/35/head 2025-12-04T09:43:53.8041108Z * [new branch] gh/rtimpe/35/orig -> origin/gh/rtimpe/35/orig 2025-12-04T09:43:53.8043403Z * [new branch] gh/rtimpe/4/base -> origin/gh/rtimpe/4/base 2025-12-04T09:43:53.8045135Z * [new branch] gh/rtimpe/4/head -> origin/gh/rtimpe/4/head 2025-12-04T09:43:53.8047970Z * [new branch] gh/ruisizhang123/1/base -> origin/gh/ruisizhang123/1/base 2025-12-04T09:43:53.8049838Z * [new branch] gh/ruisizhang123/1/head -> origin/gh/ruisizhang123/1/head 2025-12-04T09:43:53.8051612Z * [new branch] gh/ruisizhang123/1/orig -> origin/gh/ruisizhang123/1/orig 2025-12-04T09:43:53.8053975Z * [new branch] gh/ruisizhang123/4/base -> origin/gh/ruisizhang123/4/base 2025-12-04T09:43:53.8055813Z * [new branch] gh/ruisizhang123/4/head -> origin/gh/ruisizhang123/4/head 2025-12-04T09:43:53.8057970Z * [new branch] gh/ruisizhang123/4/orig -> origin/gh/ruisizhang123/4/orig 2025-12-04T09:43:53.8060217Z * [new branch] gh/ruisizhang123/5/base -> origin/gh/ruisizhang123/5/base 2025-12-04T09:43:53.8061955Z * [new branch] gh/ruisizhang123/5/head -> origin/gh/ruisizhang123/5/head 2025-12-04T09:43:53.8063870Z * [new branch] gh/ruisizhang123/5/orig -> origin/gh/ruisizhang123/5/orig 2025-12-04T09:43:53.8066197Z * [new branch] gh/ruisizhang123/6/base -> origin/gh/ruisizhang123/6/base 2025-12-04T09:43:53.8067928Z * [new branch] gh/ruisizhang123/6/head -> origin/gh/ruisizhang123/6/head 2025-12-04T09:43:53.8069669Z * [new branch] gh/ruisizhang123/6/orig -> origin/gh/ruisizhang123/6/orig 2025-12-04T09:43:53.8072040Z * [new branch] gh/ruisizhang123/7/base -> origin/gh/ruisizhang123/7/base 2025-12-04T09:43:53.8073705Z * [new branch] gh/ruisizhang123/7/head -> origin/gh/ruisizhang123/7/head 2025-12-04T09:43:53.8075419Z * [new branch] gh/ruisizhang123/7/orig -> origin/gh/ruisizhang123/7/orig 2025-12-04T09:43:53.8077663Z * [new branch] gh/ruisizhang123/8/base -> origin/gh/ruisizhang123/8/base 2025-12-04T09:43:53.8079796Z * [new branch] gh/ruisizhang123/8/head -> origin/gh/ruisizhang123/8/head 2025-12-04T09:43:53.8081563Z * [new branch] gh/ruisizhang123/8/orig -> origin/gh/ruisizhang123/8/orig 2025-12-04T09:43:53.8083992Z * [new branch] gh/ruisizhang123/9/base -> origin/gh/ruisizhang123/9/base 2025-12-04T09:43:53.8085748Z * [new branch] gh/ruisizhang123/9/head -> origin/gh/ruisizhang123/9/head 2025-12-04T09:43:53.8087491Z * [new branch] gh/ruisizhang123/9/orig -> origin/gh/ruisizhang123/9/orig 2025-12-04T09:43:53.8090328Z * [new branch] gh/seemethere/52/base -> origin/gh/seemethere/52/base 2025-12-04T09:43:53.8092066Z * [new branch] gh/seemethere/52/head -> origin/gh/seemethere/52/head 2025-12-04T09:43:53.8093829Z * [new branch] gh/seemethere/52/orig -> origin/gh/seemethere/52/orig 2025-12-04T09:43:53.8096167Z * [new branch] gh/seemethere/53/base -> origin/gh/seemethere/53/base 2025-12-04T09:43:53.8097922Z * [new branch] gh/seemethere/53/head -> origin/gh/seemethere/53/head 2025-12-04T09:43:53.8099645Z * [new branch] gh/seemethere/53/orig -> origin/gh/seemethere/53/orig 2025-12-04T09:43:53.8101998Z * [new branch] gh/seemethere/54/base -> origin/gh/seemethere/54/base 2025-12-04T09:43:53.8103827Z * [new branch] gh/seemethere/54/head -> origin/gh/seemethere/54/head 2025-12-04T09:43:53.8105859Z * [new branch] gh/seemethere/54/orig -> origin/gh/seemethere/54/orig 2025-12-04T09:43:53.8107965Z * [new branch] gh/seemethere/55/base -> origin/gh/seemethere/55/base 2025-12-04T09:43:53.8109795Z * [new branch] gh/seemethere/55/head -> origin/gh/seemethere/55/head 2025-12-04T09:43:53.8111529Z * [new branch] gh/seemethere/55/orig -> origin/gh/seemethere/55/orig 2025-12-04T09:43:53.8113803Z * [new branch] gh/seemethere/59/base -> origin/gh/seemethere/59/base 2025-12-04T09:43:53.8115518Z * [new branch] gh/seemethere/59/head -> origin/gh/seemethere/59/head 2025-12-04T09:43:53.8117633Z * [new branch] gh/seemethere/59/orig -> origin/gh/seemethere/59/orig 2025-12-04T09:43:53.8119989Z * [new branch] gh/seemethere/62/base -> origin/gh/seemethere/62/base 2025-12-04T09:43:53.8121797Z * [new branch] gh/seemethere/62/head -> origin/gh/seemethere/62/head 2025-12-04T09:43:53.8123563Z * [new branch] gh/seemethere/62/orig -> origin/gh/seemethere/62/orig 2025-12-04T09:43:53.8125954Z * [new branch] gh/seemethere/63/base -> origin/gh/seemethere/63/base 2025-12-04T09:43:53.8127689Z * [new branch] gh/seemethere/63/head -> origin/gh/seemethere/63/head 2025-12-04T09:43:53.8129427Z * [new branch] gh/seemethere/63/orig -> origin/gh/seemethere/63/orig 2025-12-04T09:43:53.8131718Z * [new branch] gh/seemethere/71/base -> origin/gh/seemethere/71/base 2025-12-04T09:43:53.8133492Z * [new branch] gh/seemethere/71/head -> origin/gh/seemethere/71/head 2025-12-04T09:43:53.8135755Z * [new branch] gh/seemethere/71/orig -> origin/gh/seemethere/71/orig 2025-12-04T09:43:53.8138134Z * [new branch] gh/seemethere/72/base -> origin/gh/seemethere/72/base 2025-12-04T09:43:53.8139883Z * [new branch] gh/seemethere/72/head -> origin/gh/seemethere/72/head 2025-12-04T09:43:53.8141945Z * [new branch] gh/seemethere/72/orig -> origin/gh/seemethere/72/orig 2025-12-04T09:43:53.8144237Z * [new branch] gh/seemethere/73/base -> origin/gh/seemethere/73/base 2025-12-04T09:43:53.8146000Z * [new branch] gh/seemethere/73/head -> origin/gh/seemethere/73/head 2025-12-04T09:43:53.8147703Z * [new branch] gh/seemethere/73/orig -> origin/gh/seemethere/73/orig 2025-12-04T09:43:53.8150114Z * [new branch] gh/seemethere/74/base -> origin/gh/seemethere/74/base 2025-12-04T09:43:53.8151781Z * [new branch] gh/seemethere/74/head -> origin/gh/seemethere/74/head 2025-12-04T09:43:53.8153534Z * [new branch] gh/seemethere/74/orig -> origin/gh/seemethere/74/orig 2025-12-04T09:43:53.8155856Z * [new branch] gh/seemethere/75/base -> origin/gh/seemethere/75/base 2025-12-04T09:43:53.8157635Z * [new branch] gh/seemethere/75/head -> origin/gh/seemethere/75/head 2025-12-04T09:43:53.8159381Z * [new branch] gh/seemethere/75/orig -> origin/gh/seemethere/75/orig 2025-12-04T09:43:53.8161730Z * [new branch] gh/seemethere/76/base -> origin/gh/seemethere/76/base 2025-12-04T09:43:53.8163446Z * [new branch] gh/seemethere/76/head -> origin/gh/seemethere/76/head 2025-12-04T09:43:53.8165244Z * [new branch] gh/seemethere/76/orig -> origin/gh/seemethere/76/orig 2025-12-04T09:43:53.8168321Z * [new branch] gh/shunting314/145/base -> origin/gh/shunting314/145/base 2025-12-04T09:43:53.8170171Z * [new branch] gh/shunting314/145/head -> origin/gh/shunting314/145/head 2025-12-04T09:43:53.8172045Z * [new branch] gh/shunting314/145/orig -> origin/gh/shunting314/145/orig 2025-12-04T09:43:53.8174759Z * [new branch] gh/shunting314/176/base -> origin/gh/shunting314/176/base 2025-12-04T09:43:53.8176616Z * [new branch] gh/shunting314/176/head -> origin/gh/shunting314/176/head 2025-12-04T09:43:53.8179016Z * [new branch] gh/shunting314/176/orig -> origin/gh/shunting314/176/orig 2025-12-04T09:43:53.8181790Z * [new branch] gh/shunting314/249/base -> origin/gh/shunting314/249/base 2025-12-04T09:43:53.8183832Z * [new branch] gh/shunting314/249/head -> origin/gh/shunting314/249/head 2025-12-04T09:43:53.8185680Z * [new branch] gh/shunting314/249/orig -> origin/gh/shunting314/249/orig 2025-12-04T09:43:53.8188039Z * [new branch] gh/shunting314/253/base -> origin/gh/shunting314/253/base 2025-12-04T09:43:53.8189776Z * [new branch] gh/shunting314/253/head -> origin/gh/shunting314/253/head 2025-12-04T09:43:53.8191480Z * [new branch] gh/shunting314/253/orig -> origin/gh/shunting314/253/orig 2025-12-04T09:43:53.8193848Z * [new branch] gh/shunting314/256/base -> origin/gh/shunting314/256/base 2025-12-04T09:43:53.8195575Z * [new branch] gh/shunting314/256/head -> origin/gh/shunting314/256/head 2025-12-04T09:43:53.8197332Z * [new branch] gh/shunting314/256/orig -> origin/gh/shunting314/256/orig 2025-12-04T09:43:53.8199906Z * [new branch] gh/shunting314/257/base -> origin/gh/shunting314/257/base 2025-12-04T09:43:53.8201709Z * [new branch] gh/shunting314/257/head -> origin/gh/shunting314/257/head 2025-12-04T09:43:53.8203466Z * [new branch] gh/shunting314/257/orig -> origin/gh/shunting314/257/orig 2025-12-04T09:43:53.8206028Z * [new branch] gh/shunting314/258/base -> origin/gh/shunting314/258/base 2025-12-04T09:43:53.8207667Z * [new branch] gh/shunting314/258/head -> origin/gh/shunting314/258/head 2025-12-04T09:43:53.8209439Z * [new branch] gh/shunting314/258/orig -> origin/gh/shunting314/258/orig 2025-12-04T09:43:53.8211661Z * [new branch] gh/shunting314/259/base -> origin/gh/shunting314/259/base 2025-12-04T09:43:53.8213468Z * [new branch] gh/shunting314/259/head -> origin/gh/shunting314/259/head 2025-12-04T09:43:53.8215178Z * [new branch] gh/shunting314/259/orig -> origin/gh/shunting314/259/orig 2025-12-04T09:43:53.8217590Z * [new branch] gh/shunting314/260/base -> origin/gh/shunting314/260/base 2025-12-04T09:43:53.8219474Z * [new branch] gh/shunting314/260/head -> origin/gh/shunting314/260/head 2025-12-04T09:43:53.8221343Z * [new branch] gh/shunting314/260/orig -> origin/gh/shunting314/260/orig 2025-12-04T09:43:53.8223867Z * [new branch] gh/shunting314/261/base -> origin/gh/shunting314/261/base 2025-12-04T09:43:53.8225857Z * [new branch] gh/shunting314/261/head -> origin/gh/shunting314/261/head 2025-12-04T09:43:53.8227631Z * [new branch] gh/shunting314/261/orig -> origin/gh/shunting314/261/orig 2025-12-04T09:43:53.8230041Z * [new branch] gh/shunting314/262/base -> origin/gh/shunting314/262/base 2025-12-04T09:43:53.8231827Z * [new branch] gh/shunting314/262/head -> origin/gh/shunting314/262/head 2025-12-04T09:43:53.8233557Z * [new branch] gh/shunting314/262/orig -> origin/gh/shunting314/262/orig 2025-12-04T09:43:53.8235949Z * [new branch] gh/shunting314/263/base -> origin/gh/shunting314/263/base 2025-12-04T09:43:53.8237832Z * [new branch] gh/shunting314/263/head -> origin/gh/shunting314/263/head 2025-12-04T09:43:53.8239555Z * [new branch] gh/shunting314/263/orig -> origin/gh/shunting314/263/orig 2025-12-04T09:43:53.8241959Z * [new branch] gh/shunting314/264/base -> origin/gh/shunting314/264/base 2025-12-04T09:43:53.8243824Z * [new branch] gh/shunting314/264/head -> origin/gh/shunting314/264/head 2025-12-04T09:43:53.8245485Z * [new branch] gh/shunting314/264/orig -> origin/gh/shunting314/264/orig 2025-12-04T09:43:53.8247899Z * [new branch] gh/shunting314/265/base -> origin/gh/shunting314/265/base 2025-12-04T09:43:53.8249570Z * [new branch] gh/shunting314/265/head -> origin/gh/shunting314/265/head 2025-12-04T09:43:53.8251494Z * [new branch] gh/shunting314/265/orig -> origin/gh/shunting314/265/orig 2025-12-04T09:43:53.8253835Z * [new branch] gh/shunting314/266/base -> origin/gh/shunting314/266/base 2025-12-04T09:43:53.8255703Z * [new branch] gh/shunting314/266/head -> origin/gh/shunting314/266/head 2025-12-04T09:43:53.8257429Z * [new branch] gh/shunting314/266/orig -> origin/gh/shunting314/266/orig 2025-12-04T09:43:53.8259923Z * [new branch] gh/shunting314/267/base -> origin/gh/shunting314/267/base 2025-12-04T09:43:53.8261787Z * [new branch] gh/shunting314/267/head -> origin/gh/shunting314/267/head 2025-12-04T09:43:53.8263543Z * [new branch] gh/shunting314/267/orig -> origin/gh/shunting314/267/orig 2025-12-04T09:43:53.8266571Z * [new branch] gh/shunting314/268/base -> origin/gh/shunting314/268/base 2025-12-04T09:43:53.8268348Z * [new branch] gh/shunting314/268/head -> origin/gh/shunting314/268/head 2025-12-04T09:43:53.8270066Z * [new branch] gh/shunting314/268/orig -> origin/gh/shunting314/268/orig 2025-12-04T09:43:53.8272528Z * [new branch] gh/shunting314/269/base -> origin/gh/shunting314/269/base 2025-12-04T09:43:53.8274319Z * [new branch] gh/shunting314/269/head -> origin/gh/shunting314/269/head 2025-12-04T09:43:53.8276095Z * [new branch] gh/shunting314/269/orig -> origin/gh/shunting314/269/orig 2025-12-04T09:43:53.8279116Z * [new branch] gh/silverguo/1/base -> origin/gh/silverguo/1/base 2025-12-04T09:43:53.8280820Z * [new branch] gh/silverguo/1/head -> origin/gh/silverguo/1/head 2025-12-04T09:43:53.8283125Z * [new branch] gh/silverguo/2/base -> origin/gh/silverguo/2/base 2025-12-04T09:43:53.8284814Z * [new branch] gh/silverguo/2/head -> origin/gh/silverguo/2/head 2025-12-04T09:43:53.8286931Z * [new branch] gh/silverguo/3/base -> origin/gh/silverguo/3/base 2025-12-04T09:43:53.8288649Z * [new branch] gh/silverguo/3/head -> origin/gh/silverguo/3/head 2025-12-04T09:43:53.8290761Z * [new branch] gh/silverguo/4/base -> origin/gh/silverguo/4/base 2025-12-04T09:43:53.8292780Z * [new branch] gh/silverguo/4/head -> origin/gh/silverguo/4/head 2025-12-04T09:43:53.8295491Z * [new branch] gh/slayton58/39/base -> origin/gh/slayton58/39/base 2025-12-04T09:43:53.8297232Z * [new branch] gh/slayton58/39/head -> origin/gh/slayton58/39/head 2025-12-04T09:43:53.8298979Z * [new branch] gh/slayton58/39/orig -> origin/gh/slayton58/39/orig 2025-12-04T09:43:53.8301289Z * [new branch] gh/slayton58/42/base -> origin/gh/slayton58/42/base 2025-12-04T09:43:53.8302975Z * [new branch] gh/slayton58/42/head -> origin/gh/slayton58/42/head 2025-12-04T09:43:53.8304980Z * [new branch] gh/slayton58/42/orig -> origin/gh/slayton58/42/orig 2025-12-04T09:43:53.8307278Z * [new branch] gh/slayton58/43/base -> origin/gh/slayton58/43/base 2025-12-04T09:43:53.8309092Z * [new branch] gh/slayton58/43/head -> origin/gh/slayton58/43/head 2025-12-04T09:43:53.8318265Z * [new branch] gh/slayton58/43/orig -> origin/gh/slayton58/43/orig 2025-12-04T09:43:53.8318525Z * [new branch] gh/slayton58/44/base -> origin/gh/slayton58/44/base 2025-12-04T09:43:53.8318840Z * [new branch] gh/slayton58/44/head -> origin/gh/slayton58/44/head 2025-12-04T09:43:53.8319005Z * [new branch] gh/slayton58/44/orig -> origin/gh/slayton58/44/orig 2025-12-04T09:43:53.8320197Z * [new branch] gh/slayton58/45/base -> origin/gh/slayton58/45/base 2025-12-04T09:43:53.8321962Z * [new branch] gh/slayton58/45/head -> origin/gh/slayton58/45/head 2025-12-04T09:43:53.8323680Z * [new branch] gh/slayton58/45/orig -> origin/gh/slayton58/45/orig 2025-12-04T09:43:53.8326088Z * [new branch] gh/slayton58/46/base -> origin/gh/slayton58/46/base 2025-12-04T09:43:53.8327860Z * [new branch] gh/slayton58/46/head -> origin/gh/slayton58/46/head 2025-12-04T09:43:53.8329616Z * [new branch] gh/slayton58/46/orig -> origin/gh/slayton58/46/orig 2025-12-04T09:43:53.8331856Z * [new branch] gh/slayton58/6/base -> origin/gh/slayton58/6/base 2025-12-04T09:43:53.8333731Z * [new branch] gh/slayton58/6/head -> origin/gh/slayton58/6/head 2025-12-04T09:43:53.8335967Z * [new branch] gh/slayton58/7/base -> origin/gh/slayton58/7/base 2025-12-04T09:43:53.8337576Z * [new branch] gh/slayton58/7/head -> origin/gh/slayton58/7/head 2025-12-04T09:43:53.8340661Z * [new branch] gh/soulitzer/269/base -> origin/gh/soulitzer/269/base 2025-12-04T09:43:53.8342382Z * [new branch] gh/soulitzer/269/head -> origin/gh/soulitzer/269/head 2025-12-04T09:43:53.8344304Z * [new branch] gh/soulitzer/269/orig -> origin/gh/soulitzer/269/orig 2025-12-04T09:43:53.8346700Z * [new branch] gh/soulitzer/276/base -> origin/gh/soulitzer/276/base 2025-12-04T09:43:53.8348434Z * [new branch] gh/soulitzer/276/head -> origin/gh/soulitzer/276/head 2025-12-04T09:43:53.8350257Z * [new branch] gh/soulitzer/276/orig -> origin/gh/soulitzer/276/orig 2025-12-04T09:43:53.8352840Z * [new branch] gh/soulitzer/287/base -> origin/gh/soulitzer/287/base 2025-12-04T09:43:53.8354526Z * [new branch] gh/soulitzer/287/head -> origin/gh/soulitzer/287/head 2025-12-04T09:43:53.8356283Z * [new branch] gh/soulitzer/287/orig -> origin/gh/soulitzer/287/orig 2025-12-04T09:43:53.8358752Z * [new branch] gh/soulitzer/296/base -> origin/gh/soulitzer/296/base 2025-12-04T09:43:53.8360905Z * [new branch] gh/soulitzer/296/head -> origin/gh/soulitzer/296/head 2025-12-04T09:43:53.8362724Z * [new branch] gh/soulitzer/296/orig -> origin/gh/soulitzer/296/orig 2025-12-04T09:43:53.8365187Z * [new branch] gh/soulitzer/299/base -> origin/gh/soulitzer/299/base 2025-12-04T09:43:53.8366983Z * [new branch] gh/soulitzer/299/head -> origin/gh/soulitzer/299/head 2025-12-04T09:43:53.8368724Z * [new branch] gh/soulitzer/299/orig -> origin/gh/soulitzer/299/orig 2025-12-04T09:43:53.8371050Z * [new branch] gh/soulitzer/300/base -> origin/gh/soulitzer/300/base 2025-12-04T09:43:53.8372802Z * [new branch] gh/soulitzer/300/head -> origin/gh/soulitzer/300/head 2025-12-04T09:43:53.8374598Z * [new branch] gh/soulitzer/300/orig -> origin/gh/soulitzer/300/orig 2025-12-04T09:43:53.8377545Z * [new branch] gh/soulitzer/301/base -> origin/gh/soulitzer/301/base 2025-12-04T09:43:53.8381109Z * [new branch] gh/soulitzer/301/head -> origin/gh/soulitzer/301/head 2025-12-04T09:43:53.8382860Z * [new branch] gh/soulitzer/301/orig -> origin/gh/soulitzer/301/orig 2025-12-04T09:43:53.8385293Z * [new branch] gh/soulitzer/313/base -> origin/gh/soulitzer/313/base 2025-12-04T09:43:53.8386973Z * [new branch] gh/soulitzer/313/head -> origin/gh/soulitzer/313/head 2025-12-04T09:43:53.8388941Z * [new branch] gh/soulitzer/313/orig -> origin/gh/soulitzer/313/orig 2025-12-04T09:43:53.8391346Z * [new branch] gh/soulitzer/319/base -> origin/gh/soulitzer/319/base 2025-12-04T09:43:53.8392997Z * [new branch] gh/soulitzer/319/head -> origin/gh/soulitzer/319/head 2025-12-04T09:43:53.8394714Z * [new branch] gh/soulitzer/319/orig -> origin/gh/soulitzer/319/orig 2025-12-04T09:43:53.8397136Z * [new branch] gh/soulitzer/320/base -> origin/gh/soulitzer/320/base 2025-12-04T09:43:53.8399766Z * [new branch] gh/soulitzer/320/head -> origin/gh/soulitzer/320/head 2025-12-04T09:43:53.8401741Z * [new branch] gh/soulitzer/320/orig -> origin/gh/soulitzer/320/orig 2025-12-04T09:43:53.8403989Z * [new branch] gh/soulitzer/336/base -> origin/gh/soulitzer/336/base 2025-12-04T09:43:53.8405697Z * [new branch] gh/soulitzer/336/head -> origin/gh/soulitzer/336/head 2025-12-04T09:43:53.8407484Z * [new branch] gh/soulitzer/336/orig -> origin/gh/soulitzer/336/orig 2025-12-04T09:43:53.8409845Z * [new branch] gh/soulitzer/347/base -> origin/gh/soulitzer/347/base 2025-12-04T09:43:53.8411428Z * [new branch] gh/soulitzer/347/head -> origin/gh/soulitzer/347/head 2025-12-04T09:43:53.8413217Z * [new branch] gh/soulitzer/347/orig -> origin/gh/soulitzer/347/orig 2025-12-04T09:43:53.8415810Z * [new branch] gh/soulitzer/349/base -> origin/gh/soulitzer/349/base 2025-12-04T09:43:53.8417555Z * [new branch] gh/soulitzer/349/head -> origin/gh/soulitzer/349/head 2025-12-04T09:43:53.8419312Z * [new branch] gh/soulitzer/349/orig -> origin/gh/soulitzer/349/orig 2025-12-04T09:43:53.8421576Z * [new branch] gh/soulitzer/350/base -> origin/gh/soulitzer/350/base 2025-12-04T09:43:53.8423332Z * [new branch] gh/soulitzer/350/head -> origin/gh/soulitzer/350/head 2025-12-04T09:43:53.8425200Z * [new branch] gh/soulitzer/350/orig -> origin/gh/soulitzer/350/orig 2025-12-04T09:43:53.8427540Z * [new branch] gh/soulitzer/351/base -> origin/gh/soulitzer/351/base 2025-12-04T09:43:53.8429279Z * [new branch] gh/soulitzer/351/head -> origin/gh/soulitzer/351/head 2025-12-04T09:43:53.8431037Z * [new branch] gh/soulitzer/351/orig -> origin/gh/soulitzer/351/orig 2025-12-04T09:43:53.8433337Z * [new branch] gh/soulitzer/353/base -> origin/gh/soulitzer/353/base 2025-12-04T09:43:53.8435137Z * [new branch] gh/soulitzer/353/head -> origin/gh/soulitzer/353/head 2025-12-04T09:43:53.8437222Z * [new branch] gh/soulitzer/353/orig -> origin/gh/soulitzer/353/orig 2025-12-04T09:43:53.8440211Z * [new branch] gh/soulitzer/358/base -> origin/gh/soulitzer/358/base 2025-12-04T09:43:53.8442017Z * [new branch] gh/soulitzer/358/head -> origin/gh/soulitzer/358/head 2025-12-04T09:43:53.8443842Z * [new branch] gh/soulitzer/358/orig -> origin/gh/soulitzer/358/orig 2025-12-04T09:43:53.8446766Z * [new branch] gh/soulitzer/359/base -> origin/gh/soulitzer/359/base 2025-12-04T09:43:53.8449013Z * [new branch] gh/soulitzer/359/head -> origin/gh/soulitzer/359/head 2025-12-04T09:43:53.8450800Z * [new branch] gh/soulitzer/359/orig -> origin/gh/soulitzer/359/orig 2025-12-04T09:43:53.8453743Z * [new branch] gh/soulitzer/374/base -> origin/gh/soulitzer/374/base 2025-12-04T09:43:53.8455444Z * [new branch] gh/soulitzer/374/head -> origin/gh/soulitzer/374/head 2025-12-04T09:43:53.8457223Z * [new branch] gh/soulitzer/374/orig -> origin/gh/soulitzer/374/orig 2025-12-04T09:43:53.8459983Z * [new branch] gh/soulitzer/375/base -> origin/gh/soulitzer/375/base 2025-12-04T09:43:53.8461694Z * [new branch] gh/soulitzer/375/head -> origin/gh/soulitzer/375/head 2025-12-04T09:43:53.8463431Z * [new branch] gh/soulitzer/375/orig -> origin/gh/soulitzer/375/orig 2025-12-04T09:43:53.8466013Z * [new branch] gh/soulitzer/380/base -> origin/gh/soulitzer/380/base 2025-12-04T09:43:53.8467690Z * [new branch] gh/soulitzer/380/head -> origin/gh/soulitzer/380/head 2025-12-04T09:43:53.8469493Z * [new branch] gh/soulitzer/380/orig -> origin/gh/soulitzer/380/orig 2025-12-04T09:43:53.8471925Z * [new branch] gh/soulitzer/385/base -> origin/gh/soulitzer/385/base 2025-12-04T09:43:53.8473610Z * [new branch] gh/soulitzer/385/head -> origin/gh/soulitzer/385/head 2025-12-04T09:43:53.8475320Z * [new branch] gh/soulitzer/385/orig -> origin/gh/soulitzer/385/orig 2025-12-04T09:43:53.8477718Z * [new branch] gh/soulitzer/386/base -> origin/gh/soulitzer/386/base 2025-12-04T09:43:53.8479762Z * [new branch] gh/soulitzer/386/head -> origin/gh/soulitzer/386/head 2025-12-04T09:43:53.8481451Z * [new branch] gh/soulitzer/386/orig -> origin/gh/soulitzer/386/orig 2025-12-04T09:43:53.8483818Z * [new branch] gh/soulitzer/387/base -> origin/gh/soulitzer/387/base 2025-12-04T09:43:53.8485529Z * [new branch] gh/soulitzer/387/head -> origin/gh/soulitzer/387/head 2025-12-04T09:43:53.8487383Z * [new branch] gh/soulitzer/387/orig -> origin/gh/soulitzer/387/orig 2025-12-04T09:43:53.8489776Z * [new branch] gh/soulitzer/388/base -> origin/gh/soulitzer/388/base 2025-12-04T09:43:53.8491473Z * [new branch] gh/soulitzer/388/head -> origin/gh/soulitzer/388/head 2025-12-04T09:43:53.8493195Z * [new branch] gh/soulitzer/388/orig -> origin/gh/soulitzer/388/orig 2025-12-04T09:43:53.8495616Z * [new branch] gh/soulitzer/389/base -> origin/gh/soulitzer/389/base 2025-12-04T09:43:53.8497341Z * [new branch] gh/soulitzer/389/head -> origin/gh/soulitzer/389/head 2025-12-04T09:43:53.8499093Z * [new branch] gh/soulitzer/389/orig -> origin/gh/soulitzer/389/orig 2025-12-04T09:43:53.8501476Z * [new branch] gh/soulitzer/390/base -> origin/gh/soulitzer/390/base 2025-12-04T09:43:53.8503289Z * [new branch] gh/soulitzer/390/head -> origin/gh/soulitzer/390/head 2025-12-04T09:43:53.8505163Z * [new branch] gh/soulitzer/390/orig -> origin/gh/soulitzer/390/orig 2025-12-04T09:43:53.8507622Z * [new branch] gh/soulitzer/391/base -> origin/gh/soulitzer/391/base 2025-12-04T09:43:53.8509334Z * [new branch] gh/soulitzer/391/head -> origin/gh/soulitzer/391/head 2025-12-04T09:43:53.8511133Z * [new branch] gh/soulitzer/391/orig -> origin/gh/soulitzer/391/orig 2025-12-04T09:43:53.8513520Z * [new branch] gh/soulitzer/392/base -> origin/gh/soulitzer/392/base 2025-12-04T09:43:53.8515233Z * [new branch] gh/soulitzer/392/head -> origin/gh/soulitzer/392/head 2025-12-04T09:43:53.8516938Z * [new branch] gh/soulitzer/392/orig -> origin/gh/soulitzer/392/orig 2025-12-04T09:43:53.8519830Z * [new branch] gh/swolchok/728/next -> origin/gh/swolchok/728/next 2025-12-04T09:43:53.8522487Z * [new branch] gh/swolchok/819/base -> origin/gh/swolchok/819/base 2025-12-04T09:43:53.8524219Z * [new branch] gh/swolchok/819/head -> origin/gh/swolchok/819/head 2025-12-04T09:43:53.8525957Z * [new branch] gh/swolchok/819/orig -> origin/gh/swolchok/819/orig 2025-12-04T09:43:53.8528373Z * [new branch] gh/swolchok/824/base -> origin/gh/swolchok/824/base 2025-12-04T09:43:53.8530330Z * [new branch] gh/swolchok/824/head -> origin/gh/swolchok/824/head 2025-12-04T09:43:53.8531914Z * [new branch] gh/swolchok/824/orig -> origin/gh/swolchok/824/orig 2025-12-04T09:43:53.8534324Z * [new branch] gh/swolchok/829/base -> origin/gh/swolchok/829/base 2025-12-04T09:43:53.8535969Z * [new branch] gh/swolchok/829/head -> origin/gh/swolchok/829/head 2025-12-04T09:43:53.8537768Z * [new branch] gh/swolchok/829/orig -> origin/gh/swolchok/829/orig 2025-12-04T09:43:53.8540158Z * [new branch] gh/swolchok/839/base -> origin/gh/swolchok/839/base 2025-12-04T09:43:53.8541730Z * [new branch] gh/swolchok/839/head -> origin/gh/swolchok/839/head 2025-12-04T09:43:53.8544137Z * [new branch] gh/swolchok/839/orig -> origin/gh/swolchok/839/orig 2025-12-04T09:43:53.8546464Z * [new branch] gh/swolchok/841/base -> origin/gh/swolchok/841/base 2025-12-04T09:43:53.8548172Z * [new branch] gh/swolchok/841/head -> origin/gh/swolchok/841/head 2025-12-04T09:43:53.8550058Z * [new branch] gh/swolchok/841/orig -> origin/gh/swolchok/841/orig 2025-12-04T09:43:53.8552487Z * [new branch] gh/swolchok/842/base -> origin/gh/swolchok/842/base 2025-12-04T09:43:53.8554209Z * [new branch] gh/swolchok/842/head -> origin/gh/swolchok/842/head 2025-12-04T09:43:53.8555935Z * [new branch] gh/swolchok/842/orig -> origin/gh/swolchok/842/orig 2025-12-04T09:43:53.8558328Z * [new branch] gh/swolchok/845/base -> origin/gh/swolchok/845/base 2025-12-04T09:43:53.8560155Z * [new branch] gh/swolchok/845/head -> origin/gh/swolchok/845/head 2025-12-04T09:43:53.8561941Z * [new branch] gh/swolchok/845/orig -> origin/gh/swolchok/845/orig 2025-12-04T09:43:53.8564321Z * [new branch] gh/swolchok/848/base -> origin/gh/swolchok/848/base 2025-12-04T09:43:53.8566107Z * [new branch] gh/swolchok/848/head -> origin/gh/swolchok/848/head 2025-12-04T09:43:53.8567855Z * [new branch] gh/swolchok/848/orig -> origin/gh/swolchok/848/orig 2025-12-04T09:43:53.8570099Z * [new branch] gh/swolchok/856/base -> origin/gh/swolchok/856/base 2025-12-04T09:43:53.8572200Z * [new branch] gh/swolchok/856/head -> origin/gh/swolchok/856/head 2025-12-04T09:43:53.8573977Z * [new branch] gh/swolchok/856/orig -> origin/gh/swolchok/856/orig 2025-12-04T09:43:53.8576322Z * [new branch] gh/swolchok/860/base -> origin/gh/swolchok/860/base 2025-12-04T09:43:53.8578204Z * [new branch] gh/swolchok/860/head -> origin/gh/swolchok/860/head 2025-12-04T09:43:53.8581180Z * [new branch] gh/swolchok/860/orig -> origin/gh/swolchok/860/orig 2025-12-04T09:43:53.8583931Z * [new branch] gh/swolchok/861/base -> origin/gh/swolchok/861/base 2025-12-04T09:43:53.8585750Z * [new branch] gh/swolchok/861/head -> origin/gh/swolchok/861/head 2025-12-04T09:43:53.8587476Z * [new branch] gh/swolchok/861/orig -> origin/gh/swolchok/861/orig 2025-12-04T09:43:53.8589821Z * [new branch] gh/swolchok/862/base -> origin/gh/swolchok/862/base 2025-12-04T09:43:53.8591539Z * [new branch] gh/swolchok/862/head -> origin/gh/swolchok/862/head 2025-12-04T09:43:53.8593258Z * [new branch] gh/swolchok/862/orig -> origin/gh/swolchok/862/orig 2025-12-04T09:43:53.8595702Z * [new branch] gh/swolchok/863/base -> origin/gh/swolchok/863/base 2025-12-04T09:43:53.8597423Z * [new branch] gh/swolchok/863/head -> origin/gh/swolchok/863/head 2025-12-04T09:43:53.8599312Z * [new branch] gh/swolchok/863/orig -> origin/gh/swolchok/863/orig 2025-12-04T09:43:53.8601796Z * [new branch] gh/swolchok/864/base -> origin/gh/swolchok/864/base 2025-12-04T09:43:53.8603416Z * [new branch] gh/swolchok/864/head -> origin/gh/swolchok/864/head 2025-12-04T09:43:53.8605215Z * [new branch] gh/swolchok/864/orig -> origin/gh/swolchok/864/orig 2025-12-04T09:43:53.8607631Z * [new branch] gh/swolchok/865/base -> origin/gh/swolchok/865/base 2025-12-04T09:43:53.8609497Z * [new branch] gh/swolchok/865/head -> origin/gh/swolchok/865/head 2025-12-04T09:43:53.8611274Z * [new branch] gh/swolchok/865/orig -> origin/gh/swolchok/865/orig 2025-12-04T09:43:53.8614137Z * [new branch] gh/swolchok/866/base -> origin/gh/swolchok/866/base 2025-12-04T09:43:53.8616110Z * [new branch] gh/swolchok/866/head -> origin/gh/swolchok/866/head 2025-12-04T09:43:53.8617813Z * [new branch] gh/swolchok/866/orig -> origin/gh/swolchok/866/orig 2025-12-04T09:43:53.8620163Z * [new branch] gh/swolchok/867/base -> origin/gh/swolchok/867/base 2025-12-04T09:43:53.8621920Z * [new branch] gh/swolchok/867/head -> origin/gh/swolchok/867/head 2025-12-04T09:43:53.8623718Z * [new branch] gh/swolchok/867/orig -> origin/gh/swolchok/867/orig 2025-12-04T09:43:53.8626209Z * [new branch] gh/swolchok/868/base -> origin/gh/swolchok/868/base 2025-12-04T09:43:53.8627955Z * [new branch] gh/swolchok/868/head -> origin/gh/swolchok/868/head 2025-12-04T09:43:53.8629689Z * [new branch] gh/swolchok/868/orig -> origin/gh/swolchok/868/orig 2025-12-04T09:43:53.8632323Z * [new branch] gh/swolchok/869/base -> origin/gh/swolchok/869/base 2025-12-04T09:43:53.8634044Z * [new branch] gh/swolchok/869/head -> origin/gh/swolchok/869/head 2025-12-04T09:43:53.8635827Z * [new branch] gh/swolchok/869/orig -> origin/gh/swolchok/869/orig 2025-12-04T09:43:53.8638276Z * [new branch] gh/swolchok/870/base -> origin/gh/swolchok/870/base 2025-12-04T09:43:53.8639953Z * [new branch] gh/swolchok/870/head -> origin/gh/swolchok/870/head 2025-12-04T09:43:53.8641694Z * [new branch] gh/swolchok/870/orig -> origin/gh/swolchok/870/orig 2025-12-04T09:43:53.8644456Z * [new branch] gh/swolchok/871/base -> origin/gh/swolchok/871/base 2025-12-04T09:43:53.8646272Z * [new branch] gh/swolchok/871/head -> origin/gh/swolchok/871/head 2025-12-04T09:43:53.8648497Z * [new branch] gh/swolchok/871/orig -> origin/gh/swolchok/871/orig 2025-12-04T09:43:53.8651409Z * [new branch] gh/teja-rao/4/base -> origin/gh/teja-rao/4/base 2025-12-04T09:43:53.8653219Z * [new branch] gh/teja-rao/4/head -> origin/gh/teja-rao/4/head 2025-12-04T09:43:53.8654965Z * [new branch] gh/teja-rao/4/orig -> origin/gh/teja-rao/4/orig 2025-12-04T09:43:53.8657750Z * [new branch] gh/tianyu-l/2/base -> origin/gh/tianyu-l/2/base 2025-12-04T09:43:53.8659481Z * [new branch] gh/tianyu-l/2/head -> origin/gh/tianyu-l/2/head 2025-12-04T09:43:53.8661143Z * [new branch] gh/tianyu-l/2/orig -> origin/gh/tianyu-l/2/orig 2025-12-04T09:43:53.8663435Z * [new branch] gh/tianyu-l/3/base -> origin/gh/tianyu-l/3/base 2025-12-04T09:43:53.8665386Z * [new branch] gh/tianyu-l/3/orig -> origin/gh/tianyu-l/3/orig 2025-12-04T09:43:53.8667641Z * [new branch] gh/tianyu-l/4/base -> origin/gh/tianyu-l/4/base 2025-12-04T09:43:53.8669311Z * [new branch] gh/tianyu-l/4/head -> origin/gh/tianyu-l/4/head 2025-12-04T09:43:53.8671070Z * [new branch] gh/tianyu-l/4/orig -> origin/gh/tianyu-l/4/orig 2025-12-04T09:43:53.8674452Z * [new branch] gh/tugsbayasgalan/10/base -> origin/gh/tugsbayasgalan/10/base 2025-12-04T09:43:53.8676316Z * [new branch] gh/tugsbayasgalan/10/head -> origin/gh/tugsbayasgalan/10/head 2025-12-04T09:43:53.8678480Z * [new branch] gh/tugsbayasgalan/10/orig -> origin/gh/tugsbayasgalan/10/orig 2025-12-04T09:43:53.8681040Z * [new branch] gh/tugsbayasgalan/13/base -> origin/gh/tugsbayasgalan/13/base 2025-12-04T09:43:53.8682752Z * [new branch] gh/tugsbayasgalan/13/head -> origin/gh/tugsbayasgalan/13/head 2025-12-04T09:43:53.8684499Z * [new branch] gh/tugsbayasgalan/13/orig -> origin/gh/tugsbayasgalan/13/orig 2025-12-04T09:43:53.8686877Z * [new branch] gh/tugsbayasgalan/17/base -> origin/gh/tugsbayasgalan/17/base 2025-12-04T09:43:53.8688561Z * [new branch] gh/tugsbayasgalan/17/head -> origin/gh/tugsbayasgalan/17/head 2025-12-04T09:43:53.8690269Z * [new branch] gh/tugsbayasgalan/17/orig -> origin/gh/tugsbayasgalan/17/orig 2025-12-04T09:43:53.8692833Z * [new branch] gh/tugsbayasgalan/2/base -> origin/gh/tugsbayasgalan/2/base 2025-12-04T09:43:53.8694461Z * [new branch] gh/tugsbayasgalan/2/head -> origin/gh/tugsbayasgalan/2/head 2025-12-04T09:43:53.8696194Z * [new branch] gh/tugsbayasgalan/2/orig -> origin/gh/tugsbayasgalan/2/orig 2025-12-04T09:43:53.8698710Z * [new branch] gh/tugsbayasgalan/28/base -> origin/gh/tugsbayasgalan/28/base 2025-12-04T09:43:53.8700525Z * [new branch] gh/tugsbayasgalan/28/head -> origin/gh/tugsbayasgalan/28/head 2025-12-04T09:43:53.8702296Z * [new branch] gh/tugsbayasgalan/28/orig -> origin/gh/tugsbayasgalan/28/orig 2025-12-04T09:43:53.8704809Z * [new branch] gh/tugsbayasgalan/32/base -> origin/gh/tugsbayasgalan/32/base 2025-12-04T09:43:53.8706515Z * [new branch] gh/tugsbayasgalan/32/head -> origin/gh/tugsbayasgalan/32/head 2025-12-04T09:43:53.8708260Z * [new branch] gh/tugsbayasgalan/32/orig -> origin/gh/tugsbayasgalan/32/orig 2025-12-04T09:43:53.8710603Z * [new branch] gh/tugsbayasgalan/35/base -> origin/gh/tugsbayasgalan/35/base 2025-12-04T09:43:53.8712532Z * [new branch] gh/tugsbayasgalan/35/head -> origin/gh/tugsbayasgalan/35/head 2025-12-04T09:43:53.8714187Z * [new branch] gh/tugsbayasgalan/35/orig -> origin/gh/tugsbayasgalan/35/orig 2025-12-04T09:43:53.8716562Z * [new branch] gh/tugsbayasgalan/36/base -> origin/gh/tugsbayasgalan/36/base 2025-12-04T09:43:53.8718288Z * [new branch] gh/tugsbayasgalan/36/head -> origin/gh/tugsbayasgalan/36/head 2025-12-04T09:43:53.8719951Z * [new branch] gh/tugsbayasgalan/36/orig -> origin/gh/tugsbayasgalan/36/orig 2025-12-04T09:43:53.8722321Z * [new branch] gh/tugsbayasgalan/37/base -> origin/gh/tugsbayasgalan/37/base 2025-12-04T09:43:53.8724111Z * [new branch] gh/tugsbayasgalan/37/head -> origin/gh/tugsbayasgalan/37/head 2025-12-04T09:43:53.8725853Z * [new branch] gh/tugsbayasgalan/37/orig -> origin/gh/tugsbayasgalan/37/orig 2025-12-04T09:43:53.8728221Z * [new branch] gh/tugsbayasgalan/43/base -> origin/gh/tugsbayasgalan/43/base 2025-12-04T09:43:53.8730019Z * [new branch] gh/tugsbayasgalan/43/head -> origin/gh/tugsbayasgalan/43/head 2025-12-04T09:43:53.8731727Z * [new branch] gh/tugsbayasgalan/43/orig -> origin/gh/tugsbayasgalan/43/orig 2025-12-04T09:43:53.8733997Z * [new branch] gh/tugsbayasgalan/48/base -> origin/gh/tugsbayasgalan/48/base 2025-12-04T09:43:53.8735760Z * [new branch] gh/tugsbayasgalan/48/head -> origin/gh/tugsbayasgalan/48/head 2025-12-04T09:43:53.8737501Z * [new branch] gh/tugsbayasgalan/48/orig -> origin/gh/tugsbayasgalan/48/orig 2025-12-04T09:43:53.8739839Z * [new branch] gh/tugsbayasgalan/51/base -> origin/gh/tugsbayasgalan/51/base 2025-12-04T09:43:53.8741695Z * [new branch] gh/tugsbayasgalan/51/head -> origin/gh/tugsbayasgalan/51/head 2025-12-04T09:43:53.8743339Z * [new branch] gh/tugsbayasgalan/51/orig -> origin/gh/tugsbayasgalan/51/orig 2025-12-04T09:43:53.8745682Z * [new branch] gh/tugsbayasgalan/52/base -> origin/gh/tugsbayasgalan/52/base 2025-12-04T09:43:53.8747577Z * [new branch] gh/tugsbayasgalan/52/head -> origin/gh/tugsbayasgalan/52/head 2025-12-04T09:43:53.8749364Z * [new branch] gh/tugsbayasgalan/52/orig -> origin/gh/tugsbayasgalan/52/orig 2025-12-04T09:43:53.8751744Z * [new branch] gh/tugsbayasgalan/53/base -> origin/gh/tugsbayasgalan/53/base 2025-12-04T09:43:53.8753456Z * [new branch] gh/tugsbayasgalan/53/head -> origin/gh/tugsbayasgalan/53/head 2025-12-04T09:43:53.8755270Z * [new branch] gh/tugsbayasgalan/53/orig -> origin/gh/tugsbayasgalan/53/orig 2025-12-04T09:43:53.8757742Z * [new branch] gh/tugsbayasgalan/55/base -> origin/gh/tugsbayasgalan/55/base 2025-12-04T09:43:53.8759511Z * [new branch] gh/tugsbayasgalan/55/head -> origin/gh/tugsbayasgalan/55/head 2025-12-04T09:43:53.8761269Z * [new branch] gh/tugsbayasgalan/55/orig -> origin/gh/tugsbayasgalan/55/orig 2025-12-04T09:43:53.8763649Z * [new branch] gh/tugsbayasgalan/59/base -> origin/gh/tugsbayasgalan/59/base 2025-12-04T09:43:53.8765591Z * [new branch] gh/tugsbayasgalan/59/head -> origin/gh/tugsbayasgalan/59/head 2025-12-04T09:43:53.8767240Z * [new branch] gh/tugsbayasgalan/59/orig -> origin/gh/tugsbayasgalan/59/orig 2025-12-04T09:43:53.8769523Z * [new branch] gh/tugsbayasgalan/6/base -> origin/gh/tugsbayasgalan/6/base 2025-12-04T09:43:53.8771282Z * [new branch] gh/tugsbayasgalan/6/head -> origin/gh/tugsbayasgalan/6/head 2025-12-04T09:43:53.8773130Z * [new branch] gh/tugsbayasgalan/6/orig -> origin/gh/tugsbayasgalan/6/orig 2025-12-04T09:43:53.8775331Z * [new branch] gh/tugsbayasgalan/60/base -> origin/gh/tugsbayasgalan/60/base 2025-12-04T09:43:53.8777020Z * [new branch] gh/tugsbayasgalan/60/head -> origin/gh/tugsbayasgalan/60/head 2025-12-04T09:43:53.8778961Z * [new branch] gh/tugsbayasgalan/60/orig -> origin/gh/tugsbayasgalan/60/orig 2025-12-04T09:43:53.8781898Z * [new branch] gh/tugsbayasgalan/61/base -> origin/gh/tugsbayasgalan/61/base 2025-12-04T09:43:53.8783667Z * [new branch] gh/tugsbayasgalan/61/head -> origin/gh/tugsbayasgalan/61/head 2025-12-04T09:43:53.8785563Z * [new branch] gh/tugsbayasgalan/61/orig -> origin/gh/tugsbayasgalan/61/orig 2025-12-04T09:43:53.8788001Z * [new branch] gh/tugsbayasgalan/63/base -> origin/gh/tugsbayasgalan/63/base 2025-12-04T09:43:53.8789739Z * [new branch] gh/tugsbayasgalan/63/head -> origin/gh/tugsbayasgalan/63/head 2025-12-04T09:43:53.8791400Z * [new branch] gh/tugsbayasgalan/63/orig -> origin/gh/tugsbayasgalan/63/orig 2025-12-04T09:43:53.8793971Z * [new branch] gh/tugsbayasgalan/67/base -> origin/gh/tugsbayasgalan/67/base 2025-12-04T09:43:53.8795795Z * [new branch] gh/tugsbayasgalan/67/head -> origin/gh/tugsbayasgalan/67/head 2025-12-04T09:43:53.8797528Z * [new branch] gh/tugsbayasgalan/67/orig -> origin/gh/tugsbayasgalan/67/orig 2025-12-04T09:43:53.8799968Z * [new branch] gh/tugsbayasgalan/68/base -> origin/gh/tugsbayasgalan/68/base 2025-12-04T09:43:53.8802189Z * [new branch] gh/tugsbayasgalan/68/head -> origin/gh/tugsbayasgalan/68/head 2025-12-04T09:43:53.8803965Z * [new branch] gh/tugsbayasgalan/68/orig -> origin/gh/tugsbayasgalan/68/orig 2025-12-04T09:43:53.8806880Z * [new branch] gh/tugsbayasgalan/7/base -> origin/gh/tugsbayasgalan/7/base 2025-12-04T09:43:53.8808587Z * [new branch] gh/tugsbayasgalan/7/head -> origin/gh/tugsbayasgalan/7/head 2025-12-04T09:43:53.8811107Z * [new branch] gh/tugsbayasgalan/7/orig -> origin/gh/tugsbayasgalan/7/orig 2025-12-04T09:43:53.8813612Z * [new branch] gh/tugsbayasgalan/70/base -> origin/gh/tugsbayasgalan/70/base 2025-12-04T09:43:53.8815438Z * [new branch] gh/tugsbayasgalan/70/head -> origin/gh/tugsbayasgalan/70/head 2025-12-04T09:43:53.8817204Z * [new branch] gh/tugsbayasgalan/70/orig -> origin/gh/tugsbayasgalan/70/orig 2025-12-04T09:43:53.8819849Z * [new branch] gh/tugsbayasgalan/71/base -> origin/gh/tugsbayasgalan/71/base 2025-12-04T09:43:53.8821719Z * [new branch] gh/tugsbayasgalan/71/head -> origin/gh/tugsbayasgalan/71/head 2025-12-04T09:43:53.8823475Z * [new branch] gh/tugsbayasgalan/71/orig -> origin/gh/tugsbayasgalan/71/orig 2025-12-04T09:43:53.8826152Z * [new branch] gh/tugsbayasgalan/72/base -> origin/gh/tugsbayasgalan/72/base 2025-12-04T09:43:53.8827952Z * [new branch] gh/tugsbayasgalan/72/head -> origin/gh/tugsbayasgalan/72/head 2025-12-04T09:43:53.8829659Z * [new branch] gh/tugsbayasgalan/72/orig -> origin/gh/tugsbayasgalan/72/orig 2025-12-04T09:43:53.8832056Z * [new branch] gh/tugsbayasgalan/73/base -> origin/gh/tugsbayasgalan/73/base 2025-12-04T09:43:53.8833845Z * [new branch] gh/tugsbayasgalan/73/head -> origin/gh/tugsbayasgalan/73/head 2025-12-04T09:43:53.8835636Z * [new branch] gh/tugsbayasgalan/73/orig -> origin/gh/tugsbayasgalan/73/orig 2025-12-04T09:43:53.8838210Z * [new branch] gh/tugsbayasgalan/74/base -> origin/gh/tugsbayasgalan/74/base 2025-12-04T09:43:53.8840043Z * [new branch] gh/tugsbayasgalan/74/head -> origin/gh/tugsbayasgalan/74/head 2025-12-04T09:43:53.8841864Z * [new branch] gh/tugsbayasgalan/74/orig -> origin/gh/tugsbayasgalan/74/orig 2025-12-04T09:43:53.8844745Z * [new branch] gh/tugsbayasgalan/75/base -> origin/gh/tugsbayasgalan/75/base 2025-12-04T09:43:53.8846595Z * [new branch] gh/tugsbayasgalan/75/head -> origin/gh/tugsbayasgalan/75/head 2025-12-04T09:43:53.8848343Z * [new branch] gh/tugsbayasgalan/75/orig -> origin/gh/tugsbayasgalan/75/orig 2025-12-04T09:43:53.8850583Z * [new branch] gh/tugsbayasgalan/76/base -> origin/gh/tugsbayasgalan/76/base 2025-12-04T09:43:53.8852889Z * [new branch] gh/tugsbayasgalan/76/head -> origin/gh/tugsbayasgalan/76/head 2025-12-04T09:43:53.8854669Z * [new branch] gh/tugsbayasgalan/76/orig -> origin/gh/tugsbayasgalan/76/orig 2025-12-04T09:43:53.8857362Z * [new branch] gh/tugsbayasgalan/77/base -> origin/gh/tugsbayasgalan/77/base 2025-12-04T09:43:53.8859062Z * [new branch] gh/tugsbayasgalan/77/head -> origin/gh/tugsbayasgalan/77/head 2025-12-04T09:43:53.8860717Z * [new branch] gh/tugsbayasgalan/77/orig -> origin/gh/tugsbayasgalan/77/orig 2025-12-04T09:43:53.8863292Z * [new branch] gh/tugsbayasgalan/78/base -> origin/gh/tugsbayasgalan/78/base 2025-12-04T09:43:53.8865261Z * [new branch] gh/tugsbayasgalan/78/head -> origin/gh/tugsbayasgalan/78/head 2025-12-04T09:43:53.8866951Z * [new branch] gh/tugsbayasgalan/78/orig -> origin/gh/tugsbayasgalan/78/orig 2025-12-04T09:43:53.8869373Z * [new branch] gh/tugsbayasgalan/79/base -> origin/gh/tugsbayasgalan/79/base 2025-12-04T09:43:53.8871149Z * [new branch] gh/tugsbayasgalan/79/head -> origin/gh/tugsbayasgalan/79/head 2025-12-04T09:43:53.8872973Z * [new branch] gh/tugsbayasgalan/79/orig -> origin/gh/tugsbayasgalan/79/orig 2025-12-04T09:43:53.8875370Z * [new branch] gh/tugsbayasgalan/8/base -> origin/gh/tugsbayasgalan/8/base 2025-12-04T09:43:53.8877012Z * [new branch] gh/tugsbayasgalan/8/head -> origin/gh/tugsbayasgalan/8/head 2025-12-04T09:43:53.8879832Z * [new branch] gh/tugsbayasgalan/8/orig -> origin/gh/tugsbayasgalan/8/orig 2025-12-04T09:43:53.8881912Z * [new branch] gh/tugsbayasgalan/80/base -> origin/gh/tugsbayasgalan/80/base 2025-12-04T09:43:53.8883550Z * [new branch] gh/tugsbayasgalan/80/head -> origin/gh/tugsbayasgalan/80/head 2025-12-04T09:43:53.8885272Z * [new branch] gh/tugsbayasgalan/80/orig -> origin/gh/tugsbayasgalan/80/orig 2025-12-04T09:43:53.8887772Z * [new branch] gh/tugsbayasgalan/81/base -> origin/gh/tugsbayasgalan/81/base 2025-12-04T09:43:53.8889479Z * [new branch] gh/tugsbayasgalan/81/head -> origin/gh/tugsbayasgalan/81/head 2025-12-04T09:43:53.8891195Z * [new branch] gh/tugsbayasgalan/81/orig -> origin/gh/tugsbayasgalan/81/orig 2025-12-04T09:43:53.8894460Z * [new branch] gh/tugsbayasgalan/82/base -> origin/gh/tugsbayasgalan/82/base 2025-12-04T09:43:53.8896424Z * [new branch] gh/tugsbayasgalan/82/head -> origin/gh/tugsbayasgalan/82/head 2025-12-04T09:43:53.8898179Z * [new branch] gh/tugsbayasgalan/82/orig -> origin/gh/tugsbayasgalan/82/orig 2025-12-04T09:43:53.8900390Z * [new branch] gh/tugsbayasgalan/83/base -> origin/gh/tugsbayasgalan/83/base 2025-12-04T09:43:53.8902227Z * [new branch] gh/tugsbayasgalan/83/head -> origin/gh/tugsbayasgalan/83/head 2025-12-04T09:43:53.8903986Z * [new branch] gh/tugsbayasgalan/83/orig -> origin/gh/tugsbayasgalan/83/orig 2025-12-04T09:43:53.8906289Z * [new branch] gh/tugsbayasgalan/84/base -> origin/gh/tugsbayasgalan/84/base 2025-12-04T09:43:53.8908121Z * [new branch] gh/tugsbayasgalan/84/head -> origin/gh/tugsbayasgalan/84/head 2025-12-04T09:43:53.8909867Z * [new branch] gh/tugsbayasgalan/84/orig -> origin/gh/tugsbayasgalan/84/orig 2025-12-04T09:43:53.8912083Z * [new branch] gh/tugsbayasgalan/85/base -> origin/gh/tugsbayasgalan/85/base 2025-12-04T09:43:53.8913851Z * [new branch] gh/tugsbayasgalan/85/head -> origin/gh/tugsbayasgalan/85/head 2025-12-04T09:43:53.8915575Z * [new branch] gh/tugsbayasgalan/85/orig -> origin/gh/tugsbayasgalan/85/orig 2025-12-04T09:43:53.8917976Z * [new branch] gh/tugsbayasgalan/86/base -> origin/gh/tugsbayasgalan/86/base 2025-12-04T09:43:53.8919890Z * [new branch] gh/tugsbayasgalan/86/head -> origin/gh/tugsbayasgalan/86/head 2025-12-04T09:43:53.8921769Z * [new branch] gh/tugsbayasgalan/86/orig -> origin/gh/tugsbayasgalan/86/orig 2025-12-04T09:43:53.8924399Z * [new branch] gh/tugsbayasgalan/87/base -> origin/gh/tugsbayasgalan/87/base 2025-12-04T09:43:53.8926167Z * [new branch] gh/tugsbayasgalan/87/head -> origin/gh/tugsbayasgalan/87/head 2025-12-04T09:43:53.8927887Z * [new branch] gh/tugsbayasgalan/87/orig -> origin/gh/tugsbayasgalan/87/orig 2025-12-04T09:43:53.8930350Z * [new branch] gh/tugsbayasgalan/88/base -> origin/gh/tugsbayasgalan/88/base 2025-12-04T09:43:53.8932099Z * [new branch] gh/tugsbayasgalan/88/head -> origin/gh/tugsbayasgalan/88/head 2025-12-04T09:43:53.8933887Z * [new branch] gh/tugsbayasgalan/88/orig -> origin/gh/tugsbayasgalan/88/orig 2025-12-04T09:43:53.8936273Z * [new branch] gh/tugsbayasgalan/89/base -> origin/gh/tugsbayasgalan/89/base 2025-12-04T09:43:53.8938085Z * [new branch] gh/tugsbayasgalan/89/head -> origin/gh/tugsbayasgalan/89/head 2025-12-04T09:43:53.8939722Z * [new branch] gh/tugsbayasgalan/89/orig -> origin/gh/tugsbayasgalan/89/orig 2025-12-04T09:43:53.8942085Z * [new branch] gh/tugsbayasgalan/9/base -> origin/gh/tugsbayasgalan/9/base 2025-12-04T09:43:53.8943800Z * [new branch] gh/tugsbayasgalan/9/head -> origin/gh/tugsbayasgalan/9/head 2025-12-04T09:43:53.8945543Z * [new branch] gh/tugsbayasgalan/9/orig -> origin/gh/tugsbayasgalan/9/orig 2025-12-04T09:43:53.8948277Z * [new branch] gh/tugsbayasgalan/90/base -> origin/gh/tugsbayasgalan/90/base 2025-12-04T09:43:53.8950014Z * [new branch] gh/tugsbayasgalan/90/head -> origin/gh/tugsbayasgalan/90/head 2025-12-04T09:43:53.8951794Z * [new branch] gh/tugsbayasgalan/90/orig -> origin/gh/tugsbayasgalan/90/orig 2025-12-04T09:43:53.8954254Z * [new branch] gh/tugsbayasgalan/91/base -> origin/gh/tugsbayasgalan/91/base 2025-12-04T09:43:53.8955939Z * [new branch] gh/tugsbayasgalan/91/head -> origin/gh/tugsbayasgalan/91/head 2025-12-04T09:43:53.8957606Z * [new branch] gh/tugsbayasgalan/91/orig -> origin/gh/tugsbayasgalan/91/orig 2025-12-04T09:43:53.8960026Z * [new branch] gh/tugsbayasgalan/92/base -> origin/gh/tugsbayasgalan/92/base 2025-12-04T09:43:53.8961824Z * [new branch] gh/tugsbayasgalan/92/head -> origin/gh/tugsbayasgalan/92/head 2025-12-04T09:43:53.8963584Z * [new branch] gh/tugsbayasgalan/92/orig -> origin/gh/tugsbayasgalan/92/orig 2025-12-04T09:43:53.8966154Z * [new branch] gh/tugsbayasgalan/93/base -> origin/gh/tugsbayasgalan/93/base 2025-12-04T09:43:53.8967974Z * [new branch] gh/tugsbayasgalan/93/head -> origin/gh/tugsbayasgalan/93/head 2025-12-04T09:43:53.8969764Z * [new branch] gh/tugsbayasgalan/93/orig -> origin/gh/tugsbayasgalan/93/orig 2025-12-04T09:43:53.8972597Z * [new branch] gh/v0i0/14/base -> origin/gh/v0i0/14/base 2025-12-04T09:43:53.8974295Z * [new branch] gh/v0i0/14/head -> origin/gh/v0i0/14/head 2025-12-04T09:43:53.8975969Z * [new branch] gh/v0i0/14/orig -> origin/gh/v0i0/14/orig 2025-12-04T09:43:53.8978344Z * [new branch] gh/v0i0/15/base -> origin/gh/v0i0/15/base 2025-12-04T09:43:53.8980215Z * [new branch] gh/v0i0/15/head -> origin/gh/v0i0/15/head 2025-12-04T09:43:53.8982039Z * [new branch] gh/v0i0/15/orig -> origin/gh/v0i0/15/orig 2025-12-04T09:43:53.8984914Z * [new branch] gh/v0i0/16/base -> origin/gh/v0i0/16/base 2025-12-04T09:43:53.8986657Z * [new branch] gh/v0i0/16/head -> origin/gh/v0i0/16/head 2025-12-04T09:43:53.8988448Z * [new branch] gh/v0i0/16/orig -> origin/gh/v0i0/16/orig 2025-12-04T09:43:53.8990828Z * [new branch] gh/v0i0/17/base -> origin/gh/v0i0/17/base 2025-12-04T09:43:53.8992576Z * [new branch] gh/v0i0/17/head -> origin/gh/v0i0/17/head 2025-12-04T09:43:53.8994310Z * [new branch] gh/v0i0/17/orig -> origin/gh/v0i0/17/orig 2025-12-04T09:43:53.8996728Z * [new branch] gh/v0i0/18/base -> origin/gh/v0i0/18/base 2025-12-04T09:43:53.8998515Z * [new branch] gh/v0i0/18/head -> origin/gh/v0i0/18/head 2025-12-04T09:43:53.9000220Z * [new branch] gh/v0i0/18/orig -> origin/gh/v0i0/18/orig 2025-12-04T09:43:53.9002697Z * [new branch] gh/v0i0/19/base -> origin/gh/v0i0/19/base 2025-12-04T09:43:53.9004431Z * [new branch] gh/v0i0/19/head -> origin/gh/v0i0/19/head 2025-12-04T09:43:53.9006196Z * [new branch] gh/v0i0/19/orig -> origin/gh/v0i0/19/orig 2025-12-04T09:43:53.9009122Z * [new branch] gh/vishal9-team/1/base -> origin/gh/vishal9-team/1/base 2025-12-04T09:43:53.9010880Z * [new branch] gh/vishal9-team/1/head -> origin/gh/vishal9-team/1/head 2025-12-04T09:43:53.9013122Z * [new branch] gh/vishal9-team/2/base -> origin/gh/vishal9-team/2/base 2025-12-04T09:43:53.9014896Z * [new branch] gh/vishal9-team/2/head -> origin/gh/vishal9-team/2/head 2025-12-04T09:43:53.9016612Z * [new branch] gh/vishal9-team/2/orig -> origin/gh/vishal9-team/2/orig 2025-12-04T09:43:53.9019099Z * [new branch] gh/vishal9-team/3/base -> origin/gh/vishal9-team/3/base 2025-12-04T09:43:53.9020777Z * [new branch] gh/vishal9-team/3/head -> origin/gh/vishal9-team/3/head 2025-12-04T09:43:53.9022636Z * [new branch] gh/vishal9-team/3/orig -> origin/gh/vishal9-team/3/orig 2025-12-04T09:43:53.9024919Z * [new branch] gh/vishal9-team/4/base -> origin/gh/vishal9-team/4/base 2025-12-04T09:43:53.9026658Z * [new branch] gh/vishal9-team/4/head -> origin/gh/vishal9-team/4/head 2025-12-04T09:43:53.9028493Z * [new branch] gh/vishal9-team/4/orig -> origin/gh/vishal9-team/4/orig 2025-12-04T09:43:53.9031063Z * [new branch] gh/vkuzo/1/next -> origin/gh/vkuzo/1/next 2025-12-04T09:43:53.9033419Z * [new branch] gh/vkuzo/2/next -> origin/gh/vkuzo/2/next 2025-12-04T09:43:53.9035817Z * [new branch] gh/vkuzo/3/next -> origin/gh/vkuzo/3/next 2025-12-04T09:43:53.9038699Z * [new branch] gh/wconstab/424/base -> origin/gh/wconstab/424/base 2025-12-04T09:43:53.9040497Z * [new branch] gh/wconstab/424/head -> origin/gh/wconstab/424/head 2025-12-04T09:43:53.9042281Z * [new branch] gh/wconstab/424/orig -> origin/gh/wconstab/424/orig 2025-12-04T09:43:53.9044713Z * [new branch] gh/wconstab/435/base -> origin/gh/wconstab/435/base 2025-12-04T09:43:53.9046528Z * [new branch] gh/wconstab/435/head -> origin/gh/wconstab/435/head 2025-12-04T09:43:53.9048291Z * [new branch] gh/wconstab/435/orig -> origin/gh/wconstab/435/orig 2025-12-04T09:43:53.9050608Z * [new branch] gh/wconstab/444/base -> origin/gh/wconstab/444/base 2025-12-04T09:43:53.9052402Z * [new branch] gh/wconstab/444/head -> origin/gh/wconstab/444/head 2025-12-04T09:43:53.9054235Z * [new branch] gh/wconstab/444/orig -> origin/gh/wconstab/444/orig 2025-12-04T09:43:53.9056725Z * [new branch] gh/wconstab/447/base -> origin/gh/wconstab/447/base 2025-12-04T09:43:53.9058405Z * [new branch] gh/wconstab/447/head -> origin/gh/wconstab/447/head 2025-12-04T09:43:53.9060138Z * [new branch] gh/wconstab/447/orig -> origin/gh/wconstab/447/orig 2025-12-04T09:43:53.9062574Z * [new branch] gh/wconstab/448/base -> origin/gh/wconstab/448/base 2025-12-04T09:43:53.9064940Z * [new branch] gh/wconstab/448/head -> origin/gh/wconstab/448/head 2025-12-04T09:43:53.9066794Z * [new branch] gh/wconstab/448/orig -> origin/gh/wconstab/448/orig 2025-12-04T09:43:53.9069030Z * [new branch] gh/wconstab/449/base -> origin/gh/wconstab/449/base 2025-12-04T09:43:53.9070878Z * [new branch] gh/wconstab/449/head -> origin/gh/wconstab/449/head 2025-12-04T09:43:53.9072642Z * [new branch] gh/wconstab/449/orig -> origin/gh/wconstab/449/orig 2025-12-04T09:43:53.9074837Z * [new branch] gh/wconstab/450/base -> origin/gh/wconstab/450/base 2025-12-04T09:43:53.9076671Z * [new branch] gh/wconstab/450/head -> origin/gh/wconstab/450/head 2025-12-04T09:43:53.9078577Z * [new branch] gh/wconstab/450/orig -> origin/gh/wconstab/450/orig 2025-12-04T09:43:53.9080927Z * [new branch] gh/wconstab/451/base -> origin/gh/wconstab/451/base 2025-12-04T09:43:53.9082801Z * [new branch] gh/wconstab/451/head -> origin/gh/wconstab/451/head 2025-12-04T09:43:53.9084955Z * [new branch] gh/wconstab/451/orig -> origin/gh/wconstab/451/orig 2025-12-04T09:43:53.9087340Z * [new branch] gh/wconstab/452/base -> origin/gh/wconstab/452/base 2025-12-04T09:43:53.9089010Z * [new branch] gh/wconstab/452/head -> origin/gh/wconstab/452/head 2025-12-04T09:43:53.9090887Z * [new branch] gh/wconstab/452/orig -> origin/gh/wconstab/452/orig 2025-12-04T09:43:53.9093033Z * [new branch] gh/wconstab/453/base -> origin/gh/wconstab/453/base 2025-12-04T09:43:53.9094835Z * [new branch] gh/wconstab/453/head -> origin/gh/wconstab/453/head 2025-12-04T09:43:53.9098182Z * [new branch] gh/wconstab/453/orig -> origin/gh/wconstab/453/orig 2025-12-04T09:43:53.9099095Z * [new branch] gh/wconstab/454/base -> origin/gh/wconstab/454/base 2025-12-04T09:43:53.9100574Z * [new branch] gh/wconstab/454/head -> origin/gh/wconstab/454/head 2025-12-04T09:43:53.9102404Z * [new branch] gh/wconstab/454/orig -> origin/gh/wconstab/454/orig 2025-12-04T09:43:53.9104833Z * [new branch] gh/wconstab/455/base -> origin/gh/wconstab/455/base 2025-12-04T09:43:53.9106585Z * [new branch] gh/wconstab/455/head -> origin/gh/wconstab/455/head 2025-12-04T09:43:53.9108321Z * [new branch] gh/wconstab/455/orig -> origin/gh/wconstab/455/orig 2025-12-04T09:43:53.9110845Z * [new branch] gh/wconstab/456/base -> origin/gh/wconstab/456/base 2025-12-04T09:43:53.9112955Z * [new branch] gh/wconstab/456/head -> origin/gh/wconstab/456/head 2025-12-04T09:43:53.9114775Z * [new branch] gh/wconstab/456/orig -> origin/gh/wconstab/456/orig 2025-12-04T09:43:53.9117124Z * [new branch] gh/wconstab/457/base -> origin/gh/wconstab/457/base 2025-12-04T09:43:53.9118973Z * [new branch] gh/wconstab/457/head -> origin/gh/wconstab/457/head 2025-12-04T09:43:53.9120632Z * [new branch] gh/wconstab/457/orig -> origin/gh/wconstab/457/orig 2025-12-04T09:43:53.9123048Z * [new branch] gh/wconstab/458/base -> origin/gh/wconstab/458/base 2025-12-04T09:43:53.9124821Z * [new branch] gh/wconstab/458/head -> origin/gh/wconstab/458/head 2025-12-04T09:43:53.9126606Z * [new branch] gh/wconstab/458/orig -> origin/gh/wconstab/458/orig 2025-12-04T09:43:53.9128912Z * [new branch] gh/wconstab/459/base -> origin/gh/wconstab/459/base 2025-12-04T09:43:53.9130737Z * [new branch] gh/wconstab/459/head -> origin/gh/wconstab/459/head 2025-12-04T09:43:53.9132405Z * [new branch] gh/wconstab/459/orig -> origin/gh/wconstab/459/orig 2025-12-04T09:43:53.9135803Z * [new branch] gh/wconstab/460/base -> origin/gh/wconstab/460/base 2025-12-04T09:43:53.9137717Z * [new branch] gh/wconstab/460/head -> origin/gh/wconstab/460/head 2025-12-04T09:43:53.9139561Z * [new branch] gh/wconstab/460/orig -> origin/gh/wconstab/460/orig 2025-12-04T09:43:53.9142080Z * [new branch] gh/wconstab/461/base -> origin/gh/wconstab/461/base 2025-12-04T09:43:53.9143974Z * [new branch] gh/wconstab/461/head -> origin/gh/wconstab/461/head 2025-12-04T09:43:53.9145760Z * [new branch] gh/wconstab/461/orig -> origin/gh/wconstab/461/orig 2025-12-04T09:43:53.9148578Z * [new branch] gh/wconstab/462/base -> origin/gh/wconstab/462/base 2025-12-04T09:43:53.9150485Z * [new branch] gh/wconstab/462/head -> origin/gh/wconstab/462/head 2025-12-04T09:43:53.9152325Z * [new branch] gh/wconstab/462/orig -> origin/gh/wconstab/462/orig 2025-12-04T09:43:53.9154884Z * [new branch] gh/wconstab/463/base -> origin/gh/wconstab/463/base 2025-12-04T09:43:53.9156733Z * [new branch] gh/wconstab/463/head -> origin/gh/wconstab/463/head 2025-12-04T09:43:53.9158489Z * [new branch] gh/wconstab/463/orig -> origin/gh/wconstab/463/orig 2025-12-04T09:43:53.9160848Z * [new branch] gh/wconstab/464/base -> origin/gh/wconstab/464/base 2025-12-04T09:43:53.9162755Z * [new branch] gh/wconstab/464/head -> origin/gh/wconstab/464/head 2025-12-04T09:43:53.9164435Z * [new branch] gh/wconstab/464/orig -> origin/gh/wconstab/464/orig 2025-12-04T09:43:53.9166803Z * [new branch] gh/wconstab/465/base -> origin/gh/wconstab/465/base 2025-12-04T09:43:53.9168665Z * [new branch] gh/wconstab/465/head -> origin/gh/wconstab/465/head 2025-12-04T09:43:53.9170367Z * [new branch] gh/wconstab/465/orig -> origin/gh/wconstab/465/orig 2025-12-04T09:43:53.9172838Z * [new branch] gh/wconstab/466/base -> origin/gh/wconstab/466/base 2025-12-04T09:43:53.9174680Z * [new branch] gh/wconstab/466/head -> origin/gh/wconstab/466/head 2025-12-04T09:43:53.9176330Z * [new branch] gh/wconstab/466/orig -> origin/gh/wconstab/466/orig 2025-12-04T09:43:53.9180669Z * [new branch] gh/wconstab/467/base -> origin/gh/wconstab/467/base 2025-12-04T09:43:53.9182495Z * [new branch] gh/wconstab/467/head -> origin/gh/wconstab/467/head 2025-12-04T09:43:53.9184424Z * [new branch] gh/wconstab/467/orig -> origin/gh/wconstab/467/orig 2025-12-04T09:43:53.9186702Z * [new branch] gh/wconstab/468/base -> origin/gh/wconstab/468/base 2025-12-04T09:43:53.9188419Z * [new branch] gh/wconstab/468/head -> origin/gh/wconstab/468/head 2025-12-04T09:43:53.9190539Z * [new branch] gh/wconstab/468/orig -> origin/gh/wconstab/468/orig 2025-12-04T09:43:53.9193521Z * [new branch] gh/weifengpy/39/base -> origin/gh/weifengpy/39/base 2025-12-04T09:43:53.9195167Z * [new branch] gh/weifengpy/39/head -> origin/gh/weifengpy/39/head 2025-12-04T09:43:53.9197017Z * [new branch] gh/weifengpy/39/orig -> origin/gh/weifengpy/39/orig 2025-12-04T09:43:53.9199474Z * [new branch] gh/weifengpy/40/base -> origin/gh/weifengpy/40/base 2025-12-04T09:43:53.9201200Z * [new branch] gh/weifengpy/40/head -> origin/gh/weifengpy/40/head 2025-12-04T09:43:53.9202944Z * [new branch] gh/weifengpy/40/orig -> origin/gh/weifengpy/40/orig 2025-12-04T09:43:53.9205412Z * [new branch] gh/weifengpy/41/base -> origin/gh/weifengpy/41/base 2025-12-04T09:43:53.9207200Z * [new branch] gh/weifengpy/41/head -> origin/gh/weifengpy/41/head 2025-12-04T09:43:53.9209095Z * [new branch] gh/weifengpy/41/orig -> origin/gh/weifengpy/41/orig 2025-12-04T09:43:53.9211965Z * [new branch] gh/williamwen42/250/base -> origin/gh/williamwen42/250/base 2025-12-04T09:43:53.9213761Z * [new branch] gh/williamwen42/250/head -> origin/gh/williamwen42/250/head 2025-12-04T09:43:53.9215553Z * [new branch] gh/williamwen42/250/orig -> origin/gh/williamwen42/250/orig 2025-12-04T09:43:53.9217974Z * [new branch] gh/williamwen42/279/base -> origin/gh/williamwen42/279/base 2025-12-04T09:43:53.9219824Z * [new branch] gh/williamwen42/279/head -> origin/gh/williamwen42/279/head 2025-12-04T09:43:53.9221700Z * [new branch] gh/williamwen42/279/orig -> origin/gh/williamwen42/279/orig 2025-12-04T09:43:53.9224082Z * [new branch] gh/williamwen42/282/base -> origin/gh/williamwen42/282/base 2025-12-04T09:43:53.9225946Z * [new branch] gh/williamwen42/282/head -> origin/gh/williamwen42/282/head 2025-12-04T09:43:53.9227734Z * [new branch] gh/williamwen42/282/orig -> origin/gh/williamwen42/282/orig 2025-12-04T09:43:53.9230083Z * [new branch] gh/williamwen42/287/base -> origin/gh/williamwen42/287/base 2025-12-04T09:43:53.9231835Z * [new branch] gh/williamwen42/287/head -> origin/gh/williamwen42/287/head 2025-12-04T09:43:53.9233594Z * [new branch] gh/williamwen42/287/orig -> origin/gh/williamwen42/287/orig 2025-12-04T09:43:53.9236110Z * [new branch] gh/williamwen42/288/base -> origin/gh/williamwen42/288/base 2025-12-04T09:43:53.9237766Z * [new branch] gh/williamwen42/288/head -> origin/gh/williamwen42/288/head 2025-12-04T09:43:53.9239478Z * [new branch] gh/williamwen42/288/orig -> origin/gh/williamwen42/288/orig 2025-12-04T09:43:53.9242005Z * [new branch] gh/williamwen42/296/base -> origin/gh/williamwen42/296/base 2025-12-04T09:43:53.9243905Z * [new branch] gh/williamwen42/296/head -> origin/gh/williamwen42/296/head 2025-12-04T09:43:53.9245748Z * [new branch] gh/williamwen42/296/orig -> origin/gh/williamwen42/296/orig 2025-12-04T09:43:53.9247975Z * [new branch] gh/williamwen42/297/base -> origin/gh/williamwen42/297/base 2025-12-04T09:43:53.9249801Z * [new branch] gh/williamwen42/297/head -> origin/gh/williamwen42/297/head 2025-12-04T09:43:53.9251569Z * [new branch] gh/williamwen42/297/orig -> origin/gh/williamwen42/297/orig 2025-12-04T09:43:53.9254387Z * [new branch] gh/williamwen42/306/base -> origin/gh/williamwen42/306/base 2025-12-04T09:43:53.9256172Z * [new branch] gh/williamwen42/306/head -> origin/gh/williamwen42/306/head 2025-12-04T09:43:53.9257936Z * [new branch] gh/williamwen42/306/orig -> origin/gh/williamwen42/306/orig 2025-12-04T09:43:53.9260353Z * [new branch] gh/williamwen42/309/base -> origin/gh/williamwen42/309/base 2025-12-04T09:43:53.9262147Z * [new branch] gh/williamwen42/309/head -> origin/gh/williamwen42/309/head 2025-12-04T09:43:53.9264111Z * [new branch] gh/williamwen42/309/orig -> origin/gh/williamwen42/309/orig 2025-12-04T09:43:53.9266430Z * [new branch] gh/williamwen42/310/base -> origin/gh/williamwen42/310/base 2025-12-04T09:43:53.9268140Z * [new branch] gh/williamwen42/310/head -> origin/gh/williamwen42/310/head 2025-12-04T09:43:53.9270070Z * [new branch] gh/williamwen42/310/orig -> origin/gh/williamwen42/310/orig 2025-12-04T09:43:53.9273499Z * [new branch] gh/williamwen42/311/base -> origin/gh/williamwen42/311/base 2025-12-04T09:43:53.9275287Z * [new branch] gh/williamwen42/311/head -> origin/gh/williamwen42/311/head 2025-12-04T09:43:53.9277280Z * [new branch] gh/williamwen42/311/orig -> origin/gh/williamwen42/311/orig 2025-12-04T09:43:53.9279733Z * [new branch] gh/williamwen42/319/base -> origin/gh/williamwen42/319/base 2025-12-04T09:43:53.9281472Z * [new branch] gh/williamwen42/319/head -> origin/gh/williamwen42/319/head 2025-12-04T09:43:53.9283158Z * [new branch] gh/williamwen42/319/orig -> origin/gh/williamwen42/319/orig 2025-12-04T09:43:53.9285537Z * [new branch] gh/williamwen42/325/base -> origin/gh/williamwen42/325/base 2025-12-04T09:43:53.9287368Z * [new branch] gh/williamwen42/325/head -> origin/gh/williamwen42/325/head 2025-12-04T09:43:53.9289193Z * [new branch] gh/williamwen42/325/orig -> origin/gh/williamwen42/325/orig 2025-12-04T09:43:53.9291442Z * [new branch] gh/williamwen42/326/base -> origin/gh/williamwen42/326/base 2025-12-04T09:43:53.9293216Z * [new branch] gh/williamwen42/326/head -> origin/gh/williamwen42/326/head 2025-12-04T09:43:53.9295010Z * [new branch] gh/williamwen42/326/orig -> origin/gh/williamwen42/326/orig 2025-12-04T09:43:53.9297462Z * [new branch] gh/williamwen42/327/base -> origin/gh/williamwen42/327/base 2025-12-04T09:43:53.9299153Z * [new branch] gh/williamwen42/327/head -> origin/gh/williamwen42/327/head 2025-12-04T09:43:53.9300968Z * [new branch] gh/williamwen42/327/orig -> origin/gh/williamwen42/327/orig 2025-12-04T09:43:53.9303402Z * [new branch] gh/williamwen42/328/base -> origin/gh/williamwen42/328/base 2025-12-04T09:43:53.9305553Z * [new branch] gh/williamwen42/328/head -> origin/gh/williamwen42/328/head 2025-12-04T09:43:53.9307138Z * [new branch] gh/williamwen42/328/orig -> origin/gh/williamwen42/328/orig 2025-12-04T09:43:53.9309997Z * [new branch] gh/williamwen42/329/base -> origin/gh/williamwen42/329/base 2025-12-04T09:43:53.9311787Z * [new branch] gh/williamwen42/329/head -> origin/gh/williamwen42/329/head 2025-12-04T09:43:53.9313582Z * [new branch] gh/williamwen42/329/orig -> origin/gh/williamwen42/329/orig 2025-12-04T09:43:53.9315958Z * [new branch] gh/williamwen42/330/base -> origin/gh/williamwen42/330/base 2025-12-04T09:43:53.9317754Z * [new branch] gh/williamwen42/330/head -> origin/gh/williamwen42/330/head 2025-12-04T09:43:53.9319485Z * [new branch] gh/williamwen42/330/orig -> origin/gh/williamwen42/330/orig 2025-12-04T09:43:53.9321856Z * [new branch] gh/williamwen42/331/base -> origin/gh/williamwen42/331/base 2025-12-04T09:43:53.9323607Z * [new branch] gh/williamwen42/331/head -> origin/gh/williamwen42/331/head 2025-12-04T09:43:53.9325389Z * [new branch] gh/williamwen42/331/orig -> origin/gh/williamwen42/331/orig 2025-12-04T09:43:53.9327667Z * [new branch] gh/williamwen42/332/base -> origin/gh/williamwen42/332/base 2025-12-04T09:43:53.9329404Z * [new branch] gh/williamwen42/332/head -> origin/gh/williamwen42/332/head 2025-12-04T09:43:53.9331139Z * [new branch] gh/williamwen42/332/orig -> origin/gh/williamwen42/332/orig 2025-12-04T09:43:53.9333763Z * [new branch] gh/williamwen42/333/base -> origin/gh/williamwen42/333/base 2025-12-04T09:43:53.9335457Z * [new branch] gh/williamwen42/333/head -> origin/gh/williamwen42/333/head 2025-12-04T09:43:53.9337217Z * [new branch] gh/williamwen42/333/orig -> origin/gh/williamwen42/333/orig 2025-12-04T09:43:53.9339622Z * [new branch] gh/williamwen42/334/base -> origin/gh/williamwen42/334/base 2025-12-04T09:43:53.9341392Z * [new branch] gh/williamwen42/334/head -> origin/gh/williamwen42/334/head 2025-12-04T09:43:53.9343249Z * [new branch] gh/williamwen42/334/orig -> origin/gh/williamwen42/334/orig 2025-12-04T09:43:53.9349006Z * [new branch] gh/williamwen42/335/base -> origin/gh/williamwen42/335/base 2025-12-04T09:43:53.9350705Z * [new branch] gh/williamwen42/335/head -> origin/gh/williamwen42/335/head 2025-12-04T09:43:53.9352670Z * [new branch] gh/williamwen42/335/orig -> origin/gh/williamwen42/335/orig 2025-12-04T09:43:53.9355086Z * [new branch] gh/williamwen42/336/base -> origin/gh/williamwen42/336/base 2025-12-04T09:43:53.9356759Z * [new branch] gh/williamwen42/336/head -> origin/gh/williamwen42/336/head 2025-12-04T09:43:53.9358460Z * [new branch] gh/williamwen42/336/orig -> origin/gh/williamwen42/336/orig 2025-12-04T09:43:53.9360763Z * [new branch] gh/williamwen42/337/base -> origin/gh/williamwen42/337/base 2025-12-04T09:43:53.9362628Z * [new branch] gh/williamwen42/337/head -> origin/gh/williamwen42/337/head 2025-12-04T09:43:53.9364340Z * [new branch] gh/williamwen42/337/orig -> origin/gh/williamwen42/337/orig 2025-12-04T09:43:53.9366764Z * [new branch] gh/williamwen42/338/base -> origin/gh/williamwen42/338/base 2025-12-04T09:43:53.9368507Z * [new branch] gh/williamwen42/338/head -> origin/gh/williamwen42/338/head 2025-12-04T09:43:53.9370338Z * [new branch] gh/williamwen42/338/orig -> origin/gh/williamwen42/338/orig 2025-12-04T09:43:53.9372618Z * [new branch] gh/williamwen42/339/base -> origin/gh/williamwen42/339/base 2025-12-04T09:43:53.9374467Z * [new branch] gh/williamwen42/339/head -> origin/gh/williamwen42/339/head 2025-12-04T09:43:53.9376158Z * [new branch] gh/williamwen42/339/orig -> origin/gh/williamwen42/339/orig 2025-12-04T09:43:53.9379574Z * [new branch] gh/williamwen42/340/base -> origin/gh/williamwen42/340/base 2025-12-04T09:43:53.9381326Z * [new branch] gh/williamwen42/340/head -> origin/gh/williamwen42/340/head 2025-12-04T09:43:53.9383011Z * [new branch] gh/williamwen42/340/orig -> origin/gh/williamwen42/340/orig 2025-12-04T09:43:53.9385711Z * [new branch] gh/williamwen42/341/base -> origin/gh/williamwen42/341/base 2025-12-04T09:43:53.9387475Z * [new branch] gh/williamwen42/341/head -> origin/gh/williamwen42/341/head 2025-12-04T09:43:53.9389203Z * [new branch] gh/williamwen42/341/orig -> origin/gh/williamwen42/341/orig 2025-12-04T09:43:53.9391521Z * [new branch] gh/williamwen42/342/base -> origin/gh/williamwen42/342/base 2025-12-04T09:43:53.9393288Z * [new branch] gh/williamwen42/342/head -> origin/gh/williamwen42/342/head 2025-12-04T09:43:53.9395044Z * [new branch] gh/williamwen42/342/orig -> origin/gh/williamwen42/342/orig 2025-12-04T09:43:53.9397453Z * [new branch] gh/williamwen42/343/base -> origin/gh/williamwen42/343/base 2025-12-04T09:43:53.9399310Z * [new branch] gh/williamwen42/343/head -> origin/gh/williamwen42/343/head 2025-12-04T09:43:53.9401150Z * [new branch] gh/williamwen42/343/orig -> origin/gh/williamwen42/343/orig 2025-12-04T09:43:53.9403600Z * [new branch] gh/williamwen42/344/base -> origin/gh/williamwen42/344/base 2025-12-04T09:43:53.9405312Z * [new branch] gh/williamwen42/344/head -> origin/gh/williamwen42/344/head 2025-12-04T09:43:53.9407046Z * [new branch] gh/williamwen42/344/orig -> origin/gh/williamwen42/344/orig 2025-12-04T09:43:53.9409513Z * [new branch] gh/williamwen42/345/base -> origin/gh/williamwen42/345/base 2025-12-04T09:43:53.9411729Z * [new branch] gh/williamwen42/345/head -> origin/gh/williamwen42/345/head 2025-12-04T09:43:53.9413512Z * [new branch] gh/williamwen42/345/orig -> origin/gh/williamwen42/345/orig 2025-12-04T09:43:53.9415940Z * [new branch] gh/williamwen42/346/base -> origin/gh/williamwen42/346/base 2025-12-04T09:43:53.9417747Z * [new branch] gh/williamwen42/346/head -> origin/gh/williamwen42/346/head 2025-12-04T09:43:53.9419542Z * [new branch] gh/williamwen42/346/orig -> origin/gh/williamwen42/346/orig 2025-12-04T09:43:53.9421971Z * [new branch] gh/williamwen42/347/base -> origin/gh/williamwen42/347/base 2025-12-04T09:43:53.9423745Z * [new branch] gh/williamwen42/347/head -> origin/gh/williamwen42/347/head 2025-12-04T09:43:53.9425682Z * [new branch] gh/williamwen42/347/orig -> origin/gh/williamwen42/347/orig 2025-12-04T09:43:53.9428029Z * [new branch] gh/williamwen42/348/base -> origin/gh/williamwen42/348/base 2025-12-04T09:43:53.9429656Z * [new branch] gh/williamwen42/348/head -> origin/gh/williamwen42/348/head 2025-12-04T09:43:53.9431468Z * [new branch] gh/williamwen42/348/orig -> origin/gh/williamwen42/348/orig 2025-12-04T09:43:53.9434111Z * [new branch] gh/williamwen42/349/base -> origin/gh/williamwen42/349/base 2025-12-04T09:43:53.9435891Z * [new branch] gh/williamwen42/349/head -> origin/gh/williamwen42/349/head 2025-12-04T09:43:53.9437617Z * [new branch] gh/williamwen42/349/orig -> origin/gh/williamwen42/349/orig 2025-12-04T09:43:53.9439966Z * [new branch] gh/williamwen42/350/base -> origin/gh/williamwen42/350/base 2025-12-04T09:43:53.9441766Z * [new branch] gh/williamwen42/350/head -> origin/gh/williamwen42/350/head 2025-12-04T09:43:53.9443602Z * [new branch] gh/williamwen42/350/orig -> origin/gh/williamwen42/350/orig 2025-12-04T09:43:53.9445951Z * [new branch] gh/williamwen42/351/base -> origin/gh/williamwen42/351/base 2025-12-04T09:43:53.9447795Z * [new branch] gh/williamwen42/351/head -> origin/gh/williamwen42/351/head 2025-12-04T09:43:53.9449692Z * [new branch] gh/williamwen42/351/orig -> origin/gh/williamwen42/351/orig 2025-12-04T09:43:53.9452046Z * [new branch] gh/williamwen42/352/base -> origin/gh/williamwen42/352/base 2025-12-04T09:43:53.9453801Z * [new branch] gh/williamwen42/352/head -> origin/gh/williamwen42/352/head 2025-12-04T09:43:53.9455511Z * [new branch] gh/williamwen42/352/orig -> origin/gh/williamwen42/352/orig 2025-12-04T09:43:53.9458058Z * [new branch] gh/williamwen42/353/base -> origin/gh/williamwen42/353/base 2025-12-04T09:43:53.9459863Z * [new branch] gh/williamwen42/353/head -> origin/gh/williamwen42/353/head 2025-12-04T09:43:53.9461753Z * [new branch] gh/williamwen42/353/orig -> origin/gh/williamwen42/353/orig 2025-12-04T09:43:53.9464155Z * [new branch] gh/williamwen42/354/base -> origin/gh/williamwen42/354/base 2025-12-04T09:43:53.9466056Z * [new branch] gh/williamwen42/354/head -> origin/gh/williamwen42/354/head 2025-12-04T09:43:53.9467740Z * [new branch] gh/williamwen42/354/orig -> origin/gh/williamwen42/354/orig 2025-12-04T09:43:53.9470142Z * [new branch] gh/williamwen42/355/base -> origin/gh/williamwen42/355/base 2025-12-04T09:43:53.9471889Z * [new branch] gh/williamwen42/355/head -> origin/gh/williamwen42/355/head 2025-12-04T09:43:53.9473720Z * [new branch] gh/williamwen42/355/orig -> origin/gh/williamwen42/355/orig 2025-12-04T09:43:53.9476610Z * [new branch] gh/williamwen42/356/base -> origin/gh/williamwen42/356/base 2025-12-04T09:43:53.9478627Z * [new branch] gh/williamwen42/356/head -> origin/gh/williamwen42/356/head 2025-12-04T09:43:53.9480356Z * [new branch] gh/williamwen42/356/orig -> origin/gh/williamwen42/356/orig 2025-12-04T09:43:53.9482836Z * [new branch] gh/williamwen42/357/base -> origin/gh/williamwen42/357/base 2025-12-04T09:43:53.9484592Z * [new branch] gh/williamwen42/357/head -> origin/gh/williamwen42/357/head 2025-12-04T09:43:53.9486297Z * [new branch] gh/williamwen42/357/orig -> origin/gh/williamwen42/357/orig 2025-12-04T09:43:53.9488833Z * [new branch] gh/williamwen42/358/base -> origin/gh/williamwen42/358/base 2025-12-04T09:43:53.9490561Z * [new branch] gh/williamwen42/358/head -> origin/gh/williamwen42/358/head 2025-12-04T09:43:53.9492482Z * [new branch] gh/williamwen42/358/orig -> origin/gh/williamwen42/358/orig 2025-12-04T09:43:53.9495324Z * [new branch] gh/xmfan/169/base -> origin/gh/xmfan/169/base 2025-12-04T09:43:53.9497172Z * [new branch] gh/xmfan/169/head -> origin/gh/xmfan/169/head 2025-12-04T09:43:53.9499428Z * [new branch] gh/xmfan/170/base -> origin/gh/xmfan/170/base 2025-12-04T09:43:53.9501030Z * [new branch] gh/xmfan/170/head -> origin/gh/xmfan/170/head 2025-12-04T09:43:53.9503326Z * [new branch] gh/xmfan/274/base -> origin/gh/xmfan/274/base 2025-12-04T09:43:53.9505240Z * [new branch] gh/xmfan/274/head -> origin/gh/xmfan/274/head 2025-12-04T09:43:53.9506964Z * [new branch] gh/xmfan/274/orig -> origin/gh/xmfan/274/orig 2025-12-04T09:43:53.9509270Z * [new branch] gh/xmfan/277/base -> origin/gh/xmfan/277/base 2025-12-04T09:43:53.9510966Z * [new branch] gh/xmfan/277/head -> origin/gh/xmfan/277/head 2025-12-04T09:43:53.9512787Z * [new branch] gh/xmfan/277/orig -> origin/gh/xmfan/277/orig 2025-12-04T09:43:53.9515248Z * [new branch] gh/xmfan/301/base -> origin/gh/xmfan/301/base 2025-12-04T09:43:53.9516838Z * [new branch] gh/xmfan/301/head -> origin/gh/xmfan/301/head 2025-12-04T09:43:53.9518597Z * [new branch] gh/xmfan/301/orig -> origin/gh/xmfan/301/orig 2025-12-04T09:43:53.9520896Z * [new branch] gh/xmfan/304/base -> origin/gh/xmfan/304/base 2025-12-04T09:43:53.9522657Z * [new branch] gh/xmfan/304/head -> origin/gh/xmfan/304/head 2025-12-04T09:43:53.9524384Z * [new branch] gh/xmfan/304/orig -> origin/gh/xmfan/304/orig 2025-12-04T09:43:53.9526671Z * [new branch] gh/xmfan/309/base -> origin/gh/xmfan/309/base 2025-12-04T09:43:53.9528384Z * [new branch] gh/xmfan/309/head -> origin/gh/xmfan/309/head 2025-12-04T09:43:53.9530115Z * [new branch] gh/xmfan/309/orig -> origin/gh/xmfan/309/orig 2025-12-04T09:43:53.9532892Z * [new branch] gh/xmfan/310/base -> origin/gh/xmfan/310/base 2025-12-04T09:43:53.9534598Z * [new branch] gh/xmfan/310/head -> origin/gh/xmfan/310/head 2025-12-04T09:43:53.9536308Z * [new branch] gh/xmfan/310/orig -> origin/gh/xmfan/310/orig 2025-12-04T09:43:53.9538642Z * [new branch] gh/xmfan/311/base -> origin/gh/xmfan/311/base 2025-12-04T09:43:53.9540296Z * [new branch] gh/xmfan/311/head -> origin/gh/xmfan/311/head 2025-12-04T09:43:53.9542010Z * [new branch] gh/xmfan/311/orig -> origin/gh/xmfan/311/orig 2025-12-04T09:43:53.9545045Z * [new branch] gh/xmfan/312/base -> origin/gh/xmfan/312/base 2025-12-04T09:43:53.9546759Z * [new branch] gh/xmfan/312/head -> origin/gh/xmfan/312/head 2025-12-04T09:43:53.9548454Z * [new branch] gh/xmfan/312/orig -> origin/gh/xmfan/312/orig 2025-12-04T09:43:53.9550865Z * [new branch] gh/xmfan/313/base -> origin/gh/xmfan/313/base 2025-12-04T09:43:53.9552568Z * [new branch] gh/xmfan/313/head -> origin/gh/xmfan/313/head 2025-12-04T09:43:53.9554232Z * [new branch] gh/xmfan/313/orig -> origin/gh/xmfan/313/orig 2025-12-04T09:43:53.9557082Z * [new branch] gh/xuanzhang816/27/base -> origin/gh/xuanzhang816/27/base 2025-12-04T09:43:53.9558807Z * [new branch] gh/xuanzhang816/27/head -> origin/gh/xuanzhang816/27/head 2025-12-04T09:43:53.9560605Z * [new branch] gh/xuanzhang816/27/orig -> origin/gh/xuanzhang816/27/orig 2025-12-04T09:43:53.9563008Z * [new branch] gh/xuanzhang816/32/base -> origin/gh/xuanzhang816/32/base 2025-12-04T09:43:53.9564733Z * [new branch] gh/xuanzhang816/32/head -> origin/gh/xuanzhang816/32/head 2025-12-04T09:43:53.9566464Z * [new branch] gh/xuanzhang816/32/orig -> origin/gh/xuanzhang816/32/orig 2025-12-04T09:43:53.9568892Z * [new branch] gh/xuanzhang816/33/base -> origin/gh/xuanzhang816/33/base 2025-12-04T09:43:53.9571024Z * [new branch] gh/xuanzhang816/33/head -> origin/gh/xuanzhang816/33/head 2025-12-04T09:43:53.9572750Z * [new branch] gh/xuanzhang816/33/orig -> origin/gh/xuanzhang816/33/orig 2025-12-04T09:43:53.9575327Z * [new branch] gh/xuanzhang816/34/base -> origin/gh/xuanzhang816/34/base 2025-12-04T09:43:53.9577080Z * [new branch] gh/xuanzhang816/34/head -> origin/gh/xuanzhang816/34/head 2025-12-04T09:43:53.9579163Z * [new branch] gh/xuanzhang816/34/orig -> origin/gh/xuanzhang816/34/orig 2025-12-04T09:43:53.9581637Z * [new branch] gh/xuanzhang816/35/base -> origin/gh/xuanzhang816/35/base 2025-12-04T09:43:53.9583370Z * [new branch] gh/xuanzhang816/35/head -> origin/gh/xuanzhang816/35/head 2025-12-04T09:43:53.9585415Z * [new branch] gh/xuanzhang816/35/orig -> origin/gh/xuanzhang816/35/orig 2025-12-04T09:43:53.9588137Z * [new branch] gh/yanbing-j/11/base -> origin/gh/yanbing-j/11/base 2025-12-04T09:43:53.9589925Z * [new branch] gh/yanbing-j/11/head -> origin/gh/yanbing-j/11/head 2025-12-04T09:43:53.9591745Z * [new branch] gh/yanbing-j/11/orig -> origin/gh/yanbing-j/11/orig 2025-12-04T09:43:53.9594078Z * [new branch] gh/yanbing-j/12/base -> origin/gh/yanbing-j/12/base 2025-12-04T09:43:53.9595810Z * [new branch] gh/yanbing-j/12/head -> origin/gh/yanbing-j/12/head 2025-12-04T09:43:53.9597480Z * [new branch] gh/yanbing-j/12/orig -> origin/gh/yanbing-j/12/orig 2025-12-04T09:43:53.9599853Z * [new branch] gh/yanbing-j/13/base -> origin/gh/yanbing-j/13/base 2025-12-04T09:43:53.9601617Z * [new branch] gh/yanbing-j/13/head -> origin/gh/yanbing-j/13/head 2025-12-04T09:43:53.9603348Z * [new branch] gh/yanbing-j/13/orig -> origin/gh/yanbing-j/13/orig 2025-12-04T09:43:53.9605662Z * [new branch] gh/yanbing-j/14/base -> origin/gh/yanbing-j/14/base 2025-12-04T09:43:53.9607422Z * [new branch] gh/yanbing-j/14/head -> origin/gh/yanbing-j/14/head 2025-12-04T09:43:53.9609185Z * [new branch] gh/yanbing-j/14/orig -> origin/gh/yanbing-j/14/orig 2025-12-04T09:43:53.9611376Z * [new branch] gh/yanbing-j/15/base -> origin/gh/yanbing-j/15/base 2025-12-04T09:43:53.9613194Z * [new branch] gh/yanbing-j/15/head -> origin/gh/yanbing-j/15/head 2025-12-04T09:43:53.9614931Z * [new branch] gh/yanbing-j/15/orig -> origin/gh/yanbing-j/15/orig 2025-12-04T09:43:53.9617177Z * [new branch] gh/yanbing-j/18/base -> origin/gh/yanbing-j/18/base 2025-12-04T09:43:53.9618889Z * [new branch] gh/yanbing-j/18/head -> origin/gh/yanbing-j/18/head 2025-12-04T09:43:53.9620633Z * [new branch] gh/yanbing-j/18/orig -> origin/gh/yanbing-j/18/orig 2025-12-04T09:43:53.9623007Z * [new branch] gh/yanbing-j/19/base -> origin/gh/yanbing-j/19/base 2025-12-04T09:43:53.9624888Z * [new branch] gh/yanbing-j/19/head -> origin/gh/yanbing-j/19/head 2025-12-04T09:43:53.9626565Z * [new branch] gh/yanbing-j/19/orig -> origin/gh/yanbing-j/19/orig 2025-12-04T09:43:53.9628870Z * [new branch] gh/yanbing-j/20/base -> origin/gh/yanbing-j/20/base 2025-12-04T09:43:53.9630655Z * [new branch] gh/yanbing-j/20/head -> origin/gh/yanbing-j/20/head 2025-12-04T09:43:53.9632444Z * [new branch] gh/yanbing-j/20/orig -> origin/gh/yanbing-j/20/orig 2025-12-04T09:43:53.9634764Z * [new branch] gh/yanbing-j/21/base -> origin/gh/yanbing-j/21/base 2025-12-04T09:43:53.9636636Z * [new branch] gh/yanbing-j/21/head -> origin/gh/yanbing-j/21/head 2025-12-04T09:43:53.9639016Z * [new branch] gh/yanbing-j/22/base -> origin/gh/yanbing-j/22/base 2025-12-04T09:43:53.9640757Z * [new branch] gh/yanbing-j/22/head -> origin/gh/yanbing-j/22/head 2025-12-04T09:43:53.9642505Z * [new branch] gh/yanbing-j/22/orig -> origin/gh/yanbing-j/22/orig 2025-12-04T09:43:53.9644836Z * [new branch] gh/yanbing-j/23/base -> origin/gh/yanbing-j/23/base 2025-12-04T09:43:53.9646609Z * [new branch] gh/yanbing-j/23/head -> origin/gh/yanbing-j/23/head 2025-12-04T09:43:53.9648273Z * [new branch] gh/yanbing-j/23/orig -> origin/gh/yanbing-j/23/orig 2025-12-04T09:43:53.9650690Z * [new branch] gh/yanbing-j/24/base -> origin/gh/yanbing-j/24/base 2025-12-04T09:43:53.9652310Z * [new branch] gh/yanbing-j/24/head -> origin/gh/yanbing-j/24/head 2025-12-04T09:43:53.9654213Z * [new branch] gh/yanbing-j/24/orig -> origin/gh/yanbing-j/24/orig 2025-12-04T09:43:53.9656386Z * [new branch] gh/yanbing-j/25/base -> origin/gh/yanbing-j/25/base 2025-12-04T09:43:53.9658096Z * [new branch] gh/yanbing-j/25/head -> origin/gh/yanbing-j/25/head 2025-12-04T09:43:53.9660220Z * [new branch] gh/yanbing-j/25/orig -> origin/gh/yanbing-j/25/orig 2025-12-04T09:43:53.9662614Z * [new branch] gh/yanbing-j/26/base -> origin/gh/yanbing-j/26/base 2025-12-04T09:43:53.9664475Z * [new branch] gh/yanbing-j/26/head -> origin/gh/yanbing-j/26/head 2025-12-04T09:43:53.9666168Z * [new branch] gh/yanbing-j/26/orig -> origin/gh/yanbing-j/26/orig 2025-12-04T09:43:53.9669085Z * [new branch] gh/yang-yu-hang/1/base -> origin/gh/yang-yu-hang/1/base 2025-12-04T09:43:53.9671048Z * [new branch] gh/yang-yu-hang/1/head -> origin/gh/yang-yu-hang/1/head 2025-12-04T09:43:53.9672910Z * [new branch] gh/yang-yu-hang/1/orig -> origin/gh/yang-yu-hang/1/orig 2025-12-04T09:43:53.9675239Z * [new branch] gh/yang-yu-hang/2/base -> origin/gh/yang-yu-hang/2/base 2025-12-04T09:43:53.9677185Z * [new branch] gh/yang-yu-hang/2/head -> origin/gh/yang-yu-hang/2/head 2025-12-04T09:43:53.9679346Z * [new branch] gh/yang-yu-hang/2/orig -> origin/gh/yang-yu-hang/2/orig 2025-12-04T09:43:53.9681571Z * [new branch] gh/yang-yu-hang/3/base -> origin/gh/yang-yu-hang/3/base 2025-12-04T09:43:53.9683409Z * [new branch] gh/yang-yu-hang/3/head -> origin/gh/yang-yu-hang/3/head 2025-12-04T09:43:53.9685228Z * [new branch] gh/yang-yu-hang/3/orig -> origin/gh/yang-yu-hang/3/orig 2025-12-04T09:43:53.9688009Z * [new branch] gh/yangw-dev/12/base -> origin/gh/yangw-dev/12/base 2025-12-04T09:43:53.9689734Z * [new branch] gh/yangw-dev/12/head -> origin/gh/yangw-dev/12/head 2025-12-04T09:43:53.9691477Z * [new branch] gh/yangw-dev/12/orig -> origin/gh/yangw-dev/12/orig 2025-12-04T09:43:53.9694025Z * [new branch] gh/yangw-dev/13/base -> origin/gh/yangw-dev/13/base 2025-12-04T09:43:53.9695868Z * [new branch] gh/yangw-dev/13/head -> origin/gh/yangw-dev/13/head 2025-12-04T09:43:53.9698008Z * [new branch] gh/yangw-dev/13/orig -> origin/gh/yangw-dev/13/orig 2025-12-04T09:43:53.9700302Z * [new branch] gh/yangw-dev/14/base -> origin/gh/yangw-dev/14/base 2025-12-04T09:43:53.9702054Z * [new branch] gh/yangw-dev/14/head -> origin/gh/yangw-dev/14/head 2025-12-04T09:43:53.9703902Z * [new branch] gh/yangw-dev/14/orig -> origin/gh/yangw-dev/14/orig 2025-12-04T09:43:53.9706413Z * [new branch] gh/yangw-dev/15/base -> origin/gh/yangw-dev/15/base 2025-12-04T09:43:53.9708177Z * [new branch] gh/yangw-dev/15/head -> origin/gh/yangw-dev/15/head 2025-12-04T09:43:53.9709827Z * [new branch] gh/yangw-dev/15/orig -> origin/gh/yangw-dev/15/orig 2025-12-04T09:43:53.9712207Z * [new branch] gh/yangw-dev/19/base -> origin/gh/yangw-dev/19/base 2025-12-04T09:43:53.9713929Z * [new branch] gh/yangw-dev/19/head -> origin/gh/yangw-dev/19/head 2025-12-04T09:43:53.9715620Z * [new branch] gh/yangw-dev/19/orig -> origin/gh/yangw-dev/19/orig 2025-12-04T09:43:53.9718027Z * [new branch] gh/yangw-dev/26/base -> origin/gh/yangw-dev/26/base 2025-12-04T09:43:53.9719767Z * [new branch] gh/yangw-dev/26/head -> origin/gh/yangw-dev/26/head 2025-12-04T09:43:53.9721507Z * [new branch] gh/yangw-dev/26/orig -> origin/gh/yangw-dev/26/orig 2025-12-04T09:43:53.9723924Z * [new branch] gh/yangw-dev/27/base -> origin/gh/yangw-dev/27/base 2025-12-04T09:43:53.9725826Z * [new branch] gh/yangw-dev/27/head -> origin/gh/yangw-dev/27/head 2025-12-04T09:43:53.9727485Z * [new branch] gh/yangw-dev/27/orig -> origin/gh/yangw-dev/27/orig 2025-12-04T09:43:53.9730274Z * [new branch] gh/ydwu4/292/base -> origin/gh/ydwu4/292/base 2025-12-04T09:43:53.9731918Z * [new branch] gh/ydwu4/292/head -> origin/gh/ydwu4/292/head 2025-12-04T09:43:53.9733689Z * [new branch] gh/ydwu4/292/orig -> origin/gh/ydwu4/292/orig 2025-12-04T09:43:53.9735968Z * [new branch] gh/ydwu4/294/base -> origin/gh/ydwu4/294/base 2025-12-04T09:43:53.9737659Z * [new branch] gh/ydwu4/294/head -> origin/gh/ydwu4/294/head 2025-12-04T09:43:53.9739396Z * [new branch] gh/ydwu4/294/orig -> origin/gh/ydwu4/294/orig 2025-12-04T09:43:53.9741881Z * [new branch] gh/ydwu4/295/base -> origin/gh/ydwu4/295/base 2025-12-04T09:43:53.9743716Z * [new branch] gh/ydwu4/295/head -> origin/gh/ydwu4/295/head 2025-12-04T09:43:53.9745498Z * [new branch] gh/ydwu4/295/orig -> origin/gh/ydwu4/295/orig 2025-12-04T09:43:53.9747683Z * [new branch] gh/ydwu4/296/base -> origin/gh/ydwu4/296/base 2025-12-04T09:43:53.9749498Z * [new branch] gh/ydwu4/296/head -> origin/gh/ydwu4/296/head 2025-12-04T09:43:53.9751671Z * [new branch] gh/ydwu4/296/orig -> origin/gh/ydwu4/296/orig 2025-12-04T09:43:53.9754133Z * [new branch] gh/ydwu4/306/base -> origin/gh/ydwu4/306/base 2025-12-04T09:43:53.9755924Z * [new branch] gh/ydwu4/306/head -> origin/gh/ydwu4/306/head 2025-12-04T09:43:53.9757771Z * [new branch] gh/ydwu4/306/orig -> origin/gh/ydwu4/306/orig 2025-12-04T09:43:53.9760044Z * [new branch] gh/ydwu4/312/base -> origin/gh/ydwu4/312/base 2025-12-04T09:43:53.9761768Z * [new branch] gh/ydwu4/312/head -> origin/gh/ydwu4/312/head 2025-12-04T09:43:53.9763500Z * [new branch] gh/ydwu4/312/orig -> origin/gh/ydwu4/312/orig 2025-12-04T09:43:53.9765845Z * [new branch] gh/ydwu4/322/base -> origin/gh/ydwu4/322/base 2025-12-04T09:43:53.9767548Z * [new branch] gh/ydwu4/322/head -> origin/gh/ydwu4/322/head 2025-12-04T09:43:53.9769264Z * [new branch] gh/ydwu4/322/orig -> origin/gh/ydwu4/322/orig 2025-12-04T09:43:53.9771614Z * [new branch] gh/ydwu4/327/base -> origin/gh/ydwu4/327/base 2025-12-04T09:43:53.9773388Z * [new branch] gh/ydwu4/327/head -> origin/gh/ydwu4/327/head 2025-12-04T09:43:53.9775094Z * [new branch] gh/ydwu4/327/orig -> origin/gh/ydwu4/327/orig 2025-12-04T09:43:53.9777578Z * [new branch] gh/ydwu4/328/base -> origin/gh/ydwu4/328/base 2025-12-04T09:43:53.9781350Z * [new branch] gh/ydwu4/328/head -> origin/gh/ydwu4/328/head 2025-12-04T09:43:53.9783042Z * [new branch] gh/ydwu4/328/orig -> origin/gh/ydwu4/328/orig 2025-12-04T09:43:53.9785348Z * [new branch] gh/ydwu4/329/base -> origin/gh/ydwu4/329/base 2025-12-04T09:43:53.9787055Z * [new branch] gh/ydwu4/329/head -> origin/gh/ydwu4/329/head 2025-12-04T09:43:53.9788768Z * [new branch] gh/ydwu4/329/orig -> origin/gh/ydwu4/329/orig 2025-12-04T09:43:53.9791225Z * [new branch] gh/ydwu4/330/base -> origin/gh/ydwu4/330/base 2025-12-04T09:43:53.9792902Z * [new branch] gh/ydwu4/330/head -> origin/gh/ydwu4/330/head 2025-12-04T09:43:53.9794653Z * [new branch] gh/ydwu4/330/orig -> origin/gh/ydwu4/330/orig 2025-12-04T09:43:53.9796892Z * [new branch] gh/ydwu4/331/base -> origin/gh/ydwu4/331/base 2025-12-04T09:43:53.9798758Z * [new branch] gh/ydwu4/331/head -> origin/gh/ydwu4/331/head 2025-12-04T09:43:53.9800362Z * [new branch] gh/ydwu4/331/orig -> origin/gh/ydwu4/331/orig 2025-12-04T09:43:53.9802733Z * [new branch] gh/ydwu4/332/base -> origin/gh/ydwu4/332/base 2025-12-04T09:43:53.9804462Z * [new branch] gh/ydwu4/332/head -> origin/gh/ydwu4/332/head 2025-12-04T09:43:53.9806223Z * [new branch] gh/ydwu4/332/orig -> origin/gh/ydwu4/332/orig 2025-12-04T09:43:53.9808451Z * [new branch] gh/ydwu4/333/base -> origin/gh/ydwu4/333/base 2025-12-04T09:43:53.9810133Z * [new branch] gh/ydwu4/333/head -> origin/gh/ydwu4/333/head 2025-12-04T09:43:53.9811825Z * [new branch] gh/ydwu4/333/orig -> origin/gh/ydwu4/333/orig 2025-12-04T09:43:53.9814046Z * [new branch] gh/ydwu4/334/base -> origin/gh/ydwu4/334/base 2025-12-04T09:43:53.9815824Z * [new branch] gh/ydwu4/334/head -> origin/gh/ydwu4/334/head 2025-12-04T09:43:53.9818090Z * [new branch] gh/ydwu4/334/orig -> origin/gh/ydwu4/334/orig 2025-12-04T09:43:53.9820363Z * [new branch] gh/ydwu4/335/base -> origin/gh/ydwu4/335/base 2025-12-04T09:43:53.9822026Z * [new branch] gh/ydwu4/335/head -> origin/gh/ydwu4/335/head 2025-12-04T09:43:53.9823822Z * [new branch] gh/ydwu4/335/orig -> origin/gh/ydwu4/335/orig 2025-12-04T09:43:53.9826761Z * [new branch] gh/ydwu4/337/base -> origin/gh/ydwu4/337/base 2025-12-04T09:43:53.9828429Z * [new branch] gh/ydwu4/337/head -> origin/gh/ydwu4/337/head 2025-12-04T09:43:53.9830268Z * [new branch] gh/ydwu4/337/orig -> origin/gh/ydwu4/337/orig 2025-12-04T09:43:53.9832626Z * [new branch] gh/ydwu4/339/base -> origin/gh/ydwu4/339/base 2025-12-04T09:43:53.9834404Z * [new branch] gh/ydwu4/339/head -> origin/gh/ydwu4/339/head 2025-12-04T09:43:53.9836059Z * [new branch] gh/ydwu4/339/orig -> origin/gh/ydwu4/339/orig 2025-12-04T09:43:53.9838936Z * [new branch] gh/yf225/133/base -> origin/gh/yf225/133/base 2025-12-04T09:43:53.9840606Z * [new branch] gh/yf225/133/head -> origin/gh/yf225/133/head 2025-12-04T09:43:53.9842925Z * [new branch] gh/yf225/93/base -> origin/gh/yf225/93/base 2025-12-04T09:43:53.9844652Z * [new branch] gh/yf225/93/head -> origin/gh/yf225/93/head 2025-12-04T09:43:53.9848011Z * [new branch] gh/yifuwang/152/base -> origin/gh/yifuwang/152/base 2025-12-04T09:43:53.9850003Z * [new branch] gh/yifuwang/152/head -> origin/gh/yifuwang/152/head 2025-12-04T09:43:53.9851842Z * [new branch] gh/yifuwang/152/orig -> origin/gh/yifuwang/152/orig 2025-12-04T09:43:53.9854128Z * [new branch] gh/yifuwang/195/base -> origin/gh/yifuwang/195/base 2025-12-04T09:43:53.9855903Z * [new branch] gh/yifuwang/195/head -> origin/gh/yifuwang/195/head 2025-12-04T09:43:53.9857659Z * [new branch] gh/yifuwang/195/orig -> origin/gh/yifuwang/195/orig 2025-12-04T09:43:53.9860497Z * [new branch] gh/yiming0416/1/base -> origin/gh/yiming0416/1/base 2025-12-04T09:43:53.9862275Z * [new branch] gh/yiming0416/1/head -> origin/gh/yiming0416/1/head 2025-12-04T09:43:53.9864547Z * [new branch] gh/yiming0416/2/base -> origin/gh/yiming0416/2/base 2025-12-04T09:43:53.9866268Z * [new branch] gh/yiming0416/2/head -> origin/gh/yiming0416/2/head 2025-12-04T09:43:53.9869223Z * [new branch] gh/yushangdi/1/base -> origin/gh/yushangdi/1/base 2025-12-04T09:43:53.9871078Z * [new branch] gh/yushangdi/1/head -> origin/gh/yushangdi/1/head 2025-12-04T09:43:53.9873325Z * [new branch] gh/yushangdi/10/base -> origin/gh/yushangdi/10/base 2025-12-04T09:43:53.9875096Z * [new branch] gh/yushangdi/10/head -> origin/gh/yushangdi/10/head 2025-12-04T09:43:53.9876773Z * [new branch] gh/yushangdi/10/orig -> origin/gh/yushangdi/10/orig 2025-12-04T09:43:53.9879910Z * [new branch] gh/yushangdi/11/base -> origin/gh/yushangdi/11/base 2025-12-04T09:43:53.9881456Z * [new branch] gh/yushangdi/11/head -> origin/gh/yushangdi/11/head 2025-12-04T09:43:53.9883183Z * [new branch] gh/yushangdi/11/orig -> origin/gh/yushangdi/11/orig 2025-12-04T09:43:53.9885423Z * [new branch] gh/yushangdi/2/base -> origin/gh/yushangdi/2/base 2025-12-04T09:43:53.9887219Z * [new branch] gh/yushangdi/2/head -> origin/gh/yushangdi/2/head 2025-12-04T09:43:53.9889528Z * [new branch] gh/yushangdi/7/base -> origin/gh/yushangdi/7/base 2025-12-04T09:43:53.9891311Z * [new branch] gh/yushangdi/7/head -> origin/gh/yushangdi/7/head 2025-12-04T09:43:53.9893063Z * [new branch] gh/yushangdi/7/orig -> origin/gh/yushangdi/7/orig 2025-12-04T09:43:53.9895686Z * [new branch] gh/yushangdi/8/base -> origin/gh/yushangdi/8/base 2025-12-04T09:43:53.9897565Z * [new branch] gh/yushangdi/8/head -> origin/gh/yushangdi/8/head 2025-12-04T09:43:53.9899346Z * [new branch] gh/yushangdi/8/orig -> origin/gh/yushangdi/8/orig 2025-12-04T09:43:53.9901560Z * [new branch] gh/yushangdi/9/base -> origin/gh/yushangdi/9/base 2025-12-04T09:43:53.9903457Z * [new branch] gh/yushangdi/9/head -> origin/gh/yushangdi/9/head 2025-12-04T09:43:53.9905288Z * [new branch] gh/yushangdi/9/orig -> origin/gh/yushangdi/9/orig 2025-12-04T09:43:53.9908081Z * [new branch] gh/zklaus/19/base -> origin/gh/zklaus/19/base 2025-12-04T09:43:53.9909820Z * [new branch] gh/zklaus/19/head -> origin/gh/zklaus/19/head 2025-12-04T09:43:53.9911598Z * [new branch] gh/zklaus/19/orig -> origin/gh/zklaus/19/orig 2025-12-04T09:43:53.9913966Z * [new branch] gh/zklaus/20/base -> origin/gh/zklaus/20/base 2025-12-04T09:43:53.9915750Z * [new branch] gh/zklaus/20/head -> origin/gh/zklaus/20/head 2025-12-04T09:43:53.9917488Z * [new branch] gh/zklaus/20/orig -> origin/gh/zklaus/20/orig 2025-12-04T09:43:53.9919819Z * [new branch] gh/zklaus/21/base -> origin/gh/zklaus/21/base 2025-12-04T09:43:53.9921551Z * [new branch] gh/zklaus/21/head -> origin/gh/zklaus/21/head 2025-12-04T09:43:53.9923358Z * [new branch] gh/zklaus/21/orig -> origin/gh/zklaus/21/orig 2025-12-04T09:43:53.9926147Z * [new branch] gh/zklaus/22/base -> origin/gh/zklaus/22/base 2025-12-04T09:43:53.9927863Z * [new branch] gh/zklaus/22/head -> origin/gh/zklaus/22/head 2025-12-04T09:43:53.9929608Z * [new branch] gh/zklaus/22/orig -> origin/gh/zklaus/22/orig 2025-12-04T09:43:53.9931873Z * [new branch] gh/zklaus/23/base -> origin/gh/zklaus/23/base 2025-12-04T09:43:53.9933611Z * [new branch] gh/zklaus/23/head -> origin/gh/zklaus/23/head 2025-12-04T09:43:53.9935389Z * [new branch] gh/zklaus/23/orig -> origin/gh/zklaus/23/orig 2025-12-04T09:43:53.9937686Z * [new branch] gh/zklaus/24/base -> origin/gh/zklaus/24/base 2025-12-04T09:43:53.9939418Z * [new branch] gh/zklaus/24/head -> origin/gh/zklaus/24/head 2025-12-04T09:43:53.9941172Z * [new branch] gh/zklaus/24/orig -> origin/gh/zklaus/24/orig 2025-12-04T09:43:53.9944398Z * [new branch] gh/zou3519/1197/base -> origin/gh/zou3519/1197/base 2025-12-04T09:43:53.9945977Z * [new branch] gh/zou3519/1197/head -> origin/gh/zou3519/1197/head 2025-12-04T09:43:53.9947679Z * [new branch] gh/zou3519/1197/orig -> origin/gh/zou3519/1197/orig 2025-12-04T09:43:53.9950312Z * [new branch] gh/zou3519/1199/base -> origin/gh/zou3519/1199/base 2025-12-04T09:43:53.9952071Z * [new branch] gh/zou3519/1199/head -> origin/gh/zou3519/1199/head 2025-12-04T09:43:53.9954279Z * [new branch] gh/zou3519/1199/orig -> origin/gh/zou3519/1199/orig 2025-12-04T09:43:53.9957134Z * [new branch] gh/zou3519/1200/base -> origin/gh/zou3519/1200/base 2025-12-04T09:43:53.9958879Z * [new branch] gh/zou3519/1200/head -> origin/gh/zou3519/1200/head 2025-12-04T09:43:53.9960589Z * [new branch] gh/zou3519/1200/orig -> origin/gh/zou3519/1200/orig 2025-12-04T09:43:53.9963201Z * [new branch] gh/zou3519/1201/base -> origin/gh/zou3519/1201/base 2025-12-04T09:43:53.9964891Z * [new branch] gh/zou3519/1201/head -> origin/gh/zou3519/1201/head 2025-12-04T09:43:53.9966555Z * [new branch] gh/zou3519/1201/orig -> origin/gh/zou3519/1201/orig 2025-12-04T09:43:53.9968773Z * [new branch] gh/zou3519/1202/base -> origin/gh/zou3519/1202/base 2025-12-04T09:43:53.9970548Z * [new branch] gh/zou3519/1202/head -> origin/gh/zou3519/1202/head 2025-12-04T09:43:53.9972287Z * [new branch] gh/zou3519/1202/orig -> origin/gh/zou3519/1202/orig 2025-12-04T09:43:53.9975114Z * [new branch] gh/zpcore/1/base -> origin/gh/zpcore/1/base 2025-12-04T09:43:53.9976838Z * [new branch] gh/zpcore/1/head -> origin/gh/zpcore/1/head 2025-12-04T09:43:53.9980096Z * [new branch] gh/zpcore/11/base -> origin/gh/zpcore/11/base 2025-12-04T09:43:53.9982011Z * [new branch] gh/zpcore/11/head -> origin/gh/zpcore/11/head 2025-12-04T09:43:53.9983819Z * [new branch] gh/zpcore/11/orig -> origin/gh/zpcore/11/orig 2025-12-04T09:43:53.9986718Z * [new branch] gh/zpcore/12/base -> origin/gh/zpcore/12/base 2025-12-04T09:43:53.9988414Z * [new branch] gh/zpcore/12/head -> origin/gh/zpcore/12/head 2025-12-04T09:43:53.9990283Z * [new branch] gh/zpcore/12/orig -> origin/gh/zpcore/12/orig 2025-12-04T09:43:53.9992782Z * [new branch] gh/zpcore/13/base -> origin/gh/zpcore/13/base 2025-12-04T09:43:53.9994414Z * [new branch] gh/zpcore/13/head -> origin/gh/zpcore/13/head 2025-12-04T09:43:53.9996102Z * [new branch] gh/zpcore/13/orig -> origin/gh/zpcore/13/orig 2025-12-04T09:43:53.9998474Z * [new branch] gh/zpcore/14/base -> origin/gh/zpcore/14/base 2025-12-04T09:43:54.0000243Z * [new branch] gh/zpcore/14/head -> origin/gh/zpcore/14/head 2025-12-04T09:43:54.0001928Z * [new branch] gh/zpcore/14/orig -> origin/gh/zpcore/14/orig 2025-12-04T09:43:54.0004444Z * [new branch] gh/zpcore/15/base -> origin/gh/zpcore/15/base 2025-12-04T09:43:54.0006121Z * [new branch] gh/zpcore/15/head -> origin/gh/zpcore/15/head 2025-12-04T09:43:54.0007936Z * [new branch] gh/zpcore/15/orig -> origin/gh/zpcore/15/orig 2025-12-04T09:43:54.0010304Z * [new branch] gh/zpcore/2/base -> origin/gh/zpcore/2/base 2025-12-04T09:43:54.0012037Z * [new branch] gh/zpcore/2/head -> origin/gh/zpcore/2/head 2025-12-04T09:43:54.0014774Z * [new branch] gh/zpcore/21/base -> origin/gh/zpcore/21/base 2025-12-04T09:43:54.0016766Z * [new branch] gh/zpcore/21/head -> origin/gh/zpcore/21/head 2025-12-04T09:43:54.0018448Z * [new branch] gh/zpcore/21/orig -> origin/gh/zpcore/21/orig 2025-12-04T09:43:54.0020989Z * [new branch] gh/zpcore/22/base -> origin/gh/zpcore/22/base 2025-12-04T09:43:54.0022669Z * [new branch] gh/zpcore/22/head -> origin/gh/zpcore/22/head 2025-12-04T09:43:54.0024622Z * [new branch] gh/zpcore/22/orig -> origin/gh/zpcore/22/orig 2025-12-04T09:43:54.0026977Z * [new branch] gh/zpcore/23/base -> origin/gh/zpcore/23/base 2025-12-04T09:43:54.0028866Z * [new branch] gh/zpcore/23/head -> origin/gh/zpcore/23/head 2025-12-04T09:43:54.0030508Z * [new branch] gh/zpcore/23/orig -> origin/gh/zpcore/23/orig 2025-12-04T09:43:54.0032977Z * [new branch] gh/zpcore/24/base -> origin/gh/zpcore/24/base 2025-12-04T09:43:54.0034620Z * [new branch] gh/zpcore/24/head -> origin/gh/zpcore/24/head 2025-12-04T09:43:54.0036324Z * [new branch] gh/zpcore/24/orig -> origin/gh/zpcore/24/orig 2025-12-04T09:43:54.0038769Z * [new branch] gh/zpcore/25/base -> origin/gh/zpcore/25/base 2025-12-04T09:43:54.0040527Z * [new branch] gh/zpcore/25/head -> origin/gh/zpcore/25/head 2025-12-04T09:43:54.0042283Z * [new branch] gh/zpcore/25/orig -> origin/gh/zpcore/25/orig 2025-12-04T09:43:54.0044710Z * [new branch] gh/zpcore/26/base -> origin/gh/zpcore/26/base 2025-12-04T09:43:54.0046504Z * [new branch] gh/zpcore/26/head -> origin/gh/zpcore/26/head 2025-12-04T09:43:54.0048242Z * [new branch] gh/zpcore/26/orig -> origin/gh/zpcore/26/orig 2025-12-04T09:43:54.0050598Z * [new branch] gh/zpcore/27/base -> origin/gh/zpcore/27/base 2025-12-04T09:43:54.0052353Z * [new branch] gh/zpcore/27/head -> origin/gh/zpcore/27/head 2025-12-04T09:43:54.0054201Z * [new branch] gh/zpcore/27/orig -> origin/gh/zpcore/27/orig 2025-12-04T09:43:54.0056963Z * [new branch] gh/zpcore/28/base -> origin/gh/zpcore/28/base 2025-12-04T09:43:54.0058970Z * [new branch] gh/zpcore/28/head -> origin/gh/zpcore/28/head 2025-12-04T09:43:54.0060663Z * [new branch] gh/zpcore/28/orig -> origin/gh/zpcore/28/orig 2025-12-04T09:43:54.0062944Z * [new branch] gh/zpcore/3/base -> origin/gh/zpcore/3/base 2025-12-04T09:43:54.0064757Z * [new branch] gh/zpcore/3/head -> origin/gh/zpcore/3/head 2025-12-04T09:43:54.0066944Z * [new branch] gh/zpcore/4/base -> origin/gh/zpcore/4/base 2025-12-04T09:43:54.0068638Z * [new branch] gh/zpcore/4/head -> origin/gh/zpcore/4/head 2025-12-04T09:43:54.0070872Z * [new branch] gh/zpcore/5/base -> origin/gh/zpcore/5/base 2025-12-04T09:43:54.0072615Z * [new branch] gh/zpcore/5/head -> origin/gh/zpcore/5/head 2025-12-04T09:43:54.0074807Z * [new branch] gh/zpcore/6/base -> origin/gh/zpcore/6/base 2025-12-04T09:43:54.0076489Z * [new branch] gh/zpcore/6/head -> origin/gh/zpcore/6/head 2025-12-04T09:43:54.0079567Z * [new branch] gh/zpcore/7/base -> origin/gh/zpcore/7/base 2025-12-04T09:43:54.0081239Z * [new branch] gh/zpcore/7/head -> origin/gh/zpcore/7/head 2025-12-04T09:43:54.0083524Z * [new branch] gh/zpcore/8/base -> origin/gh/zpcore/8/base 2025-12-04T09:43:54.0085355Z * [new branch] gh/zpcore/8/head -> origin/gh/zpcore/8/head 2025-12-04T09:43:54.0087284Z * [new branch] google-main -> origin/google-main 2025-12-04T09:43:54.0090246Z * [new branch] guangyey/external_stream -> origin/guangyey/external_stream 2025-12-04T09:43:54.0091757Z * [new branch] guangyey/test_2025 -> origin/guangyey/test_2025 2025-12-04T09:43:54.0094222Z * [new branch] guilhermeleobas/cherry-pick-55d87d9dfd9 -> origin/guilhermeleobas/cherry-pick-55d87d9dfd9 2025-12-04T09:43:54.0096559Z * [new branch] hameerabbasi/complex_tensor_subclass -> origin/hameerabbasi/complex_tensor_subclass 2025-12-04T09:43:54.0098450Z * [new branch] hameerabbasi/fix-ctensor-gradcheck-tests -> origin/hameerabbasi/fix-ctensor-gradcheck-tests 2025-12-04T09:43:54.0100176Z * [new branch] hameerabbasi/gradcheck-allclose -> origin/hameerabbasi/gradcheck-allclose 2025-12-04T09:43:54.0101806Z * [new branch] hc_baseline -> origin/hc_baseline 2025-12-04T09:43:54.0103876Z * [new branch] hhh_rand -> origin/hhh_rand 2025-12-04T09:43:54.0106221Z * [new branch] huba/f1 -> origin/huba/f1 2025-12-04T09:43:54.0108603Z * [new branch] increase-timeout-linux-jammy-cuda12_8-py3_10-gcc11-test -> origin/increase-timeout-linux-jammy-cuda12_8-py3_10-gcc11-test 2025-12-04T09:43:54.0110087Z * [new branch] inlining -> origin/inlining 2025-12-04T09:43:54.0111988Z * [new branch] inlining-ezyang -> origin/inlining-ezyang 2025-12-04T09:43:54.0113908Z * [new branch] install-torchao-0.13.0 -> origin/install-torchao-0.13.0 2025-12-04T09:43:54.0116015Z * [new branch] instrument-trunk-pull-linux-with-job-test-filters -> origin/instrument-trunk-pull-linux-with-job-test-filters 2025-12-04T09:43:54.0117534Z * [new branch] invoke-subgraph -> origin/invoke-subgraph 2025-12-04T09:43:54.0119497Z * [new branch] issue#58739 -> origin/issue#58739 2025-12-04T09:43:54.0121395Z * [new branch] jainapurva-patch-1 -> origin/jainapurva-patch-1 2025-12-04T09:43:54.0123678Z * [new branch] jathu/o3 -> origin/jathu/o3 2025-12-04T09:43:54.0125721Z * [new branch] jathu/sve -> origin/jathu/sve 2025-12-04T09:43:54.0128181Z * [new branch] jcaip/test-cusparselt-version-0.6.2 -> origin/jcaip/test-cusparselt-version-0.6.2 2025-12-04T09:43:54.0129984Z * [new branch] jcaip/update-cusparselt-0.6.2 -> origin/jcaip/update-cusparselt-0.6.2 2025-12-04T09:43:54.0132299Z * [new branch] jiannanWang/memorysnapshot_filter -> origin/jiannanWang/memorysnapshot_filter 2025-12-04T09:43:54.0134463Z * [new branch] jiannanWang/profilerstepwarning -> origin/jiannanWang/profilerstepwarning 2025-12-04T09:43:54.0136284Z * [new branch] jithunnair-amd-patch-1 -> origin/jithunnair-amd-patch-1 2025-12-04T09:43:54.0138164Z * [new branch] jithunnair-amd-patch-10 -> origin/jithunnair-amd-patch-10 2025-12-04T09:43:54.0140083Z * [new branch] jithunnair-amd-patch-2 -> origin/jithunnair-amd-patch-2 2025-12-04T09:43:54.0141891Z * [new branch] jithunnair-amd-patch-3 -> origin/jithunnair-amd-patch-3 2025-12-04T09:43:54.0143876Z * [new branch] jithunnair-amd-patch-4 -> origin/jithunnair-amd-patch-4 2025-12-04T09:43:54.0145689Z * [new branch] jithunnair-amd-patch-5 -> origin/jithunnair-amd-patch-5 2025-12-04T09:43:54.0147588Z * [new branch] jithunnair-amd-patch-6 -> origin/jithunnair-amd-patch-6 2025-12-04T09:43:54.0149336Z * [new branch] jithunnair-amd-patch-7 -> origin/jithunnair-amd-patch-7 2025-12-04T09:43:54.0151208Z * [new branch] jithunnair-amd-patch-8 -> origin/jithunnair-amd-patch-8 2025-12-04T09:43:54.0152988Z * [new branch] jithunnair-amd-patch-9 -> origin/jithunnair-amd-patch-9 2025-12-04T09:43:54.0155434Z * [new branch] justinchu/native-qdq -> origin/justinchu/native-qdq 2025-12-04T09:43:54.0158028Z * [new branch] kainan666/xlf_debug -> origin/kainan666/xlf_debug 2025-12-04T09:43:54.0159673Z * [new branch] kainan_test -> origin/kainan_test 2025-12-04T09:43:54.0161565Z * [new branch] larryliu0820-patch-1 -> origin/larryliu0820-patch-1 2025-12-04T09:43:54.0163955Z * [new branch] leslie/test_group_gemm_epilogues -> origin/leslie/test_group_gemm_epilogues 2025-12-04T09:43:54.0166432Z * [new branch] lessw2020/fix_cutlass_cache_error -> origin/lessw2020/fix_cutlass_cache_error 2025-12-04T09:43:54.0168634Z * [new branch] liaoxuan/shm_all_reduce -> origin/liaoxuan/shm_all_reduce 2025-12-04T09:43:54.0170302Z * [new branch] liaoxuan/test_fa_disable_softmax -> origin/liaoxuan/test_fa_disable_softmax 2025-12-04T09:43:54.0171964Z * [new branch] liaoxuan/test_int8_sdpa -> origin/liaoxuan/test_int8_sdpa 2025-12-04T09:43:54.0173744Z * [new branch] llama4-stable -> origin/llama4-stable 2025-12-04T09:43:54.0176574Z * [new branch] lts/release/1.8 -> origin/lts/release/1.8 2025-12-04T09:43:54.0179260Z * [new branch] lucaskabela/#94773 -> origin/lucaskabela/#94773 2025-12-04T09:43:54.0180908Z * [new branch] lucaskabela/fix_164876 -> origin/lucaskabela/fix_164876 2025-12-04T09:43:54.0182721Z * [new branch] lucaskabela/flop_counter -> origin/lucaskabela/flop_counter 2025-12-04T09:43:54.0184502Z * [new branch] lucaskabela/func_under_decomp -> origin/lucaskabela/func_under_decomp 2025-12-04T09:43:54.0186109Z * [new branch] lucaskabela/functional_in_dynamo -> origin/lucaskabela/functional_in_dynamo 2025-12-04T09:43:54.0187839Z * [new branch] lucaskabela/install_params_as_graph_attr -> origin/lucaskabela/install_params_as_graph_attr 2025-12-04T09:43:54.0189866Z * [new branch] lucaskabela/parameters_as_graph_attr -> origin/lucaskabela/parameters_as_graph_attr 2025-12-04T09:43:54.0191987Z * [new branch] lucaskabela/remove_aot_dispatcher_metadata -> origin/lucaskabela/remove_aot_dispatcher_metadata 2025-12-04T09:43:54.0193674Z * [new branch] lucaskabela/rnn_decomp -> origin/lucaskabela/rnn_decomp 2025-12-04T09:43:54.0195431Z * [new branch] lucaskabela/typing_backends -> origin/lucaskabela/typing_backends 2025-12-04T09:43:54.0197238Z * [new branch] lucaskabela/typing_ctx_manager -> origin/lucaskabela/typing_ctx_manager 2025-12-04T09:43:54.0198982Z * [new branch] lucaskabela/typing_nn_module -> origin/lucaskabela/typing_nn_module 2025-12-04T09:43:54.0200845Z * [new branch] lucaskabela/typing_user_defined -> origin/lucaskabela/typing_user_defined 2025-12-04T09:43:54.0202549Z * [new branch] lucaskabela/typing_variables -> origin/lucaskabela/typing_variables 2025-12-04T09:43:54.0204473Z * [new branch] lucaskabela/typing_variables_dicts -> origin/lucaskabela/typing_variables_dicts 2025-12-04T09:43:54.0206397Z * [new branch] lucaskabela/typing_variables_functions -> origin/lucaskabela/typing_variables_functions 2025-12-04T09:43:54.0208110Z * [new branch] lucaskabela/typing_variables_lists -> origin/lucaskabela/typing_variables_lists 2025-12-04T09:43:54.0210299Z * [new branch] lw/torch_box_by_ref -> origin/lw/torch_box_by_ref 2025-12-04T09:43:54.0212182Z * [new branch] main -> origin/main 2025-12-04T09:43:54.0214115Z * [new branch] malfet-patch-1 -> origin/malfet-patch-1 2025-12-04T09:43:54.0216034Z * [new branch] malfet-patch-2 -> origin/malfet-patch-2 2025-12-04T09:43:54.0217931Z * [new branch] malfet-patch-3 -> origin/malfet-patch-3 2025-12-04T09:43:54.0219894Z * [new branch] malfet-patch-4 -> origin/malfet-patch-4 2025-12-04T09:43:54.0222182Z * [new branch] malfet-patch-5 -> origin/malfet-patch-5 2025-12-04T09:43:54.0224101Z * [new branch] malfet-patch-6 -> origin/malfet-patch-6 2025-12-04T09:43:54.0226046Z * [new branch] malfet-patch-7 -> origin/malfet-patch-7 2025-12-04T09:43:54.0227885Z * [new branch] malfet-patch-8 -> origin/malfet-patch-8 2025-12-04T09:43:54.0230246Z * [new branch] malfet/add-3.14-ci -> origin/malfet/add-3.14-ci 2025-12-04T09:43:54.0232106Z * [new branch] malfet/be-do-not-make-typos-in-build-artifacts -> origin/malfet/be-do-not-make-typos-in-build-artifacts 2025-12-04T09:43:54.0233908Z * [new branch] malfet/be-move-more-settings-to-checkout-pytorch -> origin/malfet/be-move-more-settings-to-checkout-pytorch 2025-12-04T09:43:54.0235836Z * [new branch] malfet/be-remove-misisng-neon-headers -> origin/malfet/be-remove-misisng-neon-headers 2025-12-04T09:43:54.0237738Z * [new branch] malfet/mps-implement-col2im -> origin/malfet/mps-implement-col2im 2025-12-04T09:43:54.0240147Z * [new branch] manuel/aoti_metal_shimify-thread_safe -> origin/manuel/aoti_metal_shimify-thread_safe 2025-12-04T09:43:54.0241800Z * [new branch] manuel/inductor_link_openmp -> origin/manuel/inductor_link_openmp 2025-12-04T09:43:54.0244093Z * [new branch] masnesral/metaconda -> origin/masnesral/metaconda 2025-12-04T09:43:54.0246040Z * [new branch] mem_profiler_flaky_fix -> origin/mem_profiler_flaky_fix 2025-12-04T09:43:54.0247848Z * [new branch] mem_profiler_stack_trace -> origin/mem_profiler_stack_trace 2025-12-04T09:43:54.0249703Z * [new branch] memory_profiler_stack -> origin/memory_profiler_stack 2025-12-04T09:43:54.0251603Z * [new branch] metascroy-patch-1 -> origin/metascroy-patch-1 2025-12-04T09:43:54.0253431Z * [new branch] mingw_posix -> origin/mingw_posix 2025-12-04T09:43:54.0255867Z * [new branch] mlazos/S429861-debug -> origin/mlazos/S429861-debug 2025-12-04T09:43:54.0257583Z * [new branch] mlazos/aa -> origin/mlazos/aa 2025-12-04T09:43:54.0259364Z * [new branch] mlazos/acts -> origin/mlazos/acts 2025-12-04T09:43:54.0261070Z * [new branch] mlazos/arg-renames -> origin/mlazos/arg-renames 2025-12-04T09:43:54.0262744Z * [new branch] mlazos/bad-cudagraphs -> origin/mlazos/bad-cudagraphs 2025-12-04T09:43:54.0264622Z * [new branch] mlazos/baseline-graph-breaks -> origin/mlazos/baseline-graph-breaks 2025-12-04T09:43:54.0266236Z * [new branch] mlazos/beta-tensor -> origin/mlazos/beta-tensor 2025-12-04T09:43:54.0267900Z * [new branch] mlazos/buffers -> origin/mlazos/buffers 2025-12-04T09:43:54.0269402Z * [new branch] mlazos/buffers2 -> origin/mlazos/buffers2 2025-12-04T09:43:54.0271450Z * [new branch] mlazos/buffers3 -> origin/mlazos/buffers3 2025-12-04T09:43:54.0273487Z * [new branch] mlazos/bwd -> origin/mlazos/bwd 2025-12-04T09:43:54.0275671Z * [new branch] mlazos/combo-test -> origin/mlazos/combo-test 2025-12-04T09:43:54.0277488Z * [new branch] mlazos/ctx-cleanup -> origin/mlazos/ctx-cleanup 2025-12-04T09:43:54.0279561Z * [new branch] mlazos/cuda-cmd-log -> origin/mlazos/cuda-cmd-log 2025-12-04T09:43:54.0281520Z * [new branch] mlazos/cudagraph-tests -> origin/mlazos/cudagraph-tests 2025-12-04T09:43:54.0283292Z * [new branch] mlazos/cudagraphs-measurement -> origin/mlazos/cudagraphs-measurement 2025-12-04T09:43:54.0285088Z * [new branch] mlazos/cutlass-test -> origin/mlazos/cutlass-test 2025-12-04T09:43:54.0287031Z * [new branch] mlazos/cutlass-topo-bug -> origin/mlazos/cutlass-topo-bug 2025-12-04T09:43:54.0288776Z * [new branch] mlazos/dataclass-proxy -> origin/mlazos/dataclass-proxy 2025-12-04T09:43:54.0290523Z * [new branch] mlazos/dc-attrs -> origin/mlazos/dc-attrs 2025-12-04T09:43:54.0292407Z * [new branch] mlazos/dc-helion -> origin/mlazos/dc-helion 2025-12-04T09:43:54.0294172Z * [new branch] mlazos/dict-fix -> origin/mlazos/dict-fix 2025-12-04T09:43:54.0295903Z * [new branch] mlazos/disable-tf -> origin/mlazos/disable-tf 2025-12-04T09:43:54.0297670Z * [new branch] mlazos/dupe-fix -> origin/mlazos/dupe-fix 2025-12-04T09:43:54.0299497Z * [new branch] mlazos/dyn-batch -> origin/mlazos/dyn-batch 2025-12-04T09:43:54.0301201Z * [new branch] mlazos/evt -> origin/mlazos/evt 2025-12-04T09:43:54.0303034Z * [new branch] mlazos/extract-examples -> origin/mlazos/extract-examples 2025-12-04T09:43:54.0304930Z * [new branch] mlazos/foreach-op -> origin/mlazos/foreach-op 2025-12-04T09:43:54.0306620Z * [new branch] mlazos/fp8 -> origin/mlazos/fp8 2025-12-04T09:43:54.0308382Z * [new branch] mlazos/fp8-bias -> origin/mlazos/fp8-bias 2025-12-04T09:43:54.0310295Z * [new branch] mlazos/fp8-bias-fusion -> origin/mlazos/fp8-bias-fusion 2025-12-04T09:43:54.0312274Z * [new branch] mlazos/fp8-fixes -> origin/mlazos/fp8-fixes 2025-12-04T09:43:54.0314011Z * [new branch] mlazos/freezing -> origin/mlazos/freezing 2025-12-04T09:43:54.0315795Z * [new branch] mlazos/h-comp -> origin/mlazos/h-comp 2025-12-04T09:43:54.0317705Z * [new branch] mlazos/h-comp2 -> origin/mlazos/h-comp2 2025-12-04T09:43:54.0319467Z * [new branch] mlazos/hash-hop -> origin/mlazos/hash-hop 2025-12-04T09:43:54.0321219Z * [new branch] mlazos/hc -> origin/mlazos/hc 2025-12-04T09:43:54.0323045Z * [new branch] mlazos/hc-cycles -> origin/mlazos/hc-cycles 2025-12-04T09:43:54.0324832Z * [new branch] mlazos/hc-fixes -> origin/mlazos/hc-fixes 2025-12-04T09:43:54.0326557Z * [new branch] mlazos/hc-fixes3 -> origin/mlazos/hc-fixes3 2025-12-04T09:43:54.0328358Z * [new branch] mlazos/hc-fixes4 -> origin/mlazos/hc-fixes4 2025-12-04T09:43:54.0330071Z * [new branch] mlazos/hc-hf -> origin/mlazos/hc-hf 2025-12-04T09:43:54.0331900Z * [new branch] mlazos/hc-mut -> origin/mlazos/hc-mut 2025-12-04T09:43:54.0333650Z * [new branch] mlazos/hc10 -> origin/mlazos/hc10 2025-12-04T09:43:54.0335437Z * [new branch] mlazos/hc11 -> origin/mlazos/hc11 2025-12-04T09:43:54.0337162Z * [new branch] mlazos/hc12 -> origin/mlazos/hc12 2025-12-04T09:43:54.0338922Z * [new branch] mlazos/hc13 -> origin/mlazos/hc13 2025-12-04T09:43:54.0340654Z * [new branch] mlazos/hc14 -> origin/mlazos/hc14 2025-12-04T09:43:54.0342433Z * [new branch] mlazos/hc15 -> origin/mlazos/hc15 2025-12-04T09:43:54.0344236Z * [new branch] mlazos/hc2 -> origin/mlazos/hc2 2025-12-04T09:43:54.0346051Z * [new branch] mlazos/hc4 -> origin/mlazos/hc4 2025-12-04T09:43:54.0347795Z * [new branch] mlazos/hc5 -> origin/mlazos/hc5 2025-12-04T09:43:54.0349672Z * [new branch] mlazos/hc6 -> origin/mlazos/hc6 2025-12-04T09:43:54.0351416Z * [new branch] mlazos/hc7 -> origin/mlazos/hc7 2025-12-04T09:43:54.0353190Z * [new branch] mlazos/hc8 -> origin/mlazos/hc8 2025-12-04T09:43:54.0354920Z * [new branch] mlazos/hc9 -> origin/mlazos/hc9 2025-12-04T09:43:54.0356727Z * [new branch] mlazos/hc_baseline2 -> origin/mlazos/hc_baseline2 2025-12-04T09:43:54.0358407Z * [new branch] mlazos/inductor-streams -> origin/mlazos/inductor-streams 2025-12-04T09:43:54.0360050Z * [new branch] mlazos/main -> origin/mlazos/main 2025-12-04T09:43:54.0361870Z * [new branch] mlazos/mcg2 -> origin/mlazos/mcg2 2025-12-04T09:43:54.0364186Z * [new branch] mlazos/meta-guards -> origin/mlazos/meta-guards 2025-12-04T09:43:54.0367021Z * [new branch] mlazos/mlazos/foreach-map-adam -> origin/mlazos/mlazos/foreach-map-adam 2025-12-04T09:43:54.0368784Z * [new branch] mlazos/mlazos/tf-mode-backup -> origin/mlazos/mlazos/tf-mode-backup 2025-12-04T09:43:54.0370503Z * [new branch] mlazos/mod-fix -> origin/mlazos/mod-fix 2025-12-04T09:43:54.0372479Z * [new branch] mlazos/mode-fix -> origin/mlazos/mode-fix 2025-12-04T09:43:54.0374259Z * [new branch] mlazos/offsets -> origin/mlazos/offsets 2025-12-04T09:43:54.0375950Z * [new branch] mlazos/overguarding -> origin/mlazos/overguarding 2025-12-04T09:43:54.0377944Z * [new branch] mlazos/proxy-ctors -> origin/mlazos/proxy-ctors 2025-12-04T09:43:54.0381057Z * [new branch] mlazos/quant-fix -> origin/mlazos/quant-fix 2025-12-04T09:43:54.0382819Z * [new branch] mlazos/resnet-fix -> origin/mlazos/resnet-fix 2025-12-04T09:43:54.0384742Z * [new branch] mlazos/rm-buf-names -> origin/mlazos/rm-buf-names 2025-12-04T09:43:54.0386507Z * [new branch] mlazos/rm-code -> origin/mlazos/rm-code 2025-12-04T09:43:54.0388313Z * [new branch] mlazos/rm-spam -> origin/mlazos/rm-spam 2025-12-04T09:43:54.0390124Z * [new branch] mlazos/rtp -> origin/mlazos/rtp 2025-12-04T09:43:54.0391969Z * [new branch] mlazos/static-idx-dbg -> origin/mlazos/static-idx-dbg 2025-12-04T09:43:54.0393747Z * [new branch] mlazos/static-inputs-log -> origin/mlazos/static-inputs-log 2025-12-04T09:43:54.0395350Z * [new branch] mlazos/stests -> origin/mlazos/stests 2025-12-04T09:43:54.0397510Z * [new branch] mlazos/stream-ops -> origin/mlazos/stream-ops 2025-12-04T09:43:54.0399264Z * [new branch] mlazos/td-fix2 -> origin/mlazos/td-fix2 2025-12-04T09:43:54.0401114Z * [new branch] mlazos/tensor-hasattr2 -> origin/mlazos/tensor-hasattr2 2025-12-04T09:43:54.0402856Z * [new branch] mlazos/test -> origin/mlazos/test 2025-12-04T09:43:54.0404736Z * [new branch] mlazos/tf-mode -> origin/mlazos/tf-mode 2025-12-04T09:43:54.0406533Z * [new branch] mlazos/tf-mode-backup2 -> origin/mlazos/tf-mode-backup2 2025-12-04T09:43:54.0408299Z * [new branch] mlazos/tf-mode-reland -> origin/mlazos/tf-mode-reland 2025-12-04T09:43:54.0410136Z * [new branch] mlazos/tf-mode-reland2 -> origin/mlazos/tf-mode-reland2 2025-12-04T09:43:54.0411985Z * [new branch] mlazos/tf-mode-reland3 -> origin/mlazos/tf-mode-reland3 2025-12-04T09:43:54.0413736Z * [new branch] mlazos/triton-no-epi -> origin/mlazos/triton-no-epi 2025-12-04T09:43:54.0415527Z * [new branch] mlazos/tune-proto -> origin/mlazos/tune-proto 2025-12-04T09:43:54.0417347Z * [new branch] mlazos/tuple-fixes -> origin/mlazos/tuple-fixes 2025-12-04T09:43:54.0419268Z * [new branch] mlazos/tuple-fixes2 -> origin/mlazos/tuple-fixes2 2025-12-04T09:43:54.0421042Z * [new branch] mlazos/tuple-handling -> origin/mlazos/tuple-handling 2025-12-04T09:43:54.0422815Z * [new branch] mlazos/user-stream-base -> origin/mlazos/user-stream-base 2025-12-04T09:43:54.0424742Z * [new branch] mlazos/user-streams -> origin/mlazos/user-streams 2025-12-04T09:43:54.0426543Z * [new branch] mlazos/user-streams-backup -> origin/mlazos/user-streams-backup 2025-12-04T09:43:54.0428382Z * [new branch] mlazos/user-streams-backup2 -> origin/mlazos/user-streams-backup2 2025-12-04T09:43:54.0430112Z * [new branch] mlazos/vary-beta -> origin/mlazos/vary-beta 2025-12-04T09:43:54.0431954Z * [new branch] mlazos/vary-beta2 -> origin/mlazos/vary-beta2 2025-12-04T09:43:54.0433773Z * [new branch] mlazos/weird-perf1 -> origin/mlazos/weird-perf1 2025-12-04T09:43:54.0435603Z * [new branch] mm_out_dtype_compile -> origin/mm_out_dtype_compile 2025-12-04T09:43:54.0437392Z * [new branch] module-shim -> origin/module-shim 2025-12-04T09:43:54.0439242Z * [new branch] move_config -> origin/move_config 2025-12-04T09:43:54.0441572Z * [new branch] msaroufim/reduce -> origin/msaroufim/reduce 2025-12-04T09:43:54.0443903Z * [new branch] mtia/basic-cmake -> origin/mtia/basic-cmake 2025-12-04T09:43:54.0446767Z * [new branch] mwizak/fix-triton-block-shape -> origin/mwizak/fix-triton-block-shape 2025-12-04T09:43:54.0448532Z * [new branch] my_varlen_backup -> origin/my_varlen_backup 2025-12-04T09:43:54.0450394Z * [new branch] nativert_num_outputs -> origin/nativert_num_outputs 2025-12-04T09:43:54.0452245Z * [new branch] new-codegen -> origin/new-codegen 2025-12-04T09:43:54.0454131Z * [new branch] newtest-base -> origin/newtest-base 2025-12-04T09:43:54.0456468Z * [new branch] ngimel/addmm_dtype -> origin/ngimel/addmm_dtype 2025-12-04T09:43:54.0458197Z * [new branch] ngimel/div_inv -> origin/ngimel/div_inv 2025-12-04T09:43:54.0459919Z * [new branch] ngimel/error_index_list -> origin/ngimel/error_index_list 2025-12-04T09:43:54.0461575Z * [new branch] ngimel/gather_grid -> origin/ngimel/gather_grid 2025-12-04T09:43:54.0463303Z * [new branch] ngimel/gather_grid_release -> origin/ngimel/gather_grid_release 2025-12-04T09:43:54.0465073Z * [new branch] ngimel/gg_new -> origin/ngimel/gg_new 2025-12-04T09:43:54.0466719Z * [new branch] ngimel/hostalloc -> origin/ngimel/hostalloc 2025-12-04T09:43:54.0468858Z * [new branch] ngimel/storage_id -> origin/ngimel/storage_id 2025-12-04T09:43:54.0470618Z * [new branch] nightly -> origin/nightly 2025-12-04T09:43:54.0473096Z * [new branch] nikitaved/addmm_1_rowcol_lt_path_check -> origin/nikitaved/addmm_1_rowcol_lt_path_check 2025-12-04T09:43:54.0474802Z * [new branch] nikitaved/addmm_epilogue_fusions_2d_bias -> origin/nikitaved/addmm_epilogue_fusions_2d_bias 2025-12-04T09:43:54.0476599Z * [new branch] nikitaved/addmm_epilogue_fusions_inductor -> origin/nikitaved/addmm_epilogue_fusions_inductor 2025-12-04T09:43:54.0478614Z * [new branch] nikitaved/addmm_epilogue_fusions_scratch -> origin/nikitaved/addmm_epilogue_fusions_scratch 2025-12-04T09:43:54.0480805Z * [new branch] nikitaved/grad_addmm_epilogue_fusions -> origin/nikitaved/grad_addmm_epilogue_fusions 2025-12-04T09:43:54.0482963Z * [new branch] nikitaved/simpler_can_use_32bit_index -> origin/nikitaved/simpler_can_use_32bit_index 2025-12-04T09:43:54.0484834Z * [new branch] nikitaved/test -> origin/nikitaved/test 2025-12-04T09:43:54.0486951Z * [new branch] nmacchioni-perf-test-async-autotune -> origin/nmacchioni-perf-test-async-autotune 2025-12-04T09:43:54.0488661Z * [new branch] no_distributed_log_spew -> origin/no_distributed_log_spew 2025-12-04T09:43:54.0490523Z * [new branch] nofun-hack -> origin/nofun-hack 2025-12-04T09:43:54.0492281Z * [new branch] norm_bench -> origin/norm_bench 2025-12-04T09:43:54.0494656Z * [new branch] nullplay/fuse_matmul -> origin/nullplay/fuse_matmul 2025-12-04T09:43:54.0496847Z * [new branch] nullplay_fuse_matmul -> origin/nullplay_fuse_matmul 2025-12-04T09:43:54.0498663Z * [new branch] optimizer_test -> origin/optimizer_test 2025-12-04T09:43:54.0501541Z * [new branch] orig/release/1.10 -> origin/orig/release/1.10 2025-12-04T09:43:54.0503417Z * [new branch] orig/release/1.11 -> origin/orig/release/1.11 2025-12-04T09:43:54.0505421Z * [new branch] orig/release/1.12 -> origin/orig/release/1.12 2025-12-04T09:43:54.0507336Z * [new branch] orig/release/1.13 -> origin/orig/release/1.13 2025-12-04T09:43:54.0509132Z * [new branch] orig/release/1.6 -> origin/orig/release/1.6 2025-12-04T09:43:54.0511002Z * [new branch] orig/release/1.7 -> origin/orig/release/1.7 2025-12-04T09:43:54.0512905Z * [new branch] orig/release/1.8 -> origin/orig/release/1.8 2025-12-04T09:43:54.0514657Z * [new branch] orig/release/1.9 -> origin/orig/release/1.9 2025-12-04T09:43:54.0516428Z * [new branch] orig/release/2.0 -> origin/orig/release/2.0 2025-12-04T09:43:54.0518564Z * [new branch] orig/release/2.1 -> origin/orig/release/2.1 2025-12-04T09:43:54.0520357Z * [new branch] orig/release/2.2 -> origin/orig/release/2.2 2025-12-04T09:43:54.0522195Z * [new branch] orig/release/2.3 -> origin/orig/release/2.3 2025-12-04T09:43:54.0523850Z * [new branch] orig/release/2.4 -> origin/orig/release/2.4 2025-12-04T09:43:54.0525550Z * [new branch] orig/release/2.5 -> origin/orig/release/2.5 2025-12-04T09:43:54.0527255Z * [new branch] orig/release/2.6 -> origin/orig/release/2.6 2025-12-04T09:43:54.0529315Z * [new branch] orig/release/2.7 -> origin/orig/release/2.7 2025-12-04T09:43:54.0531576Z * [new branch] orig/release/2.8 -> origin/orig/release/2.8 2025-12-04T09:43:54.0533396Z * [new branch] orig/release/2.9 -> origin/orig/release/2.9 2025-12-04T09:43:54.0537225Z * [new branch] origin/gh/fxdawnn/1/base -> origin/origin/gh/fxdawnn/1/base 2025-12-04T09:43:54.0538904Z * [new branch] origin/gh/fxdawnn/1/orig -> origin/origin/gh/fxdawnn/1/orig 2025-12-04T09:43:54.0542147Z * [new branch] origin/gh/zpcore/14/orig -> origin/origin/gh/zpcore/14/orig 2025-12-04T09:43:54.0544293Z * [new branch] oulgen-patch-1 -> origin/oulgen-patch-1 2025-12-04T09:43:54.0546193Z * [new branch] oulgen-patch-2 -> origin/oulgen-patch-2 2025-12-04T09:43:54.0548113Z * [new branch] oulgen-patch-3 -> origin/oulgen-patch-3 2025-12-04T09:43:54.0549963Z * [new branch] oulgen-patch-4 -> origin/oulgen-patch-4 2025-12-04T09:43:54.0551806Z * [new branch] padded-tensor -> origin/padded-tensor 2025-12-04T09:43:54.0553670Z * [new branch] pca2 -> origin/pca2 2025-12-04T09:43:54.0555759Z * [new branch] per_channel_backup -> origin/per_channel_backup 2025-12-04T09:43:54.0557657Z * [new branch] perf_ops -> origin/perf_ops 2025-12-04T09:43:54.0559432Z * [new branch] perf_ops_2_9 -> origin/perf_ops_2_9 2025-12-04T09:43:54.0561304Z * [new branch] pianpwk-patch-1 -> origin/pianpwk-patch-1 2025-12-04T09:43:54.0563755Z * [new branch] pianpwk/__draft_debug_mode -> origin/pianpwk/__draft_debug_mode 2025-12-04T09:43:54.0565485Z * [new branch] pianpwk/_debug_mode_for_triton_draft -> origin/pianpwk/_debug_mode_for_triton_draft 2025-12-04T09:43:54.0567125Z * [new branch] pianpwk/_debug_nn_module_compile -> origin/pianpwk/_debug_nn_module_compile 2025-12-04T09:43:54.0568729Z * [new branch] pianpwk/_draft_triton_11_3 -> origin/pianpwk/_draft_triton_11_3 2025-12-04T09:43:54.0570428Z * [new branch] pianpwk/_manual_bucket_draft -> origin/pianpwk/_manual_bucket_draft 2025-12-04T09:43:54.0572400Z * [new branch] pianpwk/_profile_w_dispatch_keys -> origin/pianpwk/_profile_w_dispatch_keys 2025-12-04T09:43:54.0574555Z * [new branch] pianpwk/_super_draft_debug_mode -> origin/pianpwk/_super_draft_debug_mode 2025-12-04T09:43:54.0576405Z * [new branch] pianpwk/_unbacked_local_shard_size -> origin/pianpwk/_unbacked_local_shard_size 2025-12-04T09:43:54.0578347Z * [new branch] pianpwk/anomaly_tb -> origin/pianpwk/anomaly_tb 2025-12-04T09:43:54.0580095Z * [new branch] pianpwk/auto_fx_annotate -> origin/pianpwk/auto_fx_annotate 2025-12-04T09:43:54.0581960Z * [new branch] pianpwk/backed_size_oblivious_export -> origin/pianpwk/backed_size_oblivious_export 2025-12-04T09:43:54.0583659Z * [new branch] pianpwk/bert_dynamic_perf -> origin/pianpwk/bert_dynamic_perf 2025-12-04T09:43:54.0585619Z * [new branch] pianpwk/debug_fwd_stack_traces -> origin/pianpwk/debug_fwd_stack_traces 2025-12-04T09:43:54.0587367Z * [new branch] pianpwk/debug_hash_tensor -> origin/pianpwk/debug_hash_tensor 2025-12-04T09:43:54.0589201Z * [new branch] pianpwk/debug_mode_annotate -> origin/pianpwk/debug_mode_annotate 2025-12-04T09:43:54.0590918Z * [new branch] pianpwk/debug_mode_defaults -> origin/pianpwk/debug_mode_defaults 2025-12-04T09:43:54.0592746Z * [new branch] pianpwk/debug_mode_hacks -> origin/pianpwk/debug_mode_hacks 2025-12-04T09:43:54.0594500Z * [new branch] pianpwk/debug_mode_opcall_refactor -> origin/pianpwk/debug_mode_opcall_refactor 2025-12-04T09:43:54.0596259Z * [new branch] pianpwk/debug_mode_show_ids -> origin/pianpwk/debug_mode_show_ids 2025-12-04T09:43:54.0598007Z * [new branch] pianpwk/debug_mode_triton -> origin/pianpwk/debug_mode_triton 2025-12-04T09:43:54.0600329Z * [new branch] pianpwk/debug_show_stack_trace -> origin/pianpwk/debug_show_stack_trace 2025-12-04T09:43:54.0602160Z * [new branch] pianpwk/debug_wait_on_collective -> origin/pianpwk/debug_wait_on_collective 2025-12-04T09:43:54.0603981Z * [new branch] pianpwk/debugmode_compile_tf -> origin/pianpwk/debugmode_compile_tf 2025-12-04T09:43:54.0605922Z * [new branch] pianpwk/dispatch_key_debugging_for_debug -> origin/pianpwk/dispatch_key_debugging_for_debug 2025-12-04T09:43:54.0607578Z * [new branch] pianpwk/draft_debug_mode_tfcompile -> origin/pianpwk/draft_debug_mode_tfcompile 2025-12-04T09:43:54.0609282Z * [new branch] pianpwk/draft_multikernel_nn -> origin/pianpwk/draft_multikernel_nn 2025-12-04T09:43:54.0611116Z * [new branch] pianpwk/draft_multikernel_status_10_5 -> origin/pianpwk/draft_multikernel_status_10_5 2025-12-04T09:43:54.0613027Z * [new branch] pianpwk/dtensor_custom_chunk -> origin/pianpwk/dtensor_custom_chunk 2025-12-04T09:43:54.0614824Z * [new branch] pianpwk/dtensor_unbacked_keypath -> origin/pianpwk/dtensor_unbacked_keypath 2025-12-04T09:43:54.0616699Z * [new branch] pianpwk/event_list_tree -> origin/pianpwk/event_list_tree 2025-12-04T09:43:54.0618427Z * [new branch] pianpwk/false_numel_refs -> origin/pianpwk/false_numel_refs 2025-12-04T09:43:54.0620630Z * [new branch] pianpwk/maybe_guard_rel -> origin/pianpwk/maybe_guard_rel 2025-12-04T09:43:54.0622606Z * [new branch] pianpwk/multikernel_hints_draft -> origin/pianpwk/multikernel_hints_draft 2025-12-04T09:43:54.0624456Z * [new branch] pianpwk/no_size_oblivious_slice_scat -> origin/pianpwk/no_size_oblivious_slice_scat 2025-12-04T09:43:54.0626304Z * [new branch] pianpwk/oblivious_reshape_view_better -> origin/pianpwk/oblivious_reshape_view_better 2025-12-04T09:43:54.0628002Z * [new branch] pianpwk/pre_forward_hook -> origin/pianpwk/pre_forward_hook 2025-12-04T09:43:54.0629781Z * [new branch] pianpwk/skip_python_keys_alternate -> origin/pianpwk/skip_python_keys_alternate 2025-12-04T09:43:54.0631498Z * [new branch] pianpwk/skip_python_keys_in_guards -> origin/pianpwk/skip_python_keys_in_guards 2025-12-04T09:43:54.0633227Z * [new branch] pianpwk/sym_tokens_draft -> origin/pianpwk/sym_tokens_draft 2025-12-04T09:43:54.0635046Z * [new branch] pianpwk/symint_one_hot -> origin/pianpwk/symint_one_hot 2025-12-04T09:43:54.0637708Z * [new branch] pianpwk/test_pointwise_guard_or_false -> origin/pianpwk/test_pointwise_guard_or_false 2025-12-04T09:43:54.0639311Z * [new branch] pianpwk/totally_draft_sym_wrap -> origin/pianpwk/totally_draft_sym_wrap 2025-12-04T09:43:54.0640970Z * [new branch] pianpwk/try_dumb_stuff -> origin/pianpwk/try_dumb_stuff 2025-12-04T09:43:54.0642764Z * [new branch] pianpwk/try_dumb_stuff_2 -> origin/pianpwk/try_dumb_stuff_2 2025-12-04T09:43:54.0644519Z * [new branch] pianpwk/unbacked_dtensor_mm -> origin/pianpwk/unbacked_dtensor_mm 2025-12-04T09:43:54.0646304Z * [new branch] pianpwk/unbacked_tracing_12_2 -> origin/pianpwk/unbacked_tracing_12_2 2025-12-04T09:43:54.0648045Z * [new branch] pianpwk/user_symints -> origin/pianpwk/user_symints 2025-12-04T09:43:54.0649904Z * [new branch] pianpwk/wan21_reshape -> origin/pianpwk/wan21_reshape 2025-12-04T09:43:54.0652445Z * [new branch] piz/fix_partial_backward_1112 -> origin/piz/fix_partial_backward_1112 2025-12-04T09:43:54.0654549Z * [new branch] piz/prop_cache_clean -> origin/piz/prop_cache_clean 2025-12-04T09:43:54.0656280Z * [new branch] pool-separate -> origin/pool-separate 2025-12-04T09:43:54.0658134Z * [new branch] pr-156087 -> origin/pr-156087 2025-12-04T09:43:54.0660440Z * [new branch] pr/131860 -> origin/pr/131860 2025-12-04T09:43:54.0662221Z * [new branch] predispatch_to -> origin/predispatch_to 2025-12-04T09:43:54.0664077Z * [new branch] protect-c17 -> origin/protect-c17 2025-12-04T09:43:54.0666086Z * [new branch] pt-opt-cuda3 -> origin/pt-opt-cuda3 2025-12-04T09:43:54.0668421Z * [new branch] python_compiled_autograd -> origin/python_compiled_autograd 2025-12-04T09:43:54.0671039Z * [new branch] q1l1/fix_device_moved_constant_type_unknown -> origin/q1l1/fix_device_moved_constant_type_unknown 2025-12-04T09:43:54.0672796Z * [new branch] q1l1/fix_wrong_default_type_for_kernel_call_args -> origin/q1l1/fix_wrong_default_type_for_kernel_call_args 2025-12-04T09:43:54.0675392Z * [new branch] qchip/export-D54134695 -> origin/qchip/export-D54134695 2025-12-04T09:43:54.0677360Z * [new branch] quote-pytest_cache -> origin/quote-pytest_cache 2025-12-04T09:43:54.0679799Z * [new branch] reland-accgrad-stream-warn -> origin/reland-accgrad-stream-warn 2025-12-04T09:43:54.0682737Z * [new branch] release/1.10 -> origin/release/1.10 2025-12-04T09:43:54.0684459Z * [new branch] release/1.11 -> origin/release/1.11 2025-12-04T09:43:54.0686159Z * [new branch] release/1.12 -> origin/release/1.12 2025-12-04T09:43:54.0687916Z * [new branch] release/1.13 -> origin/release/1.13 2025-12-04T09:43:54.0689626Z * [new branch] release/1.4 -> origin/release/1.4 2025-12-04T09:43:54.0691236Z * [new branch] release/1.4.1 -> origin/release/1.4.1 2025-12-04T09:43:54.0692991Z * [new branch] release/1.5 -> origin/release/1.5 2025-12-04T09:43:54.0694757Z * [new branch] release/1.6 -> origin/release/1.6 2025-12-04T09:43:54.0696625Z * [new branch] release/1.7 -> origin/release/1.7 2025-12-04T09:43:54.0698507Z * [new branch] release/1.8 -> origin/release/1.8 2025-12-04T09:43:54.0700287Z * [new branch] release/1.9 -> origin/release/1.9 2025-12-04T09:43:54.0702188Z * [new branch] release/2.0 -> origin/release/2.0 2025-12-04T09:43:54.0704265Z * [new branch] release/2.1 -> origin/release/2.1 2025-12-04T09:43:54.0706170Z * [new branch] release/2.2 -> origin/release/2.2 2025-12-04T09:43:54.0708205Z * [new branch] release/2.3 -> origin/release/2.3 2025-12-04T09:43:54.0710439Z * [new branch] release/2.4 -> origin/release/2.4 2025-12-04T09:43:54.0712736Z * [new branch] release/2.5 -> origin/release/2.5 2025-12-04T09:43:54.0714615Z * [new branch] release/2.6 -> origin/release/2.6 2025-12-04T09:43:54.0716781Z * [new branch] release/2.7 -> origin/release/2.7 2025-12-04T09:43:54.0719080Z * [new branch] release/2.8 -> origin/release/2.8 2025-12-04T09:43:54.0721071Z * [new branch] release/2.9 -> origin/release/2.9 2025-12-04T09:43:54.0722823Z * [new branch] release_notes -> origin/release_notes 2025-12-04T09:43:54.0724578Z * [new branch] remove_pyinterpreter -> origin/remove_pyinterpreter 2025-12-04T09:43:54.0726580Z * [new branch] replace-pytorch-labs-20250812-195836 -> origin/replace-pytorch-labs-20250812-195836 2025-12-04T09:43:54.0728257Z * [new branch] replace-pytorch-labs-20250812-200248 -> origin/replace-pytorch-labs-20250812-200248 2025-12-04T09:43:54.0730045Z * [new branch] replace-pytorch-labs-20250812-200324 -> origin/replace-pytorch-labs-20250812-200324 2025-12-04T09:43:54.0731799Z * [new branch] replace-pytorch-labs-20250812-204020 -> origin/replace-pytorch-labs-20250812-204020 2025-12-04T09:43:54.0735266Z * [new branch] revert-131069-gh/krzysztofjordan/1/head -> origin/revert-131069-gh/krzysztofjordan/1/head 2025-12-04T09:43:54.0738550Z * [new branch] revert-131469-gh/andrewor14/51/head -> origin/revert-131469-gh/andrewor14/51/head 2025-12-04T09:43:54.0741895Z * [new branch] revert-152361-gh/fadara01/1/head -> origin/revert-152361-gh/fadara01/1/head 2025-12-04T09:43:54.0745350Z * [new branch] revert-156870-gh/skarjala/3/head -> origin/revert-156870-gh/skarjala/3/head 2025-12-04T09:43:54.0747415Z * [new branch] revert-157914-cherry-pick-157503-by-pytorch_bot_bot_ -> origin/revert-157914-cherry-pick-157503-by-pytorch_bot_bot_ 2025-12-04T09:43:54.0749194Z * [new branch] revert-hoo-invoke-subgraph -> origin/revert-hoo-invoke-subgraph 2025-12-04T09:43:54.0750983Z * [new branch] revert_always_build_distributed -> origin/revert_always_build_distributed 2025-12-04T09:43:54.0752754Z * [new branch] rms_norm_patch -> origin/rms_norm_patch 2025-12-04T09:43:54.0755710Z * [new branch] ruisi/fix_all_to_all_estimation -> origin/ruisi/fix_all_to_all_estimation 2025-12-04T09:43:54.0757336Z * [new branch] ruisi/fix_comm_estimation -> origin/ruisi/fix_comm_estimation 2025-12-04T09:43:54.0758971Z * [new branch] ruisi/fix_dynamic_shape_estimation -> origin/ruisi/fix_dynamic_shape_estimation 2025-12-04T09:43:54.0760614Z * [new branch] ruisi/fix_llama3_autobucketing -> origin/ruisi/fix_llama3_autobucketing 2025-12-04T09:43:54.0762596Z * [new branch] ruisi/fix_manual_bucketing_ep_pass -> origin/ruisi/fix_manual_bucketing_ep_pass 2025-12-04T09:43:54.0764744Z * [new branch] ruisi/manual_bucket_pass -> origin/ruisi/manual_bucket_pass 2025-12-04T09:43:54.0767291Z * [new branch] ryanguo99/cleanup-dynamo-expected-failures -> origin/ryanguo99/cleanup-dynamo-expected-failures 2025-12-04T09:43:54.0768838Z * [new branch] ryanguo99/fix-closure-var -> origin/ryanguo99/fix-closure-var 2025-12-04T09:43:54.0771194Z * [new branch] rzou/faketensor_bench -> origin/rzou/faketensor_bench 2025-12-04T09:43:54.0772771Z * [new branch] rzou/njt -> origin/rzou/njt 2025-12-04T09:43:54.0774512Z * [new branch] rzou/pca -> origin/rzou/pca 2025-12-04T09:43:54.0776208Z * [new branch] rzou/realprop -> origin/rzou/realprop 2025-12-04T09:43:54.0778084Z * [new branch] samplevllm -> origin/samplevllm 2025-12-04T09:43:54.0781131Z * [new branch] sanchitintel/weird_thing_with_test_cpu_select_algorithm -> origin/sanchitintel/weird_thing_with_test_cpu_select_algorithm 2025-12-04T09:43:54.0782809Z * [new branch] sapling-pr-archive-SS-JIA -> origin/sapling-pr-archive-SS-JIA 2025-12-04T09:43:54.0784899Z * [new branch] sapling-pr-archive-tushar00jain -> origin/sapling-pr-archive-tushar00jain 2025-12-04T09:43:54.0786603Z * [new branch] save -> origin/save 2025-12-04T09:43:54.0788463Z * [new branch] scaled_mm -> origin/scaled_mm 2025-12-04T09:43:54.0790309Z * [new branch] scan_attempt -> origin/scan_attempt 2025-12-04T09:43:54.0792864Z * [new branch] sdym/2.5.1 -> origin/sdym/2.5.1 2025-12-04T09:43:54.0794735Z * [new branch] sekyondaMeta-dynamoconfig-fix -> origin/sekyondaMeta-dynamoconfig-fix 2025-12-04T09:43:54.0797004Z * [new branch] shengf/fx-xform-perf -> origin/shengf/fx-xform-perf 2025-12-04T09:43:54.0798872Z * [new branch] shoumikhin-patch-1 -> origin/shoumikhin-patch-1 2025-12-04T09:43:54.0800583Z * [new branch] solve-accuracy-fix -> origin/solve-accuracy-fix 2025-12-04T09:43:54.0802594Z * [new branch] some_rocm_inductor_skips -> origin/some_rocm_inductor_skips 2025-12-04T09:43:54.0804980Z * [new branch] soulitzer/stash-tls-ac -> origin/soulitzer/stash-tls-ac 2025-12-04T09:43:54.0807294Z * [new branch] sparse-mm-bf16-support -> origin/sparse-mm-bf16-support 2025-12-04T09:43:54.0809195Z * [new branch] starterTaskUpdate -> origin/starterTaskUpdate 2025-12-04T09:43:54.0810978Z * [new branch] suo -> origin/suo 2025-12-04T09:43:54.0812892Z * [new branch] sve-poc -> origin/sve-poc 2025-12-04T09:43:54.0814788Z * [new branch] switch-bn -> origin/switch-bn 2025-12-04T09:43:54.0816682Z * [new branch] sy_annotation_in_autograd_hop -> origin/sy_annotation_in_autograd_hop 2025-12-04T09:43:54.0818445Z * [new branch] sy_aot_eager_record -> origin/sy_aot_eager_record 2025-12-04T09:43:54.0820283Z * [new branch] sy_custom_bucketing -> origin/sy_custom_bucketing 2025-12-04T09:43:54.0822438Z * [new branch] sy_debug_mode_test -> origin/sy_debug_mode_test 2025-12-04T09:43:54.0824208Z * [new branch] sy_deserialize -> origin/sy_deserialize 2025-12-04T09:43:54.0826002Z * [new branch] sy_dump_gm_code -> origin/sy_dump_gm_code 2025-12-04T09:43:54.0827810Z * [new branch] sy_exp -> origin/sy_exp 2025-12-04T09:43:54.0829691Z * [new branch] sy_export_annotation -> origin/sy_export_annotation 2025-12-04T09:43:54.0831556Z * [new branch] sy_invoke_subgraph -> origin/sy_invoke_subgraph 2025-12-04T09:43:54.0833379Z * [new branch] sy_kernel_bw_name -> origin/sy_kernel_bw_name 2025-12-04T09:43:54.0835116Z * [new branch] sy_multi_arch -> origin/sy_multi_arch 2025-12-04T09:43:54.0836934Z * [new branch] sy_nn_module_stack -> origin/sy_nn_module_stack 2025-12-04T09:43:54.0838772Z * [new branch] sy_original_dtensor -> origin/sy_original_dtensor 2025-12-04T09:43:54.0840597Z * [new branch] sy_profiler_cia -> origin/sy_profiler_cia 2025-12-04T09:43:54.0842491Z * [new branch] symm_mem_sync -> origin/symm_mem_sync 2025-12-04T09:43:54.0844880Z * [new branch] sympy-bottleneck-repro -> origin/sympy-bottleneck-repro 2025-12-04T09:43:54.0846805Z * [new branch] tensordict_integration -> origin/tensordict_integration 2025-12-04T09:43:54.0848737Z * [new branch] test-move-conda-builds -> origin/test-move-conda-builds 2025-12-04T09:43:54.0850402Z * [new branch] test-old -> origin/test-old 2025-12-04T09:43:54.0852866Z * [new branch] test/bmm_heur -> origin/test/bmm_heur 2025-12-04T09:43:54.0855289Z * [new branch] tianren/customOp_autotune_fix -> origin/tianren/customOp_autotune_fix 2025-12-04T09:43:54.0857070Z * [new branch] tianren/customOp_enable_max_autotune -> origin/tianren/customOp_enable_max_autotune 2025-12-04T09:43:54.0858688Z * [new branch] tianren/customOp_fusion -> origin/tianren/customOp_fusion 2025-12-04T09:43:54.0860313Z * [new branch] tianren/customop_collectiveop_benchmark -> origin/tianren/customop_collectiveop_benchmark 2025-12-04T09:43:54.0862401Z * [new branch] tianren/customop_collectiveop_benchmark_fix -> origin/tianren/customop_collectiveop_benchmark_fix 2025-12-04T09:43:54.0864656Z * [new branch] tianren/customop_dynamic_config -> origin/tianren/customop_dynamic_config 2025-12-04T09:43:54.0866423Z * [new branch] tianren/dynamic_range_input -> origin/tianren/dynamic_range_input 2025-12-04T09:43:54.0868243Z * [new branch] tianren/dynamic_range_input_fix -> origin/tianren/dynamic_range_input_fix 2025-12-04T09:43:54.0870009Z * [new branch] tianren/dynamic_range_input_merge -> origin/tianren/dynamic_range_input_merge 2025-12-04T09:43:54.0871696Z * [new branch] tianren/flex_paged_attn_fix_temp -> origin/tianren/flex_paged_attn_fix_temp 2025-12-04T09:43:54.0873516Z * [new branch] tianren/fx_codegen_dump -> origin/tianren/fx_codegen_dump 2025-12-04T09:43:54.0875274Z * [new branch] tianren/symmetric_memory -> origin/tianren/symmetric_memory 2025-12-04T09:43:54.0876971Z * [new branch] tianren/test -> origin/tianren/test 2025-12-04T09:43:54.0879806Z * [new branch] tidy_performance_cyy -> origin/tidy_performance_cyy 2025-12-04T09:43:54.0881737Z * [new branch] tmp -> origin/tmp 2025-12-04T09:43:54.0883702Z * [new branch] torchtitan_ep -> origin/torchtitan_ep 2025-12-04T09:43:54.0885571Z * [new branch] torchtitan_integration -> origin/torchtitan_integration 2025-12-04T09:43:54.0887579Z * [new branch] trace_fsdp_torchtune_lora -> origin/trace_fsdp_torchtune_lora 2025-12-04T09:43:54.0889286Z * [new branch] traceable_fsdp_unit_tests -> origin/traceable_fsdp_unit_tests 2025-12-04T09:43:54.0891027Z * [new branch] tree_loop_vec_base -> origin/tree_loop_vec_base 2025-12-04T09:43:54.0892903Z * [new branch] triton_kernel -> origin/triton_kernel 2025-12-04T09:43:54.0894800Z * [new branch] tt_pkg_1908 -> origin/tt_pkg_1908 2025-12-04T09:43:54.0896657Z * [new branch] type_dec -> origin/type_dec 2025-12-04T09:43:54.0898477Z * [new branch] udate-sphinx-dependancies -> origin/udate-sphinx-dependancies 2025-12-04T09:43:54.0900929Z * [new branch] update-audio-commit-hash/17630256502-1803-1 -> origin/update-audio-commit-hash/17630256502-1803-1 2025-12-04T09:43:54.0902666Z * [new branch] update-audio-commit-hash/19087141161-1916-1 -> origin/update-audio-commit-hash/19087141161-1916-1 2025-12-04T09:43:54.0904483Z * [new branch] update-audio-commit-hash/19250643381-1929-1 -> origin/update-audio-commit-hash/19250643381-1929-1 2025-12-04T09:43:54.0906140Z * [new branch] update-audio-commit-hash/19397724337-1935-1 -> origin/update-audio-commit-hash/19397724337-1935-1 2025-12-04T09:43:54.0908103Z * [new branch] update-audio-commit-hash/19555670148-1941-1 -> origin/update-audio-commit-hash/19555670148-1941-1 2025-12-04T09:43:54.0910082Z * [new branch] update-audio-commit-hash/19750627930-1946-1 -> origin/update-audio-commit-hash/19750627930-1946-1 2025-12-04T09:43:54.0912425Z * [new branch] update-triton-commit-hash/13663274526-1487-2 -> origin/update-triton-commit-hash/13663274526-1487-2 2025-12-04T09:43:54.0914889Z * [new branch] update-vision-commit-hash/19087141161-1916-1 -> origin/update-vision-commit-hash/19087141161-1916-1 2025-12-04T09:43:54.0916580Z * [new branch] update-vision-commit-hash/19184897099-1925-1 -> origin/update-vision-commit-hash/19184897099-1925-1 2025-12-04T09:43:54.0918186Z * [new branch] update-vision-commit-hash/19250643381-1929-1 -> origin/update-vision-commit-hash/19250643381-1929-1 2025-12-04T09:43:54.0919948Z * [new branch] update-vision-commit-hash/19381328640-1934-1 -> origin/update-vision-commit-hash/19381328640-1934-1 2025-12-04T09:43:54.0921708Z * [new branch] update-vision-commit-hash/19485237164-1938-1 -> origin/update-vision-commit-hash/19485237164-1938-1 2025-12-04T09:43:54.0924185Z * [new branch] update-vllm-commit-hash/18451675449-1879-1 -> origin/update-vllm-commit-hash/18451675449-1879-1 2025-12-04T09:43:54.0925910Z * [new branch] update-vllm-dockerfile -> origin/update-vllm-dockerfile 2025-12-04T09:43:54.0928857Z * [new branch] update-xla-commit-hash/19224287370-211-1 -> origin/update-xla-commit-hash/19224287370-211-1 2025-12-04T09:43:54.0930474Z * [new branch] update-xla-commit-hash/19422028566-212-1 -> origin/update-xla-commit-hash/19422028566-212-1 2025-12-04T09:43:54.0932161Z * [new branch] update-xla-commit-hash/19626841311-213-1 -> origin/update-xla-commit-hash/19626841311-213-1 2025-12-04T09:43:54.0934017Z * [new branch] update_docs_torch_multinomial_issue#125388 -> origin/update_docs_torch_multinomial_issue#125388 2025-12-04T09:43:54.0935816Z * [new branch] update_operator_readme -> origin/update_operator_readme 2025-12-04T09:43:54.0937806Z * [new branch] update_slow_tests_1722488736 -> origin/update_slow_tests_1722488736 2025-12-04T09:43:54.0939641Z * [new branch] update_slow_tests_1722879173 -> origin/update_slow_tests_1722879173 2025-12-04T09:43:54.0941414Z * [new branch] update_slow_tests_1762155677 -> origin/update_slow_tests_1762155677 2025-12-04T09:43:54.0943334Z * [new branch] update_slow_tests_1763365283 -> origin/update_slow_tests_1763365283 2025-12-04T09:43:54.0945238Z * [new branch] update_submodule_FBGEMM -> origin/update_submodule_FBGEMM 2025-12-04T09:43:54.0947081Z * [new branch] update_submodule_kineto -> origin/update_submodule_kineto 2025-12-04T09:43:54.0948990Z * [new branch] update_submodule_tensorpipe -> origin/update_submodule_tensorpipe 2025-12-04T09:43:54.0950879Z * [new branch] upload-tests-for-autorevert -> origin/upload-tests-for-autorevert 2025-12-04T09:43:54.0952818Z * [new branch] v0.1.2 -> origin/v0.1.2 2025-12-04T09:43:54.0954695Z * [new branch] v1.0.1 -> origin/v1.0.1 2025-12-04T09:43:54.0956591Z * [new branch] v1.0.3 -> origin/v1.0.3 2025-12-04T09:43:54.0958686Z * [new branch] v1.1.0 -> origin/v1.1.0 2025-12-04T09:43:54.0960787Z * [new branch] v1.2.0 -> origin/v1.2.0 2025-12-04T09:43:54.0962635Z * [new branch] v1.3.0 -> origin/v1.3.0 2025-12-04T09:43:54.0964560Z * [new branch] v1.3.1 -> origin/v1.3.1 2025-12-04T09:43:54.0966454Z * [new branch] validate_fn -> origin/validate_fn 2025-12-04T09:43:54.0968449Z * [new branch] validations_2.6 -> origin/validations_2.6 2025-12-04T09:43:54.0970258Z * [new branch] validations_2.8 -> origin/validations_2.8 2025-12-04T09:43:54.0972290Z * [new branch] varlen-api -> origin/varlen-api 2025-12-04T09:43:54.0974120Z * [new branch] varlen-api-backup -> origin/varlen-api-backup 2025-12-04T09:43:54.0976034Z * [new branch] varlen_batch_invariance -> origin/varlen_batch_invariance 2025-12-04T09:43:54.0978364Z * [new branch] viable/strict -> origin/viable/strict 2025-12-04T09:43:54.0980916Z * [new branch] vishal9-team/dtensor_parallelism_toy -> origin/vishal9-team/dtensor_parallelism_toy 2025-12-04T09:43:54.0982632Z * [new branch] vllmbuildci -> origin/vllmbuildci 2025-12-04T09:43:54.0984679Z * [new branch] vllmpin -> origin/vllmpin 2025-12-04T09:43:54.0986558Z * [new branch] vscode-recommend-pyrefly -> origin/vscode-recommend-pyrefly 2025-12-04T09:43:54.0988925Z * [new branch] wdvr-patch-1 -> origin/wdvr-patch-1 2025-12-04T09:43:54.0991290Z * [new branch] wdvr/iss_145259 -> origin/wdvr/iss_145259 2025-12-04T09:43:54.0993629Z * [new branch] whc/pei -> origin/whc/pei 2025-12-04T09:43:54.0995302Z * [new branch] whc/pp_fix -> origin/whc/pp_fix 2025-12-04T09:43:54.0997145Z * [new branch] whc/sharding -> origin/whc/sharding 2025-12-04T09:43:54.0998882Z * [new branch] whc/sharding2 -> origin/whc/sharding2 2025-12-04T09:43:54.1000567Z * [new branch] whc/uneven -> origin/whc/uneven 2025-12-04T09:43:54.1002441Z * [new branch] whc/uneven-merge -> origin/whc/uneven-merge 2025-12-04T09:43:54.1004244Z * [new branch] win_warnings -> origin/win_warnings 2025-12-04T09:43:54.1006052Z * [new branch] windows_libtorch_free -> origin/windows_libtorch_free 2025-12-04T09:43:54.1007897Z * [new branch] xmfan-war -> origin/xmfan-war 2025-12-04T09:43:54.1010211Z * [new branch] xmfan/ca_0516 -> origin/xmfan/ca_0516 2025-12-04T09:43:54.1011833Z * [new branch] xmfan/ca_1051b93192 -> origin/xmfan/ca_1051b93192 2025-12-04T09:43:54.1013714Z * [new branch] xmfan/ca_1a722f62c248391fc4a542e8851a5559aa356ae8 -> origin/xmfan/ca_1a722f62c248391fc4a542e8851a5559aa356ae8 2025-12-04T09:43:54.1015124Z * [new branch] xmfan/ca_5a2be192d1 -> origin/xmfan/ca_5a2be192d1 2025-12-04T09:43:54.1016706Z * [new branch] xmfan/ca_9d59b516e9 -> origin/xmfan/ca_9d59b516e9 2025-12-04T09:43:54.1018357Z * [new branch] xmfan/ca_apr8 -> origin/xmfan/ca_apr8 2025-12-04T09:43:54.1020072Z * [new branch] xmfan/ca_base -> origin/xmfan/ca_base 2025-12-04T09:43:54.1022609Z * [new branch] xmfan/ca_dynamic -> origin/xmfan/ca_dynamic 2025-12-04T09:43:54.1024890Z * [new branch] xmfan/ca_fix_dyn -> origin/xmfan/ca_fix_dyn 2025-12-04T09:43:54.1026737Z * [new branch] xmfan/ca_fix_lowering -> origin/xmfan/ca_fix_lowering 2025-12-04T09:43:54.1028524Z * [new branch] xmfan/ca_fix_polyfills -> origin/xmfan/ca_fix_polyfills 2025-12-04T09:43:54.1030218Z * [new branch] xmfan/ca_jan3 -> origin/xmfan/ca_jan3 2025-12-04T09:43:54.1031932Z * [new branch] xmfan/ca_jun18 -> origin/xmfan/ca_jun18 2025-12-04T09:43:54.1033808Z * [new branch] xmfan/ca_jun24 -> origin/xmfan/ca_jun24 2025-12-04T09:43:54.1035590Z * [new branch] xmfan/ca_nested -> origin/xmfan/ca_nested 2025-12-04T09:43:54.1037376Z * [new branch] xmfan/ca_overhead -> origin/xmfan/ca_overhead 2025-12-04T09:43:54.1039627Z * [new branch] xmfan/ca_overhead_0eba7e5451 -> origin/xmfan/ca_overhead_0eba7e5451 2025-12-04T09:43:54.1041274Z * [new branch] xmfan/cacu_jun18 -> origin/xmfan/cacu_jun18 2025-12-04T09:43:54.1043004Z * [new branch] xmfan/cacu_jun19 -> origin/xmfan/cacu_jun19 2025-12-04T09:43:54.1044765Z * [new branch] xmfan/cacu_jun4 -> origin/xmfan/cacu_jun4 2025-12-04T09:43:54.1046528Z * [new branch] xmfan/disable_duck_shape -> origin/xmfan/disable_duck_shape 2025-12-04T09:43:54.1048332Z * [new branch] xmfan/fca_cpp_node_passthrough -> origin/xmfan/fca_cpp_node_passthrough 2025-12-04T09:43:54.1050237Z * [new branch] xmfan/post_3945954741e2d37023c5d6954f9483008e0892f9 -> origin/xmfan/post_3945954741e2d37023c5d6954f9483008e0892f9 2025-12-04T09:43:54.1051970Z * [new branch] xmfan/pre_3945954741e2d37023c5d6954f9483008e0892f9 -> origin/xmfan/pre_3945954741e2d37023c5d6954f9483008e0892f9 2025-12-04T09:43:54.1053670Z * [new branch] xmfan/single_step -> origin/xmfan/single_step 2025-12-04T09:43:54.1055561Z * [new branch] xmfan/sth_0829 -> origin/xmfan/sth_0829 2025-12-04T09:43:54.1057331Z * [new branch] xmfan/test -> origin/xmfan/test 2025-12-04T09:43:54.1059699Z * [new branch] yguo/debug-0226-constexpr -> origin/yguo/debug-0226-constexpr 2025-12-04T09:43:54.1061297Z * [new branch] yguo/new_latest_changes -> origin/yguo/new_latest_changes 2025-12-04T09:43:54.1062963Z * [new branch] yguo/patch_constexpr_changes -> origin/yguo/patch_constexpr_changes 2025-12-04T09:43:54.1065462Z * [new branch] yiming/bootcamp -> origin/yiming/bootcamp 2025-12-04T09:43:54.1067085Z * [new branch] yiming/run_with_start_end_rng_hop -> origin/yiming/run_with_start_end_rng_hop 2025-12-04T09:43:54.1068870Z * [new branch] yolo-llama3 -> origin/yolo-llama3 2025-12-04T09:43:54.1071213Z * [new branch] zainr/canary-test -> origin/zainr/canary-test 2025-12-04T09:43:54.1073132Z * [new branch] zainr/cleanup-gh-runners -> origin/zainr/cleanup-gh-runners 2025-12-04T09:43:54.1074732Z * [new branch] zainr/pull-migration-c -> origin/zainr/pull-migration-c 2025-12-04T09:43:54.1076779Z * [new branch] zainr/test2 -> origin/zainr/test2 2025-12-04T09:43:54.1079659Z * [new branch] zasdfgbnm-patch-3 -> origin/zasdfgbnm-patch-3 2025-12-04T09:43:54.1081414Z * [new branch] zb2p -> origin/zb2p 2025-12-04T09:43:54.1083302Z * [new branch] zeros-and-scatter-part2 -> origin/zeros-and-scatter-part2 2025-12-04T09:43:54.1086145Z * [new branch] zhxchen17/ci/vllm_lora_oom -> origin/zhxchen17/ci/vllm_lora_oom 2025-12-04T09:43:54.1087907Z * [new branch] zhxchen17/ci/vllm_multimodal_oom -> origin/zhxchen17/ci/vllm_multimodal_oom 2025-12-04T09:43:54.1089630Z * [new branch] zhxchen17/ci/vllm_pin -> origin/zhxchen17/ci/vllm_pin 2025-12-04T09:43:54.1092024Z * [new branch] zhxchen17/dynamo/unsafe_drop_all_guards -> origin/zhxchen17/dynamo/unsafe_drop_all_guards 2025-12-04T09:43:54.1094302Z * [new branch] zhxchen17/export/call_override -> origin/zhxchen17/export/call_override 2025-12-04T09:43:54.1096031Z * [new branch] zhxchen17/export/codemod1 -> origin/zhxchen17/export/codemod1 2025-12-04T09:43:54.1097829Z * [new branch] zhxchen17/export/ctx_return -> origin/zhxchen17/export/ctx_return 2025-12-04T09:43:54.1099603Z * [new branch] zhxchen17/export/disable_side_effect_warn -> origin/zhxchen17/export/disable_side_effect_warn 2025-12-04T09:43:54.1101265Z * [new branch] zhxchen17/export/pytree_check -> origin/zhxchen17/export/pytree_check 2025-12-04T09:43:54.1103709Z * [new branch] zhxchen17/precompile/aoti -> origin/zhxchen17/precompile/aoti 2025-12-04T09:43:54.1105595Z * [new branch] zhxchen17/precompile/globals -> origin/zhxchen17/precompile/globals 2025-12-04T09:43:54.1114114Z * [new branch] zhxchen17/precompile/inductor_guards -> origin/zhxchen17/precompile/inductor_guards 2025-12-04T09:43:54.1114317Z * [new branch] zhxchen17/scratch/0 -> origin/zhxchen17/scratch/0 2025-12-04T09:43:54.1114568Z * [new branch] zhxchen17/torch_export_api_update -> origin/zhxchen17/torch_export_api_update 2025-12-04T09:43:54.1114738Z * [new branch] zhxhcen17/moodycamel -> origin/zhxhcen17/moodycamel 2025-12-04T09:43:54.1116202Z * [new branch] zxiiro/build-times -> origin/zxiiro/build-times 2025-12-04T09:43:54.1117969Z * [new branch] zxiiro/c7i.2xlarge -> origin/zxiiro/c7i.2xlarge 2025-12-04T09:43:54.1119738Z * [new branch] zxiiro/c7i.2xlarge.h100 -> origin/zxiiro/c7i.2xlarge.h100 2025-12-04T09:43:54.1121360Z * [new branch] zxiiro/main -> origin/zxiiro/main 2025-12-04T09:43:54.1123036Z * [new branch] zxiiro/risc64 -> origin/zxiiro/risc64 2025-12-04T09:43:54.1124822Z * [new branch] zxiiro/test-multicloud-arc -> origin/zxiiro/test-multicloud-arc 2025-12-04T09:43:54.1126489Z * [new tag] bc2caa7fdf006894eff7af936babde69ab5a40f8-huydhn-debug -> bc2caa7fdf006894eff7af936babde69ab5a40f8-huydhn-debug 2025-12-04T09:43:54.1127932Z * [new tag] ci/binaries/77164 -> ci/binaries/77164 2025-12-04T09:43:54.1129490Z * [new tag] ciflow/b200/115316 -> ciflow/b200/115316 2025-12-04T09:43:54.1130616Z * [new tag] ciflow/b200/160685 -> ciflow/b200/160685 2025-12-04T09:43:54.1131833Z * [new tag] ciflow/b200/161607 -> ciflow/b200/161607 2025-12-04T09:43:54.1132965Z * [new tag] ciflow/b200/161938 -> ciflow/b200/161938 2025-12-04T09:43:54.1134327Z * [new tag] ciflow/b200/167207 -> ciflow/b200/167207 2025-12-04T09:43:54.1135477Z * [new tag] ciflow/b200/167989 -> ciflow/b200/167989 2025-12-04T09:43:54.1136798Z * [new tag] ciflow/b200/168096 -> ciflow/b200/168096 2025-12-04T09:43:54.1138080Z * [new tag] ciflow/b200/168175 -> ciflow/b200/168175 2025-12-04T09:43:54.1139357Z * [new tag] ciflow/b200/168195 -> ciflow/b200/168195 2025-12-04T09:43:54.1140564Z * [new tag] ciflow/b200/169200 -> ciflow/b200/169200 2025-12-04T09:43:54.1141848Z * [new tag] ciflow/b200/169216 -> ciflow/b200/169216 2025-12-04T09:43:54.1143451Z * [new tag] ciflow/b200/169380 -> ciflow/b200/169380 2025-12-04T09:43:54.1145387Z * [new tag] ciflow/b200/169412 -> ciflow/b200/169412 2025-12-04T09:43:54.1146877Z * [new tag] ciflow/b200/169470 -> ciflow/b200/169470 2025-12-04T09:43:54.1148139Z * [new tag] ciflow/b200/169471 -> ciflow/b200/169471 2025-12-04T09:43:54.1149368Z * [new tag] ciflow/b200/169472 -> ciflow/b200/169472 2025-12-04T09:43:54.1150862Z * [new tag] ciflow/b200/169514 -> ciflow/b200/169514 2025-12-04T09:43:54.1152109Z * [new tag] ciflow/b200/169517 -> ciflow/b200/169517 2025-12-04T09:43:54.1153783Z * [new tag] ciflow/binaries/165922 -> ciflow/binaries/165922 2025-12-04T09:43:54.1154956Z * [new tag] ciflow/binaries/169510 -> ciflow/binaries/169510 2025-12-04T09:43:54.1156474Z * [new tag] ciflow/binaries_wheel/157994 -> ciflow/binaries_wheel/157994 2025-12-04T09:43:54.1157746Z * [new tag] ciflow/binaries_wheel/166829 -> ciflow/binaries_wheel/166829 2025-12-04T09:43:54.1158922Z * [new tag] ciflow/binaries_wheel/167972 -> ciflow/binaries_wheel/167972 2025-12-04T09:43:54.1160332Z * [new tag] ciflow/binaries_wheel/167981 -> ciflow/binaries_wheel/167981 2025-12-04T09:43:54.1161793Z * [new tag] ciflow/dynamo/167695 -> ciflow/dynamo/167695 2025-12-04T09:43:54.1162955Z * [new tag] ciflow/dynamo/168096 -> ciflow/dynamo/168096 2025-12-04T09:43:54.1164254Z * [new tag] ciflow/dynamo/169525 -> ciflow/dynamo/169525 2025-12-04T09:43:54.1165732Z * [new tag] ciflow/h100-cutlass-backend/161938 -> ciflow/h100-cutlass-backend/161938 2025-12-04T09:43:54.1166920Z * [new tag] ciflow/h100-cutlass-backend/161940 -> ciflow/h100-cutlass-backend/161940 2025-12-04T09:43:54.1168374Z * [new tag] ciflow/h100-distributed/168923 -> ciflow/h100-distributed/168923 2025-12-04T09:43:54.1169785Z * [new tag] ciflow/h100-symm-mem/167552 -> ciflow/h100-symm-mem/167552 2025-12-04T09:43:54.1170946Z * [new tag] ciflow/h100-symm-mem/168129 -> ciflow/h100-symm-mem/168129 2025-12-04T09:43:54.1172145Z * [new tag] ciflow/h100-symm-mem/168917 -> ciflow/h100-symm-mem/168917 2025-12-04T09:43:54.1173580Z * [new tag] ciflow/h100-symm-mem/169156 -> ciflow/h100-symm-mem/169156 2025-12-04T09:43:54.1175165Z * [new tag] ciflow/h100-symm-mem/169200 -> ciflow/h100-symm-mem/169200 2025-12-04T09:43:54.1176331Z * [new tag] ciflow/h100-symm-mem/169216 -> ciflow/h100-symm-mem/169216 2025-12-04T09:43:54.1177533Z * [new tag] ciflow/h100-symm-mem/169338 -> ciflow/h100-symm-mem/169338 2025-12-04T09:43:54.1179014Z * [new tag] ciflow/h100-symm-mem/169355 -> ciflow/h100-symm-mem/169355 2025-12-04T09:43:54.1180230Z * [new tag] ciflow/h100-symm-mem/169543 -> ciflow/h100-symm-mem/169543 2025-12-04T09:43:54.1181952Z * [new tag] ciflow/h100/115316 -> ciflow/h100/115316 2025-12-04T09:43:54.1183155Z * [new tag] ciflow/h100/160685 -> ciflow/h100/160685 2025-12-04T09:43:54.1184417Z * [new tag] ciflow/h100/160729 -> ciflow/h100/160729 2025-12-04T09:43:54.1185583Z * [new tag] ciflow/h100/161607 -> ciflow/h100/161607 2025-12-04T09:43:54.1186893Z * [new tag] ciflow/h100/161938 -> ciflow/h100/161938 2025-12-04T09:43:54.1188082Z * [new tag] ciflow/h100/167207 -> ciflow/h100/167207 2025-12-04T09:43:54.1189037Z * [new tag] ciflow/h100/167989 -> ciflow/h100/167989 2025-12-04T09:43:54.1190287Z * [new tag] ciflow/h100/168096 -> ciflow/h100/168096 2025-12-04T09:43:54.1191418Z * [new tag] ciflow/h100/168175 -> ciflow/h100/168175 2025-12-04T09:43:54.1192608Z * [new tag] ciflow/h100/168195 -> ciflow/h100/168195 2025-12-04T09:43:54.1193780Z * [new tag] ciflow/h100/168980 -> ciflow/h100/168980 2025-12-04T09:43:54.1195297Z * [new tag] ciflow/h100/169200 -> ciflow/h100/169200 2025-12-04T09:43:54.1196898Z * [new tag] ciflow/h100/169216 -> ciflow/h100/169216 2025-12-04T09:43:54.1198366Z * [new tag] ciflow/h100/169380 -> ciflow/h100/169380 2025-12-04T09:43:54.1199610Z * [new tag] ciflow/h100/169412 -> ciflow/h100/169412 2025-12-04T09:43:54.1200871Z * [new tag] ciflow/h100/169470 -> ciflow/h100/169470 2025-12-04T09:43:54.1202055Z * [new tag] ciflow/h100/169471 -> ciflow/h100/169471 2025-12-04T09:43:54.1203307Z * [new tag] ciflow/h100/169472 -> ciflow/h100/169472 2025-12-04T09:43:54.1204569Z * [new tag] ciflow/h100/169514 -> ciflow/h100/169514 2025-12-04T09:43:54.1205993Z * [new tag] ciflow/inductor-cu126/168096 -> ciflow/inductor-cu126/168096 2025-12-04T09:43:54.1207788Z * [new tag] ciflow/inductor-micro-benchmark-cpu-x86/168096 -> ciflow/inductor-micro-benchmark-cpu-x86/168096 2025-12-04T09:43:54.1209142Z * [new tag] ciflow/inductor-micro-benchmark/166165 -> ciflow/inductor-micro-benchmark/166165 2025-12-04T09:43:54.1210493Z * [new tag] ciflow/inductor-micro-benchmark/168096 -> ciflow/inductor-micro-benchmark/168096 2025-12-04T09:43:54.1211865Z * [new tag] ciflow/inductor-perf-compare/168096 -> ciflow/inductor-perf-compare/168096 2025-12-04T09:43:54.1213631Z * [new tag] ciflow/inductor-perf-test-nightly-rocm-mi300/168073 -> ciflow/inductor-perf-test-nightly-rocm-mi300/168073 2025-12-04T09:43:54.1214677Z * [new tag] ciflow/inductor-perf-test-nightly-rocm-mi300/168096 -> ciflow/inductor-perf-test-nightly-rocm-mi300/168096 2025-12-04T09:43:54.1216157Z * [new tag] ciflow/inductor-perf-test-nightly-rocm-mi300/169024 -> ciflow/inductor-perf-test-nightly-rocm-mi300/169024 2025-12-04T09:43:54.1217565Z * [new tag] ciflow/inductor-perf-test-nightly-rocm-mi355/169024 -> ciflow/inductor-perf-test-nightly-rocm-mi355/169024 2025-12-04T09:43:54.1218890Z * [new tag] ciflow/inductor-perf-test-nightly/168096 -> ciflow/inductor-perf-test-nightly/168096 2025-12-04T09:43:54.1220287Z * [new tag] ciflow/inductor-periodic/168096 -> ciflow/inductor-periodic/168096 2025-12-04T09:43:54.1221476Z * [new tag] ciflow/inductor-periodic/169024 -> ciflow/inductor-periodic/169024 2025-12-04T09:43:54.1222757Z * [new tag] ciflow/inductor-periodic/169425 -> ciflow/inductor-periodic/169425 2025-12-04T09:43:54.1224396Z * [new tag] ciflow/inductor-rocm-mi200/165545 -> ciflow/inductor-rocm-mi200/165545 2025-12-04T09:43:54.1225668Z * [new tag] ciflow/inductor-rocm-mi200/165997 -> ciflow/inductor-rocm-mi200/165997 2025-12-04T09:43:54.1226839Z * [new tag] ciflow/inductor-rocm-mi200/168096 -> ciflow/inductor-rocm-mi200/168096 2025-12-04T09:43:54.1228200Z * [new tag] ciflow/inductor-rocm-mi200/169063 -> ciflow/inductor-rocm-mi200/169063 2025-12-04T09:43:54.1229376Z * [new tag] ciflow/inductor-rocm-mi200/169425 -> ciflow/inductor-rocm-mi200/169425 2025-12-04T09:43:54.1230876Z * [new tag] ciflow/inductor-rocm-mi300/165545 -> ciflow/inductor-rocm-mi300/165545 2025-12-04T09:43:54.1231987Z * [new tag] ciflow/inductor-rocm-mi300/168096 -> ciflow/inductor-rocm-mi300/168096 2025-12-04T09:43:54.1233166Z * [new tag] ciflow/inductor-rocm-mi300/169063 -> ciflow/inductor-rocm-mi300/169063 2025-12-04T09:43:54.1234331Z * [new tag] ciflow/inductor-rocm-mi300/169425 -> ciflow/inductor-rocm-mi300/169425 2025-12-04T09:43:54.1235862Z * [new tag] ciflow/inductor-rocm/162052 -> ciflow/inductor-rocm/162052 2025-12-04T09:43:54.1237015Z * [new tag] ciflow/inductor-rocm/168971 -> ciflow/inductor-rocm/168971 2025-12-04T09:43:54.1238491Z * [new tag] ciflow/inductor-windows/168096 -> ciflow/inductor-windows/168096 2025-12-04T09:43:54.1239913Z * [new tag] ciflow/inductor/144542 -> ciflow/inductor/144542 2025-12-04T09:43:54.1241153Z * [new tag] ciflow/inductor/146506 -> ciflow/inductor/146506 2025-12-04T09:43:54.1242318Z * [new tag] ciflow/inductor/147990 -> ciflow/inductor/147990 2025-12-04T09:43:54.1243634Z * [new tag] ciflow/inductor/148294 -> ciflow/inductor/148294 2025-12-04T09:43:54.1244812Z * [new tag] ciflow/inductor/148492 -> ciflow/inductor/148492 2025-12-04T09:43:54.1246007Z * [new tag] ciflow/inductor/157149 -> ciflow/inductor/157149 2025-12-04T09:43:54.1247695Z * [new tag] ciflow/inductor/157994 -> ciflow/inductor/157994 2025-12-04T09:43:54.1248881Z * [new tag] ciflow/inductor/160685 -> ciflow/inductor/160685 2025-12-04T09:43:54.1250086Z * [new tag] ciflow/inductor/160686 -> ciflow/inductor/160686 2025-12-04T09:43:54.1251292Z * [new tag] ciflow/inductor/160687 -> ciflow/inductor/160687 2025-12-04T09:43:54.1252464Z * [new tag] ciflow/inductor/160688 -> ciflow/inductor/160688 2025-12-04T09:43:54.1254014Z * [new tag] ciflow/inductor/160706 -> ciflow/inductor/160706 2025-12-04T09:43:54.1255745Z * [new tag] ciflow/inductor/160729 -> ciflow/inductor/160729 2025-12-04T09:43:54.1257326Z * [new tag] ciflow/inductor/161938 -> ciflow/inductor/161938 2025-12-04T09:43:54.1258538Z * [new tag] ciflow/inductor/161939 -> ciflow/inductor/161939 2025-12-04T09:43:54.1259789Z * [new tag] ciflow/inductor/161940 -> ciflow/inductor/161940 2025-12-04T09:43:54.1261014Z * [new tag] ciflow/inductor/162052 -> ciflow/inductor/162052 2025-12-04T09:43:54.1262309Z * [new tag] ciflow/inductor/162275 -> ciflow/inductor/162275 2025-12-04T09:43:54.1263544Z * [new tag] ciflow/inductor/162795 -> ciflow/inductor/162795 2025-12-04T09:43:54.1265175Z * [new tag] ciflow/inductor/163245 -> ciflow/inductor/163245 2025-12-04T09:43:54.1266467Z * [new tag] ciflow/inductor/163335 -> ciflow/inductor/163335 2025-12-04T09:43:54.1267737Z * [new tag] ciflow/inductor/163503 -> ciflow/inductor/163503 2025-12-04T09:43:54.1268980Z * [new tag] ciflow/inductor/163942 -> ciflow/inductor/163942 2025-12-04T09:43:54.1270382Z * [new tag] ciflow/inductor/165270 -> ciflow/inductor/165270 2025-12-04T09:43:54.1271655Z * [new tag] ciflow/inductor/165274 -> ciflow/inductor/165274 2025-12-04T09:43:54.1272944Z * [new tag] ciflow/inductor/165322 -> ciflow/inductor/165322 2025-12-04T09:43:54.1274173Z * [new tag] ciflow/inductor/165597 -> ciflow/inductor/165597 2025-12-04T09:43:54.1275426Z * [new tag] ciflow/inductor/166063 -> ciflow/inductor/166063 2025-12-04T09:43:54.1276682Z * [new tag] ciflow/inductor/166075 -> ciflow/inductor/166075 2025-12-04T09:43:54.1278305Z * [new tag] ciflow/inductor/166165 -> ciflow/inductor/166165 2025-12-04T09:43:54.1281178Z * [new tag] ciflow/inductor/166254 -> ciflow/inductor/166254 2025-12-04T09:43:54.1282386Z * [new tag] ciflow/inductor/166483 -> ciflow/inductor/166483 2025-12-04T09:43:54.1283654Z * [new tag] ciflow/inductor/166494 -> ciflow/inductor/166494 2025-12-04T09:43:54.1284898Z * [new tag] ciflow/inductor/166545 -> ciflow/inductor/166545 2025-12-04T09:43:54.1286224Z * [new tag] ciflow/inductor/166788 -> ciflow/inductor/166788 2025-12-04T09:43:54.1287570Z * [new tag] ciflow/inductor/166846 -> ciflow/inductor/166846 2025-12-04T09:43:54.1288863Z * [new tag] ciflow/inductor/167300 -> ciflow/inductor/167300 2025-12-04T09:43:54.1290201Z * [new tag] ciflow/inductor/167407 -> ciflow/inductor/167407 2025-12-04T09:43:54.1291553Z * [new tag] ciflow/inductor/167536 -> ciflow/inductor/167536 2025-12-04T09:43:54.1292816Z * [new tag] ciflow/inductor/167552 -> ciflow/inductor/167552 2025-12-04T09:43:54.1294071Z * [new tag] ciflow/inductor/167555 -> ciflow/inductor/167555 2025-12-04T09:43:54.1295431Z * [new tag] ciflow/inductor/167583 -> ciflow/inductor/167583 2025-12-04T09:43:54.1296656Z * [new tag] ciflow/inductor/167599 -> ciflow/inductor/167599 2025-12-04T09:43:54.1297920Z * [new tag] ciflow/inductor/167647 -> ciflow/inductor/167647 2025-12-04T09:43:54.1299171Z * [new tag] ciflow/inductor/167677 -> ciflow/inductor/167677 2025-12-04T09:43:54.1300428Z * [new tag] ciflow/inductor/167680 -> ciflow/inductor/167680 2025-12-04T09:43:54.1301702Z * [new tag] ciflow/inductor/167695 -> ciflow/inductor/167695 2025-12-04T09:43:54.1302965Z * [new tag] ciflow/inductor/167742 -> ciflow/inductor/167742 2025-12-04T09:43:54.1304372Z * [new tag] ciflow/inductor/167768 -> ciflow/inductor/167768 2025-12-04T09:43:54.1305825Z * [new tag] ciflow/inductor/167773 -> ciflow/inductor/167773 2025-12-04T09:43:54.1307156Z * [new tag] ciflow/inductor/167781 -> ciflow/inductor/167781 2025-12-04T09:43:54.1308481Z * [new tag] ciflow/inductor/167880 -> ciflow/inductor/167880 2025-12-04T09:43:54.1309757Z * [new tag] ciflow/inductor/167887 -> ciflow/inductor/167887 2025-12-04T09:43:54.1311026Z * [new tag] ciflow/inductor/167972 -> ciflow/inductor/167972 2025-12-04T09:43:54.1312341Z * [new tag] ciflow/inductor/167989 -> ciflow/inductor/167989 2025-12-04T09:43:54.1313612Z * [new tag] ciflow/inductor/168002 -> ciflow/inductor/168002 2025-12-04T09:43:54.1314845Z * [new tag] ciflow/inductor/168050 -> ciflow/inductor/168050 2025-12-04T09:43:54.1316114Z * [new tag] ciflow/inductor/168051 -> ciflow/inductor/168051 2025-12-04T09:43:54.1317370Z * [new tag] ciflow/inductor/168052 -> ciflow/inductor/168052 2025-12-04T09:43:54.1318622Z * [new tag] ciflow/inductor/168073 -> ciflow/inductor/168073 2025-12-04T09:43:54.1319899Z * [new tag] ciflow/inductor/168096 -> ciflow/inductor/168096 2025-12-04T09:43:54.1321167Z * [new tag] ciflow/inductor/168114 -> ciflow/inductor/168114 2025-12-04T09:43:54.1322413Z * [new tag] ciflow/inductor/168115 -> ciflow/inductor/168115 2025-12-04T09:43:54.1323674Z * [new tag] ciflow/inductor/168127 -> ciflow/inductor/168127 2025-12-04T09:43:54.1324943Z * [new tag] ciflow/inductor/168129 -> ciflow/inductor/168129 2025-12-04T09:43:54.1326221Z * [new tag] ciflow/inductor/168157 -> ciflow/inductor/168157 2025-12-04T09:43:54.1327549Z * [new tag] ciflow/inductor/168175 -> ciflow/inductor/168175 2025-12-04T09:43:54.1328719Z * [new tag] ciflow/inductor/168185 -> ciflow/inductor/168185 2025-12-04T09:43:54.1329943Z * [new tag] ciflow/inductor/168195 -> ciflow/inductor/168195 2025-12-04T09:43:54.1331234Z * [new tag] ciflow/inductor/168209 -> ciflow/inductor/168209 2025-12-04T09:43:54.1332980Z * [new tag] ciflow/inductor/168266 -> ciflow/inductor/168266 2025-12-04T09:43:54.1334264Z * [new tag] ciflow/inductor/168316 -> ciflow/inductor/168316 2025-12-04T09:43:54.1335694Z * [new tag] ciflow/inductor/168326 -> ciflow/inductor/168326 2025-12-04T09:43:54.1336897Z * [new tag] ciflow/inductor/168368 -> ciflow/inductor/168368 2025-12-04T09:43:54.1338175Z * [new tag] ciflow/inductor/168894 -> ciflow/inductor/168894 2025-12-04T09:43:54.1339463Z * [new tag] ciflow/inductor/168934 -> ciflow/inductor/168934 2025-12-04T09:43:54.1340742Z * [new tag] ciflow/inductor/168939 -> ciflow/inductor/168939 2025-12-04T09:43:54.1341993Z * [new tag] ciflow/inductor/168946 -> ciflow/inductor/168946 2025-12-04T09:43:54.1343255Z * [new tag] ciflow/inductor/168950 -> ciflow/inductor/168950 2025-12-04T09:43:54.1344630Z * [new tag] ciflow/inductor/168951 -> ciflow/inductor/168951 2025-12-04T09:43:54.1345903Z * [new tag] ciflow/inductor/168952 -> ciflow/inductor/168952 2025-12-04T09:43:54.1347146Z * [new tag] ciflow/inductor/168955 -> ciflow/inductor/168955 2025-12-04T09:43:54.1348368Z * [new tag] ciflow/inductor/168971 -> ciflow/inductor/168971 2025-12-04T09:43:54.1349731Z * [new tag] ciflow/inductor/168979 -> ciflow/inductor/168979 2025-12-04T09:43:54.1351077Z * [new tag] ciflow/inductor/168980 -> ciflow/inductor/168980 2025-12-04T09:43:54.1352402Z * [new tag] ciflow/inductor/168983 -> ciflow/inductor/168983 2025-12-04T09:43:54.1353688Z * [new tag] ciflow/inductor/169006 -> ciflow/inductor/169006 2025-12-04T09:43:54.1354971Z * [new tag] ciflow/inductor/169023 -> ciflow/inductor/169023 2025-12-04T09:43:54.1356275Z * [new tag] ciflow/inductor/169024 -> ciflow/inductor/169024 2025-12-04T09:43:54.1357483Z * [new tag] ciflow/inductor/169025 -> ciflow/inductor/169025 2025-12-04T09:43:54.1358720Z * [new tag] ciflow/inductor/169066 -> ciflow/inductor/169066 2025-12-04T09:43:54.1359984Z * [new tag] ciflow/inductor/169091 -> ciflow/inductor/169091 2025-12-04T09:43:54.1361297Z * [new tag] ciflow/inductor/169102 -> ciflow/inductor/169102 2025-12-04T09:43:54.1362521Z * [new tag] ciflow/inductor/169103 -> ciflow/inductor/169103 2025-12-04T09:43:54.1363758Z * [new tag] ciflow/inductor/169121 -> ciflow/inductor/169121 2025-12-04T09:43:54.1364999Z * [new tag] ciflow/inductor/169134 -> ciflow/inductor/169134 2025-12-04T09:43:54.1366259Z * [new tag] ciflow/inductor/169135 -> ciflow/inductor/169135 2025-12-04T09:43:54.1367666Z * [new tag] ciflow/inductor/169141 -> ciflow/inductor/169141 2025-12-04T09:43:54.1369152Z * [new tag] ciflow/inductor/169151 -> ciflow/inductor/169151 2025-12-04T09:43:54.1370400Z * [new tag] ciflow/inductor/169161 -> ciflow/inductor/169161 2025-12-04T09:43:54.1371654Z * [new tag] ciflow/inductor/169167 -> ciflow/inductor/169167 2025-12-04T09:43:54.1373054Z * [new tag] ciflow/inductor/169177 -> ciflow/inductor/169177 2025-12-04T09:43:54.1374570Z * [new tag] ciflow/inductor/169185 -> ciflow/inductor/169185 2025-12-04T09:43:54.1375762Z * [new tag] ciflow/inductor/169196 -> ciflow/inductor/169196 2025-12-04T09:43:54.1377002Z * [new tag] ciflow/inductor/169200 -> ciflow/inductor/169200 2025-12-04T09:43:54.1378537Z * [new tag] ciflow/inductor/169204 -> ciflow/inductor/169204 2025-12-04T09:43:54.1379726Z * [new tag] ciflow/inductor/169216 -> ciflow/inductor/169216 2025-12-04T09:43:54.1380978Z * [new tag] ciflow/inductor/169219 -> ciflow/inductor/169219 2025-12-04T09:43:54.1382193Z * [new tag] ciflow/inductor/169220 -> ciflow/inductor/169220 2025-12-04T09:43:54.1383540Z * [new tag] ciflow/inductor/169230 -> ciflow/inductor/169230 2025-12-04T09:43:54.1384956Z * [new tag] ciflow/inductor/169242 -> ciflow/inductor/169242 2025-12-04T09:43:54.1386177Z * [new tag] ciflow/inductor/169245 -> ciflow/inductor/169245 2025-12-04T09:43:54.1387549Z * [new tag] ciflow/inductor/169260 -> ciflow/inductor/169260 2025-12-04T09:43:54.1388774Z * [new tag] ciflow/inductor/169282 -> ciflow/inductor/169282 2025-12-04T09:43:54.1390058Z * [new tag] ciflow/inductor/169286 -> ciflow/inductor/169286 2025-12-04T09:43:54.1391462Z * [new tag] ciflow/inductor/169299 -> ciflow/inductor/169299 2025-12-04T09:43:54.1392865Z * [new tag] ciflow/inductor/169304 -> ciflow/inductor/169304 2025-12-04T09:43:54.1394501Z * [new tag] ciflow/inductor/169305 -> ciflow/inductor/169305 2025-12-04T09:43:54.1395722Z * [new tag] ciflow/inductor/169308 -> ciflow/inductor/169308 2025-12-04T09:43:54.1396993Z * [new tag] ciflow/inductor/169319 -> ciflow/inductor/169319 2025-12-04T09:43:54.1398284Z * [new tag] ciflow/inductor/169326 -> ciflow/inductor/169326 2025-12-04T09:43:54.1399540Z * [new tag] ciflow/inductor/169332 -> ciflow/inductor/169332 2025-12-04T09:43:54.1400791Z * [new tag] ciflow/inductor/169333 -> ciflow/inductor/169333 2025-12-04T09:43:54.1402241Z * [new tag] ciflow/inductor/169336 -> ciflow/inductor/169336 2025-12-04T09:43:54.1403524Z * [new tag] ciflow/inductor/169340 -> ciflow/inductor/169340 2025-12-04T09:43:54.1404837Z * [new tag] ciflow/inductor/169341 -> ciflow/inductor/169341 2025-12-04T09:43:54.1406111Z * [new tag] ciflow/inductor/169343 -> ciflow/inductor/169343 2025-12-04T09:43:54.1407333Z * [new tag] ciflow/inductor/169346 -> ciflow/inductor/169346 2025-12-04T09:43:54.1408734Z * [new tag] ciflow/inductor/169348 -> ciflow/inductor/169348 2025-12-04T09:43:54.1410089Z * [new tag] ciflow/inductor/169350 -> ciflow/inductor/169350 2025-12-04T09:43:54.1411392Z * [new tag] ciflow/inductor/169355 -> ciflow/inductor/169355 2025-12-04T09:43:54.1412680Z * [new tag] ciflow/inductor/169370 -> ciflow/inductor/169370 2025-12-04T09:43:54.1414297Z * [new tag] ciflow/inductor/169375 -> ciflow/inductor/169375 2025-12-04T09:43:54.1415590Z * [new tag] ciflow/inductor/169389 -> ciflow/inductor/169389 2025-12-04T09:43:54.1417246Z * [new tag] ciflow/inductor/169391 -> ciflow/inductor/169391 2025-12-04T09:43:54.1418564Z * [new tag] ciflow/inductor/169393 -> ciflow/inductor/169393 2025-12-04T09:43:54.1419802Z * [new tag] ciflow/inductor/169399 -> ciflow/inductor/169399 2025-12-04T09:43:54.1421212Z * [new tag] ciflow/inductor/169400 -> ciflow/inductor/169400 2025-12-04T09:43:54.1422480Z * [new tag] ciflow/inductor/169415 -> ciflow/inductor/169415 2025-12-04T09:43:54.1423970Z * [new tag] ciflow/inductor/169417 -> ciflow/inductor/169417 2025-12-04T09:43:54.1425198Z * [new tag] ciflow/inductor/169418 -> ciflow/inductor/169418 2025-12-04T09:43:54.1426596Z * [new tag] ciflow/inductor/169430 -> ciflow/inductor/169430 2025-12-04T09:43:54.1427867Z * [new tag] ciflow/inductor/169432 -> ciflow/inductor/169432 2025-12-04T09:43:54.1429174Z * [new tag] ciflow/inductor/169436 -> ciflow/inductor/169436 2025-12-04T09:43:54.1430490Z * [new tag] ciflow/inductor/169437 -> ciflow/inductor/169437 2025-12-04T09:43:54.1431766Z * [new tag] ciflow/inductor/169438 -> ciflow/inductor/169438 2025-12-04T09:43:54.1433028Z * [new tag] ciflow/inductor/169441 -> ciflow/inductor/169441 2025-12-04T09:43:54.1434436Z * [new tag] ciflow/inductor/169446 -> ciflow/inductor/169446 2025-12-04T09:43:54.1435807Z * [new tag] ciflow/inductor/169447 -> ciflow/inductor/169447 2025-12-04T09:43:54.1437081Z * [new tag] ciflow/inductor/169452 -> ciflow/inductor/169452 2025-12-04T09:43:54.1438479Z * [new tag] ciflow/inductor/169455 -> ciflow/inductor/169455 2025-12-04T09:43:54.1439809Z * [new tag] ciflow/inductor/169459 -> ciflow/inductor/169459 2025-12-04T09:43:54.1441252Z * [new tag] ciflow/inductor/169463 -> ciflow/inductor/169463 2025-12-04T09:43:54.1442627Z * [new tag] ciflow/inductor/169476 -> ciflow/inductor/169476 2025-12-04T09:43:54.1443899Z * [new tag] ciflow/inductor/169485 -> ciflow/inductor/169485 2025-12-04T09:43:54.1445248Z * [new tag] ciflow/inductor/169493 -> ciflow/inductor/169493 2025-12-04T09:43:54.1446494Z * [new tag] ciflow/inductor/169496 -> ciflow/inductor/169496 2025-12-04T09:43:54.1447764Z * [new tag] ciflow/inductor/169497 -> ciflow/inductor/169497 2025-12-04T09:43:54.1449052Z * [new tag] ciflow/inductor/169503 -> ciflow/inductor/169503 2025-12-04T09:43:54.1450353Z * [new tag] ciflow/inductor/169504 -> ciflow/inductor/169504 2025-12-04T09:43:54.1451854Z * [new tag] ciflow/inductor/169505 -> ciflow/inductor/169505 2025-12-04T09:43:54.1453502Z * [new tag] ciflow/inductor/169508 -> ciflow/inductor/169508 2025-12-04T09:43:54.1454765Z * [new tag] ciflow/inductor/169509 -> ciflow/inductor/169509 2025-12-04T09:43:54.1456073Z * [new tag] ciflow/inductor/169513 -> ciflow/inductor/169513 2025-12-04T09:43:54.1457329Z * [new tag] ciflow/inductor/169514 -> ciflow/inductor/169514 2025-12-04T09:43:54.1458602Z * [new tag] ciflow/inductor/169515 -> ciflow/inductor/169515 2025-12-04T09:43:54.1459886Z * [new tag] ciflow/inductor/169517 -> ciflow/inductor/169517 2025-12-04T09:43:54.1461143Z * [new tag] ciflow/inductor/169519 -> ciflow/inductor/169519 2025-12-04T09:43:54.1462421Z * [new tag] ciflow/inductor/169520 -> ciflow/inductor/169520 2025-12-04T09:43:54.1463763Z * [new tag] ciflow/inductor/169521 -> ciflow/inductor/169521 2025-12-04T09:43:54.1465120Z * [new tag] ciflow/inductor/169524 -> ciflow/inductor/169524 2025-12-04T09:43:54.1466459Z * [new tag] ciflow/inductor/169527 -> ciflow/inductor/169527 2025-12-04T09:43:54.1467691Z * [new tag] ciflow/inductor/169528 -> ciflow/inductor/169528 2025-12-04T09:43:54.1469078Z * [new tag] ciflow/inductor/169532 -> ciflow/inductor/169532 2025-12-04T09:43:54.1470365Z * [new tag] ciflow/inductor/169535 -> ciflow/inductor/169535 2025-12-04T09:43:54.1471629Z * [new tag] ciflow/inductor/169536 -> ciflow/inductor/169536 2025-12-04T09:43:54.1472999Z * [new tag] ciflow/inductor/169547 -> ciflow/inductor/169547 2025-12-04T09:43:54.1474200Z * [new tag] ciflow/inductor/169548 -> ciflow/inductor/169548 2025-12-04T09:43:54.1475492Z * [new tag] ciflow/inductor/169549 -> ciflow/inductor/169549 2025-12-04T09:43:54.1476794Z * [new tag] ciflow/inductor/169551 -> ciflow/inductor/169551 2025-12-04T09:43:54.1478158Z * [new tag] ciflow/inductor/169552 -> ciflow/inductor/169552 2025-12-04T09:43:54.1479582Z * [new tag] ciflow/inductor/169553 -> ciflow/inductor/169553 2025-12-04T09:43:54.1480794Z * [new tag] ciflow/inductor/169557 -> ciflow/inductor/169557 2025-12-04T09:43:54.1482257Z * [new tag] ciflow/inductor/3b9a386 -> ciflow/inductor/3b9a386 2025-12-04T09:43:54.1483673Z * [new tag] ciflow/inductor/3d4b92b -> ciflow/inductor/3d4b92b 2025-12-04T09:43:54.1485114Z * [new tag] ciflow/inductor/d224ac7 -> ciflow/inductor/d224ac7 2025-12-04T09:43:54.1486637Z * [new tag] ciflow/linux-aarch64/157994 -> ciflow/linux-aarch64/157994 2025-12-04T09:43:54.1487790Z * [new tag] ciflow/linux-aarch64/166075 -> ciflow/linux-aarch64/166075 2025-12-04T09:43:54.1488971Z * [new tag] ciflow/linux-aarch64/166876 -> ciflow/linux-aarch64/166876 2025-12-04T09:43:54.1490229Z * [new tag] ciflow/linux-aarch64/167981 -> ciflow/linux-aarch64/167981 2025-12-04T09:43:54.1491704Z * [new tag] ciflow/mps/166254 -> ciflow/mps/166254 2025-12-04T09:43:54.1492840Z * [new tag] ciflow/mps/169017 -> ciflow/mps/169017 2025-12-04T09:43:54.1494109Z * [new tag] ciflow/mps/169372 -> ciflow/mps/169372 2025-12-04T09:43:54.1495294Z * [new tag] ciflow/mps/169478 -> ciflow/mps/169478 2025-12-04T09:43:54.1496860Z * [new tag] ciflow/op-benchmark/157994 -> ciflow/op-benchmark/157994 2025-12-04T09:43:54.1498025Z * [new tag] ciflow/op-benchmark/166075 -> ciflow/op-benchmark/166075 2025-12-04T09:43:54.1499182Z * [new tag] ciflow/op-benchmark/169544 -> ciflow/op-benchmark/169544 2025-12-04T09:43:54.1500618Z * [new tag] ciflow/periodic-rocm-mi200/165997 -> ciflow/periodic-rocm-mi200/165997 2025-12-04T09:43:54.1501949Z * [new tag] ciflow/periodic-rocm-mi200/166517 -> ciflow/periodic-rocm-mi200/166517 2025-12-04T09:43:54.1503125Z * [new tag] ciflow/periodic-rocm-mi200/169063 -> ciflow/periodic-rocm-mi200/169063 2025-12-04T09:43:54.1504492Z * [new tag] ciflow/periodic-rocm-mi200/169425 -> ciflow/periodic-rocm-mi200/169425 2025-12-04T09:43:54.1505910Z * [new tag] ciflow/periodic-rocm-mi300/166517 -> ciflow/periodic-rocm-mi300/166517 2025-12-04T09:43:54.1507104Z * [new tag] ciflow/periodic-rocm-mi300/169063 -> ciflow/periodic-rocm-mi300/169063 2025-12-04T09:43:54.1508256Z * [new tag] ciflow/periodic-rocm-mi300/169425 -> ciflow/periodic-rocm-mi300/169425 2025-12-04T09:43:54.1509871Z * [new tag] ciflow/periodic/054a2fd -> ciflow/periodic/054a2fd 2025-12-04T09:43:54.1511165Z * [new tag] ciflow/periodic/167207 -> ciflow/periodic/167207 2025-12-04T09:43:54.1512484Z * [new tag] ciflow/periodic/167978 -> ciflow/periodic/167978 2025-12-04T09:43:54.1513637Z * [new tag] ciflow/periodic/168096 -> ciflow/periodic/168096 2025-12-04T09:43:54.1514851Z * [new tag] ciflow/periodic/169286 -> ciflow/periodic/169286 2025-12-04T09:43:54.1516231Z * [new tag] ciflow/periodic/2a6d37d -> ciflow/periodic/2a6d37d 2025-12-04T09:43:54.1517591Z * [new tag] ciflow/periodic/317eeb8 -> ciflow/periodic/317eeb8 2025-12-04T09:43:54.1519061Z * [new tag] ciflow/periodic/3c32 -> ciflow/periodic/3c32 2025-12-04T09:43:54.1520386Z * [new tag] ciflow/periodic/3e98831 -> ciflow/periodic/3e98831 2025-12-04T09:43:54.1522229Z * [new tag] ciflow/periodic/7c648509a7470ace9fb2bae960dd4790f7e943e9 -> ciflow/periodic/7c648509a7470ace9fb2bae960dd4790f7e943e9 2025-12-04T09:43:54.1523771Z * [new tag] ciflow/periodic/94512-point -> ciflow/periodic/94512-point 2025-12-04T09:43:54.1525524Z * [new tag] ciflow/periodic/csl/test87519 -> ciflow/periodic/csl/test87519 2025-12-04T09:43:54.1526894Z * [new tag] ciflow/periodic/csltest88275 -> ciflow/periodic/csltest88275 2025-12-04T09:43:54.1528269Z * [new tag] ciflow/periodic/csltest88761 -> ciflow/periodic/csltest88761 2025-12-04T09:43:54.1529762Z * [new tag] ciflow/periodic/release_1.12 -> ciflow/periodic/release_1.12 2025-12-04T09:43:54.1531233Z * [new tag] ciflow/periodic/release_1.12.0 -> ciflow/periodic/release_1.12.0 2025-12-04T09:43:54.1532740Z * [new tag] ciflow/periodic/sha-ec5b83 -> ciflow/periodic/sha-ec5b83 2025-12-04T09:43:54.1534141Z * [new tag] ciflow/pull/167207 -> ciflow/pull/167207 2025-12-04T09:43:54.1535799Z * [new tag] ciflow/quantization-periodic/169207 -> ciflow/quantization-periodic/169207 2025-12-04T09:43:54.1537147Z * [new tag] ciflow/rocm-mi200/165545 -> ciflow/rocm-mi200/165545 2025-12-04T09:43:54.1538304Z * [new tag] ciflow/rocm-mi200/165997 -> ciflow/rocm-mi200/165997 2025-12-04T09:43:54.1539467Z * [new tag] ciflow/rocm-mi200/168096 -> ciflow/rocm-mi200/168096 2025-12-04T09:43:54.1540858Z * [new tag] ciflow/rocm-mi200/168275 -> ciflow/rocm-mi200/168275 2025-12-04T09:43:54.1542082Z * [new tag] ciflow/rocm-mi200/169063 -> ciflow/rocm-mi200/169063 2025-12-04T09:43:54.1543370Z * [new tag] ciflow/rocm-mi200/169356 -> ciflow/rocm-mi200/169356 2025-12-04T09:43:54.1544784Z * [new tag] ciflow/rocm-mi200/169425 -> ciflow/rocm-mi200/169425 2025-12-04T09:43:54.1546215Z * [new tag] ciflow/rocm-mi300/165545 -> ciflow/rocm-mi300/165545 2025-12-04T09:43:54.1547537Z * [new tag] ciflow/rocm-mi300/167157 -> ciflow/rocm-mi300/167157 2025-12-04T09:43:54.1548806Z * [new tag] ciflow/rocm-mi300/168096 -> ciflow/rocm-mi300/168096 2025-12-04T09:43:54.1550016Z * [new tag] ciflow/rocm-mi300/169063 -> ciflow/rocm-mi300/169063 2025-12-04T09:43:54.1551197Z * [new tag] ciflow/rocm-mi300/169425 -> ciflow/rocm-mi300/169425 2025-12-04T09:43:54.1552612Z * [new tag] ciflow/rocm-mi355/167157 -> ciflow/rocm-mi355/167157 2025-12-04T09:43:54.1553845Z * [new tag] ciflow/rocm-mi355/168275 -> ciflow/rocm-mi355/168275 2025-12-04T09:43:54.1555073Z * [new tag] ciflow/rocm-mi355/169425 -> ciflow/rocm-mi355/169425 2025-12-04T09:43:54.1556455Z * [new tag] ciflow/rocm-navi31/168275 -> ciflow/rocm-navi31/168275 2025-12-04T09:43:54.1557614Z * [new tag] ciflow/rocm-navi31/169425 -> ciflow/rocm-navi31/169425 2025-12-04T09:43:54.1559017Z * [new tag] ciflow/rocm/115316 -> ciflow/rocm/115316 2025-12-04T09:43:54.1560169Z * [new tag] ciflow/rocm/148492 -> ciflow/rocm/148492 2025-12-04T09:43:54.1561286Z * [new tag] ciflow/rocm/160685 -> ciflow/rocm/160685 2025-12-04T09:43:54.1562474Z * [new tag] ciflow/rocm/161607 -> ciflow/rocm/161607 2025-12-04T09:43:54.1563625Z * [new tag] ciflow/rocm/162052 -> ciflow/rocm/162052 2025-12-04T09:43:54.1564831Z * [new tag] ciflow/rocm/165997 -> ciflow/rocm/165997 2025-12-04T09:43:54.1566101Z * [new tag] ciflow/rocm/166165 -> ciflow/rocm/166165 2025-12-04T09:43:54.1567235Z * [new tag] ciflow/rocm/166517 -> ciflow/rocm/166517 2025-12-04T09:43:54.1568369Z * [new tag] ciflow/rocm/167207 -> ciflow/rocm/167207 2025-12-04T09:43:54.1569560Z * [new tag] ciflow/rocm/167536 -> ciflow/rocm/167536 2025-12-04T09:43:54.1570722Z * [new tag] ciflow/rocm/167781 -> ciflow/rocm/167781 2025-12-04T09:43:54.1572224Z * [new tag] ciflow/rocm/167989 -> ciflow/rocm/167989 2025-12-04T09:43:54.1573830Z * [new tag] ciflow/rocm/168073 -> ciflow/rocm/168073 2025-12-04T09:43:54.1575280Z * [new tag] ciflow/rocm/168195 -> ciflow/rocm/168195 2025-12-04T09:43:54.1576522Z * [new tag] ciflow/rocm/168939 -> ciflow/rocm/168939 2025-12-04T09:43:54.1577761Z * [new tag] ciflow/rocm/168971 -> ciflow/rocm/168971 2025-12-04T09:43:54.1579270Z * [new tag] ciflow/rocm/169024 -> ciflow/rocm/169024 2025-12-04T09:43:54.1580579Z * [new tag] ciflow/rocm/169200 -> ciflow/rocm/169200 2025-12-04T09:43:54.1581782Z * [new tag] ciflow/rocm/169216 -> ciflow/rocm/169216 2025-12-04T09:43:54.1583021Z * [new tag] ciflow/rocm/169312 -> ciflow/rocm/169312 2025-12-04T09:43:54.1584356Z * [new tag] ciflow/rocm/169380 -> ciflow/rocm/169380 2025-12-04T09:43:54.1585704Z * [new tag] ciflow/rocm/169427 -> ciflow/rocm/169427 2025-12-04T09:43:54.1586954Z * [new tag] ciflow/rocm/169455 -> ciflow/rocm/169455 2025-12-04T09:43:54.1588204Z * [new tag] ciflow/rocm/169470 -> ciflow/rocm/169470 2025-12-04T09:43:54.1589441Z * [new tag] ciflow/rocm/169471 -> ciflow/rocm/169471 2025-12-04T09:43:54.1590689Z * [new tag] ciflow/rocm/169472 -> ciflow/rocm/169472 2025-12-04T09:43:54.1591914Z * [new tag] ciflow/rocm/169514 -> ciflow/rocm/169514 2025-12-04T09:43:54.1593552Z * [new tag] ciflow/slow/01c7106 -> ciflow/slow/01c7106 2025-12-04T09:43:54.1594846Z * [new tag] ciflow/slow/0577043 -> ciflow/slow/0577043 2025-12-04T09:43:54.1596533Z * [new tag] ciflow/slow/0d5b74da0cab798fbfdb9caa53fad816999c8386-sdym -> ciflow/slow/0d5b74da0cab798fbfdb9caa53fad816999c8386-sdym 2025-12-04T09:43:54.1597623Z * [new tag] ciflow/slow/0e81104 -> ciflow/slow/0e81104 2025-12-04T09:43:54.1598786Z * [new tag] ciflow/slow/167207 -> ciflow/slow/167207 2025-12-04T09:43:54.1599966Z * [new tag] ciflow/slow/168050 -> ciflow/slow/168050 2025-12-04T09:43:54.1601426Z * [new tag] ciflow/slow/1732077 -> ciflow/slow/1732077 2025-12-04T09:43:54.1602830Z * [new tag] ciflow/slow/187eb7c -> ciflow/slow/187eb7c 2025-12-04T09:43:54.1604484Z * [new tag] ciflow/slow/1faef89 -> ciflow/slow/1faef89 2025-12-04T09:43:54.1606126Z * [new tag] ciflow/slow/3920ec1 -> ciflow/slow/3920ec1 2025-12-04T09:43:54.1607750Z * [new tag] ciflow/slow/3b7c6b2 -> ciflow/slow/3b7c6b2 2025-12-04T09:43:54.1609143Z * [new tag] ciflow/slow/59a3759 -> ciflow/slow/59a3759 2025-12-04T09:43:54.1610521Z * [new tag] ciflow/slow/70ef0bb -> ciflow/slow/70ef0bb 2025-12-04T09:43:54.1611926Z * [new tag] ciflow/slow/788ff06 -> ciflow/slow/788ff06 2025-12-04T09:43:54.1613658Z * [new tag] ciflow/slow/8751002215790a3a88750faa8f4366933e296693-sdym -> ciflow/slow/8751002215790a3a88750faa8f4366933e296693-sdym 2025-12-04T09:43:54.1614800Z * [new tag] ciflow/slow/9d85864 -> ciflow/slow/9d85864 2025-12-04T09:43:54.1616388Z * [new tag] ciflow/slow/9ffad5b -> ciflow/slow/9ffad5b 2025-12-04T09:43:54.1617667Z * [new tag] ciflow/slow/a206e8b -> ciflow/slow/a206e8b 2025-12-04T09:43:54.1619056Z * [new tag] ciflow/slow/a837609 -> ciflow/slow/a837609 2025-12-04T09:43:54.1620507Z * [new tag] ciflow/slow/af841f3 -> ciflow/slow/af841f3 2025-12-04T09:43:54.1622295Z * [new tag] ciflow/slow/da3aba1e46157c4df504b067477cdf2b3c96b194-sdym -> ciflow/slow/da3aba1e46157c4df504b067477cdf2b3c96b194-sdym 2025-12-04T09:43:54.1623616Z * [new tag] ciflow/torchbench/168175 -> ciflow/torchbench/168175 2025-12-04T09:43:54.1625167Z * [new tag] ciflow/trunk/148492 -> ciflow/trunk/148492 2025-12-04T09:43:54.1626383Z * [new tag] ciflow/trunk/157149 -> ciflow/trunk/157149 2025-12-04T09:43:54.1627547Z * [new tag] ciflow/trunk/157994 -> ciflow/trunk/157994 2025-12-04T09:43:54.1628724Z * [new tag] ciflow/trunk/159718 -> ciflow/trunk/159718 2025-12-04T09:43:54.1629908Z * [new tag] ciflow/trunk/160685 -> ciflow/trunk/160685 2025-12-04T09:43:54.1631056Z * [new tag] ciflow/trunk/160729 -> ciflow/trunk/160729 2025-12-04T09:43:54.1632217Z * [new tag] ciflow/trunk/162275 -> ciflow/trunk/162275 2025-12-04T09:43:54.1633414Z * [new tag] ciflow/trunk/162795 -> ciflow/trunk/162795 2025-12-04T09:43:54.1634596Z * [new tag] ciflow/trunk/163245 -> ciflow/trunk/163245 2025-12-04T09:43:54.1635760Z * [new tag] ciflow/trunk/163942 -> ciflow/trunk/163942 2025-12-04T09:43:54.1636980Z * [new tag] ciflow/trunk/165274 -> ciflow/trunk/165274 2025-12-04T09:43:54.1638830Z * [new tag] ciflow/trunk/165483 -> ciflow/trunk/165483 2025-12-04T09:43:54.1640458Z * [new tag] ciflow/trunk/165728 -> ciflow/trunk/165728 2025-12-04T09:43:54.1641935Z * [new tag] ciflow/trunk/165922 -> ciflow/trunk/165922 2025-12-04T09:43:54.1643190Z * [new tag] ciflow/trunk/166075 -> ciflow/trunk/166075 2025-12-04T09:43:54.1644433Z * [new tag] ciflow/trunk/166165 -> ciflow/trunk/166165 2025-12-04T09:43:54.1645688Z * [new tag] ciflow/trunk/166829 -> ciflow/trunk/166829 2025-12-04T09:43:54.1647083Z * [new tag] ciflow/trunk/166843 -> ciflow/trunk/166843 2025-12-04T09:43:54.1648330Z * [new tag] ciflow/trunk/166876 -> ciflow/trunk/166876 2025-12-04T09:43:54.1649571Z * [new tag] ciflow/trunk/167207 -> ciflow/trunk/167207 2025-12-04T09:43:54.1650821Z * [new tag] ciflow/trunk/167536 -> ciflow/trunk/167536 2025-12-04T09:43:54.1652240Z * [new tag] ciflow/trunk/167552 -> ciflow/trunk/167552 2025-12-04T09:43:54.1653450Z * [new tag] ciflow/trunk/167555 -> ciflow/trunk/167555 2025-12-04T09:43:54.1654727Z * [new tag] ciflow/trunk/167599 -> ciflow/trunk/167599 2025-12-04T09:43:54.1655987Z * [new tag] ciflow/trunk/167659 -> ciflow/trunk/167659 2025-12-04T09:43:54.1657302Z * [new tag] ciflow/trunk/167672 -> ciflow/trunk/167672 2025-12-04T09:43:54.1658572Z * [new tag] ciflow/trunk/167742 -> ciflow/trunk/167742 2025-12-04T09:43:54.1659807Z * [new tag] ciflow/trunk/167781 -> ciflow/trunk/167781 2025-12-04T09:43:54.1661287Z * [new tag] ciflow/trunk/167837 -> ciflow/trunk/167837 2025-12-04T09:43:54.1662545Z * [new tag] ciflow/trunk/167887 -> ciflow/trunk/167887 2025-12-04T09:43:54.1663885Z * [new tag] ciflow/trunk/167978 -> ciflow/trunk/167978 2025-12-04T09:43:54.1665334Z * [new tag] ciflow/trunk/168050 -> ciflow/trunk/168050 2025-12-04T09:43:54.1666514Z * [new tag] ciflow/trunk/168051 -> ciflow/trunk/168051 2025-12-04T09:43:54.1667755Z * [new tag] ciflow/trunk/168096 -> ciflow/trunk/168096 2025-12-04T09:43:54.1669004Z * [new tag] ciflow/trunk/168127 -> ciflow/trunk/168127 2025-12-04T09:43:54.1670249Z * [new tag] ciflow/trunk/168157 -> ciflow/trunk/168157 2025-12-04T09:43:54.1671941Z * [new tag] ciflow/trunk/168175 -> ciflow/trunk/168175 2025-12-04T09:43:54.1673181Z * [new tag] ciflow/trunk/168209 -> ciflow/trunk/168209 2025-12-04T09:43:54.1674552Z * [new tag] ciflow/trunk/168213 -> ciflow/trunk/168213 2025-12-04T09:43:54.1675932Z * [new tag] ciflow/trunk/168226 -> ciflow/trunk/168226 2025-12-04T09:43:54.1677230Z * [new tag] ciflow/trunk/168262 -> ciflow/trunk/168262 2025-12-04T09:43:54.1678552Z * [new tag] ciflow/trunk/168275 -> ciflow/trunk/168275 2025-12-04T09:43:54.1681788Z * [new tag] ciflow/trunk/168328 -> ciflow/trunk/168328 2025-12-04T09:43:54.1683096Z * [new tag] ciflow/trunk/168368 -> ciflow/trunk/168368 2025-12-04T09:43:54.1684319Z * [new tag] ciflow/trunk/168917 -> ciflow/trunk/168917 2025-12-04T09:43:54.1685584Z * [new tag] ciflow/trunk/168933 -> ciflow/trunk/168933 2025-12-04T09:43:54.1686912Z * [new tag] ciflow/trunk/168941 -> ciflow/trunk/168941 2025-12-04T09:43:54.1688254Z * [new tag] ciflow/trunk/168955 -> ciflow/trunk/168955 2025-12-04T09:43:54.1689484Z * [new tag] ciflow/trunk/168980 -> ciflow/trunk/168980 2025-12-04T09:43:54.1690824Z * [new tag] ciflow/trunk/169004 -> ciflow/trunk/169004 2025-12-04T09:43:54.1692116Z * [new tag] ciflow/trunk/169006 -> ciflow/trunk/169006 2025-12-04T09:43:54.1693386Z * [new tag] ciflow/trunk/169023 -> ciflow/trunk/169023 2025-12-04T09:43:54.1694647Z * [new tag] ciflow/trunk/169025 -> ciflow/trunk/169025 2025-12-04T09:43:54.1695892Z * [new tag] ciflow/trunk/169048 -> ciflow/trunk/169048 2025-12-04T09:43:54.1697142Z * [new tag] ciflow/trunk/169066 -> ciflow/trunk/169066 2025-12-04T09:43:54.1698432Z * [new tag] ciflow/trunk/169091 -> ciflow/trunk/169091 2025-12-04T09:43:54.1699648Z * [new tag] ciflow/trunk/169102 -> ciflow/trunk/169102 2025-12-04T09:43:54.1700897Z * [new tag] ciflow/trunk/169103 -> ciflow/trunk/169103 2025-12-04T09:43:54.1702297Z * [new tag] ciflow/trunk/169125 -> ciflow/trunk/169125 2025-12-04T09:43:54.1703774Z * [new tag] ciflow/trunk/169139 -> ciflow/trunk/169139 2025-12-04T09:43:54.1705156Z * [new tag] ciflow/trunk/169148 -> ciflow/trunk/169148 2025-12-04T09:43:54.1706407Z * [new tag] ciflow/trunk/169151 -> ciflow/trunk/169151 2025-12-04T09:43:54.1707719Z * [new tag] ciflow/trunk/169156 -> ciflow/trunk/169156 2025-12-04T09:43:54.1709114Z * [new tag] ciflow/trunk/169176 -> ciflow/trunk/169176 2025-12-04T09:43:54.1710401Z * [new tag] ciflow/trunk/169204 -> ciflow/trunk/169204 2025-12-04T09:43:54.1711713Z * [new tag] ciflow/trunk/169207 -> ciflow/trunk/169207 2025-12-04T09:43:54.1712998Z * [new tag] ciflow/trunk/169211 -> ciflow/trunk/169211 2025-12-04T09:43:54.1714546Z * [new tag] ciflow/trunk/169231 -> ciflow/trunk/169231 2025-12-04T09:43:54.1715930Z * [new tag] ciflow/trunk/169260 -> ciflow/trunk/169260 2025-12-04T09:43:54.1717311Z * [new tag] ciflow/trunk/169271 -> ciflow/trunk/169271 2025-12-04T09:43:54.1718563Z * [new tag] ciflow/trunk/169280 -> ciflow/trunk/169280 2025-12-04T09:43:54.1719809Z * [new tag] ciflow/trunk/169281 -> ciflow/trunk/169281 2025-12-04T09:43:54.1721020Z * [new tag] ciflow/trunk/169286 -> ciflow/trunk/169286 2025-12-04T09:43:54.1722439Z * [new tag] ciflow/trunk/169293 -> ciflow/trunk/169293 2025-12-04T09:43:54.1723659Z * [new tag] ciflow/trunk/169296 -> ciflow/trunk/169296 2025-12-04T09:43:54.1724931Z * [new tag] ciflow/trunk/169304 -> ciflow/trunk/169304 2025-12-04T09:43:54.1726169Z * [new tag] ciflow/trunk/169305 -> ciflow/trunk/169305 2025-12-04T09:43:54.1727404Z * [new tag] ciflow/trunk/169312 -> ciflow/trunk/169312 2025-12-04T09:43:54.1728944Z * [new tag] ciflow/trunk/169328 -> ciflow/trunk/169328 2025-12-04T09:43:54.1730237Z * [new tag] ciflow/trunk/169343 -> ciflow/trunk/169343 2025-12-04T09:43:54.1731491Z * [new tag] ciflow/trunk/169355 -> ciflow/trunk/169355 2025-12-04T09:43:54.1732725Z * [new tag] ciflow/trunk/169370 -> ciflow/trunk/169370 2025-12-04T09:43:54.1734098Z * [new tag] ciflow/trunk/169379 -> ciflow/trunk/169379 2025-12-04T09:43:54.1735378Z * [new tag] ciflow/trunk/169380 -> ciflow/trunk/169380 2025-12-04T09:43:54.1736609Z * [new tag] ciflow/trunk/169385 -> ciflow/trunk/169385 2025-12-04T09:43:54.1737859Z * [new tag] ciflow/trunk/169387 -> ciflow/trunk/169387 2025-12-04T09:43:54.1739220Z * [new tag] ciflow/trunk/169410 -> ciflow/trunk/169410 2025-12-04T09:43:54.1740460Z * [new tag] ciflow/trunk/169412 -> ciflow/trunk/169412 2025-12-04T09:43:54.1741703Z * [new tag] ciflow/trunk/169418 -> ciflow/trunk/169418 2025-12-04T09:43:54.1742945Z * [new tag] ciflow/trunk/169423 -> ciflow/trunk/169423 2025-12-04T09:43:54.1744290Z * [new tag] ciflow/trunk/169427 -> ciflow/trunk/169427 2025-12-04T09:43:54.1745571Z * [new tag] ciflow/trunk/169430 -> ciflow/trunk/169430 2025-12-04T09:43:54.1746836Z * [new tag] ciflow/trunk/169437 -> ciflow/trunk/169437 2025-12-04T09:43:54.1748074Z * [new tag] ciflow/trunk/169442 -> ciflow/trunk/169442 2025-12-04T09:43:54.1749334Z * [new tag] ciflow/trunk/169452 -> ciflow/trunk/169452 2025-12-04T09:43:54.1750601Z * [new tag] ciflow/trunk/169454 -> ciflow/trunk/169454 2025-12-04T09:43:54.1751848Z * [new tag] ciflow/trunk/169459 -> ciflow/trunk/169459 2025-12-04T09:43:54.1753313Z * [new tag] ciflow/trunk/169474 -> ciflow/trunk/169474 2025-12-04T09:43:54.1754567Z * [new tag] ciflow/trunk/169475 -> ciflow/trunk/169475 2025-12-04T09:43:54.1755851Z * [new tag] ciflow/trunk/169476 -> ciflow/trunk/169476 2025-12-04T09:43:54.1757621Z * [new tag] ciflow/trunk/169487 -> ciflow/trunk/169487 2025-12-04T09:43:54.1758916Z * [new tag] ciflow/trunk/169497 -> ciflow/trunk/169497 2025-12-04T09:43:54.1760182Z * [new tag] ciflow/trunk/169503 -> ciflow/trunk/169503 2025-12-04T09:43:54.1761436Z * [new tag] ciflow/trunk/169505 -> ciflow/trunk/169505 2025-12-04T09:43:54.1762694Z * [new tag] ciflow/trunk/169507 -> ciflow/trunk/169507 2025-12-04T09:43:54.1763934Z * [new tag] ciflow/trunk/169514 -> ciflow/trunk/169514 2025-12-04T09:43:54.1765301Z * [new tag] ciflow/trunk/169517 -> ciflow/trunk/169517 2025-12-04T09:43:54.1766519Z * [new tag] ciflow/trunk/169519 -> ciflow/trunk/169519 2025-12-04T09:43:54.1767710Z * [new tag] ciflow/trunk/169528 -> ciflow/trunk/169528 2025-12-04T09:43:54.1768948Z * [new tag] ciflow/trunk/169541 -> ciflow/trunk/169541 2025-12-04T09:43:54.1770290Z * [new tag] ciflow/trunk/169555 -> ciflow/trunk/169555 2025-12-04T09:43:54.1771991Z * [new tag] ciflow/unstable/123 -> ciflow/unstable/123 2025-12-04T09:43:54.1773412Z * [new tag] ciflow/vllm/165270 -> ciflow/vllm/165270 2025-12-04T09:43:54.1774618Z * [new tag] ciflow/vllm/165274 -> ciflow/vllm/165274 2025-12-04T09:43:54.1775781Z * [new tag] ciflow/vllm/166494 -> ciflow/vllm/166494 2025-12-04T09:43:54.1776963Z * [new tag] ciflow/vllm/169219 -> ciflow/vllm/169219 2025-12-04T09:43:54.1778449Z * [new tag] ciflow/vllm/169220 -> ciflow/vllm/169220 2025-12-04T09:43:54.1779929Z * [new tag] ciflow/xpu/157994 -> ciflow/xpu/157994 2025-12-04T09:43:54.1780899Z * [new tag] ciflow/xpu/159718 -> ciflow/xpu/159718 2025-12-04T09:43:54.1782138Z * [new tag] ciflow/xpu/161940 -> ciflow/xpu/161940 2025-12-04T09:43:54.1783395Z * [new tag] ciflow/xpu/163251 -> ciflow/xpu/163251 2025-12-04T09:43:54.1784772Z * [new tag] ciflow/xpu/166829 -> ciflow/xpu/166829 2025-12-04T09:43:54.1785968Z * [new tag] ciflow/xpu/166843 -> ciflow/xpu/166843 2025-12-04T09:43:54.1787138Z * [new tag] ciflow/xpu/167972 -> ciflow/xpu/167972 2025-12-04T09:43:54.1788322Z * [new tag] ciflow/xpu/167981 -> ciflow/xpu/167981 2025-12-04T09:43:54.1789487Z * [new tag] ciflow/xpu/168213 -> ciflow/xpu/168213 2025-12-04T09:43:54.1790738Z * [new tag] ciflow/xpu/168262 -> ciflow/xpu/168262 2025-12-04T09:43:54.1791919Z * [new tag] ciflow/xpu/168328 -> ciflow/xpu/168328 2025-12-04T09:43:54.1793399Z * [new tag] ciflow/xpu/168950 -> ciflow/xpu/168950 2025-12-04T09:43:54.1795155Z * [new tag] ciflow/xpu/169039 -> ciflow/xpu/169039 2025-12-04T09:43:54.1796642Z * [new tag] ciflow/xpu/169200 -> ciflow/xpu/169200 2025-12-04T09:43:54.1797877Z * [new tag] ciflow/xpu/169203 -> ciflow/xpu/169203 2025-12-04T09:43:54.1799094Z * [new tag] ciflow/xpu/169230 -> ciflow/xpu/169230 2025-12-04T09:43:54.1800381Z * [new tag] ciflow/xpu/169231 -> ciflow/xpu/169231 2025-12-04T09:43:54.1801766Z * [new tag] ciflow/xpu/169241 -> ciflow/xpu/169241 2025-12-04T09:43:54.1802988Z * [new tag] ciflow/xpu/169280 -> ciflow/xpu/169280 2025-12-04T09:43:54.1804678Z * [new tag] ciflow/xpu/169296 -> ciflow/xpu/169296 2025-12-04T09:43:54.1806066Z * [new tag] ciflow/xpu/169353 -> ciflow/xpu/169353 2025-12-04T09:43:54.1807360Z * [new tag] ciflow/xpu/169410 -> ciflow/xpu/169410 2025-12-04T09:43:54.1808576Z * [new tag] ciflow/xpu/169442 -> ciflow/xpu/169442 2025-12-04T09:43:54.1809922Z * [new tag] ciflow/xpu/169555 -> ciflow/xpu/169555 2025-12-04T09:43:54.1811150Z * [new tag] cslpull75 -> cslpull75 2025-12-04T09:43:54.1812429Z * [new tag] cslpull76 -> cslpull76 2025-12-04T09:43:54.1813639Z * [new tag] cslpull77 -> cslpull77 2025-12-04T09:43:54.1815027Z * [new tag] cslpull78 -> cslpull78 2025-12-04T09:43:54.1816431Z * [new tag] cslpull79 -> cslpull79 2025-12-04T09:43:54.1817962Z * [new tag] cslpull80 -> cslpull80 2025-12-04T09:43:54.1819334Z * [new tag] cslpull81 -> cslpull81 2025-12-04T09:43:54.1820624Z * [new tag] cslpull82 -> cslpull82 2025-12-04T09:43:54.1821939Z * [new tag] cslpull83 -> cslpull83 2025-12-04T09:43:54.1823341Z * [new tag] cslpull84 -> cslpull84 2025-12-04T09:43:54.1824737Z * [new tag] cslpull85 -> cslpull85 2025-12-04T09:43:54.1826035Z * [new tag] cslpull86 -> cslpull86 2025-12-04T09:43:54.1827339Z * [new tag] cslpull87 -> cslpull87 2025-12-04T09:43:54.1828703Z * [new tag] cslpull88 -> cslpull88 2025-12-04T09:43:54.1829948Z * [new tag] cslpull89 -> cslpull89 2025-12-04T09:43:54.1831152Z * [new tag] cslpull90 -> cslpull90 2025-12-04T09:43:54.1832762Z * [new tag] cslpull91 -> cslpull91 2025-12-04T09:43:54.1834151Z * [new tag] cslpull92 -> cslpull92 2025-12-04T09:43:54.1835507Z * [new tag] flight_5 -> flight_5 2025-12-04T09:43:54.1837053Z * [new tag] flight_5.1 -> flight_5.1 2025-12-04T09:43:54.1838361Z * [new tag] flight_5.2 -> flight_5.2 2025-12-04T09:43:54.1839778Z * [new tag] flight_5.3 -> flight_5.3 2025-12-04T09:43:54.1841130Z * [new tag] forpull1 -> forpull1 2025-12-04T09:43:54.1842697Z * [new tag] malfet/tag-2ef5611 -> malfet/tag-2ef5611 2025-12-04T09:43:54.1844012Z * [new tag] malfet/tag-317b1a0 -> malfet/tag-317b1a0 2025-12-04T09:43:54.1845372Z * [new tag] malfet/tag-ec6f767 -> malfet/tag-ec6f767 2025-12-04T09:43:54.1846721Z * [new tag] nightly-binary -> nightly-binary 2025-12-04T09:43:54.1848165Z * [new tag] sqzhang_flight4_plus -> sqzhang_flight4_plus 2025-12-04T09:43:54.1849678Z * [new tag] sqzhang_flight_3 -> sqzhang_flight_3 2025-12-04T09:43:54.1851334Z * [new tag] trunk/02d8bd6974cf84b721680d773dbdb1b6f40ce272 -> trunk/02d8bd6974cf84b721680d773dbdb1b6f40ce272 2025-12-04T09:43:54.1852640Z * [new tag] trunk/066997fb38ade71e00d78e9d572e380b5f02bd3e -> trunk/066997fb38ade71e00d78e9d572e380b5f02bd3e 2025-12-04T09:43:54.1854285Z * [new tag] trunk/076e7b19fa1d481ad778d06d2b49ba57d3ce8c88 -> trunk/076e7b19fa1d481ad778d06d2b49ba57d3ce8c88 2025-12-04T09:43:54.1855899Z * [new tag] trunk/07dcc0b83db3211653a38565a24e15acdba75654 -> trunk/07dcc0b83db3211653a38565a24e15acdba75654 2025-12-04T09:43:54.1857727Z * [new tag] trunk/082e96b68dfcd16cab7cfafc4d3d055767dab3eb -> trunk/082e96b68dfcd16cab7cfafc4d3d055767dab3eb 2025-12-04T09:43:54.1859081Z * [new tag] trunk/088048f2fea28ff7d450f65c72419ca45780d30b -> trunk/088048f2fea28ff7d450f65c72419ca45780d30b 2025-12-04T09:43:54.1860478Z * [new tag] trunk/09076941a95c76f4d9ad189d064dfd8baa39e672 -> trunk/09076941a95c76f4d9ad189d064dfd8baa39e672 2025-12-04T09:43:54.1861826Z * [new tag] trunk/0b80a4c62b94402844bf221791c096b0035c6d75 -> trunk/0b80a4c62b94402844bf221791c096b0035c6d75 2025-12-04T09:43:54.1863404Z * [new tag] trunk/0bbbdf1750567a980634ad907a325357ba8ba8f2 -> trunk/0bbbdf1750567a980634ad907a325357ba8ba8f2 2025-12-04T09:43:54.1864990Z * [new tag] trunk/0c281dd78773b2bc17c58ead0e4cd4ac46e775c5 -> trunk/0c281dd78773b2bc17c58ead0e4cd4ac46e775c5 2025-12-04T09:43:54.1866340Z * [new tag] trunk/135f3753c418a6879b1954904184937b67e61688 -> trunk/135f3753c418a6879b1954904184937b67e61688 2025-12-04T09:43:54.1867647Z * [new tag] trunk/15da21026cb13cd20257dc9e96830db108743c10 -> trunk/15da21026cb13cd20257dc9e96830db108743c10 2025-12-04T09:43:54.1869090Z * [new tag] trunk/166efdad2ac827f30fb02504c6017520257f88ec -> trunk/166efdad2ac827f30fb02504c6017520257f88ec 2025-12-04T09:43:54.1871637Z * [new tag] trunk/174272c15fae553d8488140af931f7d8050a313f -> trunk/174272c15fae553d8488140af931f7d8050a313f 2025-12-04T09:43:54.1872224Z * [new tag] trunk/18f3ca08f13b8de61307f5e8cd7d4cccb67e9d11 -> trunk/18f3ca08f13b8de61307f5e8cd7d4cccb67e9d11 2025-12-04T09:43:54.1873487Z * [new tag] trunk/1902eddfe655a15ebcf2c72bd81ade110fdeef63 -> trunk/1902eddfe655a15ebcf2c72bd81ade110fdeef63 2025-12-04T09:43:54.1874867Z * [new tag] trunk/195f92e98d3d66738577f11f22c4b5c8a1c76dd5 -> trunk/195f92e98d3d66738577f11f22c4b5c8a1c76dd5 2025-12-04T09:43:54.1876209Z * [new tag] trunk/1aa13e17de39e3c768ea7aebaad166ce72a06676 -> trunk/1aa13e17de39e3c768ea7aebaad166ce72a06676 2025-12-04T09:43:54.1877602Z * [new tag] trunk/1afe2832f58e24e54a5bfda5a5afa9b96fdea40e -> trunk/1afe2832f58e24e54a5bfda5a5afa9b96fdea40e 2025-12-04T09:43:54.1879178Z * [new tag] trunk/1c87554d74140eaee964ca8b1832cede67f5f520 -> trunk/1c87554d74140eaee964ca8b1832cede67f5f520 2025-12-04T09:43:54.1880558Z * [new tag] trunk/1ccb743b7b5be955f49736c162c4f5004b8a0dd8 -> trunk/1ccb743b7b5be955f49736c162c4f5004b8a0dd8 2025-12-04T09:43:54.1881982Z * [new tag] trunk/1cee47d6ce0a02227185b566593f002dd639ca0c -> trunk/1cee47d6ce0a02227185b566593f002dd639ca0c 2025-12-04T09:43:54.1883207Z * [new tag] trunk/1d21b4df2babe322e5d085ceb6de884eb260a62d -> trunk/1d21b4df2babe322e5d085ceb6de884eb260a62d 2025-12-04T09:43:54.1884608Z * [new tag] trunk/1e34fb2550e4aa650314f7a6d9f6daf4da7478a8 -> trunk/1e34fb2550e4aa650314f7a6d9f6daf4da7478a8 2025-12-04T09:43:54.1886021Z * [new tag] trunk/1e526fb5b1d93bfc70691c5c3955fdffc1b7b7de -> trunk/1e526fb5b1d93bfc70691c5c3955fdffc1b7b7de 2025-12-04T09:43:54.1887386Z * [new tag] trunk/1ee32a8b1f554a312d79bad01ded24f38cd95543 -> trunk/1ee32a8b1f554a312d79bad01ded24f38cd95543 2025-12-04T09:43:54.1888753Z * [new tag] trunk/201e2c4117eb9744594dad6a5c18213d7b4705d7 -> trunk/201e2c4117eb9744594dad6a5c18213d7b4705d7 2025-12-04T09:43:54.1890133Z * [new tag] trunk/2353a0f60eb4b4cb6675907a7fa9fbedc1c02e7f -> trunk/2353a0f60eb4b4cb6675907a7fa9fbedc1c02e7f 2025-12-04T09:43:54.1891697Z * [new tag] trunk/285779b1621cf9f073a062b0889a642d200308d9 -> trunk/285779b1621cf9f073a062b0889a642d200308d9 2025-12-04T09:43:54.1892963Z * [new tag] trunk/2887faaec6295d081580d09fce161201826c6d87 -> trunk/2887faaec6295d081580d09fce161201826c6d87 2025-12-04T09:43:54.1894413Z * [new tag] trunk/296e67c92635443c67b11c0ae1bd045f03ebb7bc -> trunk/296e67c92635443c67b11c0ae1bd045f03ebb7bc 2025-12-04T09:43:54.1895805Z * [new tag] trunk/29856679769b3dede478767e2fe6cfb51197cb25 -> trunk/29856679769b3dede478767e2fe6cfb51197cb25 2025-12-04T09:43:54.1897185Z * [new tag] trunk/29e5455a4740c326ab187c7aa7b5ef98034ea563 -> trunk/29e5455a4740c326ab187c7aa7b5ef98034ea563 2025-12-04T09:43:54.1898544Z * [new tag] trunk/2ac3ef882afb23136adc188975f0a8802fc68adf -> trunk/2ac3ef882afb23136adc188975f0a8802fc68adf 2025-12-04T09:43:54.1899786Z * [new tag] trunk/2bec68e73b64715354af076ad309335f943e36cd -> trunk/2bec68e73b64715354af076ad309335f943e36cd 2025-12-04T09:43:54.1901203Z * [new tag] trunk/2c87367e6f88662cd5cedbd1537748b7948c38e1 -> trunk/2c87367e6f88662cd5cedbd1537748b7948c38e1 2025-12-04T09:43:54.1902718Z * [new tag] trunk/2d1f78fe3ec13820f136a2e0336da12a25f41708 -> trunk/2d1f78fe3ec13820f136a2e0336da12a25f41708 2025-12-04T09:43:54.1904174Z * [new tag] trunk/2df6058f116a65722a0e03073402feb242572d35 -> trunk/2df6058f116a65722a0e03073402feb242572d35 2025-12-04T09:43:54.1905562Z * [new tag] trunk/2e0c2e170fe658c440775c8e5c44228aafcc47ec -> trunk/2e0c2e170fe658c440775c8e5c44228aafcc47ec 2025-12-04T09:43:54.1906998Z * [new tag] trunk/2f9b7dad7b5419b063bd0f2e204de192720ebb94 -> trunk/2f9b7dad7b5419b063bd0f2e204de192720ebb94 2025-12-04T09:43:54.1908322Z * [new tag] trunk/305168768a95d69c444df5cd334bb774edfe06f1 -> trunk/305168768a95d69c444df5cd334bb774edfe06f1 2025-12-04T09:43:54.1909790Z * [new tag] trunk/31fc12773026e8e00f054dd79ad9b2491e693b48 -> trunk/31fc12773026e8e00f054dd79ad9b2491e693b48 2025-12-04T09:43:54.1911096Z * [new tag] trunk/320de0c6b0a3e7c6d2693ea5c28d5d0156ba7991 -> trunk/320de0c6b0a3e7c6d2693ea5c28d5d0156ba7991 2025-12-04T09:43:54.1912566Z * [new tag] trunk/3418bd29475dff06695045fcdf93e7d0dac67da8 -> trunk/3418bd29475dff06695045fcdf93e7d0dac67da8 2025-12-04T09:43:54.1914057Z * [new tag] trunk/34a98608afa0cb5b48f0d6d30432fdd0a2614ddf -> trunk/34a98608afa0cb5b48f0d6d30432fdd0a2614ddf 2025-12-04T09:43:54.1915333Z * [new tag] trunk/35b7a9a26c5923d98aebaa41a031dae21788a9ee -> trunk/35b7a9a26c5923d98aebaa41a031dae21788a9ee 2025-12-04T09:43:54.1916745Z * [new tag] trunk/39d07dbf03a911bdd45d1af78d8638dc92074938 -> trunk/39d07dbf03a911bdd45d1af78d8638dc92074938 2025-12-04T09:43:54.1917985Z * [new tag] trunk/3cd98b4205ada151042cc7ff097a82d4a4b18725 -> trunk/3cd98b4205ada151042cc7ff097a82d4a4b18725 2025-12-04T09:43:54.1919444Z * [new tag] trunk/3d35fd20a78ff4d016fa80f4e5fad37191d7bcae -> trunk/3d35fd20a78ff4d016fa80f4e5fad37191d7bcae 2025-12-04T09:43:54.1920837Z * [new tag] trunk/409a5fee945c46a3edaf5df162812f201bfd7b2f -> trunk/409a5fee945c46a3edaf5df162812f201bfd7b2f 2025-12-04T09:43:54.1922359Z * [new tag] trunk/42e9005cda22da3f1c559c3649218cebd671027c -> trunk/42e9005cda22da3f1c559c3649218cebd671027c 2025-12-04T09:43:54.1923732Z * [new tag] trunk/43b94713bbf340d3c124fde02d0f73add4021247 -> trunk/43b94713bbf340d3c124fde02d0f73add4021247 2025-12-04T09:43:54.1925081Z * [new tag] trunk/44ac69388a4a5eb463dbd2a13f00d1e3b924566c -> trunk/44ac69388a4a5eb463dbd2a13f00d1e3b924566c 2025-12-04T09:43:54.1926435Z * [new tag] trunk/45d14e2497292be06ad36eaa1aaaf7c630a2586a -> trunk/45d14e2497292be06ad36eaa1aaaf7c630a2586a 2025-12-04T09:43:54.1927770Z * [new tag] trunk/45d310ad84854dff730c0b12e577d7998d978686 -> trunk/45d310ad84854dff730c0b12e577d7998d978686 2025-12-04T09:43:54.1929340Z * [new tag] trunk/47b28ddf7bd74b50fa93b307a7d3b183a6d77f54 -> trunk/47b28ddf7bd74b50fa93b307a7d3b183a6d77f54 2025-12-04T09:43:54.1930609Z * [new tag] trunk/481e5ab336275bd3acd5fa8a611b05b4469012af -> trunk/481e5ab336275bd3acd5fa8a611b05b4469012af 2025-12-04T09:43:54.1932025Z * [new tag] trunk/491731647f6b8a9345dcfb3bc9416aea254a7d96 -> trunk/491731647f6b8a9345dcfb3bc9416aea254a7d96 2025-12-04T09:43:54.1933333Z * [new tag] trunk/49a04d26088acc17d948ddd66920f3e16371e873 -> trunk/49a04d26088acc17d948ddd66920f3e16371e873 2025-12-04T09:43:54.1934768Z * [new tag] trunk/4bebc827c47d2f1f0fa1a417a5201a97aef3d985 -> trunk/4bebc827c47d2f1f0fa1a417a5201a97aef3d985 2025-12-04T09:43:54.1936057Z * [new tag] trunk/4c246677784c6a14bc2dbb9ff8773ef0a3a3222f -> trunk/4c246677784c6a14bc2dbb9ff8773ef0a3a3222f 2025-12-04T09:43:54.1937590Z * [new tag] trunk/4cfb47ff548b6d996641058cf04a70e311a4c3aa -> trunk/4cfb47ff548b6d996641058cf04a70e311a4c3aa 2025-12-04T09:43:54.1939056Z * [new tag] trunk/4e0061c1aa52f606dda8cfab0bd7591e588faf2c -> trunk/4e0061c1aa52f606dda8cfab0bd7591e588faf2c 2025-12-04T09:43:54.1940729Z * [new tag] trunk/4fefb8e7e942386ffac764a41b232241f82bea3a -> trunk/4fefb8e7e942386ffac764a41b232241f82bea3a 2025-12-04T09:43:54.1942101Z * [new tag] trunk/503b2640023521f5a35cd9a52fc8033d73a95d0d -> trunk/503b2640023521f5a35cd9a52fc8033d73a95d0d 2025-12-04T09:43:54.1943483Z * [new tag] trunk/518c2b1b3dab9a2ef2849e04b3bc2f20c1c41db9 -> trunk/518c2b1b3dab9a2ef2849e04b3bc2f20c1c41db9 2025-12-04T09:43:54.1945077Z * [new tag] trunk/5191b2fa68ba19960912bfd7fd721c79d76bb1f3 -> trunk/5191b2fa68ba19960912bfd7fd721c79d76bb1f3 2025-12-04T09:43:54.1946489Z * [new tag] trunk/52ac0f0dc4acacd219f1317fbc28ec631c01e07a -> trunk/52ac0f0dc4acacd219f1317fbc28ec631c01e07a 2025-12-04T09:43:54.1948331Z * [new tag] trunk/539ba711b029de9f191070f4f0d12f18f5b7f292 -> trunk/539ba711b029de9f191070f4f0d12f18f5b7f292 2025-12-04T09:43:54.1949772Z * [new tag] trunk/556375b55deebebbc56cb7aef81f4d52f031ba28 -> trunk/556375b55deebebbc56cb7aef81f4d52f031ba28 2025-12-04T09:43:54.1951182Z * [new tag] trunk/55c4ab554845481d0a69a3811937575fe8bb1a66 -> trunk/55c4ab554845481d0a69a3811937575fe8bb1a66 2025-12-04T09:43:54.1952568Z * [new tag] trunk/5634469fda9e5d98869c82c7d03bb08914245f96 -> trunk/5634469fda9e5d98869c82c7d03bb08914245f96 2025-12-04T09:43:54.1953817Z * [new tag] trunk/5778f6ff894686a975a9a23645178ae4c87ad5dc -> trunk/5778f6ff894686a975a9a23645178ae4c87ad5dc 2025-12-04T09:43:54.1955273Z * [new tag] trunk/587d63a3e07de5dc91065f9ef70bcacda9989068 -> trunk/587d63a3e07de5dc91065f9ef70bcacda9989068 2025-12-04T09:43:54.1956645Z * [new tag] trunk/597930f6b568852356ca9795dac76f9e4653adbd -> trunk/597930f6b568852356ca9795dac76f9e4653adbd 2025-12-04T09:43:54.1957927Z * [new tag] trunk/597df3a4e2a67b9fdbe1a89b2f4d74f822274db6 -> trunk/597df3a4e2a67b9fdbe1a89b2f4d74f822274db6 2025-12-04T09:43:54.1959426Z * [new tag] trunk/59abd50e931f4efb21b053f7a2911f5d8a49d883 -> trunk/59abd50e931f4efb21b053f7a2911f5d8a49d883 2025-12-04T09:43:54.1960877Z * [new tag] trunk/5a607febc04c3a2b5824c75f3f60307867439a2c -> trunk/5a607febc04c3a2b5824c75f3f60307867439a2c 2025-12-04T09:43:54.1962261Z * [new tag] trunk/5bf1cdf4755c54ef462b44cb8041b0a57311556b -> trunk/5bf1cdf4755c54ef462b44cb8041b0a57311556b 2025-12-04T09:43:54.1963501Z * [new tag] trunk/5f0030ba63d334d7e8c93a09e41403b89e4c573c -> trunk/5f0030ba63d334d7e8c93a09e41403b89e4c573c 2025-12-04T09:43:54.1964833Z * [new tag] trunk/5f21d27e71268464d362a96c9ac09ea475f7f202 -> trunk/5f21d27e71268464d362a96c9ac09ea475f7f202 2025-12-04T09:43:54.1966275Z * [new tag] trunk/5fafc13038c9988d9ac21fa793fbd5890604b447 -> trunk/5fafc13038c9988d9ac21fa793fbd5890604b447 2025-12-04T09:43:54.1967818Z * [new tag] trunk/61be54a31dc09b59d99b62176fb935aee0b924ef -> trunk/61be54a31dc09b59d99b62176fb935aee0b924ef 2025-12-04T09:43:54.1969174Z * [new tag] trunk/62d3ccd71484ed6a760d909b41487101bbc65719 -> trunk/62d3ccd71484ed6a760d909b41487101bbc65719 2025-12-04T09:43:54.1970563Z * [new tag] trunk/641cdb68ae27668eb441d0e49c87a0602c120c2b -> trunk/641cdb68ae27668eb441d0e49c87a0602c120c2b 2025-12-04T09:43:54.1971921Z * [new tag] trunk/65c4620d6bb0c6029f69762c22b91dda2294da9a -> trunk/65c4620d6bb0c6029f69762c22b91dda2294da9a 2025-12-04T09:43:54.1973315Z * [new tag] trunk/66004b993744b4106bf8afaba71f3c228a804206 -> trunk/66004b993744b4106bf8afaba71f3c228a804206 2025-12-04T09:43:54.1974725Z * [new tag] trunk/6658a04c7ca67acb64512341342e7b3ee13ee386 -> trunk/6658a04c7ca67acb64512341342e7b3ee13ee386 2025-12-04T09:43:54.1976126Z * [new tag] trunk/6864e309092a71f8ab0ca6a4dc7f8a4073fd31c4 -> trunk/6864e309092a71f8ab0ca6a4dc7f8a4073fd31c4 2025-12-04T09:43:54.1977575Z * [new tag] trunk/6c261c6cb07892c90ca19ed51c9705b1659a3f7d -> trunk/6c261c6cb07892c90ca19ed51c9705b1659a3f7d 2025-12-04T09:43:54.1979091Z * [new tag] trunk/6c8b6a043f1628188b6396b3a2a6e000ca68362b -> trunk/6c8b6a043f1628188b6396b3a2a6e000ca68362b 2025-12-04T09:43:54.1980428Z * [new tag] trunk/6ceb4a32f92ae67ce5d7d97931d17401ebf5ffa5 -> trunk/6ceb4a32f92ae67ce5d7d97931d17401ebf5ffa5 2025-12-04T09:43:54.1981904Z * [new tag] trunk/6e404e9b7d6f5fb0de86aa73888c3038248c17f8 -> trunk/6e404e9b7d6f5fb0de86aa73888c3038248c17f8 2025-12-04T09:43:54.1983300Z * [new tag] trunk/6ec30b490aee1db6bcdc7340abddef25784f08ec -> trunk/6ec30b490aee1db6bcdc7340abddef25784f08ec 2025-12-04T09:43:54.1984811Z * [new tag] trunk/6f2783a6c08e1db34275ff25176ffe9aebc30a71 -> trunk/6f2783a6c08e1db34275ff25176ffe9aebc30a71 2025-12-04T09:43:54.1986176Z * [new tag] trunk/6f53fefeb90ad3281119b5cfc4aa9ffd8a066e3d -> trunk/6f53fefeb90ad3281119b5cfc4aa9ffd8a066e3d 2025-12-04T09:43:54.1987595Z * [new tag] trunk/6f7dcf51e46d0c880db1a2f5c70de57adb576f4a -> trunk/6f7dcf51e46d0c880db1a2f5c70de57adb576f4a 2025-12-04T09:43:54.1989097Z * [new tag] trunk/6ff831180d2fa436c7f1c1af3adac641fce9d60e -> trunk/6ff831180d2fa436c7f1c1af3adac641fce9d60e 2025-12-04T09:43:54.1990457Z * [new tag] trunk/70076464a63ab218a7ceefb0e76ccd7131deb8f8 -> trunk/70076464a63ab218a7ceefb0e76ccd7131deb8f8 2025-12-04T09:43:54.1991856Z * [new tag] trunk/70d797a5fc109b20a517646fcaa819477cd0d485 -> trunk/70d797a5fc109b20a517646fcaa819477cd0d485 2025-12-04T09:43:54.1993204Z * [new tag] trunk/7348cb355ff0a6f79cd4871215aea72185748734 -> trunk/7348cb355ff0a6f79cd4871215aea72185748734 2025-12-04T09:43:54.1994698Z * [new tag] trunk/74fe26a1ebe32931783569f2e762e3c2c974901f -> trunk/74fe26a1ebe32931783569f2e762e3c2c974901f 2025-12-04T09:43:54.1996109Z * [new tag] trunk/76aeb8c7e0f795b3fddca134cbea9a69da3ee696 -> trunk/76aeb8c7e0f795b3fddca134cbea9a69da3ee696 2025-12-04T09:43:54.1997330Z * [new tag] trunk/7716da9fb23f27a65b41f9f016a2afadf281c18f -> trunk/7716da9fb23f27a65b41f9f016a2afadf281c18f 2025-12-04T09:43:54.1998733Z * [new tag] trunk/7741edd4ed665f3988052e260863efb508d61a03 -> trunk/7741edd4ed665f3988052e260863efb508d61a03 2025-12-04T09:43:54.2000181Z * [new tag] trunk/78adb3b3df41b45d2368b67226d2f864b78939a6 -> trunk/78adb3b3df41b45d2368b67226d2f864b78939a6 2025-12-04T09:43:54.2001598Z * [new tag] trunk/79d7b178225e5ed24d4e1db74e5abbff848f5fb7 -> trunk/79d7b178225e5ed24d4e1db74e5abbff848f5fb7 2025-12-04T09:43:54.2002881Z * [new tag] trunk/7a1e316115fc6996b3f2336822ba5d5f6179f0c3 -> trunk/7a1e316115fc6996b3f2336822ba5d5f6179f0c3 2025-12-04T09:43:54.2004205Z * [new tag] trunk/7a41b66367c38d0af3e8a90f7be48d6b281e7bca -> trunk/7a41b66367c38d0af3e8a90f7be48d6b281e7bca 2025-12-04T09:43:54.2005584Z * [new tag] trunk/7b7af390ea8541c611d1ce2018a6934188fc197b -> trunk/7b7af390ea8541c611d1ce2018a6934188fc197b 2025-12-04T09:43:54.2006976Z * [new tag] trunk/7ba4680f3755a560af81aa0f688791e367aa3609 -> trunk/7ba4680f3755a560af81aa0f688791e367aa3609 2025-12-04T09:43:54.2008448Z * [new tag] trunk/7bc2a66ded06a0b2549aa51d807edc5dc3e73d1b -> trunk/7bc2a66ded06a0b2549aa51d807edc5dc3e73d1b 2025-12-04T09:43:54.2009652Z * [new tag] trunk/7c648509a7470ace9fb2bae960dd4790f7e943e9 -> trunk/7c648509a7470ace9fb2bae960dd4790f7e943e9 2025-12-04T09:43:54.2010947Z * [new tag] trunk/7cbc2d034cecd21ab5c9707d0a9c525c17143fb8 -> trunk/7cbc2d034cecd21ab5c9707d0a9c525c17143fb8 2025-12-04T09:43:54.2012447Z * [new tag] trunk/7d1bbaf4ba301ea3fba6f3c7bc02d58f6417aaed -> trunk/7d1bbaf4ba301ea3fba6f3c7bc02d58f6417aaed 2025-12-04T09:43:54.2013957Z * [new tag] trunk/7d2a33e4ebf60b217a3cd77feae19231eb996fc8 -> trunk/7d2a33e4ebf60b217a3cd77feae19231eb996fc8 2025-12-04T09:43:54.2015216Z * [new tag] trunk/7eb625920054b1126a7d2d99818aaa188c6ba95e -> trunk/7eb625920054b1126a7d2d99818aaa188c6ba95e 2025-12-04T09:43:54.2016467Z * [new tag] trunk/7f55ba19c456a3d6cc443dd9edb6bb7cca677ead -> trunk/7f55ba19c456a3d6cc443dd9edb6bb7cca677ead 2025-12-04T09:43:54.2017853Z * [new tag] trunk/81af382128efa094d8702e18f2c133760904c718 -> trunk/81af382128efa094d8702e18f2c133760904c718 2025-12-04T09:43:54.2019498Z * [new tag] trunk/84149583d483e9c973c9a0feda70e4f3964947b0 -> trunk/84149583d483e9c973c9a0feda70e4f3964947b0 2025-12-04T09:43:54.2021143Z * [new tag] trunk/85a315917efe82c24306be805c584ec044951c75 -> trunk/85a315917efe82c24306be805c584ec044951c75 2025-12-04T09:43:54.2022493Z * [new tag] trunk/87329491c82a5f8c1cc4ec11d8f55a5de2551ece -> trunk/87329491c82a5f8c1cc4ec11d8f55a5de2551ece 2025-12-04T09:43:54.2023842Z * [new tag] trunk/892640e25aeefa8007c5af837214b4502b6b62a6 -> trunk/892640e25aeefa8007c5af837214b4502b6b62a6 2025-12-04T09:43:54.2025464Z * [new tag] trunk/89e3bbcb5b5321dc8b9520b4d5a8ee60cea1d0b4 -> trunk/89e3bbcb5b5321dc8b9520b4d5a8ee60cea1d0b4 2025-12-04T09:43:54.2026807Z * [new tag] trunk/8c73bbbb02159223c0c97d268a0a74cb78158a1c -> trunk/8c73bbbb02159223c0c97d268a0a74cb78158a1c 2025-12-04T09:43:54.2028199Z * [new tag] trunk/8d56e98c8db988a22cb2dfaeefb30bc7d2a3cc43 -> trunk/8d56e98c8db988a22cb2dfaeefb30bc7d2a3cc43 2025-12-04T09:43:54.2029666Z * [new tag] trunk/8d9dd9603e5ee26c01007f0cd4f018e584840922 -> trunk/8d9dd9603e5ee26c01007f0cd4f018e584840922 2025-12-04T09:43:54.2030993Z * [new tag] trunk/8ef0c0b02b062d75e7c9be2594914a3e784d23ca -> trunk/8ef0c0b02b062d75e7c9be2594914a3e784d23ca 2025-12-04T09:43:54.2032355Z * [new tag] trunk/90b27e7e8352cde97d32ddad24740ef819633f38 -> trunk/90b27e7e8352cde97d32ddad24740ef819633f38 2025-12-04T09:43:54.2033633Z * [new tag] trunk/90f0139e64b2951815d524b6a373bed20c4fbf90 -> trunk/90f0139e64b2951815d524b6a373bed20c4fbf90 2025-12-04T09:43:54.2034909Z * [new tag] trunk/93d0d6838c56af59b0dba794e6aa08f0c1c7799c -> trunk/93d0d6838c56af59b0dba794e6aa08f0c1c7799c 2025-12-04T09:43:54.2036408Z * [new tag] trunk/94ca8d5f1e81fea3ae488650a0fb6795049a9f87 -> trunk/94ca8d5f1e81fea3ae488650a0fb6795049a9f87 2025-12-04T09:43:54.2038280Z * [new tag] trunk/9844fbeadd5cebdf1281d6fbf79164139c352693 -> trunk/9844fbeadd5cebdf1281d6fbf79164139c352693 2025-12-04T09:43:54.2039674Z * [new tag] trunk/99024dec888ec1e50b546822a32b6fb2f35e5eaa -> trunk/99024dec888ec1e50b546822a32b6fb2f35e5eaa 2025-12-04T09:43:54.2041063Z * [new tag] trunk/9a296e640fc88aa44d275b48cd9cc30c573b169d -> trunk/9a296e640fc88aa44d275b48cd9cc30c573b169d 2025-12-04T09:43:54.2042515Z * [new tag] trunk/9b3e34d8589b29f7b4e7fab6f78711b7ca6e4639 -> trunk/9b3e34d8589b29f7b4e7fab6f78711b7ca6e4639 2025-12-04T09:43:54.2043887Z * [new tag] trunk/9cd055e547e9b67a5f9827f8999c38d7eda1bcb8 -> trunk/9cd055e547e9b67a5f9827f8999c38d7eda1bcb8 2025-12-04T09:43:54.2045274Z * [new tag] trunk/9f0df5686cb4ada94f94620acba2e3c3f363b11d -> trunk/9f0df5686cb4ada94f94620acba2e3c3f363b11d 2025-12-04T09:43:54.2046676Z * [new tag] trunk/9f7fceb887d0cfa0326a59b887821c63ff11340a -> trunk/9f7fceb887d0cfa0326a59b887821c63ff11340a 2025-12-04T09:43:54.2048067Z * [new tag] trunk/9f8ef8855d3078d70f7b782540ff2aaf158d6742 -> trunk/9f8ef8855d3078d70f7b782540ff2aaf158d6742 2025-12-04T09:43:54.2049563Z * [new tag] trunk/9fb52efc797b47a1f425a03aa5e47b866d8b1098 -> trunk/9fb52efc797b47a1f425a03aa5e47b866d8b1098 2025-12-04T09:43:54.2050966Z * [new tag] trunk/9ff4a2ebc5762d46c73e46b1b523d7ff349fedfa -> trunk/9ff4a2ebc5762d46c73e46b1b523d7ff349fedfa 2025-12-04T09:43:54.2052626Z * [new tag] trunk/a0f3937b94422354538ebbd47202d5b0e8a3fd0d -> trunk/a0f3937b94422354538ebbd47202d5b0e8a3fd0d 2025-12-04T09:43:54.2053927Z * [new tag] trunk/a15066c28b3145e6edbfc88359d0411d14cfc70c -> trunk/a15066c28b3145e6edbfc88359d0411d14cfc70c 2025-12-04T09:43:54.2055288Z * [new tag] trunk/a20f775e82564d2a9979221ed7f3b8d7cf54ce90 -> trunk/a20f775e82564d2a9979221ed7f3b8d7cf54ce90 2025-12-04T09:43:54.2056754Z * [new tag] trunk/a2973fb00ec002dd4b6bbf07385f066efb259b8c -> trunk/a2973fb00ec002dd4b6bbf07385f066efb259b8c 2025-12-04T09:43:54.2058071Z * [new tag] trunk/a7dc6dab9ad911259d4801c502907e531594db45 -> trunk/a7dc6dab9ad911259d4801c502907e531594db45 2025-12-04T09:43:54.2059532Z * [new tag] trunk/a951a9cee65c01660bbc6e6fded90ecb10fa6109 -> trunk/a951a9cee65c01660bbc6e6fded90ecb10fa6109 2025-12-04T09:43:54.2061055Z * [new tag] trunk/abfa1a6d65c7c159e35c72c25979b9da4971689e -> trunk/abfa1a6d65c7c159e35c72c25979b9da4971689e 2025-12-04T09:43:54.2062536Z * [new tag] trunk/ae3a2395bf66151078e2d201716f7d63ce1c6f3e -> trunk/ae3a2395bf66151078e2d201716f7d63ce1c6f3e 2025-12-04T09:43:54.2063879Z * [new tag] trunk/afdff7f0325080dedac44d080cb5a3b0e65e6c5e -> trunk/afdff7f0325080dedac44d080cb5a3b0e65e6c5e 2025-12-04T09:43:54.2065232Z * [new tag] trunk/b1aed4e7a72c03a38f44543aaea0dae2e9b76d48 -> trunk/b1aed4e7a72c03a38f44543aaea0dae2e9b76d48 2025-12-04T09:43:54.2066735Z * [new tag] trunk/b1decff555cd50e2123c8c6e25cc0d447c411f62 -> trunk/b1decff555cd50e2123c8c6e25cc0d447c411f62 2025-12-04T09:43:54.2068252Z * [new tag] trunk/b2b6b034c9fd08672c40e63ef243556ad4c49bd2 -> trunk/b2b6b034c9fd08672c40e63ef243556ad4c49bd2 2025-12-04T09:43:54.2069650Z * [new tag] trunk/b39813b4a04931682b0491adba2138d01d716d99 -> trunk/b39813b4a04931682b0491adba2138d01d716d99 2025-12-04T09:43:54.2071102Z * [new tag] trunk/b3a7edb2311367974cc7cd764cfb11a5d6758b24 -> trunk/b3a7edb2311367974cc7cd764cfb11a5d6758b24 2025-12-04T09:43:54.2072622Z * [new tag] trunk/b4cc1329c86acaef6d42c1fac7169b8d870ab0d7 -> trunk/b4cc1329c86acaef6d42c1fac7169b8d870ab0d7 2025-12-04T09:43:54.2074059Z * [new tag] trunk/b555c39217f765759954a4f9f9bd1e9b87bed11a -> trunk/b555c39217f765759954a4f9f9bd1e9b87bed11a 2025-12-04T09:43:54.2075601Z * [new tag] trunk/b6b6c80379388b7f9932c3e6a0f9907bf430e417 -> trunk/b6b6c80379388b7f9932c3e6a0f9907bf430e417 2025-12-04T09:43:54.2077030Z * [new tag] trunk/b6b6d912df0b6f4082f8e50b18bd1de1dd7325f4 -> trunk/b6b6d912df0b6f4082f8e50b18bd1de1dd7325f4 2025-12-04T09:43:54.2078638Z * [new tag] trunk/b7d60685f8cbc939b68a20871e90db67e729329b -> trunk/b7d60685f8cbc939b68a20871e90db67e729329b 2025-12-04T09:43:54.2081720Z * [new tag] trunk/b7f6b9a4fc6259f7af068f31868b3119bb1bac3e -> trunk/b7f6b9a4fc6259f7af068f31868b3119bb1bac3e 2025-12-04T09:43:54.2083216Z * [new tag] trunk/b8c4ba3593761e7b2a3ebd86f040fb07b47c02cf -> trunk/b8c4ba3593761e7b2a3ebd86f040fb07b47c02cf 2025-12-04T09:43:54.2084553Z * [new tag] trunk/b9c8f3a4884befb965ff42620ce44a71b04887f5 -> trunk/b9c8f3a4884befb965ff42620ce44a71b04887f5 2025-12-04T09:43:54.2086083Z * [new tag] trunk/ba1412546f3082c0958c077acc2025e4dbc33f1f -> trunk/ba1412546f3082c0958c077acc2025e4dbc33f1f 2025-12-04T09:43:54.2087496Z * [new tag] trunk/bac403c0b38c63bdbcc0c31f1c2b0bc0260f610f -> trunk/bac403c0b38c63bdbcc0c31f1c2b0bc0260f610f 2025-12-04T09:43:54.2088887Z * [new tag] trunk/bb3034198b459401fabeab254e1b99f0115046e2 -> trunk/bb3034198b459401fabeab254e1b99f0115046e2 2025-12-04T09:43:54.2090312Z * [new tag] trunk/bc39b2b3bc7a6e19a42e62bd576974035086fe55 -> trunk/bc39b2b3bc7a6e19a42e62bd576974035086fe55 2025-12-04T09:43:54.2091944Z * [new tag] trunk/bc43d5b297f207a11d83d77ddf0152bdaabe15a8 -> trunk/bc43d5b297f207a11d83d77ddf0152bdaabe15a8 2025-12-04T09:43:54.2093319Z * [new tag] trunk/bc6a4863c7246a6493d16d4ea6eee71ec07c6a09 -> trunk/bc6a4863c7246a6493d16d4ea6eee71ec07c6a09 2025-12-04T09:43:54.2094697Z * [new tag] trunk/bea4912944defdbcb8b061800caab6cbbbd01df5 -> trunk/bea4912944defdbcb8b061800caab6cbbbd01df5 2025-12-04T09:43:54.2096349Z * [new tag] trunk/c04e2c656f48d82d1521b867bbbf03967b9b7564 -> trunk/c04e2c656f48d82d1521b867bbbf03967b9b7564 2025-12-04T09:43:54.2097821Z * [new tag] trunk/c0660bcee27e7d7731634e274576a7081882bede -> trunk/c0660bcee27e7d7731634e274576a7081882bede 2025-12-04T09:43:54.2099243Z * [new tag] trunk/c178ed43d3d99cbefe84fbfb21d6f282b20d62ac -> trunk/c178ed43d3d99cbefe84fbfb21d6f282b20d62ac 2025-12-04T09:43:54.2100653Z * [new tag] trunk/c55b1e8f61d041ee436d697449eb028931d574fb -> trunk/c55b1e8f61d041ee436d697449eb028931d574fb 2025-12-04T09:43:54.2101954Z * [new tag] trunk/c6ae7579fe12fe75f1a8f7043a494c90567273f1 -> trunk/c6ae7579fe12fe75f1a8f7043a494c90567273f1 2025-12-04T09:43:54.2103620Z * [new tag] trunk/c8210e7d94bad5ae21ac389fa4ba8a463c76c4d0 -> trunk/c8210e7d94bad5ae21ac389fa4ba8a463c76c4d0 2025-12-04T09:43:54.2105141Z * [new tag] trunk/cc0853af42122f8185321f542616f4474e717f09 -> trunk/cc0853af42122f8185321f542616f4474e717f09 2025-12-04T09:43:54.2106522Z * [new tag] trunk/cddec6562eabfa390d014fa3741a5659cf9c94c9 -> trunk/cddec6562eabfa390d014fa3741a5659cf9c94c9 2025-12-04T09:43:54.2108047Z * [new tag] trunk/ce5e7e3bf1f4b69a4f4f93d288ba75b906df492a -> trunk/ce5e7e3bf1f4b69a4f4f93d288ba75b906df492a 2025-12-04T09:43:54.2109484Z * [new tag] trunk/d038b0130ec7c20ebcac219301292fd8e98a1ace -> trunk/d038b0130ec7c20ebcac219301292fd8e98a1ace 2025-12-04T09:43:54.2110837Z * [new tag] trunk/d16447dacaf2420ea175f0c275c75da951f57d39 -> trunk/d16447dacaf2420ea175f0c275c75da951f57d39 2025-12-04T09:43:54.2112252Z * [new tag] trunk/d19f1e8cab6810bb2e99141f9976665954c67a50 -> trunk/d19f1e8cab6810bb2e99141f9976665954c67a50 2025-12-04T09:43:54.2113667Z * [new tag] trunk/d1c9f03b2a5af4104721712f8cdffe9b4f340c01 -> trunk/d1c9f03b2a5af4104721712f8cdffe9b4f340c01 2025-12-04T09:43:54.2115440Z * [new tag] trunk/d40f4950f2b7f7aa380a22fe0f6166e71680fbcf -> trunk/d40f4950f2b7f7aa380a22fe0f6166e71680fbcf 2025-12-04T09:43:54.2116735Z * [new tag] trunk/d5038950bacfe36bbf24a47a455fe76901deb8e8 -> trunk/d5038950bacfe36bbf24a47a455fe76901deb8e8 2025-12-04T09:43:54.2118065Z * [new tag] trunk/d54ff42903c2ae0533931ff11d23b35f875bdb3d -> trunk/d54ff42903c2ae0533931ff11d23b35f875bdb3d 2025-12-04T09:43:54.2119795Z * [new tag] trunk/d76697633a2d2b9cced1ae21161849b33bfe7e47 -> trunk/d76697633a2d2b9cced1ae21161849b33bfe7e47 2025-12-04T09:43:54.2121010Z * [new tag] trunk/d78f52b199c547106d4cd9d2856dd0805c118bf1 -> trunk/d78f52b199c547106d4cd9d2856dd0805c118bf1 2025-12-04T09:43:54.2122536Z * [new tag] trunk/d8fd5c6eed28e5004150691d048a3f6785e19a8e -> trunk/d8fd5c6eed28e5004150691d048a3f6785e19a8e 2025-12-04T09:43:54.2123853Z * [new tag] trunk/d900f5e86745dec76713f4b0ef07005ef36b2f5a -> trunk/d900f5e86745dec76713f4b0ef07005ef36b2f5a 2025-12-04T09:43:54.2126916Z * [new tag] trunk/d973dc6b87d763859fe1c5bd1287e3b6b1c49d1b -> trunk/d973dc6b87d763859fe1c5bd1287e3b6b1c49d1b 2025-12-04T09:43:54.2128182Z * [new tag] trunk/d998c03304cb6ede76e1ed535b4ddeb6c2bf40ec -> trunk/d998c03304cb6ede76e1ed535b4ddeb6c2bf40ec 2025-12-04T09:43:54.2129808Z * [new tag] trunk/d9cb8a70833101dbbe16b99520cfbdd70d0a87bf -> trunk/d9cb8a70833101dbbe16b99520cfbdd70d0a87bf 2025-12-04T09:43:54.2131049Z * [new tag] trunk/d9d5e91b43f70eb8637af55db6856d49be391ffd -> trunk/d9d5e91b43f70eb8637af55db6856d49be391ffd 2025-12-04T09:43:54.2132392Z * [new tag] trunk/dd18a75336a4fbd7497955cc5665904724fce889 -> trunk/dd18a75336a4fbd7497955cc5665904724fce889 2025-12-04T09:43:54.2134824Z * [new tag] trunk/ded9bcd61a059bf723e6e84689552962b480ea77 -> trunk/ded9bcd61a059bf723e6e84689552962b480ea77 2025-12-04T09:43:54.2136783Z * [new tag] trunk/dfbd3714d15c37a7b83b322a6b60f997fc00f50c -> trunk/dfbd3714d15c37a7b83b322a6b60f997fc00f50c 2025-12-04T09:43:54.2138436Z * [new tag] trunk/e115f9f4e4b039f8e9a642aaa2bd8254a920541b -> trunk/e115f9f4e4b039f8e9a642aaa2bd8254a920541b 2025-12-04T09:43:54.2139543Z * [new tag] trunk/e3f24fd73ad74c6e7176687986436956c7c18235 -> trunk/e3f24fd73ad74c6e7176687986436956c7c18235 2025-12-04T09:43:54.2141179Z * [new tag] trunk/e7d24d3ff93d1503ba63860b7057438ad93f918e -> trunk/e7d24d3ff93d1503ba63860b7057438ad93f918e 2025-12-04T09:43:54.2142452Z * [new tag] trunk/ea7035f462a0d2830865ee86c832bd101e1427fc -> trunk/ea7035f462a0d2830865ee86c832bd101e1427fc 2025-12-04T09:43:54.2144314Z * [new tag] trunk/eabb7ad2128580ef674446027b95bcf4e21e8df3 -> trunk/eabb7ad2128580ef674446027b95bcf4e21e8df3 2025-12-04T09:43:54.2145585Z * [new tag] trunk/eb5c63652a33da42e7018c23df5f20a3eb4c6ccf -> trunk/eb5c63652a33da42e7018c23df5f20a3eb4c6ccf 2025-12-04T09:43:54.2147366Z * [new tag] trunk/ec2c71f5c85021b8938cdafadce24c15a36fd93e -> trunk/ec2c71f5c85021b8938cdafadce24c15a36fd93e 2025-12-04T09:43:54.2148614Z * [new tag] trunk/ecbcc3f6bf327856b435b259ac63cc2f328c4b4e -> trunk/ecbcc3f6bf327856b435b259ac63cc2f328c4b4e 2025-12-04T09:43:54.2150611Z * [new tag] trunk/ee87bbe876c42575e961b32a0827d76bc9782ca2 -> trunk/ee87bbe876c42575e961b32a0827d76bc9782ca2 2025-12-04T09:43:54.2151867Z * [new tag] trunk/ef019d1d431c4c5a95b594cb90d40a50cd00f5e4 -> trunk/ef019d1d431c4c5a95b594cb90d40a50cd00f5e4 2025-12-04T09:43:54.2153503Z * [new tag] trunk/ef8ecc13830a86c4b231f1aad9aba7851db61b53 -> trunk/ef8ecc13830a86c4b231f1aad9aba7851db61b53 2025-12-04T09:43:54.2154715Z * [new tag] trunk/f1076f5510920044912247b1abb8760cb820f598 -> trunk/f1076f5510920044912247b1abb8760cb820f598 2025-12-04T09:43:54.2156406Z * [new tag] trunk/f2d6a75a00a1d648ca9a0abc6a33e14c3dea6c40 -> trunk/f2d6a75a00a1d648ca9a0abc6a33e14c3dea6c40 2025-12-04T09:43:54.2157591Z * [new tag] trunk/f47dd0ddef1359e5b43e4b962412f67b30ecde56 -> trunk/f47dd0ddef1359e5b43e4b962412f67b30ecde56 2025-12-04T09:43:54.2159235Z * [new tag] trunk/f49d32dfa4730dcfb1b60eeeb369b5889da983c8 -> trunk/f49d32dfa4730dcfb1b60eeeb369b5889da983c8 2025-12-04T09:43:54.2160418Z * [new tag] trunk/f4dedf78fc30fd4b93975787ca6074ee89db9467 -> trunk/f4dedf78fc30fd4b93975787ca6074ee89db9467 2025-12-04T09:43:54.2162129Z * [new tag] trunk/f7c0d03819ebed05c4038f095d66d1b8c54aca17 -> trunk/f7c0d03819ebed05c4038f095d66d1b8c54aca17 2025-12-04T09:43:54.2163292Z * [new tag] trunk/f7e1bd80a063e17453c361837ba6ea2570920a73 -> trunk/f7e1bd80a063e17453c361837ba6ea2570920a73 2025-12-04T09:43:54.2164612Z * [new tag] trunk/f9bd6c53624c7c0ea3772de78498326e84c2f0e7 -> trunk/f9bd6c53624c7c0ea3772de78498326e84c2f0e7 2025-12-04T09:43:54.2166322Z * [new tag] trunk/fb5be221a46b51bfc9509013b0d85bc5a9d4f15b -> trunk/fb5be221a46b51bfc9509013b0d85bc5a9d4f15b 2025-12-04T09:43:54.2167528Z * [new tag] trunk/fdf863d5e1de3b2688c9511e96876e34581dbfd7 -> trunk/fdf863d5e1de3b2688c9511e96876e34581dbfd7 2025-12-04T09:43:54.2169508Z * [new tag] trunk/fe0e65adfc0e7ca6e5f57e6ea8b16bd5cc967307 -> trunk/fe0e65adfc0e7ca6e5f57e6ea8b16bd5cc967307 2025-12-04T09:43:54.2170758Z * [new tag] trunk/fec710bf89173f5355468a7ce1afe9157c3d9009 -> trunk/fec710bf89173f5355468a7ce1afe9157c3d9009 2025-12-04T09:43:54.2172558Z * [new tag] trunk/ffd9b0fb4355e97af82fc42cf185c3ffa0fc0a32 -> trunk/ffd9b0fb4355e97af82fc42cf185c3ffa0fc0a32 2025-12-04T09:43:54.2173443Z * [new tag] v0.1.1 -> v0.1.1 2025-12-04T09:43:54.2174941Z * [new tag] v0.1.10 -> v0.1.10 2025-12-04T09:43:54.2176257Z * [new tag] v0.1.11 -> v0.1.11 2025-12-04T09:43:54.2177740Z * [new tag] v0.1.12 -> v0.1.12 2025-12-04T09:43:54.2179443Z * [new tag] v0.1.2 -> v0.1.2 2025-12-04T09:43:54.2180785Z * [new tag] v0.1.3 -> v0.1.3 2025-12-04T09:43:54.2182054Z * [new tag] v0.1.4 -> v0.1.4 2025-12-04T09:43:54.2183407Z * [new tag] v0.1.5 -> v0.1.5 2025-12-04T09:43:54.2185363Z * [new tag] v0.1.6 -> v0.1.6 2025-12-04T09:43:54.2186629Z * [new tag] v0.1.7 -> v0.1.7 2025-12-04T09:43:54.2187907Z * [new tag] v0.1.8 -> v0.1.8 2025-12-04T09:43:54.2189269Z * [new tag] v0.1.9 -> v0.1.9 2025-12-04T09:43:54.2190580Z * [new tag] v0.2.0 -> v0.2.0 2025-12-04T09:43:54.2191973Z * [new tag] v0.3.0 -> v0.3.0 2025-12-04T09:43:54.2193448Z * [new tag] v0.3.1 -> v0.3.1 2025-12-04T09:43:54.2194855Z * [new tag] v0.4.0 -> v0.4.0 2025-12-04T09:43:54.2196162Z * [new tag] v0.4.1 -> v0.4.1 2025-12-04T09:43:54.2197522Z * [new tag] v1.0.0 -> v1.0.0 2025-12-04T09:43:54.2198848Z * [new tag] v1.0.0a0 -> v1.0.0a0 2025-12-04T09:43:54.2200316Z * [new tag] v1.0.1 -> v1.0.1 2025-12-04T09:43:54.2201721Z * [new tag] v1.0rc0 -> v1.0rc0 2025-12-04T09:43:54.2202701Z * [new tag] v1.0rc1 -> v1.0rc1 2025-12-04T09:43:54.2204222Z * [new tag] v1.1.0 -> v1.1.0 2025-12-04T09:43:54.2205749Z * [new tag] v1.1.0a0 -> v1.1.0a0 2025-12-04T09:43:54.2207431Z * [new tag] v1.10.0 -> v1.10.0 2025-12-04T09:43:54.2208826Z * [new tag] v1.10.0-rc1 -> v1.10.0-rc1 2025-12-04T09:43:54.2210150Z * [new tag] v1.10.0-rc2 -> v1.10.0-rc2 2025-12-04T09:43:54.2211137Z * [new tag] v1.10.0-rc3 -> v1.10.0-rc3 2025-12-04T09:43:54.2212677Z * [new tag] v1.10.1 -> v1.10.1 2025-12-04T09:43:54.2213883Z * [new tag] v1.10.1-rc1 -> v1.10.1-rc1 2025-12-04T09:43:54.2214878Z * [new tag] v1.10.2 -> v1.10.2 2025-12-04T09:43:54.2216197Z * [new tag] v1.10.2-rc1 -> v1.10.2-rc1 2025-12-04T09:43:54.2217585Z * [new tag] v1.11.0 -> v1.11.0 2025-12-04T09:43:54.2219004Z * [new tag] v1.11.0-rc1 -> v1.11.0-rc1 2025-12-04T09:43:54.2220407Z * [new tag] v1.11.0-rc2 -> v1.11.0-rc2 2025-12-04T09:43:54.2221847Z * [new tag] v1.11.0-rc3 -> v1.11.0-rc3 2025-12-04T09:43:54.2223243Z * [new tag] v1.11.0-rc4 -> v1.11.0-rc4 2025-12-04T09:43:54.2224790Z * [new tag] v1.11.0-rc5 -> v1.11.0-rc5 2025-12-04T09:43:54.2225815Z * [new tag] v1.11.0-rc6 -> v1.11.0-rc6 2025-12-04T09:43:54.2227145Z * [new tag] v1.11.0-rc7 -> v1.11.0-rc7 2025-12-04T09:43:54.2228712Z * [new tag] v1.12.0 -> v1.12.0 2025-12-04T09:43:54.2229745Z * [new tag] v1.12.0-rc1 -> v1.12.0-rc1 2025-12-04T09:43:54.2231322Z * [new tag] v1.12.0-rc2 -> v1.12.0-rc2 2025-12-04T09:43:54.2232704Z * [new tag] v1.12.0-rc3 -> v1.12.0-rc3 2025-12-04T09:43:54.2234083Z * [new tag] v1.12.0-rc4 -> v1.12.0-rc4 2025-12-04T09:43:54.2235406Z * [new tag] v1.12.0-rc5 -> v1.12.0-rc5 2025-12-04T09:43:54.2236979Z * [new tag] v1.12.0-rc6 -> v1.12.0-rc6 2025-12-04T09:43:54.2237985Z * [new tag] v1.12.0-rc7 -> v1.12.0-rc7 2025-12-04T09:43:54.2239277Z * [new tag] v1.12.0-rc8 -> v1.12.0-rc8 2025-12-04T09:43:54.2240249Z * [new tag] v1.12.1 -> v1.12.1 2025-12-04T09:43:54.2241849Z * [new tag] v1.12.1-rc1 -> v1.12.1-rc1 2025-12-04T09:43:54.2243243Z * [new tag] v1.12.1-rc2 -> v1.12.1-rc2 2025-12-04T09:43:54.2244751Z * [new tag] v1.12.1-rc3 -> v1.12.1-rc3 2025-12-04T09:43:54.2246115Z * [new tag] v1.12.1-rc4 -> v1.12.1-rc4 2025-12-04T09:43:54.2247240Z * [new tag] v1.12.1-rc5 -> v1.12.1-rc5 2025-12-04T09:43:54.2248693Z * [new tag] v1.13.0 -> v1.13.0 2025-12-04T09:43:54.2250020Z * [new tag] v1.13.0-rc1 -> v1.13.0-rc1 2025-12-04T09:43:54.2251449Z * [new tag] v1.13.0-rc2 -> v1.13.0-rc2 2025-12-04T09:43:54.2252741Z * [new tag] v1.13.0-rc3 -> v1.13.0-rc3 2025-12-04T09:43:54.2254217Z * [new tag] v1.13.0-rc4 -> v1.13.0-rc4 2025-12-04T09:43:54.2255214Z * [new tag] v1.13.0-rc5 -> v1.13.0-rc5 2025-12-04T09:43:54.2256525Z * [new tag] v1.13.0-rc6 -> v1.13.0-rc6 2025-12-04T09:43:54.2257978Z * [new tag] v1.13.1 -> v1.13.1 2025-12-04T09:43:54.2259209Z * [new tag] v1.13.1-rc1 -> v1.13.1-rc1 2025-12-04T09:43:54.2260450Z * [new tag] v1.2.0 -> v1.2.0 2025-12-04T09:43:54.2261798Z * [new tag] v1.2.0a0 -> v1.2.0a0 2025-12-04T09:43:54.2263152Z * [new tag] v1.3.0 -> v1.3.0 2025-12-04T09:43:54.2264630Z * [new tag] v1.3.0a0 -> v1.3.0a0 2025-12-04T09:43:54.2265620Z * [new tag] v1.3.1 -> v1.3.1 2025-12-04T09:43:54.2267167Z * [new tag] v1.4.0 -> v1.4.0 2025-12-04T09:43:54.2268517Z * [new tag] v1.4.0a0 -> v1.4.0a0 2025-12-04T09:43:54.2269532Z * [new tag] v1.4.1 -> v1.4.1 2025-12-04T09:43:54.2271503Z * [new tag] v1.5.0 -> v1.5.0 2025-12-04T09:43:54.2272888Z * [new tag] v1.5.0-rc1 -> v1.5.0-rc1 2025-12-04T09:43:54.2274277Z * [new tag] v1.5.0-rc2 -> v1.5.0-rc2 2025-12-04T09:43:54.2275828Z * [new tag] v1.5.0-rc3 -> v1.5.0-rc3 2025-12-04T09:43:54.2277087Z * [new tag] v1.5.0-rc4 -> v1.5.0-rc4 2025-12-04T09:43:54.2278276Z * [new tag] v1.5.0-rc5 -> v1.5.0-rc5 2025-12-04T09:43:54.2280048Z * [new tag] v1.5.1 -> v1.5.1 2025-12-04T09:43:54.2281067Z * [new tag] v1.5.1-rc1 -> v1.5.1-rc1 2025-12-04T09:43:54.2282371Z * [new tag] v1.6.0 -> v1.6.0 2025-12-04T09:43:54.2283794Z * [new tag] v1.6.0-rc1 -> v1.6.0-rc1 2025-12-04T09:43:54.2285352Z * [new tag] v1.6.0-rc2 -> v1.6.0-rc2 2025-12-04T09:43:54.2286564Z * [new tag] v1.6.0-rc3 -> v1.6.0-rc3 2025-12-04T09:43:54.2288015Z * [new tag] v1.6.0-rc4 -> v1.6.0-rc4 2025-12-04T09:43:54.2289382Z * [new tag] v1.6.0-rc5 -> v1.6.0-rc5 2025-12-04T09:43:54.2290764Z * [new tag] v1.6.0-rc6 -> v1.6.0-rc6 2025-12-04T09:43:54.2291826Z * [new tag] v1.6.0-rc7 -> v1.6.0-rc7 2025-12-04T09:43:54.2293418Z * [new tag] v1.7.0 -> v1.7.0 2025-12-04T09:43:54.2294760Z * [new tag] v1.7.0-rc1 -> v1.7.0-rc1 2025-12-04T09:43:54.2296178Z * [new tag] v1.7.0-rc2 -> v1.7.0-rc2 2025-12-04T09:43:54.2297607Z * [new tag] v1.7.0-rc3 -> v1.7.0-rc3 2025-12-04T09:43:54.2298634Z * [new tag] v1.7.0-rc4 -> v1.7.0-rc4 2025-12-04T09:43:54.2300179Z * [new tag] v1.7.1 -> v1.7.1 2025-12-04T09:43:54.2301642Z * [new tag] v1.7.1-rc1 -> v1.7.1-rc1 2025-12-04T09:43:54.2303090Z * [new tag] v1.7.1-rc2 -> v1.7.1-rc2 2025-12-04T09:43:54.2304345Z * [new tag] v1.7.1-rc3 -> v1.7.1-rc3 2025-12-04T09:43:54.2305746Z * [new tag] v1.8.0 -> v1.8.0 2025-12-04T09:43:54.2306913Z * [new tag] v1.8.0-rc1 -> v1.8.0-rc1 2025-12-04T09:43:54.2308370Z * [new tag] v1.8.0-rc2 -> v1.8.0-rc2 2025-12-04T09:43:54.2309703Z * [new tag] v1.8.0-rc3 -> v1.8.0-rc3 2025-12-04T09:43:54.2311005Z * [new tag] v1.8.0-rc4 -> v1.8.0-rc4 2025-12-04T09:43:54.2312020Z * [new tag] v1.8.0-rc5 -> v1.8.0-rc5 2025-12-04T09:43:54.2313328Z * [new tag] v1.8.1 -> v1.8.1 2025-12-04T09:43:54.2314686Z * [new tag] v1.8.1-rc1 -> v1.8.1-rc1 2025-12-04T09:43:54.2315739Z * [new tag] v1.8.1-rc2 -> v1.8.1-rc2 2025-12-04T09:43:54.2317053Z * [new tag] v1.8.1-rc3 -> v1.8.1-rc3 2025-12-04T09:43:54.2318839Z * [new tag] v1.8.2 -> v1.8.2 2025-12-04T09:43:54.2319843Z * [new tag] v1.8.2-rc1 -> v1.8.2-rc1 2025-12-04T09:43:54.2321317Z * [new tag] v1.9.0 -> v1.9.0 2025-12-04T09:43:54.2322706Z * [new tag] v1.9.0-rc1 -> v1.9.0-rc1 2025-12-04T09:43:54.2324187Z * [new tag] v1.9.0-rc2 -> v1.9.0-rc2 2025-12-04T09:43:54.2325572Z * [new tag] v1.9.0-rc3 -> v1.9.0-rc3 2025-12-04T09:43:54.2326729Z * [new tag] v1.9.0-rc4 -> v1.9.0-rc4 2025-12-04T09:43:54.2328113Z * [new tag] v1.9.1 -> v1.9.1 2025-12-04T09:43:54.2329650Z * [new tag] v1.9.1-rc1 -> v1.9.1-rc1 2025-12-04T09:43:54.2330828Z * [new tag] v1.9.1-rc2 -> v1.9.1-rc2 2025-12-04T09:43:54.2332265Z * [new tag] v2.0.0 -> v2.0.0 2025-12-04T09:43:54.2333585Z * [new tag] v2.0.0-rc1 -> v2.0.0-rc1 2025-12-04T09:43:54.2335018Z * [new tag] v2.0.0-rc2 -> v2.0.0-rc2 2025-12-04T09:43:54.2336408Z * [new tag] v2.0.0-rc3 -> v2.0.0-rc3 2025-12-04T09:43:54.2337746Z * [new tag] v2.0.0-rc4 -> v2.0.0-rc4 2025-12-04T09:43:54.2339140Z * [new tag] v2.0.0-rc5 -> v2.0.0-rc5 2025-12-04T09:43:54.2340512Z * [new tag] v2.0.0-rc6 -> v2.0.0-rc6 2025-12-04T09:43:54.2341844Z * [new tag] v2.0.1 -> v2.0.1 2025-12-04T09:43:54.2343309Z * [new tag] v2.0.1-rc1 -> v2.0.1-rc1 2025-12-04T09:43:54.2344561Z * [new tag] v2.0.1-rc2 -> v2.0.1-rc2 2025-12-04T09:43:54.2345946Z * [new tag] v2.0.1-rc3 -> v2.0.1-rc3 2025-12-04T09:43:54.2346962Z * [new tag] v2.0.1-rc4 -> v2.0.1-rc4 2025-12-04T09:43:54.2348906Z * [new tag] v2.1.0 -> v2.1.0 2025-12-04T09:43:54.2350248Z * [new tag] v2.1.0-rc1 -> v2.1.0-rc1 2025-12-04T09:43:54.2351653Z * [new tag] v2.1.0-rc2 -> v2.1.0-rc2 2025-12-04T09:43:54.2353112Z * [new tag] v2.1.0-rc3 -> v2.1.0-rc3 2025-12-04T09:43:54.2354541Z * [new tag] v2.1.0-rc4 -> v2.1.0-rc4 2025-12-04T09:43:54.2355932Z * [new tag] v2.1.0-rc5 -> v2.1.0-rc5 2025-12-04T09:43:54.2357099Z * [new tag] v2.1.0-rc6 -> v2.1.0-rc6 2025-12-04T09:43:54.2358989Z * [new tag] v2.1.1 -> v2.1.1 2025-12-04T09:43:54.2360451Z * [new tag] v2.1.1-rc1 -> v2.1.1-rc1 2025-12-04T09:43:54.2361845Z * [new tag] v2.1.1-rc2 -> v2.1.1-rc2 2025-12-04T09:43:54.2363348Z * [new tag] v2.1.1-rc3 -> v2.1.1-rc3 2025-12-04T09:43:54.2364782Z * [new tag] v2.1.1-rc4 -> v2.1.1-rc4 2025-12-04T09:43:54.2366091Z * [new tag] v2.1.1-rc5 -> v2.1.1-rc5 2025-12-04T09:43:54.2367110Z * [new tag] v2.1.1-rc6 -> v2.1.1-rc6 2025-12-04T09:43:54.2368618Z * [new tag] v2.1.2 -> v2.1.2 2025-12-04T09:43:54.2370061Z * [new tag] v2.1.2-rc1 -> v2.1.2-rc1 2025-12-04T09:43:54.2371490Z * [new tag] v2.1.2-rc2 -> v2.1.2-rc2 2025-12-04T09:43:54.2372510Z * [new tag] v2.1.2-rc3 -> v2.1.2-rc3 2025-12-04T09:43:54.2374093Z * [new tag] v2.2.0 -> v2.2.0 2025-12-04T09:43:54.2375501Z * [new tag] v2.2.0-rc1 -> v2.2.0-rc1 2025-12-04T09:43:54.2376814Z * [new tag] v2.2.0-rc2 -> v2.2.0-rc2 2025-12-04T09:43:54.2378217Z * [new tag] v2.2.0-rc3 -> v2.2.0-rc3 2025-12-04T09:43:54.2379898Z * [new tag] v2.2.0-rc4 -> v2.2.0-rc4 2025-12-04T09:43:54.2381137Z * [new tag] v2.2.0-rc5 -> v2.2.0-rc5 2025-12-04T09:43:54.2382473Z * [new tag] v2.2.0-rc6 -> v2.2.0-rc6 2025-12-04T09:43:54.2383506Z * [new tag] v2.2.0-rc7 -> v2.2.0-rc7 2025-12-04T09:43:54.2384982Z * [new tag] v2.2.0-rc8 -> v2.2.0-rc8 2025-12-04T09:43:54.2386423Z * [new tag] v2.2.1 -> v2.2.1 2025-12-04T09:43:54.2387812Z * [new tag] v2.2.1-rc1 -> v2.2.1-rc1 2025-12-04T09:43:54.2388861Z * [new tag] v2.2.1-rc2 -> v2.2.1-rc2 2025-12-04T09:43:54.2390246Z * [new tag] v2.2.1-rc3 -> v2.2.1-rc3 2025-12-04T09:43:54.2391439Z * [new tag] v2.2.2 -> v2.2.2 2025-12-04T09:43:54.2392864Z * [new tag] v2.2.2-rc1 -> v2.2.2-rc1 2025-12-04T09:43:54.2394057Z * [new tag] v2.2.2-rc2 -> v2.2.2-rc2 2025-12-04T09:43:54.2395082Z * [new tag] v2.2.2-rc3 -> v2.2.2-rc3 2025-12-04T09:43:54.2396848Z * [new tag] v2.3.0 -> v2.3.0 2025-12-04T09:43:54.2397894Z * [new tag] v2.3.0-rc1 -> v2.3.0-rc1 2025-12-04T09:43:54.2399469Z * [new tag] v2.3.0-rc10 -> v2.3.0-rc10 2025-12-04T09:43:54.2400836Z * [new tag] v2.3.0-rc11 -> v2.3.0-rc11 2025-12-04T09:43:54.2401888Z * [new tag] v2.3.0-rc12 -> v2.3.0-rc12 2025-12-04T09:43:54.2403484Z * [new tag] v2.3.0-rc2 -> v2.3.0-rc2 2025-12-04T09:43:54.2404860Z * [new tag] v2.3.0-rc3 -> v2.3.0-rc3 2025-12-04T09:43:54.2406248Z * [new tag] v2.3.0-rc4 -> v2.3.0-rc4 2025-12-04T09:43:54.2407593Z * [new tag] v2.3.0-rc5 -> v2.3.0-rc5 2025-12-04T09:43:54.2408946Z * [new tag] v2.3.0-rc6 -> v2.3.0-rc6 2025-12-04T09:43:54.2410372Z * [new tag] v2.3.0-rc7 -> v2.3.0-rc7 2025-12-04T09:43:54.2411751Z * [new tag] v2.3.0-rc8 -> v2.3.0-rc8 2025-12-04T09:43:54.2412955Z * [new tag] v2.3.0-rc9 -> v2.3.0-rc9 2025-12-04T09:43:54.2413976Z * [new tag] v2.3.1 -> v2.3.1 2025-12-04T09:43:54.2415551Z * [new tag] v2.3.1-rc1 -> v2.3.1-rc1 2025-12-04T09:43:54.2416908Z * [new tag] v2.3.1-rc2 -> v2.3.1-rc2 2025-12-04T09:43:54.2418465Z * [new tag] v2.3.1-rc3 -> v2.3.1-rc3 2025-12-04T09:43:54.2419844Z * [new tag] v2.4.0 -> v2.4.0 2025-12-04T09:43:54.2421196Z * [new tag] v2.4.0-rc1 -> v2.4.0-rc1 2025-12-04T09:43:54.2422568Z * [new tag] v2.4.0-rc2 -> v2.4.0-rc2 2025-12-04T09:43:54.2423977Z * [new tag] v2.4.0-rc3 -> v2.4.0-rc3 2025-12-04T09:43:54.2425339Z * [new tag] v2.4.0-rc4 -> v2.4.0-rc4 2025-12-04T09:43:54.2426780Z * [new tag] v2.4.0-rc5 -> v2.4.0-rc5 2025-12-04T09:43:54.2428152Z * [new tag] v2.4.0-rc6 -> v2.4.0-rc6 2025-12-04T09:43:54.2429545Z * [new tag] v2.4.0-rc7 -> v2.4.0-rc7 2025-12-04T09:43:54.2430848Z * [new tag] v2.4.0-rc8 -> v2.4.0-rc8 2025-12-04T09:43:54.2432237Z * [new tag] v2.4.0-rc9 -> v2.4.0-rc9 2025-12-04T09:43:54.2433288Z * [new tag] v2.4.1 -> v2.4.1 2025-12-04T09:43:54.2434872Z * [new tag] v2.4.1-rc1 -> v2.4.1-rc1 2025-12-04T09:43:54.2436290Z * [new tag] v2.4.1-rc2 -> v2.4.1-rc2 2025-12-04T09:43:54.2437742Z * [new tag] v2.4.1-rc3 -> v2.4.1-rc3 2025-12-04T09:43:54.2439088Z * [new tag] v2.5.0 -> v2.5.0 2025-12-04T09:43:54.2440445Z * [new tag] v2.5.0-rc1 -> v2.5.0-rc1 2025-12-04T09:43:54.2441477Z * [new tag] v2.5.0-rc10 -> v2.5.0-rc10 2025-12-04T09:43:54.2443004Z * [new tag] v2.5.0-rc2 -> v2.5.0-rc2 2025-12-04T09:43:54.2444335Z * [new tag] v2.5.0-rc3 -> v2.5.0-rc3 2025-12-04T09:43:54.2446102Z * [new tag] v2.5.0-rc4 -> v2.5.0-rc4 2025-12-04T09:43:54.2447510Z * [new tag] v2.5.0-rc5 -> v2.5.0-rc5 2025-12-04T09:43:54.2448897Z * [new tag] v2.5.0-rc6 -> v2.5.0-rc6 2025-12-04T09:43:54.2450317Z * [new tag] v2.5.0-rc7 -> v2.5.0-rc7 2025-12-04T09:43:54.2451718Z * [new tag] v2.5.0-rc8 -> v2.5.0-rc8 2025-12-04T09:43:54.2453189Z * [new tag] v2.5.0-rc9 -> v2.5.0-rc9 2025-12-04T09:43:54.2454095Z * [new tag] v2.5.1 -> v2.5.1 2025-12-04T09:43:54.2455416Z * [new tag] v2.5.1-rc1 -> v2.5.1-rc1 2025-12-04T09:43:54.2456630Z * [new tag] v2.6.0 -> v2.6.0 2025-12-04T09:43:54.2458024Z * [new tag] v2.6.0-rc1 -> v2.6.0-rc1 2025-12-04T09:43:54.2459412Z * [new tag] v2.6.0-rc2 -> v2.6.0-rc2 2025-12-04T09:43:54.2460810Z * [new tag] v2.6.0-rc3 -> v2.6.0-rc3 2025-12-04T09:43:54.2462177Z * [new tag] v2.6.0-rc4 -> v2.6.0-rc4 2025-12-04T09:43:54.2463833Z * [new tag] v2.6.0-rc5 -> v2.6.0-rc5 2025-12-04T09:43:54.2465390Z * [new tag] v2.6.0-rc6 -> v2.6.0-rc6 2025-12-04T09:43:54.2466811Z * [new tag] v2.6.0-rc7 -> v2.6.0-rc7 2025-12-04T09:43:54.2468331Z * [new tag] v2.6.0-rc8 -> v2.6.0-rc8 2025-12-04T09:43:54.2469708Z * [new tag] v2.6.0-rc9 -> v2.6.0-rc9 2025-12-04T09:43:54.2471334Z * [new tag] v2.7.0 -> v2.7.0 2025-12-04T09:43:54.2472796Z * [new tag] v2.7.0-rc1 -> v2.7.0-rc1 2025-12-04T09:43:54.2473841Z * [new tag] v2.7.0-rc10 -> v2.7.0-rc10 2025-12-04T09:43:54.2475378Z * [new tag] v2.7.0-rc2 -> v2.7.0-rc2 2025-12-04T09:43:54.2476849Z * [new tag] v2.7.0-rc3 -> v2.7.0-rc3 2025-12-04T09:43:54.2478428Z * [new tag] v2.7.0-rc4 -> v2.7.0-rc4 2025-12-04T09:43:54.2482080Z * [new tag] v2.7.0-rc5 -> v2.7.0-rc5 2025-12-04T09:43:54.2483431Z * [new tag] v2.7.0-rc6 -> v2.7.0-rc6 2025-12-04T09:43:54.2485008Z * [new tag] v2.7.0-rc7 -> v2.7.0-rc7 2025-12-04T09:43:54.2486442Z * [new tag] v2.7.0-rc8 -> v2.7.0-rc8 2025-12-04T09:43:54.2487838Z * [new tag] v2.7.0-rc9 -> v2.7.0-rc9 2025-12-04T09:43:54.2489015Z * [new tag] v2.7.1 -> v2.7.1 2025-12-04T09:43:54.2490408Z * [new tag] v2.7.1-rc1 -> v2.7.1-rc1 2025-12-04T09:43:54.2491815Z * [new tag] v2.7.1-rc2 -> v2.7.1-rc2 2025-12-04T09:43:54.2493310Z * [new tag] v2.7.1-rc3 -> v2.7.1-rc3 2025-12-04T09:43:54.2494739Z * [new tag] v2.7.1-rc4 -> v2.7.1-rc4 2025-12-04T09:43:54.2496124Z * [new tag] v2.7.1-rc5 -> v2.7.1-rc5 2025-12-04T09:43:54.2497364Z * [new tag] v2.8.0 -> v2.8.0 2025-12-04T09:43:54.2498758Z * [new tag] v2.8.0-rc1 -> v2.8.0-rc1 2025-12-04T09:43:54.2500084Z * [new tag] v2.8.0-rc2 -> v2.8.0-rc2 2025-12-04T09:43:54.2501611Z * [new tag] v2.8.0-rc3 -> v2.8.0-rc3 2025-12-04T09:43:54.2503059Z * [new tag] v2.8.0-rc4 -> v2.8.0-rc4 2025-12-04T09:43:54.2504674Z * [new tag] v2.8.0-rc5 -> v2.8.0-rc5 2025-12-04T09:43:54.2506080Z * [new tag] v2.8.0-rc6 -> v2.8.0-rc6 2025-12-04T09:43:54.2507514Z * [new tag] v2.8.0-rc7 -> v2.8.0-rc7 2025-12-04T09:43:54.2508908Z * [new tag] v2.8.0-rc8 -> v2.8.0-rc8 2025-12-04T09:43:54.2510385Z * [new tag] v2.9.0 -> v2.9.0 2025-12-04T09:43:54.2511809Z * [new tag] v2.9.0-rc1 -> v2.9.0-rc1 2025-12-04T09:43:54.2513356Z * [new tag] v2.9.0-rc10 -> v2.9.0-rc10 2025-12-04T09:43:54.2514726Z * [new tag] v2.9.0-rc11 -> v2.9.0-rc11 2025-12-04T09:43:54.2516300Z * [new tag] v2.9.0-rc2 -> v2.9.0-rc2 2025-12-04T09:43:54.2517696Z * [new tag] v2.9.0-rc3 -> v2.9.0-rc3 2025-12-04T09:43:54.2519147Z * [new tag] v2.9.0-rc4 -> v2.9.0-rc4 2025-12-04T09:43:54.2520553Z * [new tag] v2.9.0-rc5 -> v2.9.0-rc5 2025-12-04T09:43:54.2522194Z * [new tag] v2.9.0-rc6 -> v2.9.0-rc6 2025-12-04T09:43:54.2523643Z * [new tag] v2.9.0-rc7 -> v2.9.0-rc7 2025-12-04T09:43:54.2525216Z * [new tag] v2.9.0-rc8 -> v2.9.0-rc8 2025-12-04T09:43:54.2526434Z * [new tag] v2.9.0-rc9 -> v2.9.0-rc9 2025-12-04T09:43:54.2527655Z * [new tag] v2.9.1 -> v2.9.1 2025-12-04T09:43:54.2529074Z * [new tag] v2.9.1-rc1 -> v2.9.1-rc1 2025-12-04T09:43:54.2530580Z * [new tag] v2.9.1-rc2 -> v2.9.1-rc2 2025-12-04T09:43:54.2532473Z * [new tag] viable/strict/1759343184 -> viable/strict/1759343184 2025-12-04T09:43:54.2533803Z * [new tag] viable/strict/1759346540 -> viable/strict/1759346540 2025-12-04T09:43:54.2535173Z * [new tag] viable/strict/1759348181 -> viable/strict/1759348181 2025-12-04T09:43:54.2536523Z * [new tag] viable/strict/1759350324 -> viable/strict/1759350324 2025-12-04T09:43:54.2537832Z * [new tag] viable/strict/1759351793 -> viable/strict/1759351793 2025-12-04T09:43:54.2539226Z * [new tag] viable/strict/1759353844 -> viable/strict/1759353844 2025-12-04T09:43:54.2540542Z * [new tag] viable/strict/1759355374 -> viable/strict/1759355374 2025-12-04T09:43:54.2541857Z * [new tag] viable/strict/1759357472 -> viable/strict/1759357472 2025-12-04T09:43:54.2543174Z * [new tag] viable/strict/1759361002 -> viable/strict/1759361002 2025-12-04T09:43:54.2544977Z * [new tag] viable/strict/1759362585 -> viable/strict/1759362585 2025-12-04T09:43:54.2546553Z * [new tag] viable/strict/1759365359 -> viable/strict/1759365359 2025-12-04T09:43:54.2547966Z * [new tag] viable/strict/1759370089 -> viable/strict/1759370089 2025-12-04T09:43:54.2549378Z * [new tag] viable/strict/1759377554 -> viable/strict/1759377554 2025-12-04T09:43:54.2550794Z * [new tag] viable/strict/1759379133 -> viable/strict/1759379133 2025-12-04T09:43:54.2552193Z * [new tag] viable/strict/1759389871 -> viable/strict/1759389871 2025-12-04T09:43:54.2553649Z * [new tag] viable/strict/1759393562 -> viable/strict/1759393562 2025-12-04T09:43:54.2555068Z * [new tag] viable/strict/1759395076 -> viable/strict/1759395076 2025-12-04T09:43:54.2556473Z * [new tag] viable/strict/1759398579 -> viable/strict/1759398579 2025-12-04T09:43:54.2557932Z * [new tag] viable/strict/1759404142 -> viable/strict/1759404142 2025-12-04T09:43:54.2559373Z * [new tag] viable/strict/1759405773 -> viable/strict/1759405773 2025-12-04T09:43:54.2560729Z * [new tag] viable/strict/1759408041 -> viable/strict/1759408041 2025-12-04T09:43:54.2562090Z * [new tag] viable/strict/1759411593 -> viable/strict/1759411593 2025-12-04T09:43:54.2563463Z * [new tag] viable/strict/1759427395 -> viable/strict/1759427395 2025-12-04T09:43:54.2564843Z * [new tag] viable/strict/1759434582 -> viable/strict/1759434582 2025-12-04T09:43:54.2566254Z * [new tag] viable/strict/1759436720 -> viable/strict/1759436720 2025-12-04T09:43:54.2567739Z * [new tag] viable/strict/1759440219 -> viable/strict/1759440219 2025-12-04T09:43:54.2569048Z * [new tag] viable/strict/1759441948 -> viable/strict/1759441948 2025-12-04T09:43:54.2570485Z * [new tag] viable/strict/1759443860 -> viable/strict/1759443860 2025-12-04T09:43:54.2579325Z * [new tag] viable/strict/1759445377 -> viable/strict/1759445377 2025-12-04T09:43:54.2579770Z * [new tag] viable/strict/1759447415 -> viable/strict/1759447415 2025-12-04T09:43:54.2580312Z * [new tag] viable/strict/1759451750 -> viable/strict/1759451750 2025-12-04T09:43:54.2580654Z * [new tag] viable/strict/1759453910 -> viable/strict/1759453910 2025-12-04T09:43:54.2580986Z * [new tag] viable/strict/1759456483 -> viable/strict/1759456483 2025-12-04T09:43:54.2581314Z * [new tag] viable/strict/1759459279 -> viable/strict/1759459279 2025-12-04T09:43:54.2581652Z * [new tag] viable/strict/1759460742 -> viable/strict/1759460742 2025-12-04T09:43:54.2582215Z * [new tag] viable/strict/1759462025 -> viable/strict/1759462025 2025-12-04T09:43:54.2583856Z * [new tag] viable/strict/1759469086 -> viable/strict/1759469086 2025-12-04T09:43:54.2585280Z * [new tag] viable/strict/1759470581 -> viable/strict/1759470581 2025-12-04T09:43:54.2586608Z * [new tag] viable/strict/1759472786 -> viable/strict/1759472786 2025-12-04T09:43:54.2588013Z * [new tag] viable/strict/1759476294 -> viable/strict/1759476294 2025-12-04T09:43:54.2589363Z * [new tag] viable/strict/1759479963 -> viable/strict/1759479963 2025-12-04T09:43:54.2590717Z * [new tag] viable/strict/1759492177 -> viable/strict/1759492177 2025-12-04T09:43:54.2592125Z * [new tag] viable/strict/1759519278 -> viable/strict/1759519278 2025-12-04T09:43:54.2593564Z * [new tag] viable/strict/1759524580 -> viable/strict/1759524580 2025-12-04T09:43:54.2594937Z * [new tag] viable/strict/1759528193 -> viable/strict/1759528193 2025-12-04T09:43:54.2596510Z * [new tag] viable/strict/1759533797 -> viable/strict/1759533797 2025-12-04T09:43:54.2597908Z * [new tag] viable/strict/1759542780 -> viable/strict/1759542780 2025-12-04T09:43:54.2599437Z * [new tag] viable/strict/1759549779 -> viable/strict/1759549779 2025-12-04T09:43:54.2600800Z * [new tag] viable/strict/1759555455 -> viable/strict/1759555455 2025-12-04T09:43:54.2602194Z * [new tag] viable/strict/1759559176 -> viable/strict/1759559176 2025-12-04T09:43:54.2603625Z * [new tag] viable/strict/1759560629 -> viable/strict/1759560629 2025-12-04T09:43:54.2605023Z * [new tag] viable/strict/1759569848 -> viable/strict/1759569848 2025-12-04T09:43:54.2606525Z * [new tag] viable/strict/1759571382 -> viable/strict/1759571382 2025-12-04T09:43:54.2607993Z * [new tag] viable/strict/1759573474 -> viable/strict/1759573474 2025-12-04T09:43:54.2609440Z * [new tag] viable/strict/1759618187 -> viable/strict/1759618187 2025-12-04T09:43:54.2610847Z * [new tag] viable/strict/1759626742 -> viable/strict/1759626742 2025-12-04T09:43:54.2612271Z * [new tag] viable/strict/1759632427 -> viable/strict/1759632427 2025-12-04T09:43:54.2613651Z * [new tag] viable/strict/1759634971 -> viable/strict/1759634971 2025-12-04T09:43:54.2615590Z * [new tag] viable/strict/1759661382 -> viable/strict/1759661382 2025-12-04T09:43:54.2616995Z * [new tag] viable/strict/1759663294 -> viable/strict/1759663294 2025-12-04T09:43:54.2618285Z * [new tag] viable/strict/1759708178 -> viable/strict/1759708178 2025-12-04T09:43:54.2619810Z * [new tag] viable/strict/1759715695 -> viable/strict/1759715695 2025-12-04T09:43:54.2621109Z * [new tag] viable/strict/1759728293 -> viable/strict/1759728293 2025-12-04T09:43:54.2622462Z * [new tag] viable/strict/1759735513 -> viable/strict/1759735513 2025-12-04T09:43:54.2624028Z * [new tag] viable/strict/1759739177 -> viable/strict/1759739177 2025-12-04T09:43:54.2625476Z * [new tag] viable/strict/1759758635 -> viable/strict/1759758635 2025-12-04T09:43:54.2626879Z * [new tag] viable/strict/1759765784 -> viable/strict/1759765784 2025-12-04T09:43:54.2628221Z * [new tag] viable/strict/1759767948 -> viable/strict/1759767948 2025-12-04T09:43:54.2629626Z * [new tag] viable/strict/1759771461 -> viable/strict/1759771461 2025-12-04T09:43:54.2630864Z * [new tag] viable/strict/1759776706 -> viable/strict/1759776706 2025-12-04T09:43:54.2632382Z * [new tag] viable/strict/1759782317 -> viable/strict/1759782317 2025-12-04T09:43:54.2633793Z * [new tag] viable/strict/1759783777 -> viable/strict/1759783777 2025-12-04T09:43:54.2635268Z * [new tag] viable/strict/1759785815 -> viable/strict/1759785815 2025-12-04T09:43:54.2636747Z * [new tag] viable/strict/1759789459 -> viable/strict/1759789459 2025-12-04T09:43:54.2638154Z * [new tag] viable/strict/1759790974 -> viable/strict/1759790974 2025-12-04T09:43:54.2639399Z * [new tag] viable/strict/1759794583 -> viable/strict/1759794583 2025-12-04T09:43:54.2640854Z * [new tag] viable/strict/1759797408 -> viable/strict/1759797408 2025-12-04T09:43:54.2642253Z * [new tag] viable/strict/1759799518 -> viable/strict/1759799518 2025-12-04T09:43:54.2643690Z * [new tag] viable/strict/1759804909 -> viable/strict/1759804909 2025-12-04T09:43:54.2645157Z * [new tag] viable/strict/1759807643 -> viable/strict/1759807643 2025-12-04T09:43:54.2646583Z * [new tag] viable/strict/1759809089 -> viable/strict/1759809089 2025-12-04T09:43:54.2647964Z * [new tag] viable/strict/1759811145 -> viable/strict/1759811145 2025-12-04T09:43:54.2649367Z * [new tag] viable/strict/1759812581 -> viable/strict/1759812581 2025-12-04T09:43:54.2650716Z * [new tag] viable/strict/1759814683 -> viable/strict/1759814683 2025-12-04T09:43:54.2652133Z * [new tag] viable/strict/1759821889 -> viable/strict/1759821889 2025-12-04T09:43:54.2653592Z * [new tag] viable/strict/1759823376 -> viable/strict/1759823376 2025-12-04T09:43:54.2654996Z * [new tag] viable/strict/1759827107 -> viable/strict/1759827107 2025-12-04T09:43:54.2656365Z * [new tag] viable/strict/1759830577 -> viable/strict/1759830577 2025-12-04T09:43:54.2657987Z * [new tag] viable/strict/1759832720 -> viable/strict/1759832720 2025-12-04T09:43:54.2659254Z * [new tag] viable/strict/1759842063 -> viable/strict/1759842063 2025-12-04T09:43:54.2660682Z * [new tag] viable/strict/1759847121 -> viable/strict/1759847121 2025-12-04T09:43:54.2662347Z * [new tag] viable/strict/1759850721 -> viable/strict/1759850721 2025-12-04T09:43:54.2663813Z * [new tag] viable/strict/1759857870 -> viable/strict/1759857870 2025-12-04T09:43:54.2665304Z * [new tag] viable/strict/1759863143 -> viable/strict/1759863143 2025-12-04T09:43:54.2666671Z * [new tag] viable/strict/1759875874 -> viable/strict/1759875874 2025-12-04T09:43:54.2667907Z * [new tag] viable/strict/1759877385 -> viable/strict/1759877385 2025-12-04T09:43:54.2669294Z * [new tag] viable/strict/1759883801 -> viable/strict/1759883801 2025-12-04T09:43:54.2670740Z * [new tag] viable/strict/1759885922 -> viable/strict/1759885922 2025-12-04T09:43:54.2672081Z * [new tag] viable/strict/1759888488 -> viable/strict/1759888488 2025-12-04T09:43:54.2673474Z * [new tag] viable/strict/1759895471 -> viable/strict/1759895471 2025-12-04T09:43:54.2674858Z * [new tag] viable/strict/1759904803 -> viable/strict/1759904803 2025-12-04T09:43:54.2676458Z * [new tag] viable/strict/1759908300 -> viable/strict/1759908300 2025-12-04T09:43:54.2678005Z * [new tag] viable/strict/1759915520 -> viable/strict/1759915520 2025-12-04T09:43:54.2679635Z * [new tag] viable/strict/1759916978 -> viable/strict/1759916978 2025-12-04T09:43:54.2680907Z * [new tag] viable/strict/1759930024 -> viable/strict/1759930024 2025-12-04T09:43:54.2682314Z * [new tag] viable/strict/1759948122 -> viable/strict/1759948122 2025-12-04T09:43:54.2683785Z * [new tag] viable/strict/1759952983 -> viable/strict/1759952983 2025-12-04T09:43:54.2685203Z * [new tag] viable/strict/1759955121 -> viable/strict/1759955121 2025-12-04T09:43:54.2686596Z * [new tag] viable/strict/1759962298 -> viable/strict/1759962298 2025-12-04T09:43:54.2687971Z * [new tag] viable/strict/1759965837 -> viable/strict/1759965837 2025-12-04T09:43:54.2689460Z * [new tag] viable/strict/1759970213 -> viable/strict/1759970213 2025-12-04T09:43:54.2690903Z * [new tag] viable/strict/1759974894 -> viable/strict/1759974894 2025-12-04T09:43:54.2692358Z * [new tag] viable/strict/1759977763 -> viable/strict/1759977763 2025-12-04T09:43:54.2693783Z * [new tag] viable/strict/1759979241 -> viable/strict/1759979241 2025-12-04T09:43:54.2695187Z * [new tag] viable/strict/1759985417 -> viable/strict/1759985417 2025-12-04T09:43:54.2696584Z * [new tag] viable/strict/1759987490 -> viable/strict/1759987490 2025-12-04T09:43:54.2698031Z * [new tag] viable/strict/1759996180 -> viable/strict/1759996180 2025-12-04T09:43:54.2699418Z * [new tag] viable/strict/1760065682 -> viable/strict/1760065682 2025-12-04T09:43:54.2700848Z * [new tag] viable/strict/1760066894 -> viable/strict/1760066894 2025-12-04T09:43:54.2702262Z * [new tag] viable/strict/1760070345 -> viable/strict/1760070345 2025-12-04T09:43:54.2703718Z * [new tag] viable/strict/1760089782 -> viable/strict/1760089782 2025-12-04T09:43:54.2705196Z * [new tag] viable/strict/1760091921 -> viable/strict/1760091921 2025-12-04T09:43:54.2707103Z * [new tag] viable/strict/1760127924 -> viable/strict/1760127924 2025-12-04T09:43:54.2708489Z * [new tag] viable/strict/1760129489 -> viable/strict/1760129489 2025-12-04T09:43:54.2709983Z * [new tag] viable/strict/1760132980 -> viable/strict/1760132980 2025-12-04T09:43:54.2711507Z * [new tag] viable/strict/1760135060 -> viable/strict/1760135060 2025-12-04T09:43:54.2712903Z * [new tag] viable/strict/1760215782 -> viable/strict/1760215782 2025-12-04T09:43:54.2714322Z * [new tag] viable/strict/1760273849 -> viable/strict/1760273849 2025-12-04T09:43:54.2715655Z * [new tag] viable/strict/1760275517 -> viable/strict/1760275517 2025-12-04T09:43:54.2717146Z * [new tag] viable/strict/1760276979 -> viable/strict/1760276979 2025-12-04T09:43:54.2718451Z * [new tag] viable/strict/1760279007 -> viable/strict/1760279007 2025-12-04T09:43:54.2719734Z * [new tag] viable/strict/1760286328 -> viable/strict/1760286328 2025-12-04T09:43:54.2721149Z * [new tag] viable/strict/1760493304 -> viable/strict/1760493304 2025-12-04T09:43:54.2722761Z * [new tag] viable/strict/1760496298 -> viable/strict/1760496298 2025-12-04T09:43:54.2723840Z * [new tag] viable/strict/1760518396 -> viable/strict/1760518396 2025-12-04T09:43:54.2725387Z * [new tag] viable/strict/1760534864 -> viable/strict/1760534864 2025-12-04T09:43:54.2726803Z * [new tag] viable/strict/1760549062 -> viable/strict/1760549062 2025-12-04T09:43:54.2728359Z * [new tag] viable/strict/1760552799 -> viable/strict/1760552799 2025-12-04T09:43:54.2729768Z * [new tag] viable/strict/1760554355 -> viable/strict/1760554355 2025-12-04T09:43:54.2731155Z * [new tag] viable/strict/1760556275 -> viable/strict/1760556275 2025-12-04T09:43:54.2732568Z * [new tag] viable/strict/1760564979 -> viable/strict/1760564979 2025-12-04T09:43:54.2734037Z * [new tag] viable/strict/1760567049 -> viable/strict/1760567049 2025-12-04T09:43:54.2735728Z * [new tag] viable/strict/1760568585 -> viable/strict/1760568585 2025-12-04T09:43:54.2737177Z * [new tag] viable/strict/1760570630 -> viable/strict/1760570630 2025-12-04T09:43:54.2738614Z * [new tag] viable/strict/1760572180 -> viable/strict/1760572180 2025-12-04T09:43:54.2740036Z * [new tag] viable/strict/1760575094 -> viable/strict/1760575094 2025-12-04T09:43:54.2741626Z * [new tag] viable/strict/1760579709 -> viable/strict/1760579709 2025-12-04T09:43:54.2743439Z * [new tag] viable/strict/1760582614 -> viable/strict/1760582614 2025-12-04T09:43:54.2745008Z * [new tag] viable/strict/1760586815 -> viable/strict/1760586815 2025-12-04T09:43:54.2746283Z * [new tag] viable/strict/1760588829 -> viable/strict/1760588829 2025-12-04T09:43:54.2747656Z * [new tag] viable/strict/1760590200 -> viable/strict/1760590200 2025-12-04T09:43:54.2749197Z * [new tag] viable/strict/1760592311 -> viable/strict/1760592311 2025-12-04T09:43:54.2750595Z * [new tag] viable/strict/1760619733 -> viable/strict/1760619733 2025-12-04T09:43:54.2751891Z * [new tag] viable/strict/1760628335 -> viable/strict/1760628335 2025-12-04T09:43:54.2753266Z * [new tag] viable/strict/1760635490 -> viable/strict/1760635490 2025-12-04T09:43:54.2754692Z * [new tag] viable/strict/1760640743 -> viable/strict/1760640743 2025-12-04T09:43:54.2756075Z * [new tag] viable/strict/1760642528 -> viable/strict/1760642528 2025-12-04T09:43:54.2757457Z * [new tag] viable/strict/1760646330 -> viable/strict/1760646330 2025-12-04T09:43:54.2758836Z * [new tag] viable/strict/1760666101 -> viable/strict/1760666101 2025-12-04T09:43:54.2760323Z * [new tag] viable/strict/1760668990 -> viable/strict/1760668990 2025-12-04T09:43:54.2761715Z * [new tag] viable/strict/1760670600 -> viable/strict/1760670600 2025-12-04T09:43:54.2763112Z * [new tag] viable/strict/1760671704 -> viable/strict/1760671704 2025-12-04T09:43:54.2764504Z * [new tag] viable/strict/1760673121 -> viable/strict/1760673121 2025-12-04T09:43:54.2765884Z * [new tag] viable/strict/1760675352 -> viable/strict/1760675352 2025-12-04T09:43:54.2767336Z * [new tag] viable/strict/1760696731 -> viable/strict/1760696731 2025-12-04T09:43:54.2769868Z * [new tag] viable/strict/1760723515 -> viable/strict/1760723515 2025-12-04T09:43:54.2771251Z * [new tag] viable/strict/1760727234 -> viable/strict/1760727234 2025-12-04T09:43:54.2772658Z * [new tag] viable/strict/1760730578 -> viable/strict/1760730578 2025-12-04T09:43:54.2774054Z * [new tag] viable/strict/1760732726 -> viable/strict/1760732726 2025-12-04T09:43:54.2775697Z * [new tag] viable/strict/1760734180 -> viable/strict/1760734180 2025-12-04T09:43:54.2777171Z * [new tag] viable/strict/1760736251 -> viable/strict/1760736251 2025-12-04T09:43:54.2778875Z * [new tag] viable/strict/1760737772 -> viable/strict/1760737772 2025-12-04T09:43:54.2780343Z * [new tag] viable/strict/1760758005 -> viable/strict/1760758005 2025-12-04T09:43:54.2781625Z * [new tag] viable/strict/1760761532 -> viable/strict/1760761532 2025-12-04T09:43:54.2783170Z * [new tag] viable/strict/1760802581 -> viable/strict/1760802581 2025-12-04T09:43:54.2784742Z * [new tag] viable/strict/1760827772 -> viable/strict/1760827772 2025-12-04T09:43:54.2786154Z * [new tag] viable/strict/1760834524 -> viable/strict/1760834524 2025-12-04T09:43:54.2787615Z * [new tag] viable/strict/1760845009 -> viable/strict/1760845009 2025-12-04T09:43:54.2789077Z * [new tag] viable/strict/1760876836 -> viable/strict/1760876836 2025-12-04T09:43:54.2790487Z * [new tag] viable/strict/1760880329 -> viable/strict/1760880329 2025-12-04T09:43:54.2791933Z * [new tag] viable/strict/1760888987 -> viable/strict/1760888987 2025-12-04T09:43:54.2793334Z * [new tag] viable/strict/1760912664 -> viable/strict/1760912664 2025-12-04T09:43:54.2794737Z * [new tag] viable/strict/1760925321 -> viable/strict/1760925321 2025-12-04T09:43:54.2796097Z * [new tag] viable/strict/1760931488 -> viable/strict/1760931488 2025-12-04T09:43:54.2797647Z * [new tag] viable/strict/1760932693 -> viable/strict/1760932693 2025-12-04T09:43:54.2799005Z * [new tag] viable/strict/1761004184 -> viable/strict/1761004184 2025-12-04T09:43:54.2800783Z * [new tag] viable/strict/1761014748 -> viable/strict/1761014748 2025-12-04T09:43:54.2802199Z * [new tag] viable/strict/1761017491 -> viable/strict/1761017491 2025-12-04T09:43:54.2803646Z * [new tag] viable/strict/1761018806 -> viable/strict/1761018806 2025-12-04T09:43:54.2805110Z * [new tag] viable/strict/1761020754 -> viable/strict/1761020754 2025-12-04T09:43:54.2806510Z * [new tag] viable/strict/1761024303 -> viable/strict/1761024303 2025-12-04T09:43:54.2807870Z * [new tag] viable/strict/1761029582 -> viable/strict/1761029582 2025-12-04T09:43:54.2809353Z * [new tag] viable/strict/1761031535 -> viable/strict/1761031535 2025-12-04T09:43:54.2810710Z * [new tag] viable/strict/1761035196 -> viable/strict/1761035196 2025-12-04T09:43:54.2812201Z * [new tag] viable/strict/1761045825 -> viable/strict/1761045825 2025-12-04T09:43:54.2813646Z * [new tag] viable/strict/1761054796 -> viable/strict/1761054796 2025-12-04T09:43:54.2815114Z * [new tag] viable/strict/1761060314 -> viable/strict/1761060314 2025-12-04T09:43:54.2816581Z * [new tag] viable/strict/1761071198 -> viable/strict/1761071198 2025-12-04T09:43:54.2817997Z * [new tag] viable/strict/1761074628 -> viable/strict/1761074628 2025-12-04T09:43:54.2819440Z * [new tag] viable/strict/1761078351 -> viable/strict/1761078351 2025-12-04T09:43:54.2820808Z * [new tag] viable/strict/1761079822 -> viable/strict/1761079822 2025-12-04T09:43:54.2822192Z * [new tag] viable/strict/1761081873 -> viable/strict/1761081873 2025-12-04T09:43:54.2823696Z * [new tag] viable/strict/1761083392 -> viable/strict/1761083392 2025-12-04T09:43:54.2825206Z * [new tag] viable/strict/1761085465 -> viable/strict/1761085465 2025-12-04T09:43:54.2826710Z * [new tag] viable/strict/1761089099 -> viable/strict/1761089099 2025-12-04T09:43:54.2828219Z * [new tag] viable/strict/1761095535 -> viable/strict/1761095535 2025-12-04T09:43:54.2829472Z * [new tag] viable/strict/1761098119 -> viable/strict/1761098119 2025-12-04T09:43:54.2831277Z * [new tag] viable/strict/1761101330 -> viable/strict/1761101330 2025-12-04T09:43:54.2832696Z * [new tag] viable/strict/1761114425 -> viable/strict/1761114425 2025-12-04T09:43:54.2834121Z * [new tag] viable/strict/1761116036 -> viable/strict/1761116036 2025-12-04T09:43:54.2835551Z * [new tag] viable/strict/1761119379 -> viable/strict/1761119379 2025-12-04T09:43:54.2836959Z * [new tag] viable/strict/1761121601 -> viable/strict/1761121601 2025-12-04T09:43:54.2838388Z * [new tag] viable/strict/1761123234 -> viable/strict/1761123234 2025-12-04T09:43:54.2839780Z * [new tag] viable/strict/1761126621 -> viable/strict/1761126621 2025-12-04T09:43:54.2841222Z * [new tag] viable/strict/1761132259 -> viable/strict/1761132259 2025-12-04T09:43:54.2842692Z * [new tag] viable/strict/1761146746 -> viable/strict/1761146746 2025-12-04T09:43:54.2844128Z * [new tag] viable/strict/1761164752 -> viable/strict/1761164752 2025-12-04T09:43:54.2845481Z * [new tag] viable/strict/1761166198 -> viable/strict/1761166198 2025-12-04T09:43:54.2846938Z * [new tag] viable/strict/1761175424 -> viable/strict/1761175424 2025-12-04T09:43:54.2848356Z * [new tag] viable/strict/1761176983 -> viable/strict/1761176983 2025-12-04T09:43:54.2849875Z * [new tag] viable/strict/1761179891 -> viable/strict/1761179891 2025-12-04T09:43:54.2851274Z * [new tag] viable/strict/1761181930 -> viable/strict/1761181930 2025-12-04T09:43:54.2852727Z * [new tag] viable/strict/1761184516 -> viable/strict/1761184516 2025-12-04T09:43:54.2854159Z * [new tag] viable/strict/1761190179 -> viable/strict/1761190179 2025-12-04T09:43:54.2855583Z * [new tag] viable/strict/1761193558 -> viable/strict/1761193558 2025-12-04T09:43:54.2856978Z * [new tag] viable/strict/1761207990 -> viable/strict/1761207990 2025-12-04T09:43:54.2858435Z * [new tag] viable/strict/1761229539 -> viable/strict/1761229539 2025-12-04T09:43:54.2860011Z * [new tag] viable/strict/1761244031 -> viable/strict/1761244031 2025-12-04T09:43:54.2861470Z * [new tag] viable/strict/1761248986 -> viable/strict/1761248986 2025-12-04T09:43:54.2862884Z * [new tag] viable/strict/1761259791 -> viable/strict/1761259791 2025-12-04T09:43:54.2864425Z * [new tag] viable/strict/1761266139 -> viable/strict/1761266139 2025-12-04T09:43:54.2865866Z * [new tag] viable/strict/1761268316 -> viable/strict/1761268316 2025-12-04T09:43:54.2867270Z * [new tag] viable/strict/1761273805 -> viable/strict/1761273805 2025-12-04T09:43:54.2868666Z * [new tag] viable/strict/1761275261 -> viable/strict/1761275261 2025-12-04T09:43:54.2870108Z * [new tag] viable/strict/1761277913 -> viable/strict/1761277913 2025-12-04T09:43:54.2871583Z * [new tag] viable/strict/1761290701 -> viable/strict/1761290701 2025-12-04T09:43:54.2873032Z * [new tag] viable/strict/1761294396 -> viable/strict/1761294396 2025-12-04T09:43:54.2874425Z * [new tag] viable/strict/1761303047 -> viable/strict/1761303047 2025-12-04T09:43:54.2875859Z * [new tag] viable/strict/1761335388 -> viable/strict/1761335388 2025-12-04T09:43:54.2877326Z * [new tag] viable/strict/1761337551 -> viable/strict/1761337551 2025-12-04T09:43:54.2881359Z * [new tag] viable/strict/1761339007 -> viable/strict/1761339007 2025-12-04T09:43:54.2882671Z * [new tag] viable/strict/1761341050 -> viable/strict/1761341050 2025-12-04T09:43:54.2884051Z * [new tag] viable/strict/1761346188 -> viable/strict/1761346188 2025-12-04T09:43:54.2885638Z * [new tag] viable/strict/1761349792 -> viable/strict/1761349792 2025-12-04T09:43:54.2887064Z * [new tag] viable/strict/1761352620 -> viable/strict/1761352620 2025-12-04T09:43:54.2888551Z * [new tag] viable/strict/1761354730 -> viable/strict/1761354730 2025-12-04T09:43:54.2889961Z * [new tag] viable/strict/1761357298 -> viable/strict/1761357298 2025-12-04T09:43:54.2891431Z * [new tag] viable/strict/1761360201 -> viable/strict/1761360201 2025-12-04T09:43:54.2892815Z * [new tag] viable/strict/1761361753 -> viable/strict/1761361753 2025-12-04T09:43:54.2894237Z * [new tag] viable/strict/1761364351 -> viable/strict/1761364351 2025-12-04T09:43:54.2896018Z * [new tag] viable/strict/1761366338 -> viable/strict/1761366338 2025-12-04T09:43:54.2897531Z * [new tag] viable/strict/1761367802 -> viable/strict/1761367802 2025-12-04T09:43:54.2898976Z * [new tag] viable/strict/1761369889 -> viable/strict/1761369889 2025-12-04T09:43:54.2900364Z * [new tag] viable/strict/1761371385 -> viable/strict/1761371385 2025-12-04T09:43:54.2901801Z * [new tag] viable/strict/1761373581 -> viable/strict/1761373581 2025-12-04T09:43:54.2903335Z * [new tag] viable/strict/1761375054 -> viable/strict/1761375054 2025-12-04T09:43:54.2904891Z * [new tag] viable/strict/1761421785 -> viable/strict/1761421785 2025-12-04T09:43:54.2906335Z * [new tag] viable/strict/1761434614 -> viable/strict/1761434614 2025-12-04T09:43:54.2908105Z * [new tag] viable/strict/1761439254 -> viable/strict/1761439254 2025-12-04T09:43:54.2909637Z * [new tag] viable/strict/1761454187 -> viable/strict/1761454187 2025-12-04T09:43:54.2911080Z * [new tag] viable/strict/1761459991 -> viable/strict/1761459991 2025-12-04T09:43:54.2912673Z * [new tag] viable/strict/1761470668 -> viable/strict/1761470668 2025-12-04T09:43:54.2914399Z * [new tag] viable/strict/1761472188 -> viable/strict/1761472188 2025-12-04T09:43:54.2915849Z * [new tag] viable/strict/1761503178 -> viable/strict/1761503178 2025-12-04T09:43:54.2917263Z * [new tag] viable/strict/1761517492 -> viable/strict/1761517492 2025-12-04T09:43:54.2918671Z * [new tag] viable/strict/1761518981 -> viable/strict/1761518981 2025-12-04T09:43:54.2920276Z * [new tag] viable/strict/1761533609 -> viable/strict/1761533609 2025-12-04T09:43:54.2921634Z * [new tag] viable/strict/1761546438 -> viable/strict/1761546438 2025-12-04T09:43:54.2923094Z * [new tag] viable/strict/1761548133 -> viable/strict/1761548133 2025-12-04T09:43:54.2924738Z * [new tag] viable/strict/1761555186 -> viable/strict/1761555186 2025-12-04T09:43:54.2926208Z * [new tag] viable/strict/1761557178 -> viable/strict/1761557178 2025-12-04T09:43:54.2927765Z * [new tag] viable/strict/1761560772 -> viable/strict/1761560772 2025-12-04T09:43:54.2929191Z * [new tag] viable/strict/1761562266 -> viable/strict/1761562266 2025-12-04T09:43:54.2930647Z * [new tag] viable/strict/1761564260 -> viable/strict/1761564260 2025-12-04T09:43:54.2932084Z * [new tag] viable/strict/1761568072 -> viable/strict/1761568072 2025-12-04T09:43:54.2933519Z * [new tag] viable/strict/1761571683 -> viable/strict/1761571683 2025-12-04T09:43:54.2934923Z * [new tag] viable/strict/1761580199 -> viable/strict/1761580199 2025-12-04T09:43:54.2936302Z * [new tag] viable/strict/1761587383 -> viable/strict/1761587383 2025-12-04T09:43:54.2937776Z * [new tag] viable/strict/1761591165 -> viable/strict/1761591165 2025-12-04T09:43:54.2939188Z * [new tag] viable/strict/1761594575 -> viable/strict/1761594575 2025-12-04T09:43:54.2940595Z * [new tag] viable/strict/1761596710 -> viable/strict/1761596710 2025-12-04T09:43:54.2942101Z * [new tag] viable/strict/1761598189 -> viable/strict/1761598189 2025-12-04T09:43:54.2943508Z * [new tag] viable/strict/1761600254 -> viable/strict/1761600254 2025-12-04T09:43:54.2945139Z * [new tag] viable/strict/1761603879 -> viable/strict/1761603879 2025-12-04T09:43:54.2946600Z * [new tag] viable/strict/1761605429 -> viable/strict/1761605429 2025-12-04T09:43:54.2948066Z * [new tag] viable/strict/1761607468 -> viable/strict/1761607468 2025-12-04T09:43:54.2949494Z * [new tag] viable/strict/1761608983 -> viable/strict/1761608983 2025-12-04T09:43:54.2950934Z * [new tag] viable/strict/1761611846 -> viable/strict/1761611846 2025-12-04T09:43:54.2952387Z * [new tag] viable/strict/1761613922 -> viable/strict/1761613922 2025-12-04T09:43:54.2953694Z * [new tag] viable/strict/1761616504 -> viable/strict/1761616504 2025-12-04T09:43:54.2955034Z * [new tag] viable/strict/1761619599 -> viable/strict/1761619599 2025-12-04T09:43:54.2956432Z * [new tag] viable/strict/1761686693 -> viable/strict/1761686693 2025-12-04T09:43:54.2957854Z * [new tag] viable/strict/1761688179 -> viable/strict/1761688179 2025-12-04T09:43:54.2959272Z * [new tag] viable/strict/1761691973 -> viable/strict/1761691973 2025-12-04T09:43:54.2961048Z * [new tag] viable/strict/1761693884 -> viable/strict/1761693884 2025-12-04T09:43:54.2962511Z * [new tag] viable/strict/1761695389 -> viable/strict/1761695389 2025-12-04T09:43:54.2963938Z * [new tag] viable/strict/1761698408 -> viable/strict/1761698408 2025-12-04T09:43:54.2965364Z * [new tag] viable/strict/1761702931 -> viable/strict/1761702931 2025-12-04T09:43:54.2966859Z * [new tag] viable/strict/1761706307 -> viable/strict/1761706307 2025-12-04T09:43:54.2968308Z * [new tag] viable/strict/1761709065 -> viable/strict/1761709065 2025-12-04T09:43:54.2969856Z * [new tag] viable/strict/1761710285 -> viable/strict/1761710285 2025-12-04T09:43:54.2971277Z * [new tag] viable/strict/1761711983 -> viable/strict/1761711983 2025-12-04T09:43:54.2972800Z * [new tag] viable/strict/1761713514 -> viable/strict/1761713514 2025-12-04T09:43:54.2974423Z * [new tag] viable/strict/1761715523 -> viable/strict/1761715523 2025-12-04T09:43:54.2975897Z * [new tag] viable/strict/1761727973 -> viable/strict/1761727973 2025-12-04T09:43:54.2977425Z * [new tag] viable/strict/1761751558 -> viable/strict/1761751558 2025-12-04T09:43:54.2979155Z * [new tag] viable/strict/1761755187 -> viable/strict/1761755187 2025-12-04T09:43:54.2980679Z * [new tag] viable/strict/1761756826 -> viable/strict/1761756826 2025-12-04T09:43:54.2982201Z * [new tag] viable/strict/1761769551 -> viable/strict/1761769551 2025-12-04T09:43:54.2983854Z * [new tag] viable/strict/1761771032 -> viable/strict/1761771032 2025-12-04T09:43:54.2985304Z * [new tag] viable/strict/1761773101 -> viable/strict/1761773101 2025-12-04T09:43:54.2986735Z * [new tag] viable/strict/1761781792 -> viable/strict/1761781792 2025-12-04T09:43:54.2988372Z * [new tag] viable/strict/1761784788 -> viable/strict/1761784788 2025-12-04T09:43:54.2989751Z * [new tag] viable/strict/1761786740 -> viable/strict/1761786740 2025-12-04T09:43:54.2991750Z * [new tag] viable/strict/1761789332 -> viable/strict/1761789332 2025-12-04T09:43:54.2993562Z * [new tag] viable/strict/1761792569 -> viable/strict/1761792569 2025-12-04T09:43:54.2995024Z * [new tag] viable/strict/1761795289 -> viable/strict/1761795289 2025-12-04T09:43:54.2996596Z * [new tag] viable/strict/1761798345 -> viable/strict/1761798345 2025-12-04T09:43:54.2998006Z * [new tag] viable/strict/1761799827 -> viable/strict/1761799827 2025-12-04T09:43:54.2999503Z * [new tag] viable/strict/1761805604 -> viable/strict/1761805604 2025-12-04T09:43:54.3000881Z * [new tag] viable/strict/1761807202 -> viable/strict/1761807202 2025-12-04T09:43:54.3002392Z * [new tag] viable/strict/1761809094 -> viable/strict/1761809094 2025-12-04T09:43:54.3003854Z * [new tag] viable/strict/1761810576 -> viable/strict/1761810576 2025-12-04T09:43:54.3005341Z * [new tag] viable/strict/1761812771 -> viable/strict/1761812771 2025-12-04T09:43:54.3006844Z * [new tag] viable/strict/1761814363 -> viable/strict/1761814363 2025-12-04T09:43:54.3008295Z * [new tag] viable/strict/1761857410 -> viable/strict/1761857410 2025-12-04T09:43:54.3009783Z * [new tag] viable/strict/1761860985 -> viable/strict/1761860985 2025-12-04T09:43:54.3011267Z * [new tag] viable/strict/1761863094 -> viable/strict/1761863094 2025-12-04T09:43:54.3012710Z * [new tag] viable/strict/1761864590 -> viable/strict/1761864590 2025-12-04T09:43:54.3014219Z * [new tag] viable/strict/1761866675 -> viable/strict/1761866675 2025-12-04T09:43:54.3015853Z * [new tag] viable/strict/1761868178 -> viable/strict/1761868178 2025-12-04T09:43:54.3017367Z * [new tag] viable/strict/1761871111 -> viable/strict/1761871111 2025-12-04T09:43:54.3018842Z * [new tag] viable/strict/1761873126 -> viable/strict/1761873126 2025-12-04T09:43:54.3020373Z * [new tag] viable/strict/1761875714 -> viable/strict/1761875714 2025-12-04T09:43:54.3021968Z * [new tag] viable/strict/1761878924 -> viable/strict/1761878924 2025-12-04T09:43:54.3024268Z * [new tag] viable/strict/1761881727 -> viable/strict/1761881727 2025-12-04T09:43:54.3025816Z * [new tag] viable/strict/1761882959 -> viable/strict/1761882959 2025-12-04T09:43:54.3027012Z * [new tag] viable/strict/1761886268 -> viable/strict/1761886268 2025-12-04T09:43:54.3028530Z * [new tag] viable/strict/1761893641 -> viable/strict/1761893641 2025-12-04T09:43:54.3029991Z * [new tag] viable/strict/1761931517 -> viable/strict/1761931517 2025-12-04T09:43:54.3031499Z * [new tag] viable/strict/1761933080 -> viable/strict/1761933080 2025-12-04T09:43:54.3033024Z * [new tag] viable/strict/1761935217 -> viable/strict/1761935217 2025-12-04T09:43:54.3034527Z * [new tag] viable/strict/1761938533 -> viable/strict/1761938533 2025-12-04T09:43:54.3036106Z * [new tag] viable/strict/1761940184 -> viable/strict/1761940184 2025-12-04T09:43:54.3037568Z * [new tag] viable/strict/1761942338 -> viable/strict/1761942338 2025-12-04T09:43:54.3039031Z * [new tag] viable/strict/1761946100 -> viable/strict/1761946100 2025-12-04T09:43:54.3040564Z * [new tag] viable/strict/1761947374 -> viable/strict/1761947374 2025-12-04T09:43:54.3042030Z * [new tag] viable/strict/1761950978 -> viable/strict/1761950978 2025-12-04T09:43:54.3043573Z * [new tag] viable/strict/1761957727 -> viable/strict/1761957727 2025-12-04T09:43:54.3044959Z * [new tag] viable/strict/1761959532 -> viable/strict/1761959532 2025-12-04T09:43:54.3046499Z * [new tag] viable/strict/1761965366 -> viable/strict/1761965366 2025-12-04T09:43:54.3048038Z * [new tag] viable/strict/1761968066 -> viable/strict/1761968066 2025-12-04T09:43:54.3049517Z * [new tag] viable/strict/1761969322 -> viable/strict/1761969322 2025-12-04T09:43:54.3050956Z * [new tag] viable/strict/1761974723 -> viable/strict/1761974723 2025-12-04T09:43:54.3052610Z * [new tag] viable/strict/1761981837 -> viable/strict/1761981837 2025-12-04T09:43:54.3054114Z * [new tag] viable/strict/1761985546 -> viable/strict/1761985546 2025-12-04T09:43:54.3055605Z * [new tag] viable/strict/1761987030 -> viable/strict/1761987030 2025-12-04T09:43:54.3057106Z * [new tag] viable/strict/1762003554 -> viable/strict/1762003554 2025-12-04T09:43:54.3058666Z * [new tag] viable/strict/1762021560 -> viable/strict/1762021560 2025-12-04T09:43:54.3060074Z * [new tag] viable/strict/1762032190 -> viable/strict/1762032190 2025-12-04T09:43:54.3061579Z * [new tag] viable/strict/1762040981 -> viable/strict/1762040981 2025-12-04T09:43:54.3063070Z * [new tag] viable/strict/1762048525 -> viable/strict/1762048525 2025-12-04T09:43:54.3064721Z * [new tag] viable/strict/1762104223 -> viable/strict/1762104223 2025-12-04T09:43:54.3066141Z * [new tag] viable/strict/1762105778 -> viable/strict/1762105778 2025-12-04T09:43:54.3067632Z * [new tag] viable/strict/1762115109 -> viable/strict/1762115109 2025-12-04T09:43:54.3069171Z * [new tag] viable/strict/1762125840 -> viable/strict/1762125840 2025-12-04T09:43:54.3070518Z * [new tag] viable/strict/1762127377 -> viable/strict/1762127377 2025-12-04T09:43:54.3072327Z * [new tag] viable/strict/1762134925 -> viable/strict/1762134925 2025-12-04T09:43:54.3073739Z * [new tag] viable/strict/1762138338 -> viable/strict/1762138338 2025-12-04T09:43:54.3075271Z * [new tag] viable/strict/1762148993 -> viable/strict/1762148993 2025-12-04T09:43:54.3076777Z * [new tag] viable/strict/1762152871 -> viable/strict/1762152871 2025-12-04T09:43:54.3078468Z * [new tag] viable/strict/1762156183 -> viable/strict/1762156183 2025-12-04T09:43:54.3080089Z * [new tag] viable/strict/1762163457 -> viable/strict/1762163457 2025-12-04T09:43:54.3081540Z * [new tag] viable/strict/1762165569 -> viable/strict/1762165569 2025-12-04T09:43:54.3083009Z * [new tag] viable/strict/1762169035 -> viable/strict/1762169035 2025-12-04T09:43:54.3084486Z * [new tag] viable/strict/1762174936 -> viable/strict/1762174936 2025-12-04T09:43:54.3086013Z * [new tag] viable/strict/1762194412 -> viable/strict/1762194412 2025-12-04T09:43:54.3087478Z * [new tag] viable/strict/1762195876 -> viable/strict/1762195876 2025-12-04T09:43:54.3089406Z * [new tag] viable/strict/1762197788 -> viable/strict/1762197788 2025-12-04T09:43:54.3090952Z * [new tag] viable/strict/1762199389 -> viable/strict/1762199389 2025-12-04T09:43:54.3092614Z * [new tag] viable/strict/1762206585 -> viable/strict/1762206585 2025-12-04T09:43:54.3094181Z * [new tag] viable/strict/1762210184 -> viable/strict/1762210184 2025-12-04T09:43:54.3095549Z * [new tag] viable/strict/1762218736 -> viable/strict/1762218736 2025-12-04T09:43:54.3097066Z * [new tag] viable/strict/1762224529 -> viable/strict/1762224529 2025-12-04T09:43:54.3098719Z * [new tag] viable/strict/1762227253 -> viable/strict/1762227253 2025-12-04T09:43:54.3099970Z * [new tag] viable/strict/1762228515 -> viable/strict/1762228515 2025-12-04T09:43:54.3101496Z * [new tag] viable/strict/1762230349 -> viable/strict/1762230349 2025-12-04T09:43:54.3103051Z * [new tag] viable/strict/1762231859 -> viable/strict/1762231859 2025-12-04T09:43:54.3104615Z * [new tag] viable/strict/1762233925 -> viable/strict/1762233925 2025-12-04T09:43:54.3106206Z * [new tag] viable/strict/1762237630 -> viable/strict/1762237630 2025-12-04T09:43:54.3107553Z * [new tag] viable/strict/1762253522 -> viable/strict/1762253522 2025-12-04T09:43:54.3109194Z * [new tag] viable/strict/1762278588 -> viable/strict/1762278588 2025-12-04T09:43:54.3110741Z * [new tag] viable/strict/1762284203 -> viable/strict/1762284203 2025-12-04T09:43:54.3112231Z * [new tag] viable/strict/1762289446 -> viable/strict/1762289446 2025-12-04T09:43:54.3113707Z * [new tag] viable/strict/1762291515 -> viable/strict/1762291515 2025-12-04T09:43:54.3115163Z * [new tag] viable/strict/1762295100 -> viable/strict/1762295100 2025-12-04T09:43:54.3116536Z * [new tag] viable/strict/1762296590 -> viable/strict/1762296590 2025-12-04T09:43:54.3117881Z * [new tag] viable/strict/1762300179 -> viable/strict/1762300179 2025-12-04T09:43:54.3119235Z * [new tag] viable/strict/1762303207 -> viable/strict/1762303207 2025-12-04T09:43:54.3120777Z * [new tag] viable/strict/1762386584 -> viable/strict/1762386584 2025-12-04T09:43:54.3122302Z * [new tag] viable/strict/1762391537 -> viable/strict/1762391537 2025-12-04T09:43:54.3123688Z * [new tag] viable/strict/1762394119 -> viable/strict/1762394119 2025-12-04T09:43:54.3125400Z * [new tag] viable/strict/1762397437 -> viable/strict/1762397437 2025-12-04T09:43:54.3126907Z * [new tag] viable/strict/1762400256 -> viable/strict/1762400256 2025-12-04T09:43:54.3128396Z * [new tag] viable/strict/1762401469 -> viable/strict/1762401469 2025-12-04T09:43:54.3130013Z * [new tag] viable/strict/1762408195 -> viable/strict/1762408195 2025-12-04T09:43:54.3131678Z * [new tag] viable/strict/1762410411 -> viable/strict/1762410411 2025-12-04T09:43:54.3133116Z * [new tag] viable/strict/1762417613 -> viable/strict/1762417613 2025-12-04T09:43:54.3134596Z * [new tag] viable/strict/1762419198 -> viable/strict/1762419198 2025-12-04T09:43:54.3136140Z * [new tag] viable/strict/1762422656 -> viable/strict/1762422656 2025-12-04T09:43:54.3137904Z * [new tag] viable/strict/1762424746 -> viable/strict/1762424746 2025-12-04T09:43:54.3139397Z * [new tag] viable/strict/1762446386 -> viable/strict/1762446386 2025-12-04T09:43:54.3140827Z * [new tag] viable/strict/1762449912 -> viable/strict/1762449912 2025-12-04T09:43:54.3142345Z * [new tag] viable/strict/1762457031 -> viable/strict/1762457031 2025-12-04T09:43:54.3143875Z * [new tag] viable/strict/1762462441 -> viable/strict/1762462441 2025-12-04T09:43:54.3145423Z * [new tag] viable/strict/1762467909 -> viable/strict/1762467909 2025-12-04T09:43:54.3146949Z * [new tag] viable/strict/1762471493 -> viable/strict/1762471493 2025-12-04T09:43:54.3148488Z * [new tag] viable/strict/1762475990 -> viable/strict/1762475990 2025-12-04T09:43:54.3150012Z * [new tag] viable/strict/1762477933 -> viable/strict/1762477933 2025-12-04T09:43:54.3151498Z * [new tag] viable/strict/1762491053 -> viable/strict/1762491053 2025-12-04T09:43:54.3153084Z * [new tag] viable/strict/1762493118 -> viable/strict/1762493118 2025-12-04T09:43:54.3154509Z * [new tag] viable/strict/1762498442 -> viable/strict/1762498442 2025-12-04T09:43:54.3156018Z * [new tag] viable/strict/1762501778 -> viable/strict/1762501778 2025-12-04T09:43:54.3157491Z * [new tag] viable/strict/1762504001 -> viable/strict/1762504001 2025-12-04T09:43:54.3159075Z * [new tag] viable/strict/1762505583 -> viable/strict/1762505583 2025-12-04T09:43:54.3160698Z * [new tag] viable/strict/1762507523 -> viable/strict/1762507523 2025-12-04T09:43:54.3162209Z * [new tag] viable/strict/1762511140 -> viable/strict/1762511140 2025-12-04T09:43:54.3163805Z * [new tag] viable/strict/1762512632 -> viable/strict/1762512632 2025-12-04T09:43:54.3165313Z * [new tag] viable/strict/1762520467 -> viable/strict/1762520467 2025-12-04T09:43:54.3166811Z * [new tag] viable/strict/1762522016 -> viable/strict/1762522016 2025-12-04T09:43:54.3168251Z * [new tag] viable/strict/1762530591 -> viable/strict/1762530591 2025-12-04T09:43:54.3169838Z * [new tag] viable/strict/1762543405 -> viable/strict/1762543405 2025-12-04T09:43:54.3171148Z * [new tag] viable/strict/1762544998 -> viable/strict/1762544998 2025-12-04T09:43:54.3172632Z * [new tag] viable/strict/1762552182 -> viable/strict/1762552182 2025-12-04T09:43:54.3174116Z * [new tag] viable/strict/1762554297 -> viable/strict/1762554297 2025-12-04T09:43:54.3175493Z * [new tag] viable/strict/1762559381 -> viable/strict/1762559381 2025-12-04T09:43:54.3177019Z * [new tag] viable/strict/1762562222 -> viable/strict/1762562222 2025-12-04T09:43:54.3178735Z * [new tag] viable/strict/1762564319 -> viable/strict/1762564319 2025-12-04T09:43:54.3180088Z * [new tag] viable/strict/1762566904 -> viable/strict/1762566904 2025-12-04T09:43:54.3181589Z * [new tag] viable/strict/1762569781 -> viable/strict/1762569781 2025-12-04T09:43:54.3183063Z * [new tag] viable/strict/1762575940 -> viable/strict/1762575940 2025-12-04T09:43:54.3184692Z * [new tag] viable/strict/1762580974 -> viable/strict/1762580974 2025-12-04T09:43:54.3186683Z * [new tag] viable/strict/1762583185 -> viable/strict/1762583185 2025-12-04T09:43:54.3188158Z * [new tag] viable/strict/1762586647 -> viable/strict/1762586647 2025-12-04T09:43:54.3189641Z * [new tag] viable/strict/1762588183 -> viable/strict/1762588183 2025-12-04T09:43:54.3191098Z * [new tag] viable/strict/1762593886 -> viable/strict/1762593886 2025-12-04T09:43:54.3192582Z * [new tag] viable/strict/1762650743 -> viable/strict/1762650743 2025-12-04T09:43:54.3194161Z * [new tag] viable/strict/1762653328 -> viable/strict/1762653328 2025-12-04T09:43:54.3195625Z * [new tag] viable/strict/1762659342 -> viable/strict/1762659342 2025-12-04T09:43:54.3197105Z * [new tag] viable/strict/1762662360 -> viable/strict/1762662360 2025-12-04T09:43:54.3198619Z * [new tag] viable/strict/1762667377 -> viable/strict/1762667377 2025-12-04T09:43:54.3200093Z * [new tag] viable/strict/1762671090 -> viable/strict/1762671090 2025-12-04T09:43:54.3201626Z * [new tag] viable/strict/1762680284 -> viable/strict/1762680284 2025-12-04T09:43:54.3203131Z * [new tag] viable/strict/1762683900 -> viable/strict/1762683900 2025-12-04T09:43:54.3204648Z * [new tag] viable/strict/1762705541 -> viable/strict/1762705541 2025-12-04T09:43:54.3206106Z * [new tag] viable/strict/1762709004 -> viable/strict/1762709004 2025-12-04T09:43:54.3207737Z * [new tag] viable/strict/1762746004 -> viable/strict/1762746004 2025-12-04T09:43:54.3209260Z * [new tag] viable/strict/1762748799 -> viable/strict/1762748799 2025-12-04T09:43:54.3210713Z * [new tag] viable/strict/1762759504 -> viable/strict/1762759504 2025-12-04T09:43:54.3212302Z * [new tag] viable/strict/1762760973 -> viable/strict/1762760973 2025-12-04T09:43:54.3213781Z * [new tag] viable/strict/1762775374 -> viable/strict/1762775374 2025-12-04T09:43:54.3215317Z * [new tag] viable/strict/1762777661 -> viable/strict/1762777661 2025-12-04T09:43:54.3216772Z * [new tag] viable/strict/1762779774 -> viable/strict/1762779774 2025-12-04T09:43:54.3218410Z * [new tag] viable/strict/1762781259 -> viable/strict/1762781259 2025-12-04T09:43:54.3220019Z * [new tag] viable/strict/1762793628 -> viable/strict/1762793628 2025-12-04T09:43:54.3221465Z * [new tag] viable/strict/1762800711 -> viable/strict/1762800711 2025-12-04T09:43:54.3222936Z * [new tag] viable/strict/1762809894 -> viable/strict/1762809894 2025-12-04T09:43:54.3224632Z * [new tag] viable/strict/1762811384 -> viable/strict/1762811384 2025-12-04T09:43:54.3226179Z * [new tag] viable/strict/1762813841 -> viable/strict/1762813841 2025-12-04T09:43:54.3227603Z * [new tag] viable/strict/1762815047 -> viable/strict/1762815047 2025-12-04T09:43:54.3229218Z * [new tag] viable/strict/1762817094 -> viable/strict/1762817094 2025-12-04T09:43:54.3230696Z * [new tag] viable/strict/1762818582 -> viable/strict/1762818582 2025-12-04T09:43:54.3232237Z * [new tag] viable/strict/1762821623 -> viable/strict/1762821623 2025-12-04T09:43:54.3233553Z * [new tag] viable/strict/1762823531 -> viable/strict/1762823531 2025-12-04T09:43:54.3235068Z * [new tag] viable/strict/1762849583 -> viable/strict/1762849583 2025-12-04T09:43:54.3236585Z * [new tag] viable/strict/1762851200 -> viable/strict/1762851200 2025-12-04T09:43:54.3238091Z * [new tag] viable/strict/1762854603 -> viable/strict/1762854603 2025-12-04T09:43:54.3239594Z * [new tag] viable/strict/1762858276 -> viable/strict/1762858276 2025-12-04T09:43:54.3241102Z * [new tag] viable/strict/1762860891 -> viable/strict/1762860891 2025-12-04T09:43:54.3243078Z * [new tag] viable/strict/1762866174 -> viable/strict/1762866174 2025-12-04T09:43:54.3244567Z * [new tag] viable/strict/1762867653 -> viable/strict/1762867653 2025-12-04T09:43:54.3246049Z * [new tag] viable/strict/1762872669 -> viable/strict/1762872669 2025-12-04T09:43:54.3247400Z * [new tag] viable/strict/1762878380 -> viable/strict/1762878380 2025-12-04T09:43:54.3248932Z * [new tag] viable/strict/1762889003 -> viable/strict/1762889003 2025-12-04T09:43:54.3250453Z * [new tag] viable/strict/1762890589 -> viable/strict/1762890589 2025-12-04T09:43:54.3251920Z * [new tag] viable/strict/1762892743 -> viable/strict/1762892743 2025-12-04T09:43:54.3253426Z * [new tag] viable/strict/1762894271 -> viable/strict/1762894271 2025-12-04T09:43:54.3254763Z * [new tag] viable/strict/1762896287 -> viable/strict/1762896287 2025-12-04T09:43:54.3256306Z * [new tag] viable/strict/1762915871 -> viable/strict/1762915871 2025-12-04T09:43:54.3257837Z * [new tag] viable/strict/1762918569 -> viable/strict/1762918569 2025-12-04T09:43:54.3259195Z * [new tag] viable/strict/1762919776 -> viable/strict/1762919776 2025-12-04T09:43:54.3260735Z * [new tag] viable/strict/1762923072 -> viable/strict/1762923072 2025-12-04T09:43:54.3262332Z * [new tag] viable/strict/1762928826 -> viable/strict/1762928826 2025-12-04T09:43:54.3263895Z * [new tag] viable/strict/1762930451 -> viable/strict/1762930451 2025-12-04T09:43:54.3265447Z * [new tag] viable/strict/1762933780 -> viable/strict/1762933780 2025-12-04T09:43:54.3267012Z * [new tag] viable/strict/1762937638 -> viable/strict/1762937638 2025-12-04T09:43:54.3268653Z * [new tag] viable/strict/1762939545 -> viable/strict/1762939545 2025-12-04T09:43:54.3270217Z * [new tag] viable/strict/1762962692 -> viable/strict/1762962692 2025-12-04T09:43:54.3271662Z * [new tag] viable/strict/1762979143 -> viable/strict/1762979143 2025-12-04T09:43:54.3273155Z * [new tag] viable/strict/1762984188 -> viable/strict/1762984188 2025-12-04T09:43:54.3274542Z * [new tag] viable/strict/1762986306 -> viable/strict/1762986306 2025-12-04T09:43:54.3276032Z * [new tag] viable/strict/1762989903 -> viable/strict/1762989903 2025-12-04T09:43:54.3277512Z * [new tag] viable/strict/1762991377 -> viable/strict/1762991377 2025-12-04T09:43:54.3281557Z * [new tag] viable/strict/1762998921 -> viable/strict/1762998921 2025-12-04T09:43:54.3283202Z * [new tag] viable/strict/1763002287 -> viable/strict/1763002287 2025-12-04T09:43:54.3284722Z * [new tag] viable/strict/1763016840 -> viable/strict/1763016840 2025-12-04T09:43:54.3286669Z * [new tag] viable/strict/1763020180 -> viable/strict/1763020180 2025-12-04T09:43:54.3288217Z * [new tag] viable/strict/1763027421 -> viable/strict/1763027421 2025-12-04T09:43:54.3289766Z * [new tag] viable/strict/1763031120 -> viable/strict/1763031120 2025-12-04T09:43:54.3291183Z * [new tag] viable/strict/1763036861 -> viable/strict/1763036861 2025-12-04T09:43:54.3292703Z * [new tag] viable/strict/1763038993 -> viable/strict/1763038993 2025-12-04T09:43:54.3294203Z * [new tag] viable/strict/1763054703 -> viable/strict/1763054703 2025-12-04T09:43:54.3295611Z * [new tag] viable/strict/1763067061 -> viable/strict/1763067061 2025-12-04T09:43:54.3297124Z * [new tag] viable/strict/1763070847 -> viable/strict/1763070847 2025-12-04T09:43:54.3298614Z * [new tag] viable/strict/1763072706 -> viable/strict/1763072706 2025-12-04T09:43:54.3300161Z * [new tag] viable/strict/1763076302 -> viable/strict/1763076302 2025-12-04T09:43:54.3301806Z * [new tag] viable/strict/1763080816 -> viable/strict/1763080816 2025-12-04T09:43:54.3303314Z * [new tag] viable/strict/1763082732 -> viable/strict/1763082732 2025-12-04T09:43:54.3304922Z * [new tag] viable/strict/1763085329 -> viable/strict/1763085329 2025-12-04T09:43:54.3306410Z * [new tag] viable/strict/1763088623 -> viable/strict/1763088623 2025-12-04T09:43:54.3307966Z * [new tag] viable/strict/1763091402 -> viable/strict/1763091402 2025-12-04T09:43:54.3309460Z * [new tag] viable/strict/1763092602 -> viable/strict/1763092602 2025-12-04T09:43:54.3310962Z * [new tag] viable/strict/1763094355 -> viable/strict/1763094355 2025-12-04T09:43:54.3312470Z * [new tag] viable/strict/1763099390 -> viable/strict/1763099390 2025-12-04T09:43:54.3313956Z * [new tag] viable/strict/1763101608 -> viable/strict/1763101608 2025-12-04T09:43:54.3315452Z * [new tag] viable/strict/1763105102 -> viable/strict/1763105102 2025-12-04T09:43:54.3316978Z * [new tag] viable/strict/1763112347 -> viable/strict/1763112347 2025-12-04T09:43:54.3318502Z * [new tag] viable/strict/1763119471 -> viable/strict/1763119471 2025-12-04T09:43:54.3320047Z * [new tag] viable/strict/1763126835 -> viable/strict/1763126835 2025-12-04T09:43:54.3321307Z * [new tag] viable/strict/1763149779 -> viable/strict/1763149779 2025-12-04T09:43:54.3322801Z * [new tag] viable/strict/1763164178 -> viable/strict/1763164178 2025-12-04T09:43:54.3324303Z * [new tag] viable/strict/1763167104 -> viable/strict/1763167104 2025-12-04T09:43:54.3325725Z * [new tag] viable/strict/1763169132 -> viable/strict/1763169132 2025-12-04T09:43:54.3327230Z * [new tag] viable/strict/1763171708 -> viable/strict/1763171708 2025-12-04T09:43:54.3328678Z * [new tag] viable/strict/1763174759 -> viable/strict/1763174759 2025-12-04T09:43:54.3330239Z * [new tag] viable/strict/1763180744 -> viable/strict/1763180744 2025-12-04T09:43:54.3331744Z * [new tag] viable/strict/1763182227 -> viable/strict/1763182227 2025-12-04T09:43:54.3333191Z * [new tag] viable/strict/1763184309 -> viable/strict/1763184309 2025-12-04T09:43:54.3335099Z * [new tag] viable/strict/1763187991 -> viable/strict/1763187991 2025-12-04T09:43:54.3336589Z * [new tag] viable/strict/1763191445 -> viable/strict/1763191445 2025-12-04T09:43:54.3338347Z * [new tag] viable/strict/1763195152 -> viable/strict/1763195152 2025-12-04T09:43:54.3339709Z * [new tag] viable/strict/1763205769 -> viable/strict/1763205769 2025-12-04T09:43:54.3341146Z * [new tag] viable/strict/1763246990 -> viable/strict/1763246990 2025-12-04T09:43:54.3342862Z * [new tag] viable/strict/1763261578 -> viable/strict/1763261578 2025-12-04T09:43:54.3344255Z * [new tag] viable/strict/1763286573 -> viable/strict/1763286573 2025-12-04T09:43:54.3345692Z * [new tag] viable/strict/1763292167 -> viable/strict/1763292167 2025-12-04T09:43:54.3347238Z * [new tag] viable/strict/1763333386 -> viable/strict/1763333386 2025-12-04T09:43:54.3348789Z * [new tag] viable/strict/1763340082 -> viable/strict/1763340082 2025-12-04T09:43:54.3350724Z * [new tag] viable/strict/1763364324 -> viable/strict/1763364324 2025-12-04T09:43:54.3352206Z * [new tag] viable/strict/1763371569 -> viable/strict/1763371569 2025-12-04T09:43:54.3353697Z * [new tag] viable/strict/1763373067 -> viable/strict/1763373067 2025-12-04T09:43:54.3355270Z * [new tag] viable/strict/1763375157 -> viable/strict/1763375157 2025-12-04T09:43:54.3356751Z * [new tag] viable/strict/1763382462 -> viable/strict/1763382462 2025-12-04T09:43:54.3358298Z * [new tag] viable/strict/1763394661 -> viable/strict/1763394661 2025-12-04T09:43:54.3359919Z * [new tag] viable/strict/1763396797 -> viable/strict/1763396797 2025-12-04T09:43:54.3361489Z * [new tag] viable/strict/1763398542 -> viable/strict/1763398542 2025-12-04T09:43:54.3362918Z * [new tag] viable/strict/1763401807 -> viable/strict/1763401807 2025-12-04T09:43:54.3364273Z * [new tag] viable/strict/1763414698 -> viable/strict/1763414698 2025-12-04T09:43:54.3365789Z * [new tag] viable/strict/1763419807 -> viable/strict/1763419807 2025-12-04T09:43:54.3367397Z * [new tag] viable/strict/1763426369 -> viable/strict/1763426369 2025-12-04T09:43:54.3368883Z * [new tag] viable/strict/1763428331 -> viable/strict/1763428331 2025-12-04T09:43:54.3370373Z * [new tag] viable/strict/1763430922 -> viable/strict/1763430922 2025-12-04T09:43:54.3371752Z * [new tag] viable/strict/1763434184 -> viable/strict/1763434184 2025-12-04T09:43:54.3373247Z * [new tag] viable/strict/1763439973 -> viable/strict/1763439973 2025-12-04T09:43:54.3374907Z * [new tag] viable/strict/1763444995 -> viable/strict/1763444995 2025-12-04T09:43:54.3376273Z * [new tag] viable/strict/1763447206 -> viable/strict/1763447206 2025-12-04T09:43:54.3377952Z * [new tag] viable/strict/1763448826 -> viable/strict/1763448826 2025-12-04T09:43:54.3379725Z * [new tag] viable/strict/1763450717 -> viable/strict/1763450717 2025-12-04T09:43:54.3381163Z * [new tag] viable/strict/1763452183 -> viable/strict/1763452183 2025-12-04T09:43:54.3382731Z * [new tag] viable/strict/1763457945 -> viable/strict/1763457945 2025-12-04T09:43:54.3384818Z * [new tag] viable/strict/1763459439 -> viable/strict/1763459439 2025-12-04T09:43:54.3386178Z * [new tag] viable/strict/1763461556 -> viable/strict/1763461556 2025-12-04T09:43:54.3387646Z * [new tag] viable/strict/1763463103 -> viable/strict/1763463103 2025-12-04T09:43:54.3389180Z * [new tag] viable/strict/1763465100 -> viable/strict/1763465100 2025-12-04T09:43:54.3390542Z * [new tag] viable/strict/1763468866 -> viable/strict/1763468866 2025-12-04T09:43:54.3392059Z * [new tag] viable/strict/1763493823 -> viable/strict/1763493823 2025-12-04T09:43:54.3393407Z * [new tag] viable/strict/1763496249 -> viable/strict/1763496249 2025-12-04T09:43:54.3394894Z * [new tag] viable/strict/1763502620 -> viable/strict/1763502620 2025-12-04T09:43:54.3396495Z * [new tag] viable/strict/1763504715 -> viable/strict/1763504715 2025-12-04T09:43:54.3398002Z * [new tag] viable/strict/1763506208 -> viable/strict/1763506208 2025-12-04T09:43:54.3399494Z * [new tag] viable/strict/1763520590 -> viable/strict/1763520590 2025-12-04T09:43:54.3401034Z * [new tag] viable/strict/1763523357 -> viable/strict/1763523357 2025-12-04T09:43:54.3402558Z * [new tag] viable/strict/1763529922 -> viable/strict/1763529922 2025-12-04T09:43:54.3404123Z * [new tag] viable/strict/1763531408 -> viable/strict/1763531408 2025-12-04T09:43:54.3405603Z * [new tag] viable/strict/1763533622 -> viable/strict/1763533622 2025-12-04T09:43:54.3407081Z * [new tag] viable/strict/1763538576 -> viable/strict/1763538576 2025-12-04T09:43:54.3408710Z * [new tag] viable/strict/1763545823 -> viable/strict/1763545823 2025-12-04T09:43:54.3410033Z * [new tag] viable/strict/1763547951 -> viable/strict/1763547951 2025-12-04T09:43:54.3411559Z * [new tag] viable/strict/1763551477 -> viable/strict/1763551477 2025-12-04T09:43:54.3413116Z * [new tag] viable/strict/1763552982 -> viable/strict/1763552982 2025-12-04T09:43:54.3414726Z * [new tag] viable/strict/1763594698 -> viable/strict/1763594698 2025-12-04T09:43:54.3416248Z * [new tag] viable/strict/1763596178 -> viable/strict/1763596178 2025-12-04T09:43:54.3417815Z * [new tag] viable/strict/1763599155 -> viable/strict/1763599155 2025-12-04T09:43:54.3419282Z * [new tag] viable/strict/1763603717 -> viable/strict/1763603717 2025-12-04T09:43:54.3420850Z * [new tag] viable/strict/1763606923 -> viable/strict/1763606923 2025-12-04T09:43:54.3422324Z * [new tag] viable/strict/1763609715 -> viable/strict/1763609715 2025-12-04T09:43:54.3423877Z * [new tag] viable/strict/1763612757 -> viable/strict/1763612757 2025-12-04T09:43:54.3425385Z * [new tag] viable/strict/1763616325 -> viable/strict/1763616325 2025-12-04T09:43:54.3426882Z * [new tag] viable/strict/1763623509 -> viable/strict/1763623509 2025-12-04T09:43:54.3428498Z * [new tag] viable/strict/1763624984 -> viable/strict/1763624984 2025-12-04T09:43:54.3430107Z * [new tag] viable/strict/1763628796 -> viable/strict/1763628796 2025-12-04T09:43:54.3431478Z * [new tag] viable/strict/1763634343 -> viable/strict/1763634343 2025-12-04T09:43:54.3432898Z * [new tag] viable/strict/1763635867 -> viable/strict/1763635867 2025-12-04T09:43:54.3434543Z * [new tag] viable/strict/1763639382 -> viable/strict/1763639382 2025-12-04T09:43:54.3436049Z * [new tag] viable/strict/1763646626 -> viable/strict/1763646626 2025-12-04T09:43:54.3437630Z * [new tag] viable/strict/1763655997 -> viable/strict/1763655997 2025-12-04T09:43:54.3439146Z * [new tag] viable/strict/1763659444 -> viable/strict/1763659444 2025-12-04T09:43:54.3440611Z * [new tag] viable/strict/1763660992 -> viable/strict/1763660992 2025-12-04T09:43:54.3442041Z * [new tag] viable/strict/1763663201 -> viable/strict/1763663201 2025-12-04T09:43:54.3443596Z * [new tag] viable/strict/1763670362 -> viable/strict/1763670362 2025-12-04T09:43:54.3444948Z * [new tag] viable/strict/1763675378 -> viable/strict/1763675378 2025-12-04T09:43:54.3446403Z * [new tag] viable/strict/1763693343 -> viable/strict/1763693343 2025-12-04T09:43:54.3447854Z * [new tag] viable/strict/1763696088 -> viable/strict/1763696088 2025-12-04T09:43:54.3449462Z * [new tag] viable/strict/1763697343 -> viable/strict/1763697343 2025-12-04T09:43:54.3450917Z * [new tag] viable/strict/1763699165 -> viable/strict/1763699165 2025-12-04T09:43:54.3452373Z * [new tag] viable/strict/1763700660 -> viable/strict/1763700660 2025-12-04T09:43:54.3453830Z * [new tag] viable/strict/1763704209 -> viable/strict/1763704209 2025-12-04T09:43:54.3455363Z * [new tag] viable/strict/1763706411 -> viable/strict/1763706411 2025-12-04T09:43:54.3456825Z * [new tag] viable/strict/1763708082 -> viable/strict/1763708082 2025-12-04T09:43:54.3458212Z * [new tag] viable/strict/1763711381 -> viable/strict/1763711381 2025-12-04T09:43:54.3459619Z * [new tag] viable/strict/1763713593 -> viable/strict/1763713593 2025-12-04T09:43:54.3461073Z * [new tag] viable/strict/1763715201 -> viable/strict/1763715201 2025-12-04T09:43:54.3462552Z * [new tag] viable/strict/1763733017 -> viable/strict/1763733017 2025-12-04T09:43:54.3464246Z * [new tag] viable/strict/1763735108 -> viable/strict/1763735108 2025-12-04T09:43:54.3465688Z * [new tag] viable/strict/1763749579 -> viable/strict/1763749579 2025-12-04T09:43:54.3467172Z * [new tag] viable/strict/1763751113 -> viable/strict/1763751113 2025-12-04T09:43:54.3468694Z * [new tag] viable/strict/1763753035 -> viable/strict/1763753035 2025-12-04T09:43:54.3470248Z * [new tag] viable/strict/1763754578 -> viable/strict/1763754578 2025-12-04T09:43:54.3471772Z * [new tag] viable/strict/1763756748 -> viable/strict/1763756748 2025-12-04T09:43:54.3473223Z * [new tag] viable/strict/1763758205 -> viable/strict/1763758205 2025-12-04T09:43:54.3474595Z * [new tag] viable/strict/1763764050 -> viable/strict/1763764050 2025-12-04T09:43:54.3476059Z * [new tag] viable/strict/1763771887 -> viable/strict/1763771887 2025-12-04T09:43:54.3477732Z * [new tag] viable/strict/1763773920 -> viable/strict/1763773920 2025-12-04T09:43:54.3479444Z * [new tag] viable/strict/1763776501 -> viable/strict/1763776501 2025-12-04T09:43:54.3481357Z * [new tag] viable/strict/1763779437 -> viable/strict/1763779437 2025-12-04T09:43:54.3482990Z * [new tag] viable/strict/1763781038 -> viable/strict/1763781038 2025-12-04T09:43:54.3484454Z * [new tag] viable/strict/1763782245 -> viable/strict/1763782245 2025-12-04T09:43:54.3485990Z * [new tag] viable/strict/1763785568 -> viable/strict/1763785568 2025-12-04T09:43:54.3487427Z * [new tag] viable/strict/1763787006 -> viable/strict/1763787006 2025-12-04T09:43:54.3488948Z * [new tag] viable/strict/1763789103 -> viable/strict/1763789103 2025-12-04T09:43:54.3490417Z * [new tag] viable/strict/1763790578 -> viable/strict/1763790578 2025-12-04T09:43:54.3491897Z * [new tag] viable/strict/1763796275 -> viable/strict/1763796275 2025-12-04T09:43:54.3493537Z * [new tag] viable/strict/1763801465 -> viable/strict/1763801465 2025-12-04T09:43:54.3494990Z * [new tag] viable/strict/1763803522 -> viable/strict/1763803522 2025-12-04T09:43:54.3496486Z * [new tag] viable/strict/1763808581 -> viable/strict/1763808581 2025-12-04T09:43:54.3497990Z * [new tag] viable/strict/1763840977 -> viable/strict/1763840977 2025-12-04T09:43:54.3499426Z * [new tag] viable/strict/1763846659 -> viable/strict/1763846659 2025-12-04T09:43:54.3500819Z * [new tag] viable/strict/1763872065 -> viable/strict/1763872065 2025-12-04T09:43:54.3502368Z * [new tag] viable/strict/1763873648 -> viable/strict/1763873648 2025-12-04T09:43:54.3503982Z * [new tag] viable/strict/1763875506 -> viable/strict/1763875506 2025-12-04T09:43:54.3505316Z * [new tag] viable/strict/1763889904 -> viable/strict/1763889904 2025-12-04T09:43:54.3506771Z * [new tag] viable/strict/1763930999 -> viable/strict/1763930999 2025-12-04T09:43:54.3508242Z * [new tag] viable/strict/1763944964 -> viable/strict/1763944964 2025-12-04T09:43:54.3509681Z * [new tag] viable/strict/1763958474 -> viable/strict/1763958474 2025-12-04T09:43:54.3511112Z * [new tag] viable/strict/1763967263 -> viable/strict/1763967263 2025-12-04T09:43:54.3512644Z * [new tag] viable/strict/1763972803 -> viable/strict/1763972803 2025-12-04T09:43:54.3514111Z * [new tag] viable/strict/1763976376 -> viable/strict/1763976376 2025-12-04T09:43:54.3515672Z * [new tag] viable/strict/1763989404 -> viable/strict/1763989404 2025-12-04T09:43:54.3517098Z * [new tag] viable/strict/1763990887 -> viable/strict/1763990887 2025-12-04T09:43:54.3518561Z * [new tag] viable/strict/1764019919 -> viable/strict/1764019919 2025-12-04T09:43:54.3520094Z * [new tag] viable/strict/1764023134 -> viable/strict/1764023134 2025-12-04T09:43:54.3521457Z * [new tag] viable/strict/1764024593 -> viable/strict/1764024593 2025-12-04T09:43:54.3522919Z * [new tag] viable/strict/1764026706 -> viable/strict/1764026706 2025-12-04T09:43:54.3524625Z * [new tag] viable/strict/1764031139 -> viable/strict/1764031139 2025-12-04T09:43:54.3526161Z * [new tag] viable/strict/1764033131 -> viable/strict/1764033131 2025-12-04T09:43:54.3527512Z * [new tag] viable/strict/1764035725 -> viable/strict/1764035725 2025-12-04T09:43:54.3528842Z * [new tag] viable/strict/1764624265 -> viable/strict/1764624265 2025-12-04T09:43:54.3530173Z * [new tag] viable/strict/1764631514 -> viable/strict/1764631514 2025-12-04T09:43:54.3531492Z * [new tag] viable/strict/1764632987 -> viable/strict/1764632987 2025-12-04T09:43:54.3532822Z * [new tag] viable/strict/1764636063 -> viable/strict/1764636063 2025-12-04T09:43:54.3534165Z * [new tag] viable/strict/1764643975 -> viable/strict/1764643975 2025-12-04T09:43:54.3535477Z * [new tag] viable/strict/1764646859 -> viable/strict/1764646859 2025-12-04T09:43:54.3536939Z * [new tag] viable/strict/1764653120 -> viable/strict/1764653120 2025-12-04T09:43:54.3538208Z * [new tag] viable/strict/1764654632 -> viable/strict/1764654632 2025-12-04T09:43:54.3539536Z * [new tag] viable/strict/1764656821 -> viable/strict/1764656821 2025-12-04T09:43:54.3540851Z * [new tag] viable/strict/1764658557 -> viable/strict/1764658557 2025-12-04T09:43:54.3542195Z * [new tag] viable/strict/1764660333 -> viable/strict/1764660333 2025-12-04T09:43:54.3543514Z * [new tag] viable/strict/1764661812 -> viable/strict/1764661812 2025-12-04T09:43:54.3545002Z * [new tag] viable/strict/1764664023 -> viable/strict/1764664023 2025-12-04T09:43:54.3546320Z * [new tag] viable/strict/1764669150 -> viable/strict/1764669150 2025-12-04T09:43:54.3547639Z * [new tag] viable/strict/1764680709 -> viable/strict/1764680709 2025-12-04T09:43:54.3548954Z * [new tag] viable/strict/1764687619 -> viable/strict/1764687619 2025-12-04T09:43:54.3550282Z * [new tag] viable/strict/1764696355 -> viable/strict/1764696355 2025-12-04T09:43:54.3551618Z * [new tag] viable/strict/1764701767 -> viable/strict/1764701767 2025-12-04T09:43:54.3552960Z * [new tag] viable/strict/1764710768 -> viable/strict/1764710768 2025-12-04T09:43:54.3554294Z * [new tag] viable/strict/1764716202 -> viable/strict/1764716202 2025-12-04T09:43:54.3555616Z * [new tag] viable/strict/1764793566 -> viable/strict/1764793566 2025-12-04T09:43:54.3557036Z * [new tag] viable/strict/1764797093 -> viable/strict/1764797093 2025-12-04T09:43:54.3558331Z * [new tag] viable/strict/1764800729 -> viable/strict/1764800729 2025-12-04T09:43:54.3559797Z * [new tag] whc_flight_1 -> whc_flight_1 2025-12-04T09:43:54.3561216Z * [new tag] whc_flight_2 -> whc_flight_2 2025-12-04T09:43:54.3562745Z * [new tag] whc_flight_4 -> whc_flight_4 2025-12-04T09:43:54.4611006Z [command]/usr/bin/git rev-parse --verify --quiet ffd9b0fb4355e97af82fc42cf185c3ffa0fc0a32^{object} 2025-12-04T09:43:54.4642052Z ffd9b0fb4355e97af82fc42cf185c3ffa0fc0a32 2025-12-04T09:43:54.4646272Z ##[endgroup] 2025-12-04T09:43:54.4646553Z ##[group]Determining the checkout info 2025-12-04T09:43:54.4647379Z ##[endgroup] 2025-12-04T09:43:54.4651143Z [command]/usr/bin/git sparse-checkout disable 2025-12-04T09:43:54.4687707Z [command]/usr/bin/git config --local --unset-all extensions.worktreeConfig 2025-12-04T09:43:54.4715291Z ##[group]Checking out the ref 2025-12-04T09:43:54.4718442Z [command]/usr/bin/git checkout --progress --force ffd9b0fb4355e97af82fc42cf185c3ffa0fc0a32 2025-12-04T09:43:55.4981420Z Updating files: 71% (14291/20121) 2025-12-04T09:43:55.5068273Z Updating files: 72% (14488/20121) 2025-12-04T09:43:55.5271653Z Updating files: 73% (14689/20121) 2025-12-04T09:43:55.5520240Z Updating files: 74% (14890/20121) 2025-12-04T09:43:55.6013612Z Updating files: 75% (15091/20121) 2025-12-04T09:43:55.6178054Z Updating files: 76% (15292/20121) 2025-12-04T09:43:55.6338103Z Updating files: 77% (15494/20121) 2025-12-04T09:43:55.6558434Z Updating files: 78% (15695/20121) 2025-12-04T09:43:55.6827776Z Updating files: 79% (15896/20121) 2025-12-04T09:43:55.7144407Z Updating files: 80% (16097/20121) 2025-12-04T09:43:55.7435321Z Updating files: 81% (16299/20121) 2025-12-04T09:43:55.7667478Z Updating files: 82% (16500/20121) 2025-12-04T09:43:55.7848116Z Updating files: 83% (16701/20121) 2025-12-04T09:43:55.8013696Z Updating files: 84% (16902/20121) 2025-12-04T09:43:55.8197467Z Updating files: 85% (17103/20121) 2025-12-04T09:43:55.8373966Z Updating files: 86% (17305/20121) 2025-12-04T09:43:55.8538108Z Updating files: 87% (17506/20121) 2025-12-04T09:43:55.8679991Z Updating files: 88% (17707/20121) 2025-12-04T09:43:55.8842436Z Updating files: 89% (17908/20121) 2025-12-04T09:43:55.9035845Z Updating files: 90% (18109/20121) 2025-12-04T09:43:55.9181483Z Updating files: 91% (18311/20121) 2025-12-04T09:43:55.9357746Z Updating files: 92% (18512/20121) 2025-12-04T09:43:55.9560725Z Updating files: 93% (18713/20121) 2025-12-04T09:43:55.9778721Z Updating files: 94% (18914/20121) 2025-12-04T09:43:55.9975065Z Updating files: 95% (19115/20121) 2025-12-04T09:43:56.0157070Z Updating files: 96% (19317/20121) 2025-12-04T09:43:56.0342956Z Updating files: 97% (19518/20121) 2025-12-04T09:43:56.0630014Z Updating files: 98% (19719/20121) 2025-12-04T09:43:56.0826268Z Updating files: 99% (19920/20121) 2025-12-04T09:43:56.0826530Z Updating files: 100% (20121/20121) 2025-12-04T09:43:56.0826802Z Updating files: 100% (20121/20121), done. 2025-12-04T09:43:56.1060774Z Note: switching to 'ffd9b0fb4355e97af82fc42cf185c3ffa0fc0a32'. 2025-12-04T09:43:56.1061069Z 2025-12-04T09:43:56.1061298Z You are in 'detached HEAD' state. You can look around, make experimental 2025-12-04T09:43:56.1061791Z changes and commit them, and you can discard any commits you make in this 2025-12-04T09:43:56.1062275Z state without impacting any branches by switching back to a branch. 2025-12-04T09:43:56.1062551Z 2025-12-04T09:43:56.1062736Z If you want to create a new branch to retain commits you create, you may 2025-12-04T09:43:56.1063181Z do so (now or later) by using -c with the switch command. Example: 2025-12-04T09:43:56.1063445Z 2025-12-04T09:43:56.1063655Z git switch -c 2025-12-04T09:43:56.1063836Z 2025-12-04T09:43:56.1063939Z Or undo this operation with: 2025-12-04T09:43:56.1064107Z 2025-12-04T09:43:56.1064190Z git switch - 2025-12-04T09:43:56.1064322Z 2025-12-04T09:43:56.1064532Z Turn off this advice by setting config variable advice.detachedHead to false 2025-12-04T09:43:56.1064839Z 2025-12-04T09:43:56.1066390Z HEAD is now at ffd9b0fb435 Resolve collective autotuning test failure on arm (#168919) 2025-12-04T09:43:56.1193500Z ##[endgroup] 2025-12-04T09:43:56.1193889Z ##[group]Setting up auth for fetching submodules 2025-12-04T09:43:56.1200154Z [command]/usr/bin/git config --global http.https://github.com/.extraheader AUTHORIZATION: basic *** 2025-12-04T09:43:56.1254304Z [command]/usr/bin/git config --global --unset-all url.https://github.com/.insteadOf 2025-12-04T09:43:56.1282034Z [command]/usr/bin/git config --global --add url.https://github.com/.insteadOf git@github.com: 2025-12-04T09:43:56.1308404Z [command]/usr/bin/git config --global --add url.https://github.com/.insteadOf org-21003710@github.com: 2025-12-04T09:43:56.1334432Z ##[endgroup] 2025-12-04T09:43:56.1334797Z ##[group]Fetching submodules 2025-12-04T09:43:56.1338048Z [command]/usr/bin/git submodule sync --recursive 2025-12-04T09:43:56.1702836Z [command]/usr/bin/git -c protocol.version=2 submodule update --init --force --recursive 2025-12-04T09:43:56.2058676Z Submodule 'android/libs/fbjni' (https://github.com/facebookincubator/fbjni.git) registered for path 'android/libs/fbjni' 2025-12-04T09:43:56.2473142Z Submodule 'third_party/NNPACK_deps/FP16' (https://github.com/Maratyszcza/FP16.git) registered for path 'third_party/FP16' 2025-12-04T09:43:56.2477570Z Submodule 'third_party/NNPACK_deps/FXdiv' (https://github.com/Maratyszcza/FXdiv.git) registered for path 'third_party/FXdiv' 2025-12-04T09:43:56.2482577Z Submodule 'third_party/NNPACK' (https://github.com/Maratyszcza/NNPACK.git) registered for path 'third_party/NNPACK' 2025-12-04T09:43:56.2486460Z Submodule 'third_party/NVTX' (https://github.com/NVIDIA/NVTX.git) registered for path 'third_party/NVTX' 2025-12-04T09:43:56.2490809Z Submodule 'third_party/VulkanMemoryAllocator' (https://github.com/GPUOpen-LibrariesAndSDKs/VulkanMemoryAllocator.git) registered for path 'third_party/VulkanMemoryAllocator' 2025-12-04T09:43:56.2494726Z Submodule 'third_party/XNNPACK' (https://github.com/google/XNNPACK.git) registered for path 'third_party/XNNPACK' 2025-12-04T09:43:56.2499221Z Submodule 'third_party/aiter' (https://github.com/ROCm/aiter.git) registered for path 'third_party/aiter' 2025-12-04T09:43:56.2503331Z Submodule 'third_party/benchmark' (https://github.com/google/benchmark.git) registered for path 'third_party/benchmark' 2025-12-04T09:43:56.2508023Z Submodule 'third_party/composable_kernel' (https://github.com/ROCm/composable_kernel.git) registered for path 'third_party/composable_kernel' 2025-12-04T09:43:56.2512478Z Submodule 'third_party/cpp-httplib' (https://github.com/yhirose/cpp-httplib.git) registered for path 'third_party/cpp-httplib' 2025-12-04T09:43:56.2517026Z Submodule 'third_party/cpuinfo' (https://github.com/pytorch/cpuinfo.git) registered for path 'third_party/cpuinfo' 2025-12-04T09:43:56.2521541Z Submodule 'third_party/cudnn_frontend' (https://github.com/NVIDIA/cudnn-frontend.git) registered for path 'third_party/cudnn_frontend' 2025-12-04T09:43:56.2526235Z Submodule 'third_party/cutlass' (https://github.com/NVIDIA/cutlass.git) registered for path 'third_party/cutlass' 2025-12-04T09:43:56.2531120Z Submodule 'third_party/fbgemm' (https://github.com/pytorch/fbgemm) registered for path 'third_party/fbgemm' 2025-12-04T09:43:56.2536911Z Submodule 'third_party/flash-attention' (https://github.com/Dao-AILab/flash-attention.git) registered for path 'third_party/flash-attention' 2025-12-04T09:43:56.2541765Z Submodule 'third_party/flatbuffers' (https://github.com/google/flatbuffers.git) registered for path 'third_party/flatbuffers' 2025-12-04T09:43:56.2546940Z Submodule 'third_party/fmt' (https://github.com/fmtlib/fmt.git) registered for path 'third_party/fmt' 2025-12-04T09:43:56.2554983Z Submodule 'third_party/gemmlowp/gemmlowp' (https://github.com/google/gemmlowp.git) registered for path 'third_party/gemmlowp/gemmlowp' 2025-12-04T09:43:56.2560272Z Submodule 'third_party/gloo' (https://github.com/pytorch/gloo) registered for path 'third_party/gloo' 2025-12-04T09:43:56.2565615Z Submodule 'third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/googletest' 2025-12-04T09:43:56.2570869Z Submodule 'third_party/ideep' (https://github.com/intel/ideep) registered for path 'third_party/ideep' 2025-12-04T09:43:56.2576289Z Submodule 'third_party/ittapi' (https://github.com/intel/ittapi.git) registered for path 'third_party/ittapi' 2025-12-04T09:43:56.2582146Z Submodule 'third_party/kineto' (https://github.com/pytorch/kineto) registered for path 'third_party/kineto' 2025-12-04T09:43:56.2587944Z Submodule 'third_party/kleidiai' (https://github.com/ARM-software/kleidiai.git) registered for path 'third_party/kleidiai' 2025-12-04T09:43:56.2593405Z Submodule 'third_party/mimalloc' (https://github.com/microsoft/mimalloc.git) registered for path 'third_party/mimalloc' 2025-12-04T09:43:56.2599262Z Submodule 'third_party/nlohmann' (https://github.com/nlohmann/json.git) registered for path 'third_party/nlohmann' 2025-12-04T09:43:56.2605395Z Submodule 'third_party/onnx' (https://github.com/onnx/onnx.git) registered for path 'third_party/onnx' 2025-12-04T09:43:56.2611098Z Submodule 'third_party/opentelemetry-cpp' (https://github.com/open-telemetry/opentelemetry-cpp.git) registered for path 'third_party/opentelemetry-cpp' 2025-12-04T09:43:56.2616840Z Submodule 'third_party/pocketfft' (https://github.com/mreineck/pocketfft) registered for path 'third_party/pocketfft' 2025-12-04T09:43:56.2622917Z Submodule 'third_party/protobuf' (https://github.com/protocolbuffers/protobuf.git) registered for path 'third_party/protobuf' 2025-12-04T09:43:56.2629173Z Submodule 'third_party/NNPACK_deps/psimd' (https://github.com/Maratyszcza/psimd.git) registered for path 'third_party/psimd' 2025-12-04T09:43:56.2635392Z Submodule 'third_party/NNPACK_deps/pthreadpool' (https://github.com/Maratyszcza/pthreadpool.git) registered for path 'third_party/pthreadpool' 2025-12-04T09:43:56.2641659Z Submodule 'third_party/pybind11' (https://github.com/pybind/pybind11.git) registered for path 'third_party/pybind11' 2025-12-04T09:43:56.2648415Z Submodule 'third_party/python-peachpy' (https://github.com/malfet/PeachPy.git) registered for path 'third_party/python-peachpy' 2025-12-04T09:43:56.2657734Z Submodule 'third_party/sleef' (https://github.com/shibatch/sleef) registered for path 'third_party/sleef' 2025-12-04T09:43:56.2664154Z Submodule 'third_party/tensorpipe' (https://github.com/pytorch/tensorpipe.git) registered for path 'third_party/tensorpipe' 2025-12-04T09:43:56.2699988Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/android/libs/fbjni'... 2025-12-04T09:43:56.4820205Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/FXdiv'... 2025-12-04T09:43:56.4820987Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/FP16'... 2025-12-04T09:43:56.4821656Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/NNPACK'... 2025-12-04T09:43:56.4853520Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fmt'... 2025-12-04T09:43:59.2970546Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/benchmark'... 2025-12-04T09:43:59.2972088Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/NVTX'... 2025-12-04T09:43:59.2973150Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/gloo'... 2025-12-04T09:43:59.2974377Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/gemmlowp/gemmlowp'... 2025-12-04T09:43:59.2975571Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/flash-attention'... 2025-12-04T09:43:59.2976745Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/cpuinfo'... 2025-12-04T09:43:59.2978141Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/cpp-httplib'... 2025-12-04T09:43:59.2979137Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/ideep'... 2025-12-04T09:43:59.2980019Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/ittapi'... 2025-12-04T09:43:59.2980922Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kleidiai'... 2025-12-04T09:43:59.2981841Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/pocketfft'... 2025-12-04T09:43:59.2982776Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/cudnn_frontend'... 2025-12-04T09:43:59.2983792Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/psimd'... 2025-12-04T09:43:59.2984675Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/googletest'... 2025-12-04T09:43:59.2985573Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/mimalloc'... 2025-12-04T09:43:59.2986492Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/pthreadpool'... 2025-12-04T09:43:59.2987482Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/VulkanMemoryAllocator'... 2025-12-04T09:43:59.3972309Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/flatbuffers'... 2025-12-04T09:43:59.4462905Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto'... 2025-12-04T09:43:59.6963733Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/python-peachpy'... 2025-12-04T09:43:59.7964968Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/onnx'... 2025-12-04T09:44:02.0626120Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe'... 2025-12-04T09:44:02.0627112Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/sleef'... 2025-12-04T09:44:02.0628075Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/pybind11'... 2025-12-04T09:44:02.0628942Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/cutlass'... 2025-12-04T09:44:02.0629803Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm'... 2025-12-04T09:44:02.1627551Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/XNNPACK'... 2025-12-04T09:44:18.6015744Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/composable_kernel'... 2025-12-04T09:44:18.6016358Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/nlohmann'... 2025-12-04T09:44:18.6016920Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp'... 2025-12-04T09:44:18.6017750Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/aiter'... 2025-12-04T09:44:18.6018375Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/protobuf'... 2025-12-04T09:44:18.6444465Z Submodule path 'android/libs/fbjni': checked out '7e1e1fe3858c63c251c637ae41a20de425dde96f' 2025-12-04T09:44:18.6963522Z Submodule path 'third_party/FP16': checked out '4dfe081cf6bcd15db339cf2680b9281b8451eeb3' 2025-12-04T09:44:18.7101739Z Submodule path 'third_party/FXdiv': checked out 'b408327ac2a15ec3e43352421954f5b1967701d1' 2025-12-04T09:44:18.7422805Z Submodule path 'third_party/NNPACK': checked out 'c07e3a0400713d546e0dea2d5466dd22ea389c73' 2025-12-04T09:44:18.8379053Z Submodule path 'third_party/NVTX': checked out '3ebbc93ded7285963bff932c678fa367eb393ba6' 2025-12-04T09:44:18.8983234Z Submodule path 'third_party/VulkanMemoryAllocator': checked out '1d8f600fd424278486eade7ed3e877c99f0846b1' 2025-12-04T09:44:19.8367958Z Submodule path 'third_party/XNNPACK': checked out '51a0103656eff6fc9bfd39a4597923c4b542c883' 2025-12-04T09:44:20.0376887Z Submodule path 'third_party/aiter': checked out '01aae101b9e5e94d6c16a9514c9fb8df99c93150' 2025-12-04T09:44:20.0402929Z Submodule '3rdparty/composable_kernel' (https://github.com/ROCm/composable_kernel.git) registered for path 'third_party/aiter/3rdparty/composable_kernel' 2025-12-04T09:44:20.0435634Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/aiter/3rdparty/composable_kernel'... 2025-12-04T09:44:24.4380912Z Submodule path 'third_party/aiter/3rdparty/composable_kernel': checked out 'cffe8fa2a442ac8e80dd236a1a5d24fe3d7e0cbf' 2025-12-04T09:44:24.4678555Z Submodule path 'third_party/benchmark': checked out '299e5928955cc62af9968370293b916f5130916f' 2025-12-04T09:44:24.9067587Z Submodule path 'third_party/composable_kernel': checked out '7fe50dc3da2069d6645d9deb8c017a876472a977' 2025-12-04T09:44:24.9631053Z Submodule path 'third_party/cpp-httplib': checked out '89c932f313c6437c38f2982869beacc89c2f2246' 2025-12-04T09:44:25.0667955Z Submodule path 'third_party/cpuinfo': checked out 'f858c30bcb16f8effd5ff46996f0514539e17abc' 2025-12-04T09:44:25.1210276Z Submodule path 'third_party/cudnn_frontend': checked out '0b1577c8c83401237d601d0d0db5210506705396' 2025-12-04T09:44:25.8756012Z Submodule path 'third_party/cutlass': checked out 'f88806b1e31dfa579842638740216dd41fc6c588' 2025-12-04T09:44:26.0671722Z Submodule path 'third_party/fbgemm': checked out 'c0b988d39a9e47c794d699f29930ed4d7c7e13a4' 2025-12-04T09:44:26.0698975Z Submodule 'external/asmjit' (https://github.com/asmjit/asmjit.git) registered for path 'third_party/fbgemm/external/asmjit' 2025-12-04T09:44:26.0702501Z Submodule 'external/composable_kernel' (https://github.com/ROCm/composable_kernel.git) registered for path 'third_party/fbgemm/external/composable_kernel' 2025-12-04T09:44:26.0706300Z Submodule 'external/cpuinfo' (https://github.com/pytorch/cpuinfo) registered for path 'third_party/fbgemm/external/cpuinfo' 2025-12-04T09:44:26.0710122Z Submodule 'external/cutlass' (https://github.com/jwfromm/cutlass) registered for path 'third_party/fbgemm/external/cutlass' 2025-12-04T09:44:26.0713950Z Submodule 'external/googletest' (https://github.com/google/googletest) registered for path 'third_party/fbgemm/external/googletest' 2025-12-04T09:44:26.0718118Z Submodule 'external/hipify_torch' (https://github.com/ROCmSoftwarePlatform/hipify_torch.git) registered for path 'third_party/fbgemm/external/hipify_torch' 2025-12-04T09:44:26.0722943Z Submodule 'external/json' (https://github.com/nlohmann/json.git) registered for path 'third_party/fbgemm/external/json' 2025-12-04T09:44:26.0758127Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/external/asmjit'... 2025-12-04T09:44:27.1892373Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/external/hipify_torch'... 2025-12-04T09:44:27.1893079Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/external/cpuinfo'... 2025-12-04T09:44:27.1893946Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/external/googletest'... 2025-12-04T09:44:27.2893131Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/external/composable_kernel'... 2025-12-04T09:44:30.3461363Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/external/cutlass'... 2025-12-04T09:44:30.4463003Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/external/json'... 2025-12-04T09:44:32.3564771Z Submodule path 'third_party/fbgemm/external/asmjit': checked out 'a3199e8857792cd10b7589ff5d58343d2c9008ea' 2025-12-04T09:44:32.7938808Z Submodule path 'third_party/fbgemm/external/composable_kernel': checked out '7fe50dc3da2069d6645d9deb8c017a876472a977' 2025-12-04T09:44:32.9005826Z Submodule path 'third_party/fbgemm/external/cpuinfo': checked out '6543fec09b2f04ac4a666882998b534afc9c1349' 2025-12-04T09:44:33.6433282Z Submodule path 'third_party/fbgemm/external/cutlass': checked out '98125ce499b0fdf7ffbe0e3052f5b8709f4840f8' 2025-12-04T09:44:33.6945369Z Submodule path 'third_party/fbgemm/external/googletest': checked out '52eb8108c5bdec04579160ae17225d66034bd723' 2025-12-04T09:44:33.7098425Z Submodule path 'third_party/fbgemm/external/hipify_torch': checked out '63b6a7b541fa7f08f8475ca7d74054db36ff2691' 2025-12-04T09:44:33.8387062Z Submodule path 'third_party/fbgemm/external/json': checked out '9cca280a4d0ccf0c08f47a99aa71d1b0e52f8d03' 2025-12-04T09:44:33.9268647Z Submodule path 'third_party/flash-attention': checked out '979702c87a8713a8e0a5e9fee122b90d2ef13be5' 2025-12-04T09:44:33.9291793Z Submodule 'csrc/composable_kernel' (https://github.com/ROCm/composable_kernel.git) registered for path 'third_party/flash-attention/csrc/composable_kernel' 2025-12-04T09:44:33.9295274Z Submodule 'csrc/cutlass' (https://github.com/NVIDIA/cutlass.git) registered for path 'third_party/flash-attention/csrc/cutlass' 2025-12-04T09:44:33.9328162Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/flash-attention/csrc/composable_kernel'... 2025-12-04T09:44:37.8670552Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/flash-attention/csrc/cutlass'... 2025-12-04T09:44:38.1770912Z Submodule path 'third_party/flash-attention/csrc/composable_kernel': checked out '888317e698e9803c62bd38568abc9e05d7709f33' 2025-12-04T09:44:38.8298135Z Submodule path 'third_party/flash-attention/csrc/cutlass': checked out 'c506e16788cb08416a4a57e11a9067beeee29420' 2025-12-04T09:44:39.0025402Z Submodule path 'third_party/flatbuffers': checked out 'a2cd1ea3b6d3fee220106b5fed3f7ce8da9eb757' 2025-12-04T09:44:39.0359283Z Submodule path 'third_party/fmt': checked out '407c905e45ad75fc29bf0f9bb7c5c2fd3475976f' 2025-12-04T09:44:39.0793486Z Submodule path 'third_party/gemmlowp/gemmlowp': checked out '3fb5c176c17c765a3492cd2f0321b0dab712f350' 2025-12-04T09:44:39.1129136Z Submodule path 'third_party/gloo': checked out '54cbae0d3a67fa890b4c3d9ee162b7860315e341' 2025-12-04T09:44:39.1626025Z Submodule path 'third_party/googletest': checked out '52eb8108c5bdec04579160ae17225d66034bd723' 2025-12-04T09:44:39.1788810Z Submodule path 'third_party/ideep': checked out '719d8e6cd7f7a0e01b155657526d693acf97c2b3' 2025-12-04T09:44:39.1809268Z Submodule 'mkl-dnn' (https://github.com/intel/mkl-dnn.git) registered for path 'third_party/ideep/mkl-dnn' 2025-12-04T09:44:39.1840488Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/ideep/mkl-dnn'... 2025-12-04T09:44:55.5729680Z Submodule path 'third_party/ideep/mkl-dnn': checked out '8d263e693366ef8db40acc569cc7d8edf644556d' 2025-12-04T09:44:55.5988433Z Submodule path 'third_party/ittapi': checked out 'dec1d23ca65ab069d225dfe40dea14f455170959' 2025-12-04T09:44:55.6875220Z Submodule path 'third_party/kineto': checked out '31f85df8fbd89c188f14ef10f1ec65379786b943' 2025-12-04T09:44:55.6897908Z Submodule 'libkineto/third_party/dynolog' (https://github.com/facebookincubator/dynolog.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog' 2025-12-04T09:44:55.6901826Z Submodule 'libkineto/third_party/fmt' (https://github.com/fmtlib/fmt.git) registered for path 'third_party/kineto/libkineto/third_party/fmt' 2025-12-04T09:44:55.6906281Z Submodule 'libkineto/third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/kineto/libkineto/third_party/googletest' 2025-12-04T09:44:55.6939541Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog'... 2025-12-04T09:44:56.3756961Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/fmt'... 2025-12-04T09:44:56.8032803Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/googletest'... 2025-12-04T09:44:56.8988104Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog': checked out 'd2ffe0a4e3acace628db49974246b66fc3e85fb1' 2025-12-04T09:44:56.9016407Z Submodule 'third_party/DCGM' (https://github.com/NVIDIA/DCGM.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2025-12-04T09:44:56.9020268Z Submodule 'third_party/cpr' (https://github.com/libcpr/cpr.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2025-12-04T09:44:56.9024396Z Submodule 'third_party/fmt' (https://github.com/fmtlib/fmt.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2025-12-04T09:44:56.9028430Z Submodule 'third_party/gflags' (https://github.com/gflags/gflags.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2025-12-04T09:44:56.9032522Z Submodule 'third_party/glog' (https://github.com/google/glog.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2025-12-04T09:44:56.9037016Z Submodule 'third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2025-12-04T09:44:56.9041403Z Submodule 'third_party/json' (https://github.com/nlohmann/json.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2025-12-04T09:44:56.9045727Z Submodule 'third_party/pfs' (https://github.com/dtrugman/pfs.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2025-12-04T09:44:56.9050228Z Submodule 'third_party/prometheus-cpp' (https://github.com/jupp0r/prometheus-cpp.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/prometheus-cpp' 2025-12-04T09:44:56.9085290Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM'... 2025-12-04T09:44:58.5713273Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/pfs'... 2025-12-04T09:44:58.5714397Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/prometheus-cpp'... 2025-12-04T09:44:58.5715483Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/gflags'... 2025-12-04T09:44:58.5716460Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/cpr'... 2025-12-04T09:44:58.5717846Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/glog'... 2025-12-04T09:44:58.5718896Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/googletest'... 2025-12-04T09:44:58.5719900Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/fmt'... 2025-12-04T09:44:58.6714166Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/json'... 2025-12-04T09:45:03.3278961Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM': checked out 'ffde4e54bc7249a6039a5e6b45b395141e1217f9' 2025-12-04T09:45:03.3508003Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr': checked out '871ed52d350214a034f6ef8a3b8f51c5ce1bd400' 2025-12-04T09:45:03.3920107Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt': checked out 'cd4af11efc9c622896a3e4cb599fa28668ca3d05' 2025-12-04T09:45:03.4099280Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags': checked out 'e171aa2d15ed9eb17054558e0b3a6a413bb01067' 2025-12-04T09:45:03.4121137Z Submodule 'doc' (https://github.com/gflags/gflags.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2025-12-04T09:45:03.4155303Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc'... 2025-12-04T09:45:03.6706709Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc': checked out '8411df715cf522606e3b1aca386ddfc0b63d34b4' 2025-12-04T09:45:03.6941763Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog': checked out 'b33e3bad4c46c8a6345525fd822af355e5ef9446' 2025-12-04T09:45:03.7444580Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest': checked out '52eb8108c5bdec04579160ae17225d66034bd723' 2025-12-04T09:45:03.8592485Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/json': checked out '4f8fba14066156b73f1189a2b8bd568bde5284c5' 2025-12-04T09:45:03.8806471Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs': checked out 'f68a2fa8ea36c783bdd760371411fcb495aa3150' 2025-12-04T09:45:03.9029007Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/prometheus-cpp': checked out 'b1234816facfdda29845c46696a02998a4af115a' 2025-12-04T09:45:03.9051011Z Submodule 'civetweb' (https://github.com/civetweb/civetweb.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/prometheus-cpp/3rdparty/civetweb' 2025-12-04T09:45:03.9054898Z Submodule 'googletest' (https://github.com/google/googletest.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/prometheus-cpp/3rdparty/googletest' 2025-12-04T09:45:03.9088925Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/prometheus-cpp/3rdparty/civetweb'... 2025-12-04T09:45:06.1998720Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/prometheus-cpp/3rdparty/googletest'... 2025-12-04T09:45:06.4556743Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/prometheus-cpp/3rdparty/civetweb': checked out 'd7ba35bbb649209c66e582d5a0244ba988a15159' 2025-12-04T09:45:06.5068309Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/prometheus-cpp/3rdparty/googletest': checked out 'e2239ee6043f73722e7aa812a459f54a28552929' 2025-12-04T09:45:06.5435575Z Submodule path 'third_party/kineto/libkineto/third_party/fmt': checked out '40626af88bd7df9a5fb80be7b25ac85b122d6c21' 2025-12-04T09:45:06.5937297Z Submodule path 'third_party/kineto/libkineto/third_party/googletest': checked out '52eb8108c5bdec04579160ae17225d66034bd723' 2025-12-04T09:45:06.6589469Z Submodule path 'third_party/kleidiai': checked out 'd7770c89632329a9914ef1a90289917597639cbe' 2025-12-04T09:45:06.7052979Z Submodule path 'third_party/mimalloc': checked out 'fbd8b99c2b828428947d70fdc046bb55609be93e' 2025-12-04T09:45:06.8267862Z Submodule path 'third_party/nlohmann': checked out '55f93686c01528224f448c19128836e7df245f72' 2025-12-04T09:45:07.4164599Z Submodule path 'third_party/onnx': checked out 'e709452ef2bbc1d113faf678c24e6d3467696e83' 2025-12-04T09:45:07.4204352Z Submodule 'third_party/pybind11' (https://github.com/pybind/pybind11.git) registered for path 'third_party/onnx/third_party/pybind11' 2025-12-04T09:45:07.4238578Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/onnx/third_party/pybind11'... 2025-12-04T09:45:08.1453688Z Submodule path 'third_party/onnx/third_party/pybind11': checked out 'a2e59f0e7065404b44dfe92a28aca47ba1378dc4' 2025-12-04T09:45:08.2387038Z Submodule path 'third_party/opentelemetry-cpp': checked out 'a799f4aed9c94b765dcdaabaeab7d5e7e2310878' 2025-12-04T09:45:08.2410884Z Submodule 'third_party/benchmark' (https://github.com/google/benchmark) registered for path 'third_party/opentelemetry-cpp/third_party/benchmark' 2025-12-04T09:45:08.2414205Z Submodule 'third_party/googletest' (https://github.com/google/googletest) registered for path 'third_party/opentelemetry-cpp/third_party/googletest' 2025-12-04T09:45:08.2417842Z Submodule 'third_party/ms-gsl' (https://github.com/microsoft/GSL) registered for path 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2025-12-04T09:45:08.2421806Z Submodule 'third_party/nlohmann-json' (https://github.com/nlohmann/json) registered for path 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2025-12-04T09:45:08.2425786Z Submodule 'third_party/opentelemetry-proto' (https://github.com/open-telemetry/opentelemetry-proto) registered for path 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2025-12-04T09:45:08.2429839Z Submodule 'third_party/opentracing-cpp' (https://github.com/opentracing/opentracing-cpp.git) registered for path 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2025-12-04T09:45:08.2434077Z Submodule 'third_party/prometheus-cpp' (https://github.com/jupp0r/prometheus-cpp) registered for path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2025-12-04T09:45:08.2438185Z Submodule 'tools/vcpkg' (https://github.com/Microsoft/vcpkg) registered for path 'third_party/opentelemetry-cpp/tools/vcpkg' 2025-12-04T09:45:08.2471520Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/benchmark'... 2025-12-04T09:45:08.6172275Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/opentracing-cpp'... 2025-12-04T09:45:08.6173327Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/opentelemetry-proto'... 2025-12-04T09:45:08.6174140Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/prometheus-cpp'... 2025-12-04T09:45:08.6174877Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/ms-gsl'... 2025-12-04T09:45:08.7174031Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/googletest'... 2025-12-04T09:45:09.2549941Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/nlohmann-json'... 2025-12-04T09:45:15.1597985Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/tools/vcpkg'... 2025-12-04T09:45:15.7942826Z Submodule path 'third_party/opentelemetry-cpp/third_party/benchmark': checked out 'd572f4777349d43653b21d6c2fc63020ab326db2' 2025-12-04T09:45:15.8400871Z Submodule path 'third_party/opentelemetry-cpp/third_party/googletest': checked out 'b796f7d44681514f58a683a3a71ff17c94edb0c1' 2025-12-04T09:45:15.8595212Z Submodule path 'third_party/opentelemetry-cpp/third_party/ms-gsl': checked out '6f4529395c5b7c2d661812257cd6780c67e54afa' 2025-12-04T09:45:15.9839116Z Submodule path 'third_party/opentelemetry-cpp/third_party/nlohmann-json': checked out 'bc889afb4c5bf1c0d8ee29ef35eaaf4c8bef8a5d' 2025-12-04T09:45:16.0021482Z Submodule path 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto': checked out '4ca4f0335c63cda7ab31ea7ed70d6553aee14dce' 2025-12-04T09:45:16.0213051Z Submodule path 'third_party/opentelemetry-cpp/third_party/opentracing-cpp': checked out '06b57f48ded1fa3bdd3d4346f6ef29e40e08eaf5' 2025-12-04T09:45:16.0426037Z Submodule path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp': checked out 'c9ffcdda9086ffd9e1283ea7a0276d831f3c8a8d' 2025-12-04T09:45:16.0447321Z Submodule 'civetweb' (https://github.com/civetweb/civetweb.git) registered for path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2025-12-04T09:45:16.0451010Z Submodule 'googletest' (https://github.com/google/googletest.git) registered for path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2025-12-04T09:45:16.0483418Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb'... 2025-12-04T09:45:17.7922225Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest'... 2025-12-04T09:45:18.0457535Z Submodule path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb': checked out 'eefb26f82b233268fc98577d265352720d477ba4' 2025-12-04T09:45:18.0979023Z Submodule path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest': checked out 'e2239ee6043f73722e7aa812a459f54a28552929' 2025-12-04T09:45:18.7984302Z Submodule path 'third_party/opentelemetry-cpp/tools/vcpkg': checked out '8eb57355a4ffb410a2e94c07b4dca2dffbee8e50' 2025-12-04T09:45:18.8141103Z Submodule path 'third_party/pocketfft': checked out '0fa0ef591e38c2758e3184c6c23e497b9f732ffa' 2025-12-04T09:45:19.1169239Z Submodule path 'third_party/protobuf': checked out 'd1eca4e4b421cd2997495c4b4e65cea6be4e9b8a' 2025-12-04T09:45:19.1196441Z Submodule 'third_party/benchmark' (https://github.com/google/benchmark.git) registered for path 'third_party/protobuf/third_party/benchmark' 2025-12-04T09:45:19.1200199Z Submodule 'third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/protobuf/third_party/googletest' 2025-12-04T09:45:19.1233106Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/protobuf/third_party/benchmark'... 2025-12-04T09:45:19.6363410Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/protobuf/third_party/googletest'... 2025-12-04T09:45:19.9566590Z Submodule path 'third_party/protobuf/third_party/benchmark': checked out '5b7683f49e1e9223cf9927b24f6fd3d6bd82e3f8' 2025-12-04T09:45:20.0320370Z Submodule path 'third_party/protobuf/third_party/googletest': checked out '5ec7f0c4a113e2f18ac2c6cc7df51ad6afc24081' 2025-12-04T09:45:20.0450436Z Submodule path 'third_party/psimd': checked out '072586a71b55b7f8c584153d223e95687148a900' 2025-12-04T09:45:20.0608438Z Submodule path 'third_party/pthreadpool': checked out '4fe0e1e183925bf8cfa6aae24237e724a96479b8' 2025-12-04T09:45:20.1105776Z Submodule path 'third_party/pybind11': checked out 'f5fbe867d2d26e4a0a9177a51f6e568868ad3dc8' 2025-12-04T09:45:20.1434773Z Submodule path 'third_party/python-peachpy': checked out 'f45429b087dd7d5bc78bb40dc7cf06425c252d67' 2025-12-04T09:45:20.1916468Z Submodule path 'third_party/sleef': checked out '5a1d179df9cf652951b59010a2d2075372d67f68' 2025-12-04T09:45:20.2281288Z Submodule path 'third_party/tensorpipe': checked out '2b4cd91092d335a697416b2a3cb398283246849d' 2025-12-04T09:45:20.2302618Z Submodule 'third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/tensorpipe/third_party/googletest' 2025-12-04T09:45:20.2306829Z Submodule 'third_party/libnop' (https://github.com/google/libnop.git) registered for path 'third_party/tensorpipe/third_party/libnop' 2025-12-04T09:45:20.2310170Z Submodule 'third_party/libuv' (https://github.com/libuv/libuv.git) registered for path 'third_party/tensorpipe/third_party/libuv' 2025-12-04T09:45:20.2314036Z Submodule 'third_party/pybind11' (https://github.com/pybind/pybind11.git) registered for path 'third_party/tensorpipe/third_party/pybind11' 2025-12-04T09:45:20.2346215Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/googletest'... 2025-12-04T09:45:21.2174256Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/libnop'... 2025-12-04T09:45:21.2175170Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/pybind11'... 2025-12-04T09:45:21.2520625Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/libuv'... 2025-12-04T09:45:21.3119107Z Submodule path 'third_party/tensorpipe/third_party/googletest': checked out 'aee0f9d9b5b87796ee8a0ab26b7587ec30e8858e' 2025-12-04T09:45:21.3320315Z Submodule path 'third_party/tensorpipe/third_party/libnop': checked out '910b55815be16109f04f4180e9adee14fb4ce281' 2025-12-04T09:45:21.4118190Z Submodule path 'third_party/tensorpipe/third_party/libuv': checked out '5152db2cbfeb5582e9c27c5ea1dba2cd9e10759b' 2025-12-04T09:45:21.4463457Z Submodule path 'third_party/tensorpipe/third_party/pybind11': checked out 'a23996fce38ff6ccfbcdc09f1e63f2c4be5ea2ef' 2025-12-04T09:45:21.4484940Z Submodule 'tools/clang' (https://github.com/wjakob/clang-cindex-python3) registered for path 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2025-12-04T09:45:21.4517373Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/pybind11/tools/clang'... 2025-12-04T09:45:21.6524926Z Submodule path 'third_party/tensorpipe/third_party/pybind11/tools/clang': checked out '6a00cbc4a9b8e68b71caf7f774b3f9c753ae84d5' 2025-12-04T09:45:21.6571980Z [command]/usr/bin/git submodule foreach --recursive git config --local gc.auto 0 2025-12-04T09:45:21.6931368Z Entering 'android/libs/fbjni' 2025-12-04T09:45:21.6986538Z Entering 'third_party/FP16' 2025-12-04T09:45:21.7039927Z Entering 'third_party/FXdiv' 2025-12-04T09:45:21.7091977Z Entering 'third_party/NNPACK' 2025-12-04T09:45:21.7147526Z Entering 'third_party/NVTX' 2025-12-04T09:45:21.7201387Z Entering 'third_party/VulkanMemoryAllocator' 2025-12-04T09:45:21.7257609Z Entering 'third_party/XNNPACK' 2025-12-04T09:45:21.7324139Z Entering 'third_party/aiter' 2025-12-04T09:45:21.7379889Z Entering 'third_party/aiter/3rdparty/composable_kernel' 2025-12-04T09:45:21.7442085Z Entering 'third_party/benchmark' 2025-12-04T09:45:21.7493365Z Entering 'third_party/composable_kernel' 2025-12-04T09:45:21.7554300Z Entering 'third_party/cpp-httplib' 2025-12-04T09:45:21.7606964Z Entering 'third_party/cpuinfo' 2025-12-04T09:45:21.7671498Z Entering 'third_party/cudnn_frontend' 2025-12-04T09:45:21.7722792Z Entering 'third_party/cutlass' 2025-12-04T09:45:21.7786338Z Entering 'third_party/fbgemm' 2025-12-04T09:45:21.7840569Z Entering 'third_party/fbgemm/external/asmjit' 2025-12-04T09:45:21.7892232Z Entering 'third_party/fbgemm/external/composable_kernel' 2025-12-04T09:45:21.7952183Z Entering 'third_party/fbgemm/external/cpuinfo' 2025-12-04T09:45:21.8006610Z Entering 'third_party/fbgemm/external/cutlass' 2025-12-04T09:45:21.8070375Z Entering 'third_party/fbgemm/external/googletest' 2025-12-04T09:45:21.8120385Z Entering 'third_party/fbgemm/external/hipify_torch' 2025-12-04T09:45:21.8170245Z Entering 'third_party/fbgemm/external/json' 2025-12-04T09:45:21.8225455Z Entering 'third_party/flash-attention' 2025-12-04T09:45:21.8278454Z Entering 'third_party/flash-attention/csrc/composable_kernel' 2025-12-04T09:45:21.8335726Z Entering 'third_party/flash-attention/csrc/cutlass' 2025-12-04T09:45:21.8398600Z Entering 'third_party/flatbuffers' 2025-12-04T09:45:21.8453321Z Entering 'third_party/fmt' 2025-12-04T09:45:21.8503354Z Entering 'third_party/gemmlowp/gemmlowp' 2025-12-04T09:45:21.8555917Z Entering 'third_party/gloo' 2025-12-04T09:45:21.8607734Z Entering 'third_party/googletest' 2025-12-04T09:45:21.8663629Z Entering 'third_party/ideep' 2025-12-04T09:45:21.8714562Z Entering 'third_party/ideep/mkl-dnn' 2025-12-04T09:45:21.8771449Z Entering 'third_party/ittapi' 2025-12-04T09:45:21.8824161Z Entering 'third_party/kineto' 2025-12-04T09:45:21.8877531Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2025-12-04T09:45:21.8928199Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2025-12-04T09:45:21.8982820Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2025-12-04T09:45:21.9036292Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2025-12-04T09:45:21.9090371Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2025-12-04T09:45:21.9139875Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2025-12-04T09:45:21.9195468Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2025-12-04T09:45:21.9249090Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2025-12-04T09:45:21.9301542Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2025-12-04T09:45:21.9353851Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2025-12-04T09:45:21.9405021Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/prometheus-cpp' 2025-12-04T09:45:21.9456393Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/prometheus-cpp/3rdparty/civetweb' 2025-12-04T09:45:21.9512757Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/prometheus-cpp/3rdparty/googletest' 2025-12-04T09:45:21.9570076Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2025-12-04T09:45:21.9622040Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2025-12-04T09:45:21.9678059Z Entering 'third_party/kleidiai' 2025-12-04T09:45:21.9736499Z Entering 'third_party/mimalloc' 2025-12-04T09:45:21.9790438Z Entering 'third_party/nlohmann' 2025-12-04T09:45:21.9843020Z Entering 'third_party/onnx' 2025-12-04T09:45:21.9907740Z Entering 'third_party/onnx/third_party/pybind11' 2025-12-04T09:45:21.9963518Z Entering 'third_party/opentelemetry-cpp' 2025-12-04T09:45:22.0017110Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2025-12-04T09:45:22.0070210Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2025-12-04T09:45:22.0121154Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2025-12-04T09:45:22.0169850Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2025-12-04T09:45:22.0221998Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2025-12-04T09:45:22.0274043Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2025-12-04T09:45:22.0325546Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2025-12-04T09:45:22.0376029Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2025-12-04T09:45:22.0430297Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2025-12-04T09:45:22.0483303Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2025-12-04T09:45:22.0552001Z Entering 'third_party/pocketfft' 2025-12-04T09:45:22.0602812Z Entering 'third_party/protobuf' 2025-12-04T09:45:22.0655442Z Entering 'third_party/protobuf/third_party/benchmark' 2025-12-04T09:45:22.0708015Z Entering 'third_party/protobuf/third_party/googletest' 2025-12-04T09:45:22.0762930Z Entering 'third_party/psimd' 2025-12-04T09:45:22.0817753Z Entering 'third_party/pthreadpool' 2025-12-04T09:45:22.0869964Z Entering 'third_party/pybind11' 2025-12-04T09:45:22.0927562Z Entering 'third_party/python-peachpy' 2025-12-04T09:45:22.0982510Z Entering 'third_party/sleef' 2025-12-04T09:45:22.1035662Z Entering 'third_party/tensorpipe' 2025-12-04T09:45:22.1085926Z Entering 'third_party/tensorpipe/third_party/googletest' 2025-12-04T09:45:22.1137940Z Entering 'third_party/tensorpipe/third_party/libnop' 2025-12-04T09:45:22.1188295Z Entering 'third_party/tensorpipe/third_party/libuv' 2025-12-04T09:45:22.1239840Z Entering 'third_party/tensorpipe/third_party/pybind11' 2025-12-04T09:45:22.1289271Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2025-12-04T09:45:22.1361394Z ##[endgroup] 2025-12-04T09:45:22.1361862Z ##[group]Persisting credentials for submodules 2025-12-04T09:45:22.1368649Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'url\.https\:\/\/github\.com\/\.insteadOf' && git config --local --unset-all 'url.https://github.com/.insteadOf' || :" 2025-12-04T09:45:22.1727186Z Entering 'android/libs/fbjni' 2025-12-04T09:45:22.1797944Z Entering 'third_party/FP16' 2025-12-04T09:45:22.1870802Z Entering 'third_party/FXdiv' 2025-12-04T09:45:22.1941841Z Entering 'third_party/NNPACK' 2025-12-04T09:45:22.2009608Z Entering 'third_party/NVTX' 2025-12-04T09:45:22.2081058Z Entering 'third_party/VulkanMemoryAllocator' 2025-12-04T09:45:22.2149753Z Entering 'third_party/XNNPACK' 2025-12-04T09:45:22.2231127Z Entering 'third_party/aiter' 2025-12-04T09:45:22.2299992Z Entering 'third_party/aiter/3rdparty/composable_kernel' 2025-12-04T09:45:22.2377662Z Entering 'third_party/benchmark' 2025-12-04T09:45:22.2451876Z Entering 'third_party/composable_kernel' 2025-12-04T09:45:22.2528661Z Entering 'third_party/cpp-httplib' 2025-12-04T09:45:22.2603589Z Entering 'third_party/cpuinfo' 2025-12-04T09:45:22.2679399Z Entering 'third_party/cudnn_frontend' 2025-12-04T09:45:22.2752456Z Entering 'third_party/cutlass' 2025-12-04T09:45:22.2828424Z Entering 'third_party/fbgemm' 2025-12-04T09:45:22.2904199Z Entering 'third_party/fbgemm/external/asmjit' 2025-12-04T09:45:22.2970218Z Entering 'third_party/fbgemm/external/composable_kernel' 2025-12-04T09:45:22.3047646Z Entering 'third_party/fbgemm/external/cpuinfo' 2025-12-04T09:45:22.3123202Z Entering 'third_party/fbgemm/external/cutlass' 2025-12-04T09:45:22.3200809Z Entering 'third_party/fbgemm/external/googletest' 2025-12-04T09:45:22.3269230Z Entering 'third_party/fbgemm/external/hipify_torch' 2025-12-04T09:45:22.3341707Z Entering 'third_party/fbgemm/external/json' 2025-12-04T09:45:22.3414809Z Entering 'third_party/flash-attention' 2025-12-04T09:45:22.3487865Z Entering 'third_party/flash-attention/csrc/composable_kernel' 2025-12-04T09:45:22.3563246Z Entering 'third_party/flash-attention/csrc/cutlass' 2025-12-04T09:45:22.3641611Z Entering 'third_party/flatbuffers' 2025-12-04T09:45:22.3722233Z Entering 'third_party/fmt' 2025-12-04T09:45:22.3791273Z Entering 'third_party/gemmlowp/gemmlowp' 2025-12-04T09:45:22.3860164Z Entering 'third_party/gloo' 2025-12-04T09:45:22.3929345Z Entering 'third_party/googletest' 2025-12-04T09:45:22.3999202Z Entering 'third_party/ideep' 2025-12-04T09:45:22.4067260Z Entering 'third_party/ideep/mkl-dnn' 2025-12-04T09:45:22.4145380Z Entering 'third_party/ittapi' 2025-12-04T09:45:22.4218085Z Entering 'third_party/kineto' 2025-12-04T09:45:22.4288633Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2025-12-04T09:45:22.4358652Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2025-12-04T09:45:22.4430646Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2025-12-04T09:45:22.4501470Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2025-12-04T09:45:22.4570834Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2025-12-04T09:45:22.4638284Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2025-12-04T09:45:22.4712233Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2025-12-04T09:45:22.4783070Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2025-12-04T09:45:22.4852309Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2025-12-04T09:45:22.4922360Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2025-12-04T09:45:22.4992158Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/prometheus-cpp' 2025-12-04T09:45:22.5059501Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/prometheus-cpp/3rdparty/civetweb' 2025-12-04T09:45:22.5131924Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/prometheus-cpp/3rdparty/googletest' 2025-12-04T09:45:22.5207173Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2025-12-04T09:45:22.5278261Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2025-12-04T09:45:22.5353128Z Entering 'third_party/kleidiai' 2025-12-04T09:45:22.5422380Z Entering 'third_party/mimalloc' 2025-12-04T09:45:22.5491725Z Entering 'third_party/nlohmann' 2025-12-04T09:45:22.5561625Z Entering 'third_party/onnx' 2025-12-04T09:45:22.5649886Z Entering 'third_party/onnx/third_party/pybind11' 2025-12-04T09:45:22.5728576Z Entering 'third_party/opentelemetry-cpp' 2025-12-04T09:45:22.5805683Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2025-12-04T09:45:22.5871657Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2025-12-04T09:45:22.5939679Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2025-12-04T09:45:22.6013475Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2025-12-04T09:45:22.6081727Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2025-12-04T09:45:22.6148779Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2025-12-04T09:45:22.6218974Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2025-12-04T09:45:22.6287370Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2025-12-04T09:45:22.6359069Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2025-12-04T09:45:22.6438861Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2025-12-04T09:45:22.6527928Z Entering 'third_party/pocketfft' 2025-12-04T09:45:22.6603133Z Entering 'third_party/protobuf' 2025-12-04T09:45:22.6672769Z Entering 'third_party/protobuf/third_party/benchmark' 2025-12-04T09:45:22.6740297Z Entering 'third_party/protobuf/third_party/googletest' 2025-12-04T09:45:22.6815315Z Entering 'third_party/psimd' 2025-12-04T09:45:22.6888304Z Entering 'third_party/pthreadpool' 2025-12-04T09:45:22.6961366Z Entering 'third_party/pybind11' 2025-12-04T09:45:22.7030856Z Entering 'third_party/python-peachpy' 2025-12-04T09:45:22.7099611Z Entering 'third_party/sleef' 2025-12-04T09:45:22.7170458Z Entering 'third_party/tensorpipe' 2025-12-04T09:45:22.7239440Z Entering 'third_party/tensorpipe/third_party/googletest' 2025-12-04T09:45:22.7308715Z Entering 'third_party/tensorpipe/third_party/libnop' 2025-12-04T09:45:22.7382322Z Entering 'third_party/tensorpipe/third_party/libuv' 2025-12-04T09:45:22.7450059Z Entering 'third_party/tensorpipe/third_party/pybind11' 2025-12-04T09:45:22.7517046Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2025-12-04T09:45:22.7615134Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local 'http.https://github.com/.extraheader' 'AUTHORIZATION: basic ***' && git config --local --show-origin --name-only --get-regexp remote.origin.url" 2025-12-04T09:45:22.7966024Z Entering 'android/libs/fbjni' 2025-12-04T09:45:22.8035690Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/android/libs/fbjni/config remote.origin.url 2025-12-04T09:45:22.8057779Z Entering 'third_party/FP16' 2025-12-04T09:45:22.8121711Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/FP16/config remote.origin.url 2025-12-04T09:45:22.8142855Z Entering 'third_party/FXdiv' 2025-12-04T09:45:22.8206396Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/FXdiv/config remote.origin.url 2025-12-04T09:45:22.8228225Z Entering 'third_party/NNPACK' 2025-12-04T09:45:22.8293659Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK/config remote.origin.url 2025-12-04T09:45:22.8317322Z Entering 'third_party/NVTX' 2025-12-04T09:45:22.8385373Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NVTX/config remote.origin.url 2025-12-04T09:45:22.8407762Z Entering 'third_party/VulkanMemoryAllocator' 2025-12-04T09:45:22.8475575Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/VulkanMemoryAllocator/config remote.origin.url 2025-12-04T09:45:22.8498552Z Entering 'third_party/XNNPACK' 2025-12-04T09:45:22.8562619Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/XNNPACK/config remote.origin.url 2025-12-04T09:45:22.8598738Z Entering 'third_party/aiter' 2025-12-04T09:45:22.8662193Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/aiter/config remote.origin.url 2025-12-04T09:45:22.8683036Z Entering 'third_party/aiter/3rdparty/composable_kernel' 2025-12-04T09:45:22.8746879Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/aiter/modules/3rdparty/composable_kernel/config remote.origin.url 2025-12-04T09:45:22.8778170Z Entering 'third_party/benchmark' 2025-12-04T09:45:22.8841757Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/benchmark/config remote.origin.url 2025-12-04T09:45:22.8868645Z Entering 'third_party/composable_kernel' 2025-12-04T09:45:22.8934411Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/composable_kernel/config remote.origin.url 2025-12-04T09:45:22.8964240Z Entering 'third_party/cpp-httplib' 2025-12-04T09:45:22.9029331Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cpp-httplib/config remote.origin.url 2025-12-04T09:45:22.9050621Z Entering 'third_party/cpuinfo' 2025-12-04T09:45:22.9115072Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cpuinfo/config remote.origin.url 2025-12-04T09:45:22.9137544Z Entering 'third_party/cudnn_frontend' 2025-12-04T09:45:22.9205512Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cudnn_frontend/config remote.origin.url 2025-12-04T09:45:22.9227737Z Entering 'third_party/cutlass' 2025-12-04T09:45:22.9292620Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cutlass/config remote.origin.url 2025-12-04T09:45:22.9321954Z Entering 'third_party/fbgemm' 2025-12-04T09:45:22.9385533Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/config remote.origin.url 2025-12-04T09:45:22.9408563Z Entering 'third_party/fbgemm/external/asmjit' 2025-12-04T09:45:22.9473112Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/external/asmjit/config remote.origin.url 2025-12-04T09:45:22.9493873Z Entering 'third_party/fbgemm/external/composable_kernel' 2025-12-04T09:45:22.9558227Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/external/composable_kernel/config remote.origin.url 2025-12-04T09:45:22.9588039Z Entering 'third_party/fbgemm/external/cpuinfo' 2025-12-04T09:45:22.9653332Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/external/cpuinfo/config remote.origin.url 2025-12-04T09:45:22.9674131Z Entering 'third_party/fbgemm/external/cutlass' 2025-12-04T09:45:22.9744553Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/external/cutlass/config remote.origin.url 2025-12-04T09:45:22.9774102Z Entering 'third_party/fbgemm/external/googletest' 2025-12-04T09:45:22.9844576Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/external/googletest/config remote.origin.url 2025-12-04T09:45:22.9866064Z Entering 'third_party/fbgemm/external/hipify_torch' 2025-12-04T09:45:22.9934760Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/external/hipify_torch/config remote.origin.url 2025-12-04T09:45:22.9956301Z Entering 'third_party/fbgemm/external/json' 2025-12-04T09:45:23.0021488Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/external/json/config remote.origin.url 2025-12-04T09:45:23.0045551Z Entering 'third_party/flash-attention' 2025-12-04T09:45:23.0115373Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/flash-attention/config remote.origin.url 2025-12-04T09:45:23.0137092Z Entering 'third_party/flash-attention/csrc/composable_kernel' 2025-12-04T09:45:23.0202289Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/flash-attention/modules/csrc/composable_kernel/config remote.origin.url 2025-12-04T09:45:23.0228259Z Entering 'third_party/flash-attention/csrc/cutlass' 2025-12-04T09:45:23.0295022Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/flash-attention/modules/csrc/cutlass/config remote.origin.url 2025-12-04T09:45:23.0326522Z Entering 'third_party/flatbuffers' 2025-12-04T09:45:23.0391794Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/flatbuffers/config remote.origin.url 2025-12-04T09:45:23.0417389Z Entering 'third_party/fmt' 2025-12-04T09:45:23.0485919Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fmt/config remote.origin.url 2025-12-04T09:45:23.0508269Z Entering 'third_party/gemmlowp/gemmlowp' 2025-12-04T09:45:23.0574535Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/gemmlowp/gemmlowp/config remote.origin.url 2025-12-04T09:45:23.0597078Z Entering 'third_party/gloo' 2025-12-04T09:45:23.0666207Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/gloo/config remote.origin.url 2025-12-04T09:45:23.0688619Z Entering 'third_party/googletest' 2025-12-04T09:45:23.0754058Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/googletest/config remote.origin.url 2025-12-04T09:45:23.0780941Z Entering 'third_party/ideep' 2025-12-04T09:45:23.0845244Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/ideep/config remote.origin.url 2025-12-04T09:45:23.0865560Z Entering 'third_party/ideep/mkl-dnn' 2025-12-04T09:45:23.0930144Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/ideep/modules/mkl-dnn/config remote.origin.url 2025-12-04T09:45:23.0958988Z Entering 'third_party/ittapi' 2025-12-04T09:45:23.1024525Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/ittapi/config remote.origin.url 2025-12-04T09:45:23.1046405Z Entering 'third_party/kineto' 2025-12-04T09:45:23.1115036Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/config remote.origin.url 2025-12-04T09:45:23.1135996Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2025-12-04T09:45:23.1201834Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/config remote.origin.url 2025-12-04T09:45:23.1220900Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2025-12-04T09:45:23.1288407Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/DCGM/config remote.origin.url 2025-12-04T09:45:23.1311083Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2025-12-04T09:45:23.1376798Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/cpr/config remote.origin.url 2025-12-04T09:45:23.1397367Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2025-12-04T09:45:23.1467645Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/fmt/config remote.origin.url 2025-12-04T09:45:23.1488116Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2025-12-04T09:45:23.1554606Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/gflags/config remote.origin.url 2025-12-04T09:45:23.1575096Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2025-12-04T09:45:23.1641767Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/gflags/modules/doc/config remote.origin.url 2025-12-04T09:45:23.1665839Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2025-12-04T09:45:23.1728876Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/glog/config remote.origin.url 2025-12-04T09:45:23.1750632Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2025-12-04T09:45:23.1815657Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/googletest/config remote.origin.url 2025-12-04T09:45:23.1837960Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2025-12-04T09:45:23.1907711Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/json/config remote.origin.url 2025-12-04T09:45:23.1930685Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2025-12-04T09:45:23.1996295Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/pfs/config remote.origin.url 2025-12-04T09:45:23.2018047Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/prometheus-cpp' 2025-12-04T09:45:23.2086652Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/prometheus-cpp/config remote.origin.url 2025-12-04T09:45:23.2107068Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/prometheus-cpp/3rdparty/civetweb' 2025-12-04T09:45:23.2178851Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/prometheus-cpp/modules/civetweb/config remote.origin.url 2025-12-04T09:45:23.2202160Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/prometheus-cpp/3rdparty/googletest' 2025-12-04T09:45:23.2266273Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/prometheus-cpp/modules/googletest/config remote.origin.url 2025-12-04T09:45:23.2292523Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2025-12-04T09:45:23.2355128Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/fmt/config remote.origin.url 2025-12-04T09:45:23.2376928Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2025-12-04T09:45:23.2445041Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/googletest/config remote.origin.url 2025-12-04T09:45:23.2469841Z Entering 'third_party/kleidiai' 2025-12-04T09:45:23.2535525Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kleidiai/config remote.origin.url 2025-12-04T09:45:23.2558923Z Entering 'third_party/mimalloc' 2025-12-04T09:45:23.2627974Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/mimalloc/config remote.origin.url 2025-12-04T09:45:23.2649808Z Entering 'third_party/nlohmann' 2025-12-04T09:45:23.2713815Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/nlohmann/config remote.origin.url 2025-12-04T09:45:23.2737269Z Entering 'third_party/onnx' 2025-12-04T09:45:23.2804522Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx/config remote.origin.url 2025-12-04T09:45:23.2840426Z Entering 'third_party/onnx/third_party/pybind11' 2025-12-04T09:45:23.2903960Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx/modules/third_party/pybind11/config remote.origin.url 2025-12-04T09:45:23.2927949Z Entering 'third_party/opentelemetry-cpp' 2025-12-04T09:45:23.2998140Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/config remote.origin.url 2025-12-04T09:45:23.3019654Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2025-12-04T09:45:23.3085110Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/benchmark/config remote.origin.url 2025-12-04T09:45:23.3106857Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2025-12-04T09:45:23.3171196Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/googletest/config remote.origin.url 2025-12-04T09:45:23.3192050Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2025-12-04T09:45:23.3256289Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/ms-gsl/config remote.origin.url 2025-12-04T09:45:23.3278242Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2025-12-04T09:45:23.3346647Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/nlohmann-json/config remote.origin.url 2025-12-04T09:45:23.3369121Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2025-12-04T09:45:23.3435124Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/opentelemetry-proto/config remote.origin.url 2025-12-04T09:45:23.3456048Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2025-12-04T09:45:23.3521857Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/opentracing-cpp/config remote.origin.url 2025-12-04T09:45:23.3541757Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2025-12-04T09:45:23.3604651Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/prometheus-cpp/config remote.origin.url 2025-12-04T09:45:23.3624708Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2025-12-04T09:45:23.3686266Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/prometheus-cpp/modules/civetweb/config remote.origin.url 2025-12-04T09:45:23.3709398Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2025-12-04T09:45:23.3773648Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/prometheus-cpp/modules/googletest/config remote.origin.url 2025-12-04T09:45:23.3796561Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2025-12-04T09:45:23.3858569Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/tools/vcpkg/config remote.origin.url 2025-12-04T09:45:23.3899600Z Entering 'third_party/pocketfft' 2025-12-04T09:45:23.3964358Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/pocketfft/config remote.origin.url 2025-12-04T09:45:23.3992476Z Entering 'third_party/protobuf' 2025-12-04T09:45:23.4056825Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/protobuf/config remote.origin.url 2025-12-04T09:45:23.4078310Z Entering 'third_party/protobuf/third_party/benchmark' 2025-12-04T09:45:23.4144976Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/protobuf/modules/third_party/benchmark/config remote.origin.url 2025-12-04T09:45:23.4166626Z Entering 'third_party/protobuf/third_party/googletest' 2025-12-04T09:45:23.4228863Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/protobuf/modules/third_party/googletest/config remote.origin.url 2025-12-04T09:45:23.4253170Z Entering 'third_party/psimd' 2025-12-04T09:45:23.4316933Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/psimd/config remote.origin.url 2025-12-04T09:45:23.4338899Z Entering 'third_party/pthreadpool' 2025-12-04T09:45:23.4404789Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/pthreadpool/config remote.origin.url 2025-12-04T09:45:23.4426609Z Entering 'third_party/pybind11' 2025-12-04T09:45:23.4491396Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/pybind11/config remote.origin.url 2025-12-04T09:45:23.4512310Z Entering 'third_party/python-peachpy' 2025-12-04T09:45:23.4575179Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/python-peachpy/config remote.origin.url 2025-12-04T09:45:23.4597693Z Entering 'third_party/sleef' 2025-12-04T09:45:23.4666920Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/sleef/config remote.origin.url 2025-12-04T09:45:23.4689486Z Entering 'third_party/tensorpipe' 2025-12-04T09:45:23.4752128Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/config remote.origin.url 2025-12-04T09:45:23.4771557Z Entering 'third_party/tensorpipe/third_party/googletest' 2025-12-04T09:45:23.4835719Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/googletest/config remote.origin.url 2025-12-04T09:45:23.4857349Z Entering 'third_party/tensorpipe/third_party/libnop' 2025-12-04T09:45:23.4923667Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/libnop/config remote.origin.url 2025-12-04T09:45:23.4945944Z Entering 'third_party/tensorpipe/third_party/libuv' 2025-12-04T09:45:23.5011273Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/libuv/config remote.origin.url 2025-12-04T09:45:23.5032342Z Entering 'third_party/tensorpipe/third_party/pybind11' 2025-12-04T09:45:23.5094969Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/pybind11/config remote.origin.url 2025-12-04T09:45:23.5114868Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2025-12-04T09:45:23.5178416Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/pybind11/modules/tools/clang/config remote.origin.url 2025-12-04T09:45:23.6257153Z [command]/usr/bin/git submodule foreach --recursive git config --local --add 'url.https://github.com/.insteadOf' 'git@github.com:' 2025-12-04T09:45:23.6616199Z Entering 'android/libs/fbjni' 2025-12-04T09:45:23.6671763Z Entering 'third_party/FP16' 2025-12-04T09:45:23.6723729Z Entering 'third_party/FXdiv' 2025-12-04T09:45:23.6773922Z Entering 'third_party/NNPACK' 2025-12-04T09:45:23.6824588Z Entering 'third_party/NVTX' 2025-12-04T09:45:23.6877054Z Entering 'third_party/VulkanMemoryAllocator' 2025-12-04T09:45:23.6927375Z Entering 'third_party/XNNPACK' 2025-12-04T09:45:23.6998334Z Entering 'third_party/aiter' 2025-12-04T09:45:23.7051317Z Entering 'third_party/aiter/3rdparty/composable_kernel' 2025-12-04T09:45:23.7111476Z Entering 'third_party/benchmark' 2025-12-04T09:45:23.7162357Z Entering 'third_party/composable_kernel' 2025-12-04T09:45:23.7223964Z Entering 'third_party/cpp-httplib' 2025-12-04T09:45:23.7277096Z Entering 'third_party/cpuinfo' 2025-12-04T09:45:23.7331105Z Entering 'third_party/cudnn_frontend' 2025-12-04T09:45:23.7382842Z Entering 'third_party/cutlass' 2025-12-04T09:45:23.7443858Z Entering 'third_party/fbgemm' 2025-12-04T09:45:23.7496637Z Entering 'third_party/fbgemm/external/asmjit' 2025-12-04T09:45:23.7549154Z Entering 'third_party/fbgemm/external/composable_kernel' 2025-12-04T09:45:23.7607058Z Entering 'third_party/fbgemm/external/cpuinfo' 2025-12-04T09:45:23.7662216Z Entering 'third_party/fbgemm/external/cutlass' 2025-12-04T09:45:23.7725087Z Entering 'third_party/fbgemm/external/googletest' 2025-12-04T09:45:23.7777528Z Entering 'third_party/fbgemm/external/hipify_torch' 2025-12-04T09:45:23.7833120Z Entering 'third_party/fbgemm/external/json' 2025-12-04T09:45:23.7887741Z Entering 'third_party/flash-attention' 2025-12-04T09:45:23.7940349Z Entering 'third_party/flash-attention/csrc/composable_kernel' 2025-12-04T09:45:23.7998640Z Entering 'third_party/flash-attention/csrc/cutlass' 2025-12-04T09:45:23.8059178Z Entering 'third_party/flatbuffers' 2025-12-04T09:45:23.8115150Z Entering 'third_party/fmt' 2025-12-04T09:45:23.8169578Z Entering 'third_party/gemmlowp/gemmlowp' 2025-12-04T09:45:23.8221364Z Entering 'third_party/gloo' 2025-12-04T09:45:23.8273153Z Entering 'third_party/googletest' 2025-12-04T09:45:23.8324074Z Entering 'third_party/ideep' 2025-12-04T09:45:23.8374688Z Entering 'third_party/ideep/mkl-dnn' 2025-12-04T09:45:23.8439016Z Entering 'third_party/ittapi' 2025-12-04T09:45:23.8491778Z Entering 'third_party/kineto' 2025-12-04T09:45:23.8544112Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2025-12-04T09:45:23.8594301Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2025-12-04T09:45:23.8646065Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2025-12-04T09:45:23.8700260Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2025-12-04T09:45:23.8753673Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2025-12-04T09:45:23.8805629Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2025-12-04T09:45:23.8862141Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2025-12-04T09:45:23.8915876Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2025-12-04T09:45:23.8971255Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2025-12-04T09:45:23.9023858Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2025-12-04T09:45:23.9078412Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/prometheus-cpp' 2025-12-04T09:45:23.9128577Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/prometheus-cpp/3rdparty/civetweb' 2025-12-04T09:45:23.9183783Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/prometheus-cpp/3rdparty/googletest' 2025-12-04T09:45:23.9244558Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2025-12-04T09:45:23.9302771Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2025-12-04T09:45:23.9362586Z Entering 'third_party/kleidiai' 2025-12-04T09:45:23.9413351Z Entering 'third_party/mimalloc' 2025-12-04T09:45:23.9466550Z Entering 'third_party/nlohmann' 2025-12-04T09:45:23.9519332Z Entering 'third_party/onnx' 2025-12-04T09:45:23.9586123Z Entering 'third_party/onnx/third_party/pybind11' 2025-12-04T09:45:23.9645109Z Entering 'third_party/opentelemetry-cpp' 2025-12-04T09:45:23.9699897Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2025-12-04T09:45:23.9751016Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2025-12-04T09:45:23.9802715Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2025-12-04T09:45:23.9852460Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2025-12-04T09:45:23.9905515Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2025-12-04T09:45:23.9957372Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2025-12-04T09:45:24.0010111Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2025-12-04T09:45:24.0058758Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2025-12-04T09:45:24.0113393Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2025-12-04T09:45:24.0169771Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2025-12-04T09:45:24.0242055Z Entering 'third_party/pocketfft' 2025-12-04T09:45:24.0302907Z Entering 'third_party/protobuf' 2025-12-04T09:45:24.0359091Z Entering 'third_party/protobuf/third_party/benchmark' 2025-12-04T09:45:24.0410801Z Entering 'third_party/protobuf/third_party/googletest' 2025-12-04T09:45:24.0464507Z Entering 'third_party/psimd' 2025-12-04T09:45:24.0519535Z Entering 'third_party/pthreadpool' 2025-12-04T09:45:24.0571268Z Entering 'third_party/pybind11' 2025-12-04T09:45:24.0626674Z Entering 'third_party/python-peachpy' 2025-12-04T09:45:24.0680198Z Entering 'third_party/sleef' 2025-12-04T09:45:24.0732433Z Entering 'third_party/tensorpipe' 2025-12-04T09:45:24.0789204Z Entering 'third_party/tensorpipe/third_party/googletest' 2025-12-04T09:45:24.0840796Z Entering 'third_party/tensorpipe/third_party/libnop' 2025-12-04T09:45:24.0892948Z Entering 'third_party/tensorpipe/third_party/libuv' 2025-12-04T09:45:24.0945440Z Entering 'third_party/tensorpipe/third_party/pybind11' 2025-12-04T09:45:24.0997025Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2025-12-04T09:45:24.1072644Z [command]/usr/bin/git submodule foreach --recursive git config --local --add 'url.https://github.com/.insteadOf' 'org-21003710@github.com:' 2025-12-04T09:45:24.1436199Z Entering 'android/libs/fbjni' 2025-12-04T09:45:24.1486989Z Entering 'third_party/FP16' 2025-12-04T09:45:24.1538998Z Entering 'third_party/FXdiv' 2025-12-04T09:45:24.1591687Z Entering 'third_party/NNPACK' 2025-12-04T09:45:24.1643805Z Entering 'third_party/NVTX' 2025-12-04T09:45:24.1694962Z Entering 'third_party/VulkanMemoryAllocator' 2025-12-04T09:45:24.1749132Z Entering 'third_party/XNNPACK' 2025-12-04T09:45:24.1813472Z Entering 'third_party/aiter' 2025-12-04T09:45:24.1866151Z Entering 'third_party/aiter/3rdparty/composable_kernel' 2025-12-04T09:45:24.1927285Z Entering 'third_party/benchmark' 2025-12-04T09:45:24.1984028Z Entering 'third_party/composable_kernel' 2025-12-04T09:45:24.2052545Z Entering 'third_party/cpp-httplib' 2025-12-04T09:45:24.2104518Z Entering 'third_party/cpuinfo' 2025-12-04T09:45:24.2158971Z Entering 'third_party/cudnn_frontend' 2025-12-04T09:45:24.2209813Z Entering 'third_party/cutlass' 2025-12-04T09:45:24.2272032Z Entering 'third_party/fbgemm' 2025-12-04T09:45:24.2325202Z Entering 'third_party/fbgemm/external/asmjit' 2025-12-04T09:45:24.2376942Z Entering 'third_party/fbgemm/external/composable_kernel' 2025-12-04T09:45:24.2437307Z Entering 'third_party/fbgemm/external/cpuinfo' 2025-12-04T09:45:24.2493759Z Entering 'third_party/fbgemm/external/cutlass' 2025-12-04T09:45:24.2551164Z Entering 'third_party/fbgemm/external/googletest' 2025-12-04T09:45:24.2601153Z Entering 'third_party/fbgemm/external/hipify_torch' 2025-12-04T09:45:24.2650972Z Entering 'third_party/fbgemm/external/json' 2025-12-04T09:45:24.2705506Z Entering 'third_party/flash-attention' 2025-12-04T09:45:24.2756460Z Entering 'third_party/flash-attention/csrc/composable_kernel' 2025-12-04T09:45:24.2814625Z Entering 'third_party/flash-attention/csrc/cutlass' 2025-12-04T09:45:24.2875847Z Entering 'third_party/flatbuffers' 2025-12-04T09:45:24.2933718Z Entering 'third_party/fmt' 2025-12-04T09:45:24.2986113Z Entering 'third_party/gemmlowp/gemmlowp' 2025-12-04T09:45:24.3041382Z Entering 'third_party/gloo' 2025-12-04T09:45:24.3096529Z Entering 'third_party/googletest' 2025-12-04T09:45:24.3150221Z Entering 'third_party/ideep' 2025-12-04T09:45:24.3200585Z Entering 'third_party/ideep/mkl-dnn' 2025-12-04T09:45:24.3258834Z Entering 'third_party/ittapi' 2025-12-04T09:45:24.3312039Z Entering 'third_party/kineto' 2025-12-04T09:45:24.3398503Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2025-12-04T09:45:24.3446119Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2025-12-04T09:45:24.3500483Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2025-12-04T09:45:24.3553468Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2025-12-04T09:45:24.3604581Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2025-12-04T09:45:24.3657149Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2025-12-04T09:45:24.3713452Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2025-12-04T09:45:24.3763216Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2025-12-04T09:45:24.3816864Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2025-12-04T09:45:24.3873272Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2025-12-04T09:45:24.3927417Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/prometheus-cpp' 2025-12-04T09:45:24.3981159Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/prometheus-cpp/3rdparty/civetweb' 2025-12-04T09:45:24.4034678Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/prometheus-cpp/3rdparty/googletest' 2025-12-04T09:45:24.4094318Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2025-12-04T09:45:24.4146078Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2025-12-04T09:45:24.4202143Z Entering 'third_party/kleidiai' 2025-12-04T09:45:24.4258292Z Entering 'third_party/mimalloc' 2025-12-04T09:45:24.4314362Z Entering 'third_party/nlohmann' 2025-12-04T09:45:24.4366325Z Entering 'third_party/onnx' 2025-12-04T09:45:24.4433775Z Entering 'third_party/onnx/third_party/pybind11' 2025-12-04T09:45:24.4488137Z Entering 'third_party/opentelemetry-cpp' 2025-12-04T09:45:24.4541959Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2025-12-04T09:45:24.4592539Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2025-12-04T09:45:24.4643516Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2025-12-04T09:45:24.4697731Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2025-12-04T09:45:24.4750890Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2025-12-04T09:45:24.4802167Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2025-12-04T09:45:24.4852179Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2025-12-04T09:45:24.4901935Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2025-12-04T09:45:24.4954411Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2025-12-04T09:45:24.5007276Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2025-12-04T09:45:24.5083062Z Entering 'third_party/pocketfft' 2025-12-04T09:45:24.5138170Z Entering 'third_party/protobuf' 2025-12-04T09:45:24.5202205Z Entering 'third_party/protobuf/third_party/benchmark' 2025-12-04T09:45:24.5252647Z Entering 'third_party/protobuf/third_party/googletest' 2025-12-04T09:45:24.5307224Z Entering 'third_party/psimd' 2025-12-04T09:45:24.5360540Z Entering 'third_party/pthreadpool' 2025-12-04T09:45:24.5415252Z Entering 'third_party/pybind11' 2025-12-04T09:45:24.5469445Z Entering 'third_party/python-peachpy' 2025-12-04T09:45:24.5522194Z Entering 'third_party/sleef' 2025-12-04T09:45:24.5575271Z Entering 'third_party/tensorpipe' 2025-12-04T09:45:24.5627875Z Entering 'third_party/tensorpipe/third_party/googletest' 2025-12-04T09:45:24.5679253Z Entering 'third_party/tensorpipe/third_party/libnop' 2025-12-04T09:45:24.5730467Z Entering 'third_party/tensorpipe/third_party/libuv' 2025-12-04T09:45:24.5782378Z Entering 'third_party/tensorpipe/third_party/pybind11' 2025-12-04T09:45:24.5830679Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2025-12-04T09:45:24.5902700Z ##[endgroup] 2025-12-04T09:45:24.5948007Z [command]/usr/bin/git log -1 --format=%H 2025-12-04T09:45:24.5971589Z ffd9b0fb4355e97af82fc42cf185c3ffa0fc0a32 2025-12-04T09:45:24.6090528Z ##[group]Run cd "${GITHUB_WORKSPACE}" 2025-12-04T09:45:24.6090836Z cd "${GITHUB_WORKSPACE}" 2025-12-04T09:45:24.6091082Z # Clean stale submodule dirs 2025-12-04T09:45:24.6091321Z if [ -z "${NO_SUDO}" ]; then 2025-12-04T09:45:24.6091601Z  sudo git submodule foreach --recursive git clean -ffdx 2025-12-04T09:45:24.6092102Z else 2025-12-04T09:45:24.6092333Z  git submodule foreach --recursive git clean -ffdx 2025-12-04T09:45:24.6092600Z fi 2025-12-04T09:45:24.6102068Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-12-04T09:45:24.6102349Z env: 2025-12-04T09:45:24.6102512Z GIT_DEFAULT_BRANCH: main 2025-12-04T09:45:24.6102694Z NO_SUDO: true 2025-12-04T09:45:24.6102860Z ##[endgroup] 2025-12-04T09:45:24.6486006Z Entering 'android/libs/fbjni' 2025-12-04T09:45:24.6529907Z Entering 'third_party/FP16' 2025-12-04T09:45:24.6567149Z Entering 'third_party/FXdiv' 2025-12-04T09:45:24.6609555Z Entering 'third_party/NNPACK' 2025-12-04T09:45:24.6653611Z Entering 'third_party/NVTX' 2025-12-04T09:45:24.6701617Z Entering 'third_party/VulkanMemoryAllocator' 2025-12-04T09:45:24.6744882Z Entering 'third_party/XNNPACK' 2025-12-04T09:45:24.6881086Z Entering 'third_party/aiter' 2025-12-04T09:45:24.6931736Z Entering 'third_party/aiter/3rdparty/composable_kernel' 2025-12-04T09:45:24.7065546Z Entering 'third_party/benchmark' 2025-12-04T09:45:24.7106523Z Entering 'third_party/composable_kernel' 2025-12-04T09:45:24.7248932Z Entering 'third_party/cpp-httplib' 2025-12-04T09:45:24.7291264Z Entering 'third_party/cpuinfo' 2025-12-04T09:45:24.7336232Z Entering 'third_party/cudnn_frontend' 2025-12-04T09:45:24.7381347Z Entering 'third_party/cutlass' 2025-12-04T09:45:24.7499158Z Entering 'third_party/fbgemm' 2025-12-04T09:45:24.7573138Z Entering 'third_party/fbgemm/external/asmjit' 2025-12-04T09:45:24.7612194Z Entering 'third_party/fbgemm/external/composable_kernel' 2025-12-04T09:45:24.7757029Z Entering 'third_party/fbgemm/external/cpuinfo' 2025-12-04T09:45:24.7806549Z Entering 'third_party/fbgemm/external/cutlass' 2025-12-04T09:45:24.7925634Z Entering 'third_party/fbgemm/external/googletest' 2025-12-04T09:45:24.7968966Z Entering 'third_party/fbgemm/external/hipify_torch' 2025-12-04T09:45:24.8007806Z Entering 'third_party/fbgemm/external/json' 2025-12-04T09:45:24.8065670Z Entering 'third_party/flash-attention' 2025-12-04T09:45:24.8117139Z Entering 'third_party/flash-attention/csrc/composable_kernel' 2025-12-04T09:45:24.8237517Z Entering 'third_party/flash-attention/csrc/cutlass' 2025-12-04T09:45:24.8348749Z Entering 'third_party/flatbuffers' 2025-12-04T09:45:24.8446162Z Entering 'third_party/fmt' 2025-12-04T09:45:24.8489145Z Entering 'third_party/gemmlowp/gemmlowp' 2025-12-04T09:45:24.8531000Z Entering 'third_party/gloo' 2025-12-04T09:45:24.8572345Z Entering 'third_party/googletest' 2025-12-04T09:45:24.8620900Z Entering 'third_party/ideep' 2025-12-04T09:45:24.8658489Z Entering 'third_party/ideep/mkl-dnn' 2025-12-04T09:45:24.8761111Z Entering 'third_party/ittapi' 2025-12-04T09:45:24.8808728Z Entering 'third_party/kineto' 2025-12-04T09:45:24.8859362Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2025-12-04T09:45:24.8907923Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2025-12-04T09:45:24.8965943Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2025-12-04T09:45:24.9008145Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2025-12-04T09:45:24.9053189Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2025-12-04T09:45:24.9090593Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2025-12-04T09:45:24.9130555Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2025-12-04T09:45:24.9170913Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2025-12-04T09:45:24.9220408Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2025-12-04T09:45:24.9271804Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2025-12-04T09:45:24.9312941Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/prometheus-cpp' 2025-12-04T09:45:24.9351651Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/prometheus-cpp/3rdparty/civetweb' 2025-12-04T09:45:24.9412579Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/prometheus-cpp/3rdparty/googletest' 2025-12-04T09:45:24.9465165Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2025-12-04T09:45:24.9507972Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2025-12-04T09:45:24.9551687Z Entering 'third_party/kleidiai' 2025-12-04T09:45:24.9600652Z Entering 'third_party/mimalloc' 2025-12-04T09:45:24.9641540Z Entering 'third_party/nlohmann' 2025-12-04T09:45:24.9698270Z Entering 'third_party/onnx' 2025-12-04T09:45:25.0131225Z Entering 'third_party/onnx/third_party/pybind11' 2025-12-04T09:45:25.0177007Z Entering 'third_party/opentelemetry-cpp' 2025-12-04T09:45:25.0248239Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2025-12-04T09:45:25.0288722Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2025-12-04T09:45:25.0330974Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2025-12-04T09:45:25.0369996Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2025-12-04T09:45:25.0422522Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2025-12-04T09:45:25.0461175Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2025-12-04T09:45:25.0500070Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2025-12-04T09:45:25.0539077Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2025-12-04T09:45:25.0600974Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2025-12-04T09:45:25.0645541Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2025-12-04T09:45:25.0987465Z Entering 'third_party/pocketfft' 2025-12-04T09:45:25.1032246Z Entering 'third_party/protobuf' 2025-12-04T09:45:25.1131857Z Entering 'third_party/protobuf/third_party/benchmark' 2025-12-04T09:45:25.1170696Z Entering 'third_party/protobuf/third_party/googletest' 2025-12-04T09:45:25.1220311Z Entering 'third_party/psimd' 2025-12-04T09:45:25.1259099Z Entering 'third_party/pthreadpool' 2025-12-04T09:45:25.1301870Z Entering 'third_party/pybind11' 2025-12-04T09:45:25.1350137Z Entering 'third_party/python-peachpy' 2025-12-04T09:45:25.1391325Z Entering 'third_party/sleef' 2025-12-04T09:45:25.1435315Z Entering 'third_party/tensorpipe' 2025-12-04T09:45:25.1481423Z Entering 'third_party/tensorpipe/third_party/googletest' 2025-12-04T09:45:25.1522226Z Entering 'third_party/tensorpipe/third_party/libnop' 2025-12-04T09:45:25.1560499Z Entering 'third_party/tensorpipe/third_party/libuv' 2025-12-04T09:45:25.1607750Z Entering 'third_party/tensorpipe/third_party/pybind11' 2025-12-04T09:45:25.1649569Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2025-12-04T09:45:25.1795213Z Prepare all required actions 2025-12-04T09:45:25.1795680Z Getting action download info 2025-12-04T09:45:25.3371467Z ##[group]Run ./.github/actions/setup-linux 2025-12-04T09:45:25.3371693Z env: 2025-12-04T09:45:25.3371851Z GIT_DEFAULT_BRANCH: main 2025-12-04T09:45:25.3372037Z ##[endgroup] 2025-12-04T09:45:25.3404047Z ##[group]Run set -euo pipefail 2025-12-04T09:45:25.3404288Z set -euo pipefail 2025-12-04T09:45:25.3404498Z function get_ec2_metadata() { 2025-12-04T09:45:25.3404771Z  # Pulled from instance metadata endpoint for EC2 2025-12-04T09:45:25.3405213Z  # see https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/instancedata-data-retrieval.html 2025-12-04T09:45:25.3405623Z  category=$1 2025-12-04T09:45:25.3405885Z  # If it is GCP runner (runner name contains gcp), do not run this 2025-12-04T09:45:25.3406195Z  runner_name_str=i-02fefd73b1a8f16c5 2025-12-04T09:45:25.3406481Z  if [[ -f /.inarc ]]; then 2025-12-04T09:45:25.3406744Z  echo "ARC Runner, no info on ec2 metadata" 2025-12-04T09:45:25.3407033Z  elif [[ $runner_name_str == *"gcp"* ]]; then 2025-12-04T09:45:25.3407366Z  echo "Runner is from Google Cloud Platform, No info on ec2 metadata" 2025-12-04T09:45:25.3407851Z  else 2025-12-04T09:45:25.3408455Z  curl -H "X-aws-ec2-metadata-token: $(curl -s -X PUT "http://169.254.169.254/latest/api/token" -H "X-aws-ec2-metadata-token-ttl-seconds: 30")" -fsSL "http://169.254.169.254/latest/meta-data/${category}" 2025-12-04T09:45:25.3409087Z  fi 2025-12-04T09:45:25.3409244Z } 2025-12-04T09:45:25.3409439Z echo "ami-id: $(get_ec2_metadata ami-id)" 2025-12-04T09:45:25.3409734Z echo "instance-id: $(get_ec2_metadata instance-id)" 2025-12-04T09:45:25.3410070Z echo "instance-type: $(get_ec2_metadata instance-type)" 2025-12-04T09:45:25.3410375Z echo "system info $(uname -a)" 2025-12-04T09:45:25.3418614Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-12-04T09:45:25.3418892Z env: 2025-12-04T09:45:25.3419049Z GIT_DEFAULT_BRANCH: main 2025-12-04T09:45:25.3419228Z ##[endgroup] 2025-12-04T09:45:25.3568836Z ami-id: ami-08982f1c5bf93d976 2025-12-04T09:45:25.3691810Z instance-id: i-02fefd73b1a8f16c5 2025-12-04T09:45:25.3799055Z instance-type: g6.4xlarge 2025-12-04T09:45:25.3811405Z system info Linux ip-10-0-57-72.ec2.internal 6.1.150-174.273.amzn2023.x86_64 #1 SMP PREEMPT_DYNAMIC Tue Sep 9 12:21:26 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux 2025-12-04T09:45:25.3831361Z ##[group]Run if [ -f /usr/bin/nvidia-smi ]; then nvidia-smi; fi 2025-12-04T09:45:25.3831713Z if [ -f /usr/bin/nvidia-smi ]; then nvidia-smi; fi 2025-12-04T09:45:25.3839675Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-12-04T09:45:25.3839969Z env: 2025-12-04T09:45:25.3840126Z GIT_DEFAULT_BRANCH: main 2025-12-04T09:45:25.3840318Z ##[endgroup] 2025-12-04T09:45:26.8286400Z Thu Dec 4 09:45:26 2025 2025-12-04T09:45:26.8286815Z +-----------------------------------------------------------------------------------------+ 2025-12-04T09:45:26.8287316Z | NVIDIA-SMI 580.82.07 Driver Version: 580.82.07 CUDA Version: 13.0 | 2025-12-04T09:45:26.8287772Z +-----------------------------------------+------------------------+----------------------+ 2025-12-04T09:45:26.8288272Z | GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC | 2025-12-04T09:45:26.8288803Z | Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. | 2025-12-04T09:45:26.8289203Z | | | MIG M. | 2025-12-04T09:45:26.8289499Z |=========================================+========================+======================| 2025-12-04T09:45:26.8364113Z | 0 NVIDIA L4 Off | 00000000:35:00.0 Off | 0 | 2025-12-04T09:45:26.8364958Z | N/A 35C P0 29W / 72W | 0MiB / 23034MiB | 4% Default | 2025-12-04T09:45:26.8365353Z | | | N/A | 2025-12-04T09:45:26.8365726Z +-----------------------------------------+------------------------+----------------------+ 2025-12-04T09:45:26.8366011Z 2025-12-04T09:45:26.8366168Z +-----------------------------------------------------------------------------------------+ 2025-12-04T09:45:26.8366560Z | Processes: | 2025-12-04T09:45:26.8366992Z | GPU GI CI PID Type Process name GPU Memory | 2025-12-04T09:45:26.8367369Z | ID ID Usage | 2025-12-04T09:45:26.8367665Z |=========================================================================================| 2025-12-04T09:45:26.8368806Z | No running processes found | 2025-12-04T09:45:26.8369247Z +-----------------------------------------------------------------------------------------+ 2025-12-04T09:45:27.1640059Z ##[group]Run echo "IN_CONTAINER_RUNNER=$(if [ -f /.inarc ] || [ -f /.incontainer ]; then echo true ; else echo false; fi)" >> "$GITHUB_OUTPUT" 2025-12-04T09:45:27.1640890Z echo "IN_CONTAINER_RUNNER=$(if [ -f /.inarc ] || [ -f /.incontainer ]; then echo true ; else echo false; fi)" >> "$GITHUB_OUTPUT" 2025-12-04T09:45:27.1651910Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-12-04T09:45:27.1652184Z env: 2025-12-04T09:45:27.1652341Z GIT_DEFAULT_BRANCH: main 2025-12-04T09:45:27.1652534Z ##[endgroup] 2025-12-04T09:45:27.1707902Z ##[group]Run if systemctl is-active --quiet docker; then 2025-12-04T09:45:27.1708232Z if systemctl is-active --quiet docker; then 2025-12-04T09:45:27.1708518Z  echo "Docker daemon is running..."; 2025-12-04T09:45:27.1708765Z else 2025-12-04T09:45:27.1709015Z  echo "Starting docker daemon..." && sudo systemctl start docker; 2025-12-04T09:45:27.1709319Z fi 2025-12-04T09:45:27.1716300Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-12-04T09:45:27.1716578Z env: 2025-12-04T09:45:27.1716738Z GIT_DEFAULT_BRANCH: main 2025-12-04T09:45:27.1716926Z ##[endgroup] 2025-12-04T09:45:27.1812495Z Docker daemon is running... 2025-12-04T09:45:27.1847471Z ##[group]Run nick-fields/retry@v3.0.0 2025-12-04T09:45:27.1847685Z with: 2025-12-04T09:45:27.1847836Z shell: bash 2025-12-04T09:45:27.1848001Z timeout_minutes: 5 2025-12-04T09:45:27.1848177Z max_attempts: 3 2025-12-04T09:45:27.1848346Z retry_wait_seconds: 30 2025-12-04T09:45:27.1849970Z command: AWS_ACCOUNT_ID=$(aws sts get-caller-identity|grep Account|cut -f4 -d\") aws ecr get-login-password --region "$AWS_DEFAULT_REGION" | docker login --username AWS \ --password-stdin "$AWS_ACCOUNT_ID.dkr.ecr.$AWS_DEFAULT_REGION.amazonaws.com" # For LF Runners we need to make sure we also login to Meta's ECR docker registry too. META_AWS_ACCOUNT_ID=308535385114 if [ "$AWS_ACCOUNT_ID" != "$META_AWS_ACCOUNT_ID" ] ; then aws ecr get-login-password --region "$AWS_DEFAULT_REGION" | docker login --username AWS \ --password-stdin "$META_AWS_ACCOUNT_ID.dkr.ecr.$AWS_DEFAULT_REGION.amazonaws.com" fi 2025-12-04T09:45:27.1851646Z polling_interval_seconds: 1 2025-12-04T09:45:27.1851869Z warning_on_retry: true 2025-12-04T09:45:27.1852050Z continue_on_error: false 2025-12-04T09:45:27.1852232Z env: 2025-12-04T09:45:27.1852396Z GIT_DEFAULT_BRANCH: main 2025-12-04T09:45:27.1852579Z AWS_RETRY_MODE: standard 2025-12-04T09:45:27.1852760Z AWS_MAX_ATTEMPTS: 5 2025-12-04T09:45:27.1864053Z AWS_DEFAULT_REGION: us-east-1 2025-12-04T09:45:27.1864276Z ##[endgroup] 2025-12-04T09:45:28.2224685Z WARNING! Your password will be stored unencrypted in /home/ec2-user/.docker/config.json. 2025-12-04T09:45:28.2225812Z Configure a credential helper to remove this warning. See 2025-12-04T09:45:28.2226703Z https://docs.docker.com/engine/reference/commandline/login/#credentials-store 2025-12-04T09:45:28.2227290Z 2025-12-04T09:45:28.2227406Z Login Succeeded 2025-12-04T09:45:28.2633091Z Command completed after 1 attempt(s). 2025-12-04T09:45:28.2693347Z ##[group]Run env | grep '^GITHUB' >> "/tmp/github_env_${GITHUB_RUN_ID}" 2025-12-04T09:45:28.2693745Z env | grep '^GITHUB' >> "/tmp/github_env_${GITHUB_RUN_ID}" 2025-12-04T09:45:28.2694067Z env | grep '^CI' >> "/tmp/github_env_${GITHUB_RUN_ID}" 2025-12-04T09:45:28.2703334Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-12-04T09:45:28.2703722Z env: 2025-12-04T09:45:28.2703888Z GIT_DEFAULT_BRANCH: main 2025-12-04T09:45:28.2704080Z ##[endgroup] 2025-12-04T09:45:28.2793893Z ##[group]Run # ignore expansion of "docker ps -q" since it could be empty 2025-12-04T09:45:28.2794301Z # ignore expansion of "docker ps -q" since it could be empty 2025-12-04T09:45:28.2794616Z # shellcheck disable=SC2046 2025-12-04T09:45:28.2794865Z docker stop $(docker ps -q) || true 2025-12-04T09:45:28.2795111Z # Prune all of the docker images 2025-12-04T09:45:28.2795337Z docker system prune -af 2025-12-04T09:45:28.2802537Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-12-04T09:45:28.2802982Z env: 2025-12-04T09:45:28.2803141Z GIT_DEFAULT_BRANCH: main 2025-12-04T09:45:28.2803329Z ##[endgroup] 2025-12-04T09:45:28.3084533Z "docker stop" requires at least 1 argument. 2025-12-04T09:45:28.3084931Z See 'docker stop --help'. 2025-12-04T09:45:28.3085093Z 2025-12-04T09:45:28.3085249Z Usage: docker stop [OPTIONS] CONTAINER [CONTAINER...] 2025-12-04T09:45:28.3085490Z 2025-12-04T09:45:28.3085589Z Stop one or more running containers 2025-12-04T09:45:28.3423095Z Total reclaimed space: 0B 2025-12-04T09:45:28.3568526Z ##[group]Run pytorch/test-infra/.github/actions/calculate-docker-image@main 2025-12-04T09:45:28.3568902Z with: 2025-12-04T09:45:28.3569484Z docker-image-name: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/ci-image:pytorch-linux-jammy-cuda12.8-cudnn9-py3-gcc11-f0cd68561080d537ef3d3d6f81b25a6416ad600a 2025-12-04T09:45:28.3570135Z use-custom-docker-registry: true 2025-12-04T09:45:28.3570383Z docker-build-dir: .ci/docker 2025-12-04T09:45:28.3570619Z docker-build-script: ./build.sh 2025-12-04T09:45:28.3570830Z working-directory: . 2025-12-04T09:45:28.3571090Z docker-registry: 308535385114.dkr.ecr.us-east-1.amazonaws.com 2025-12-04T09:45:28.3571393Z force-push: false 2025-12-04T09:45:28.3571558Z env: 2025-12-04T09:45:28.3571713Z GIT_DEFAULT_BRANCH: main 2025-12-04T09:45:28.3571899Z ##[endgroup] 2025-12-04T09:45:28.3588313Z ##[group]Run set -ex 2025-12-04T09:45:28.3588523Z set -ex 2025-12-04T09:45:28.3588681Z  2025-12-04T09:45:28.3588996Z # If the docker build directory or the build script doesn't exist, the action will 2025-12-04T09:45:28.3589466Z # gracefully return the docker image name as it is. Pulling docker image in Linux 2025-12-04T09:45:28.3589877Z # job could then download the pre-built image as usual 2025-12-04T09:45:28.3590371Z if [[ -d "${DOCKER_BUILD_DIR}" ]] && [[ -f "${DOCKER_BUILD_DIR}/${DOCKER_BUILD_SCRIPT}" ]] && [[ "${USE_CUSTOM_DOCKER_REGISTRY}" == "true" ]]; then 2025-12-04T09:45:28.3590830Z  echo "skip=false" >> "${GITHUB_OUTPUT}" 2025-12-04T09:45:28.3591076Z else 2025-12-04T09:45:28.3591270Z  echo "skip=true" >> "${GITHUB_OUTPUT}" 2025-12-04T09:45:28.3591584Z  echo "docker-image=${DOCKER_IMAGE_NAME}" >> "${GITHUB_OUTPUT}" 2025-12-04T09:45:28.3591870Z  2025-12-04T09:45:28.3592268Z  echo "Not using custom ECR registry. Either it was not requested or there is no Docker build script in the ${REPO_NAME} repo..." 2025-12-04T09:45:28.3592720Z  exit 0 2025-12-04T09:45:28.3592878Z fi 2025-12-04T09:45:28.3593034Z  2025-12-04T09:45:28.3593281Z if [[ "${DOCKER_IMAGE_NAME}" == *"${DOCKER_REGISTRY}/${REPO_NAME}"* ]]; then 2025-12-04T09:45:28.3593720Z  # The docker image name already includes the ECR prefix and tag, so we can just 2025-12-04T09:45:28.3594095Z  # use it as it is, but first let's extract the tag 2025-12-04T09:45:28.3594443Z  DOCKER_TAG=$(echo "${DOCKER_IMAGE_NAME}" | awk -F '[:,]' '{print $2}') 2025-12-04T09:45:28.3594804Z  echo "docker-tag=${DOCKER_TAG}" >> "${GITHUB_OUTPUT}" 2025-12-04T09:45:28.3595137Z  echo "docker-image=${DOCKER_IMAGE_NAME}" >> "${GITHUB_OUTPUT}" 2025-12-04T09:45:28.3595420Z else 2025-12-04T09:45:28.3595610Z  if [[ "${DOCKER_IMAGE_NAME}" == *:* ]]; then 2025-12-04T09:45:28.3595881Z  CUSTOM_TAG_PREFIX=${DOCKER_IMAGE_NAME#*:} 2025-12-04T09:45:28.3596151Z  DOCKER_IMAGE_NAME=${DOCKER_IMAGE_NAME%%:*} 2025-12-04T09:45:28.3596381Z  fi 2025-12-04T09:45:28.3596703Z  DOCKER_TAG=${CUSTOM_TAG_PREFIX:+${CUSTOM_TAG_PREFIX}-}$(git rev-parse HEAD:"${DOCKER_BUILD_DIR}") 2025-12-04T09:45:28.3597118Z  echo "docker-tag=${DOCKER_TAG}" >> "${GITHUB_OUTPUT}" 2025-12-04T09:45:28.3597563Z  echo "docker-image=${DOCKER_REGISTRY}/${REPO_NAME}/${DOCKER_IMAGE_NAME}:${DOCKER_TAG}" >> "${GITHUB_OUTPUT}" 2025-12-04T09:45:28.3598227Z  echo "custom-tag-prefix=${CUSTOM_TAG_PREFIX}" >> "${GITHUB_OUTPUT}" 2025-12-04T09:45:28.3598527Z fi 2025-12-04T09:45:28.3606330Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-12-04T09:45:28.3606610Z env: 2025-12-04T09:45:28.3606770Z GIT_DEFAULT_BRANCH: main 2025-12-04T09:45:28.3606959Z REPO_NAME: pytorch 2025-12-04T09:45:28.3607718Z DOCKER_IMAGE_NAME: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/ci-image:pytorch-linux-jammy-cuda12.8-cudnn9-py3-gcc11-f0cd68561080d537ef3d3d6f81b25a6416ad600a 2025-12-04T09:45:28.3608338Z DOCKER_BUILD_DIR: .ci/docker 2025-12-04T09:45:28.3608546Z DOCKER_BUILD_SCRIPT: ./build.sh 2025-12-04T09:45:28.3608814Z DOCKER_REGISTRY: 308535385114.dkr.ecr.us-east-1.amazonaws.com 2025-12-04T09:45:28.3609110Z USE_CUSTOM_DOCKER_REGISTRY: true 2025-12-04T09:45:28.3609333Z CUSTOM_TAG_PREFIX: 2025-12-04T09:45:28.3609504Z ##[endgroup] 2025-12-04T09:45:28.3637636Z + [[ -d .ci/docker ]] 2025-12-04T09:45:28.3637940Z + [[ -f .ci/docker/./build.sh ]] 2025-12-04T09:45:28.3638210Z + [[ true == \t\r\u\e ]] 2025-12-04T09:45:28.3638446Z + echo skip=false 2025-12-04T09:45:28.3639317Z + [[ 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/ci-image:pytorch-linux-jammy-cuda12.8-cudnn9-py3-gcc11-f0cd68561080d537ef3d3d6f81b25a6416ad600a == *\3\0\8\5\3\5\3\8\5\1\1\4\.\d\k\r\.\e\c\r\.\u\s\-\e\a\s\t\-\1\.\a\m\a\z\o\n\a\w\s\.\c\o\m\/\p\y\t\o\r\c\h* ]] 2025-12-04T09:45:28.3645224Z ++ echo 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/ci-image:pytorch-linux-jammy-cuda12.8-cudnn9-py3-gcc11-f0cd68561080d537ef3d3d6f81b25a6416ad600a 2025-12-04T09:45:28.3645836Z ++ awk -F '[:,]' '{print $2}' 2025-12-04T09:45:28.3669988Z + DOCKER_TAG=pytorch-linux-jammy-cuda12.8-cudnn9-py3-gcc11-f0cd68561080d537ef3d3d6f81b25a6416ad600a 2025-12-04T09:45:28.3670932Z + echo docker-tag=pytorch-linux-jammy-cuda12.8-cudnn9-py3-gcc11-f0cd68561080d537ef3d3d6f81b25a6416ad600a 2025-12-04T09:45:28.3671999Z + echo docker-image=308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/ci-image:pytorch-linux-jammy-cuda12.8-cudnn9-py3-gcc11-f0cd68561080d537ef3d3d6f81b25a6416ad600a 2025-12-04T09:45:28.3693414Z ##[group]Run set +e 2025-12-04T09:45:28.3693631Z set +e 2025-12-04T09:45:28.3693797Z set -x 2025-12-04T09:45:28.3693955Z  2025-12-04T09:45:28.3694117Z login() { 2025-12-04T09:45:28.3694462Z  aws ecr get-login-password --region us-east-1 | docker login -u AWS --password-stdin "$1" 2025-12-04T09:45:28.3694828Z } 2025-12-04T09:45:28.3694988Z  2025-12-04T09:45:28.3695129Z retry () { 2025-12-04T09:45:28.3695325Z  $* || (sleep 1 && $*) || (sleep 2 && $*) 2025-12-04T09:45:28.3695558Z } 2025-12-04T09:45:28.3695699Z  2025-12-04T09:45:28.3695864Z retry login "${DOCKER_REGISTRY}" 2025-12-04T09:45:28.3696081Z  2025-12-04T09:45:28.3696237Z START_TIME=$(date +%s) 2025-12-04T09:45:28.3696443Z # Wait up to 120 minutes 2025-12-04T09:45:28.3696713Z while [[ $(( $(date +%s) - 7200 )) -lt $START_TIME ]]; do 2025-12-04T09:45:28.3697068Z  # Check if image already exists, if it does then skip building it 2025-12-04T09:45:28.3697405Z  if docker manifest inspect "${DOCKER_IMAGE}"; then 2025-12-04T09:45:28.3697659Z  exit 0 2025-12-04T09:45:28.3697825Z  fi 2025-12-04T09:45:28.3697969Z  2025-12-04T09:45:28.3698239Z  # NB: This flag is used by Docker build workflow to push the image to ECR, so we can 2025-12-04T09:45:28.3698718Z  # use this to differentiate between the Docker build and regular build jobs. For the 2025-12-04T09:45:28.3699185Z  # latter, it will wait for the Docker images to become available before continuing 2025-12-04T09:45:28.3699547Z  if [ "${DOCKER_PUSH:-false}" == "true" ]; then 2025-12-04T09:45:28.3699841Z  # It's a Docker build job, let's build the image 2025-12-04T09:45:28.3700258Z  break 2025-12-04T09:45:28.3700430Z  else 2025-12-04T09:45:28.3700671Z  # It's a regular build job, wait for the image to become available 2025-12-04T09:45:28.3700967Z  sleep 300 2025-12-04T09:45:28.3701146Z  fi 2025-12-04T09:45:28.3701292Z done 2025-12-04T09:45:28.3701456Z  2025-12-04T09:45:28.3701698Z # NB: This part requires a full checkout. Otherwise, the merge base will 2025-12-04T09:45:28.3702231Z # be empty. The default action would be to continue rebuild the image 2025-12-04T09:45:28.3702610Z if [[ "$BASE_REVISION" = "$(git rev-parse HEAD)" ]]; then 2025-12-04T09:45:28.3702932Z  # if we're on the base branch then use the parent commit 2025-12-04T09:45:28.3703211Z  MERGE_BASE=$(git rev-parse HEAD~) 2025-12-04T09:45:28.3703432Z else 2025-12-04T09:45:28.3703777Z  # otherwise we're on a PR, so use the most recent base commit 2025-12-04T09:45:28.3704108Z  MERGE_BASE=$(git merge-base HEAD "$BASE_REVISION") 2025-12-04T09:45:28.3704361Z fi 2025-12-04T09:45:28.3704510Z  2025-12-04T09:45:28.3704673Z if [[ -z "${MERGE_BASE}" ]]; then 2025-12-04T09:45:28.3704922Z  echo "rebuild=true" >> "${GITHUB_OUTPUT}" 2025-12-04T09:45:28.3705149Z  2025-12-04T09:45:28.3705476Z  echo "Finding merge base only works with full checkout, please set fetch-depth to 0, continuing ..." 2025-12-04T09:45:28.3705854Z  exit 0 2025-12-04T09:45:28.3706010Z fi 2025-12-04T09:45:28.3706158Z  2025-12-04T09:45:28.3706367Z if ! git rev-parse "${MERGE_BASE}:${DOCKER_BUILD_DIR}"; then 2025-12-04T09:45:28.3706846Z  echo "Directory '${DOCKER_BUILD_DIR}' not found in commit $MERGE_BASE, you should rebase onto a more recent commit" 2025-12-04T09:45:28.3707253Z  exit 1 2025-12-04T09:45:28.3707409Z fi 2025-12-04T09:45:28.3707552Z  2025-12-04T09:45:28.3707817Z PREVIOUS_DOCKER_TAG=$(git rev-parse "${MERGE_BASE}:${DOCKER_BUILD_DIR}") 2025-12-04T09:45:28.3708272Z # If no image exists but the hash is the same as the previous hash then we should error out here 2025-12-04T09:45:28.3708678Z if [[ "${PREVIOUS_DOCKER_TAG}" == "${DOCKER_TAG}" ]]; then 2025-12-04T09:45:28.3709144Z  echo "WARNING: Something has gone wrong and the previous image isn't available for the merge-base of your branch" 2025-12-04T09:45:28.3709678Z  echo " Will re-build docker image to store in local cache, TTS may be longer" 2025-12-04T09:45:28.3709991Z fi 2025-12-04T09:45:28.3710132Z  2025-12-04T09:45:28.3710316Z echo "rebuild=true" >> "${GITHUB_OUTPUT}" 2025-12-04T09:45:28.3717328Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-12-04T09:45:28.3717595Z env: 2025-12-04T09:45:28.3717757Z GIT_DEFAULT_BRANCH: main 2025-12-04T09:45:28.3717958Z DOCKER_BUILD_DIR: .ci/docker 2025-12-04T09:45:28.3718214Z BASE_REVISION: ffd9b0fb4355e97af82fc42cf185c3ffa0fc0a32 2025-12-04T09:45:28.3718856Z DOCKER_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/ci-image:pytorch-linux-jammy-cuda12.8-cudnn9-py3-gcc11-f0cd68561080d537ef3d3d6f81b25a6416ad600a 2025-12-04T09:45:28.3719644Z DOCKER_TAG: pytorch-linux-jammy-cuda12.8-cudnn9-py3-gcc11-f0cd68561080d537ef3d3d6f81b25a6416ad600a 2025-12-04T09:45:28.3720133Z DOCKER_REGISTRY: 308535385114.dkr.ecr.us-east-1.amazonaws.com 2025-12-04T09:45:28.3720421Z DOCKER_PUSH: 2025-12-04T09:45:28.3720591Z ##[endgroup] 2025-12-04T09:45:28.3746439Z + retry login 308535385114.dkr.ecr.us-east-1.amazonaws.com 2025-12-04T09:45:28.3746847Z + login 308535385114.dkr.ecr.us-east-1.amazonaws.com 2025-12-04T09:45:28.3749385Z + aws ecr get-login-password --region us-east-1 2025-12-04T09:45:28.3750377Z + docker login -u AWS --password-stdin 308535385114.dkr.ecr.us-east-1.amazonaws.com 2025-12-04T09:45:28.8476859Z WARNING! Your password will be stored unencrypted in /home/ec2-user/.docker/config.json. 2025-12-04T09:45:28.8477787Z Configure a credential helper to remove this warning. See 2025-12-04T09:45:28.8478564Z https://docs.docker.com/engine/reference/commandline/login/#credentials-store 2025-12-04T09:45:28.8479129Z 2025-12-04T09:45:28.8479367Z Login Succeeded 2025-12-04T09:45:28.8498314Z ++ date +%s 2025-12-04T09:45:28.8510591Z + START_TIME=1764841528 2025-12-04T09:45:28.8514151Z ++ date +%s 2025-12-04T09:45:28.8526090Z + [[ 1764834328 -lt 1764841528 ]] 2025-12-04T09:45:28.8526925Z + docker manifest inspect 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/ci-image:pytorch-linux-jammy-cuda12.8-cudnn9-py3-gcc11-f0cd68561080d537ef3d3d6f81b25a6416ad600a 2025-12-04T09:45:29.0734177Z { 2025-12-04T09:45:29.0734567Z "schemaVersion": 2, 2025-12-04T09:45:29.0734976Z "mediaType": "application/vnd.docker.distribution.manifest.v2+json", 2025-12-04T09:45:29.0735363Z "config": { 2025-12-04T09:45:29.0735667Z "mediaType": "application/vnd.docker.container.image.v1+json", 2025-12-04T09:45:29.0736046Z "size": 34864, 2025-12-04T09:45:29.0736416Z "digest": "sha256:add7313791033822205cdb3cf32096534b2cfaa4855bd48119b59000bfe00301" 2025-12-04T09:45:29.0736823Z }, 2025-12-04T09:45:29.0736990Z "layers": [ 2025-12-04T09:45:29.0737168Z { 2025-12-04T09:45:29.0737447Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-12-04T09:45:29.0737808Z "size": 30447951, 2025-12-04T09:45:29.0738183Z "digest": "sha256:63e5bc7682b85ae57a1221210f64d62e7a90b0a30f19af4ca734b8242ae49d63" 2025-12-04T09:45:29.0738576Z }, 2025-12-04T09:45:29.0738738Z { 2025-12-04T09:45:29.0739032Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-12-04T09:45:29.0739387Z "size": 1554, 2025-12-04T09:45:29.0739743Z "digest": "sha256:0678d56345c994444b77bb70b1177189d23e794748b1d75ffc45d227c7dea94a" 2025-12-04T09:45:29.0740137Z }, 2025-12-04T09:45:29.0740288Z { 2025-12-04T09:45:29.0740511Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-12-04T09:45:29.0740817Z "size": 313275661, 2025-12-04T09:45:29.0741119Z "digest": "sha256:45f5c9ddfce78349dff3d5edfbaa0310ae17311f66abdcd7e00fa21b500e801c" 2025-12-04T09:45:29.0741445Z }, 2025-12-04T09:45:29.0741577Z { 2025-12-04T09:45:29.0741798Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-12-04T09:45:29.0742072Z "size": 787, 2025-12-04T09:45:29.0742356Z "digest": "sha256:086b1df51ac1162d9c45698e9dfaf91c6c222c8bd9ab01797ac8f9344bc8044f" 2025-12-04T09:45:29.0742680Z }, 2025-12-04T09:45:29.0742806Z { 2025-12-04T09:45:29.0743027Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-12-04T09:45:29.0743304Z "size": 106, 2025-12-04T09:45:29.0743700Z "digest": "sha256:fe8a7b64bf98352f89057bcba66beef2fb44cc05fbd3606abccd8e86cf476234" 2025-12-04T09:45:29.0744029Z }, 2025-12-04T09:45:29.0744159Z { 2025-12-04T09:45:29.0744374Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-12-04T09:45:29.0744650Z "size": 703, 2025-12-04T09:45:29.0744913Z "digest": "sha256:7680723e9a578033dd106b45784c639f06cc8adb1f5239ec513d9de01087c1af" 2025-12-04T09:45:29.0745218Z }, 2025-12-04T09:45:29.0745336Z { 2025-12-04T09:45:29.0745550Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-12-04T09:45:29.0745826Z "size": 1216, 2025-12-04T09:45:29.0746095Z "digest": "sha256:9c5027aeeb4e3101f48c1d2e400c387110e1009e42497ee801f1b4b7f7edb5c0" 2025-12-04T09:45:29.0746477Z }, 2025-12-04T09:45:29.0746610Z { 2025-12-04T09:45:29.0746826Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-12-04T09:45:29.0747113Z "size": 483, 2025-12-04T09:45:29.0747390Z "digest": "sha256:9a56521103600bd37a1e7c1191b5136c2d738c092f8a6701499f7068a32c2628" 2025-12-04T09:45:29.0747703Z }, 2025-12-04T09:45:29.0747827Z { 2025-12-04T09:45:29.0748050Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-12-04T09:45:29.0748333Z "size": 110361875, 2025-12-04T09:45:29.0748883Z "digest": "sha256:375c4427e9141269458333b1463fdb219e736fd6231ec1c56c625c48437ace77" 2025-12-04T09:45:29.0749197Z }, 2025-12-04T09:45:29.0749326Z { 2025-12-04T09:45:29.0749538Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-12-04T09:45:29.0749818Z "size": 4961, 2025-12-04T09:45:29.0750112Z "digest": "sha256:a86faaa7dbdd70e678e5ea20072637ee42618921ca8f80ca089f789325d4b0c2" 2025-12-04T09:45:29.0750423Z }, 2025-12-04T09:45:29.0750553Z { 2025-12-04T09:45:29.0750907Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-12-04T09:45:29.0751211Z "size": 1755, 2025-12-04T09:45:29.0751498Z "digest": "sha256:fb7848686804957915d98f8655ef6da0fe4c521b50a82aefdebf475983505a15" 2025-12-04T09:45:29.0751821Z }, 2025-12-04T09:45:29.0751951Z { 2025-12-04T09:45:29.0752168Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-12-04T09:45:29.0752454Z "size": 724, 2025-12-04T09:45:29.0752728Z "digest": "sha256:3541df015cdb7e8925273399d28e56c31b3c9196f00439ac2925537b173b1f84" 2025-12-04T09:45:29.0753037Z }, 2025-12-04T09:45:29.0753167Z { 2025-12-04T09:45:29.0753388Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-12-04T09:45:29.0753716Z "size": 543, 2025-12-04T09:45:29.0753995Z "digest": "sha256:79dc80f426b29d4ae9157b967050b03e66aa0c4b1295b944a1dd70106be87066" 2025-12-04T09:45:29.0754318Z }, 2025-12-04T09:45:29.0754442Z { 2025-12-04T09:45:29.0754659Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-12-04T09:45:29.0754945Z "size": 3185190117, 2025-12-04T09:45:29.0755245Z "digest": "sha256:a13fcc1b90bb9c251ebe7ef2a03c4cb3afa1c8bdafe84f5f85136773059a3735" 2025-12-04T09:45:29.0755571Z }, 2025-12-04T09:45:29.0755704Z { 2025-12-04T09:45:29.0755932Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-12-04T09:45:29.0756203Z "size": 32, 2025-12-04T09:45:29.0756478Z "digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1" 2025-12-04T09:45:29.0756798Z }, 2025-12-04T09:45:29.0756918Z { 2025-12-04T09:45:29.0757132Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-12-04T09:45:29.0757408Z "size": 396, 2025-12-04T09:45:29.0757669Z "digest": "sha256:549db4d6c618ecd9534658a233e3c90508f82d8735f965c2786b2eaa078869e5" 2025-12-04T09:45:29.0758006Z }, 2025-12-04T09:45:29.0758156Z { 2025-12-04T09:45:29.0758377Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-12-04T09:45:29.0758665Z "size": 236860, 2025-12-04T09:45:29.0758953Z "digest": "sha256:5c63528cb580001e65104f4cb0809bf0673a00f989a7db42fd6d86aa1ec27cee" 2025-12-04T09:45:29.0759269Z }, 2025-12-04T09:45:29.0759410Z { 2025-12-04T09:45:29.0759634Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-12-04T09:45:29.0759912Z "size": 231, 2025-12-04T09:45:29.0760206Z "digest": "sha256:75bd83b989a44e4d4119a3f972891025eb0e9ce95cfbe4a0ca5cdbe7130028d6" 2025-12-04T09:45:29.0760536Z }, 2025-12-04T09:45:29.0760660Z { 2025-12-04T09:45:29.0760878Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-12-04T09:45:29.0761158Z "size": 3043497, 2025-12-04T09:45:29.0761436Z "digest": "sha256:de6e78970f517178cb91f36cd02bd9ca7b72a08fb82a0f9007516026f258c035" 2025-12-04T09:45:29.0761744Z }, 2025-12-04T09:45:29.0761871Z { 2025-12-04T09:45:29.0762082Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-12-04T09:45:29.0762351Z "size": 1472, 2025-12-04T09:45:29.0762635Z "digest": "sha256:e13ed7c7e4736e81dc21af755b3363eb26e4d3b2f1ca988dfe65effa47d8fa42" 2025-12-04T09:45:29.0762996Z }, 2025-12-04T09:45:29.0763182Z { 2025-12-04T09:45:29.0763672Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-12-04T09:45:29.0764008Z "size": 481, 2025-12-04T09:45:29.0764369Z "digest": "sha256:6e2949bcb74152577a0f20c38bcb6dd80f5e68427e3e531a80e08c9ecc73a979" 2025-12-04T09:45:29.0775809Z }, 2025-12-04T09:45:29.0776111Z { 2025-12-04T09:45:29.0776364Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-12-04T09:45:29.0776673Z "size": 202, 2025-12-04T09:45:29.0776973Z "digest": "sha256:14d69d9aaec70287efd2fd35c4f93e43a29a4098458cc9fca1c93f02ad7356cb" 2025-12-04T09:45:29.0777304Z }, 2025-12-04T09:45:29.0777440Z { 2025-12-04T09:45:29.0777672Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-12-04T09:45:29.0778321Z "size": 607, 2025-12-04T09:45:29.0778803Z "digest": "sha256:5c02769dd8e5bba2f7f5fd84bde9595fcb3bdbffcae497503fa846f9b5e78bf5" 2025-12-04T09:45:29.0779149Z }, 2025-12-04T09:45:29.0779287Z { 2025-12-04T09:45:29.0779525Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-12-04T09:45:29.0779820Z "size": 7889619584, 2025-12-04T09:45:29.0780119Z "digest": "sha256:35041ce524ac4afec40ecd73b1393c830614f1f79d43a6439767a6c7d5b7027b" 2025-12-04T09:45:29.0780434Z }, 2025-12-04T09:45:29.0780568Z { 2025-12-04T09:45:29.0780801Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-12-04T09:45:29.0781086Z "size": 830, 2025-12-04T09:45:29.0781376Z "digest": "sha256:2fa92dc5885e080e049ceb4139288b6c0e39fab34256945708b08ea55a1f7a0b" 2025-12-04T09:45:29.0781697Z }, 2025-12-04T09:45:29.0781821Z { 2025-12-04T09:45:29.0782060Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-12-04T09:45:29.0782347Z "size": 33451739, 2025-12-04T09:45:29.0782642Z "digest": "sha256:2b85eafbd92a0e70a0a70154ad8bf4584095e576d95873368f30373f5966714a" 2025-12-04T09:45:29.0782956Z }, 2025-12-04T09:45:29.0783086Z { 2025-12-04T09:45:29.0783307Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-12-04T09:45:29.0783642Z "size": 104, 2025-12-04T09:45:29.0783929Z "digest": "sha256:ff755a4ddad7880f23c6b767d432d6f1eafdb62b3ea18f8a98e22c441c099fcb" 2025-12-04T09:45:29.0784256Z }, 2025-12-04T09:45:29.0784385Z { 2025-12-04T09:45:29.0784608Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-12-04T09:45:29.0784892Z "size": 1496, 2025-12-04T09:45:29.0785160Z "digest": "sha256:09eb41bdf42d8605b57b2363348154140904dec914b34a67298b82122bfce2b3" 2025-12-04T09:45:29.0785469Z }, 2025-12-04T09:45:29.0785598Z { 2025-12-04T09:45:29.0785806Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-12-04T09:45:29.0786096Z "size": 458787828, 2025-12-04T09:45:29.0786382Z "digest": "sha256:11ede4d59e935e62f41b33220fe871794ab5e57ce724173b713368977683bcf6" 2025-12-04T09:45:29.0786704Z }, 2025-12-04T09:45:29.0786831Z { 2025-12-04T09:45:29.0787053Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-12-04T09:45:29.0787335Z "size": 164, 2025-12-04T09:45:29.0787602Z "digest": "sha256:1283cd8f801a142172f3ab76fd472df8583223d9437de3e4d18d8cf98ea3fa98" 2025-12-04T09:45:29.0787907Z }, 2025-12-04T09:45:29.0788032Z { 2025-12-04T09:45:29.0788240Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-12-04T09:45:29.0788524Z "size": 346, 2025-12-04T09:45:29.0788797Z "digest": "sha256:024fa855425fa524ad4500660cf61d53be62b99556d31b8b280d14caba434a35" 2025-12-04T09:45:29.0789103Z }, 2025-12-04T09:45:29.0789245Z { 2025-12-04T09:45:29.0789465Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-12-04T09:45:29.0789741Z "size": 32, 2025-12-04T09:45:29.0790014Z "digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1" 2025-12-04T09:45:29.0790334Z }, 2025-12-04T09:45:29.0790465Z { 2025-12-04T09:45:29.0790676Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-12-04T09:45:29.0790955Z "size": 106, 2025-12-04T09:45:29.0791229Z "digest": "sha256:303e6747a62efecf5efa1f97d0e66b40a3b39da8d79a51f75b89f4c92ae7ec52" 2025-12-04T09:45:29.0791541Z }, 2025-12-04T09:45:29.0791673Z { 2025-12-04T09:45:29.0791889Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-12-04T09:45:29.0792162Z "size": 424, 2025-12-04T09:45:29.0792579Z "digest": "sha256:3017cdf4838bcc9a33daebc07487f8ae1f6bd6e7ce8322c14f5480e8db9ef90e" 2025-12-04T09:45:29.0792901Z }, 2025-12-04T09:45:29.0793035Z { 2025-12-04T09:45:29.0793254Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-12-04T09:45:29.0793533Z "size": 19309374, 2025-12-04T09:45:29.0793828Z "digest": "sha256:6b6cd1c358e886dc6ed7fd46ac4bcc1a0a73b7b1301739ea1953478ee5d83f50" 2025-12-04T09:45:29.0794143Z }, 2025-12-04T09:45:29.0794273Z { 2025-12-04T09:45:29.0794571Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-12-04T09:45:29.0794858Z "size": 108, 2025-12-04T09:45:29.0795134Z "digest": "sha256:b2dd045011241d1cf8889e2a7369d9fe4844dfe15529b520ccd6a59bd3c1532e" 2025-12-04T09:45:29.0795455Z }, 2025-12-04T09:45:29.0795576Z { 2025-12-04T09:45:29.0795796Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-12-04T09:45:29.0796078Z "size": 827, 2025-12-04T09:45:29.0796346Z "digest": "sha256:55adc51fe5897031d4cf2f2b8fd162213f6e46a52848630c616606271b97952e" 2025-12-04T09:45:29.0796664Z }, 2025-12-04T09:45:29.0796793Z { 2025-12-04T09:45:29.0797000Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-12-04T09:45:29.0797273Z "size": 724, 2025-12-04T09:45:29.0797547Z "digest": "sha256:3541df015cdb7e8925273399d28e56c31b3c9196f00439ac2925537b173b1f84" 2025-12-04T09:45:29.0797855Z }, 2025-12-04T09:45:29.0797975Z { 2025-12-04T09:45:29.0798190Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-12-04T09:45:29.0798473Z "size": 149, 2025-12-04T09:45:29.0798747Z "digest": "sha256:a43ca0e4b837964b12b7469194cfe939c26de027298040028975324dce25938a" 2025-12-04T09:45:29.0799063Z }, 2025-12-04T09:45:29.0799195Z { 2025-12-04T09:45:29.0799414Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-12-04T09:45:29.0799697Z "size": 138, 2025-12-04T09:45:29.0799982Z "digest": "sha256:b7212f17fd1404837fcfdd086dd0e2667931e4db377d45d8d89a44390c84e11d" 2025-12-04T09:45:29.0800297Z }, 2025-12-04T09:45:29.0800427Z { 2025-12-04T09:45:29.0800646Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-12-04T09:45:29.0800926Z "size": 141, 2025-12-04T09:45:29.0801197Z "digest": "sha256:083e42cac090e6486c35f392b64ee54448f5e4aa947003aeb3e1f92c8ea5c099" 2025-12-04T09:45:29.0801513Z }, 2025-12-04T09:45:29.0801650Z { 2025-12-04T09:45:29.0801867Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-12-04T09:45:29.0802148Z "size": 32, 2025-12-04T09:45:29.0802424Z "digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1" 2025-12-04T09:45:29.0802738Z }, 2025-12-04T09:45:29.0802870Z { 2025-12-04T09:45:29.0803088Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-12-04T09:45:29.0803359Z "size": 223, 2025-12-04T09:45:29.0803633Z "digest": "sha256:0a00b784a4aac341795729b254f7edd09e811b7f51d0c58e0e6bfeeee6940503" 2025-12-04T09:45:29.0803958Z }, 2025-12-04T09:45:29.0804080Z { 2025-12-04T09:45:29.0804308Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-12-04T09:45:29.0804586Z "size": 255, 2025-12-04T09:45:29.0804857Z "digest": "sha256:c6173c779f7ba143a21214ea5f032b141863a37ceb4c0ac01d3248c216ce5241" 2025-12-04T09:45:29.0805169Z }, 2025-12-04T09:45:29.0805298Z { 2025-12-04T09:45:29.0805515Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-12-04T09:45:29.0805793Z "size": 145520672, 2025-12-04T09:45:29.0806086Z "digest": "sha256:ed3d1e3387b924585c332bf1bc252fa159cd0d25256a874043ff0141b1ab5ff7" 2025-12-04T09:45:29.0806403Z }, 2025-12-04T09:45:29.0806531Z { 2025-12-04T09:45:29.0806749Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-12-04T09:45:29.0807038Z "size": 106, 2025-12-04T09:45:29.0807301Z "digest": "sha256:b29343478586aeee19d2a622661716f6f1591280c890f49b727a8da13a610784" 2025-12-04T09:45:29.0807614Z }, 2025-12-04T09:45:29.0807830Z { 2025-12-04T09:45:29.0808040Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-12-04T09:45:29.0808323Z "size": 312293530, 2025-12-04T09:45:29.0808621Z "digest": "sha256:c6f0520487fb506bc4601fd84d5f28d8a76b203e004731e4b2067c2ab1a14e0b" 2025-12-04T09:45:29.0808935Z }, 2025-12-04T09:45:29.0809057Z { 2025-12-04T09:45:29.0809276Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-12-04T09:45:29.0809555Z "size": 3058011133, 2025-12-04T09:45:29.0809921Z "digest": "sha256:148171691cd4c4d20310d490d4b4dd903490d04ea07fb8f7e668a28768683e9a" 2025-12-04T09:45:29.0810235Z }, 2025-12-04T09:45:29.0810359Z { 2025-12-04T09:45:29.0810566Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-12-04T09:45:29.0810840Z "size": 129, 2025-12-04T09:45:29.0811115Z "digest": "sha256:2c666d30ed77fff9ff1167d41cd645dad98280fcbe941f5bc3828c7ae66b1287" 2025-12-04T09:45:29.0811430Z }, 2025-12-04T09:45:29.0811557Z { 2025-12-04T09:45:29.0811778Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-12-04T09:45:29.0812065Z "size": 880, 2025-12-04T09:45:29.0812334Z "digest": "sha256:5d8d3a0a98e012c5068e0f3bae5a03e3148ecf2d063634eee4c9241a1e3fdfb5" 2025-12-04T09:45:29.0812647Z }, 2025-12-04T09:45:29.0812773Z { 2025-12-04T09:45:29.0812983Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-12-04T09:45:29.0813258Z "size": 724, 2025-12-04T09:45:29.0813524Z "digest": "sha256:3541df015cdb7e8925273399d28e56c31b3c9196f00439ac2925537b173b1f84" 2025-12-04T09:45:29.0813827Z }, 2025-12-04T09:45:29.0813955Z { 2025-12-04T09:45:29.0814167Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-12-04T09:45:29.0814438Z "size": 139, 2025-12-04T09:45:29.0814704Z "digest": "sha256:b06bafce9e817295d8127207747c80aa18e04392ff0875844fc30a1e794a8a0c" 2025-12-04T09:45:29.0815014Z }, 2025-12-04T09:45:29.0815136Z { 2025-12-04T09:45:29.0815348Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-12-04T09:45:29.0815639Z "size": 32, 2025-12-04T09:45:29.0815909Z "digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1" 2025-12-04T09:45:29.0816240Z }, 2025-12-04T09:45:29.0816371Z { 2025-12-04T09:45:29.0816585Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-12-04T09:45:29.0816857Z "size": 159, 2025-12-04T09:45:29.0817133Z "digest": "sha256:15e0d7e4590d3d8f598d05aec3a92f891bf8b4605bcc38cc2de852b6014ef8f3" 2025-12-04T09:45:29.0817453Z }, 2025-12-04T09:45:29.0817576Z { 2025-12-04T09:45:29.0817791Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-12-04T09:45:29.0818069Z "size": 1011, 2025-12-04T09:45:29.0818347Z "digest": "sha256:a514bd1add3164d8d7ca99aa19294c4ed8b97b074635d98714c4f598a959f4cd" 2025-12-04T09:45:29.0818665Z }, 2025-12-04T09:45:29.0818793Z { 2025-12-04T09:45:29.0819000Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-12-04T09:45:29.0819281Z "size": 724, 2025-12-04T09:45:29.0819545Z "digest": "sha256:3541df015cdb7e8925273399d28e56c31b3c9196f00439ac2925537b173b1f84" 2025-12-04T09:45:29.0819855Z }, 2025-12-04T09:45:29.0819977Z { 2025-12-04T09:45:29.0820203Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-12-04T09:45:29.0820476Z "size": 134, 2025-12-04T09:45:29.0820738Z "digest": "sha256:57b84ee6000204f27a1d9bca199b19be4c86ecd324540dbdf239c56a6c3b34ea" 2025-12-04T09:45:29.0821051Z }, 2025-12-04T09:45:29.0821177Z { 2025-12-04T09:45:29.0821386Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-12-04T09:45:29.0821663Z "size": 32, 2025-12-04T09:45:29.0821948Z "digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1" 2025-12-04T09:45:29.0822258Z }, 2025-12-04T09:45:29.0822388Z { 2025-12-04T09:45:29.0822599Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-12-04T09:45:29.0822870Z "size": 157, 2025-12-04T09:45:29.0823238Z "digest": "sha256:b8babeff6d817a5961dddc15c6bdfdbd05da187fae75d5804015f99fd7c066d8" 2025-12-04T09:45:29.0823620Z }, 2025-12-04T09:45:29.0823750Z { 2025-12-04T09:45:29.0823956Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-12-04T09:45:29.0824230Z "size": 602, 2025-12-04T09:45:29.0824502Z "digest": "sha256:83779ddf6a85ab387f64a45f274cba245b69e4fd1931ff0b5d7d3efd4b7a43bc" 2025-12-04T09:45:29.0824816Z }, 2025-12-04T09:45:29.0824944Z { 2025-12-04T09:45:29.0825238Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-12-04T09:45:29.0825513Z "size": 724, 2025-12-04T09:45:29.0825777Z "digest": "sha256:3541df015cdb7e8925273399d28e56c31b3c9196f00439ac2925537b173b1f84" 2025-12-04T09:45:29.0826087Z }, 2025-12-04T09:45:29.0826211Z { 2025-12-04T09:45:29.0826425Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-12-04T09:45:29.0826702Z "size": 155, 2025-12-04T09:45:29.0826965Z "digest": "sha256:8b7620c0d736cc79381207ce5afe2af90f0cd7f0cd394577d2c9520d7f74762f" 2025-12-04T09:45:29.0827298Z }, 2025-12-04T09:45:29.0827427Z { 2025-12-04T09:45:29.0827642Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-12-04T09:45:29.0827914Z "size": 32, 2025-12-04T09:45:29.0828195Z "digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1" 2025-12-04T09:45:29.0828511Z }, 2025-12-04T09:45:29.0828637Z { 2025-12-04T09:45:29.0828852Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-12-04T09:45:29.0829128Z "size": 188, 2025-12-04T09:45:29.0829396Z "digest": "sha256:3bcfa090e4efd3677425f76baea9f1e0c50a75d8c6b5713ec05310f1dff24539" 2025-12-04T09:45:29.0829710Z }, 2025-12-04T09:45:29.0829837Z { 2025-12-04T09:45:29.0830047Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-12-04T09:45:29.0830324Z "size": 1370, 2025-12-04T09:45:29.0830606Z "digest": "sha256:eb0504ec4d9218a79896b604f73dc0ea5a0f96266ad9c2cdbbbe5f0f18222694" 2025-12-04T09:45:29.0830931Z }, 2025-12-04T09:45:29.0831051Z { 2025-12-04T09:45:29.0831265Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-12-04T09:45:29.0831538Z "size": 32, 2025-12-04T09:45:29.0831802Z "digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1" 2025-12-04T09:45:29.0832122Z }, 2025-12-04T09:45:29.0832262Z { 2025-12-04T09:45:29.0832470Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-12-04T09:45:29.0832743Z "size": 136, 2025-12-04T09:45:29.0833019Z "digest": "sha256:15d0fec09d7b196a1462d51516ee90fc3443ba178d3e56d59cacf32146b4321d" 2025-12-04T09:45:29.0833326Z }, 2025-12-04T09:45:29.0833451Z { 2025-12-04T09:45:29.0833668Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-12-04T09:45:29.0833936Z "size": 528, 2025-12-04T09:45:29.0834208Z "digest": "sha256:cca81fcc62a949959ca4dd3c9056fb293d548ef8607127eeeef6cfd3a8897ca8" 2025-12-04T09:45:29.0834525Z }, 2025-12-04T09:45:29.0834656Z { 2025-12-04T09:45:29.0834861Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-12-04T09:45:29.0835137Z "size": 32, 2025-12-04T09:45:29.0835405Z "digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1" 2025-12-04T09:45:29.0835718Z }, 2025-12-04T09:45:29.0835843Z { 2025-12-04T09:45:29.0836056Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-12-04T09:45:29.0836324Z "size": 104, 2025-12-04T09:45:29.0836600Z "digest": "sha256:b0b8f9b5c6ab98db9cd830dc584e1b6aec9add139e4cc48d8c243d36691e25b4" 2025-12-04T09:45:29.0836922Z }, 2025-12-04T09:45:29.0837046Z { 2025-12-04T09:45:29.0837257Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-12-04T09:45:29.0837532Z "size": 435, 2025-12-04T09:45:29.0837793Z "digest": "sha256:0606ca4d47a8a70e91e92b03ca51a85e731641b09342136a54ef2f2a6d9dfb44" 2025-12-04T09:45:29.0838100Z }, 2025-12-04T09:45:29.0838225Z { 2025-12-04T09:45:29.0838547Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-12-04T09:45:29.0838810Z "size": 32, 2025-12-04T09:45:29.0839078Z "digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1" 2025-12-04T09:45:29.0839386Z }, 2025-12-04T09:45:29.0839504Z { 2025-12-04T09:45:29.0839713Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-12-04T09:45:29.0839982Z "size": 109, 2025-12-04T09:45:29.0840320Z "digest": "sha256:2f80a4e1b3b95ed67bb781ea787e8a63e46de79117d9d8e65c257072b38afa2d" 2025-12-04T09:45:29.0840638Z }, 2025-12-04T09:45:29.0840776Z { 2025-12-04T09:45:29.0840983Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-12-04T09:45:29.0841257Z "size": 1896, 2025-12-04T09:45:29.0841524Z "digest": "sha256:35c916fb1bd057e517dcab78c3a2a018e68096d8993892ad84f47562d37ae352" 2025-12-04T09:45:29.0841833Z }, 2025-12-04T09:45:29.0841952Z { 2025-12-04T09:45:29.0842160Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-12-04T09:45:29.0842441Z "size": 197526165, 2025-12-04T09:45:29.0842711Z "digest": "sha256:195537b7dafc96192f768323b1a8cc2a914d41959849b73198579576b0872a44" 2025-12-04T09:45:29.0843016Z }, 2025-12-04T09:45:29.0843136Z { 2025-12-04T09:45:29.0843339Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-12-04T09:45:29.0843607Z "size": 106, 2025-12-04T09:45:29.0843870Z "digest": "sha256:dc454fd3967e5735b2498b7f1d958a2c626987d5e4ce225ca98da3cd945b59f3" 2025-12-04T09:45:29.0844177Z }, 2025-12-04T09:45:29.0844303Z { 2025-12-04T09:45:29.0844511Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-12-04T09:45:29.0844781Z "size": 165, 2025-12-04T09:45:29.0845045Z "digest": "sha256:701b34f115fa897181c046dc37288e87cbc3ad74c36a9e2224b5bfe7c5703afb" 2025-12-04T09:45:29.0845359Z }, 2025-12-04T09:45:29.0845495Z { 2025-12-04T09:45:29.0845703Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-12-04T09:45:29.0845983Z "size": 7944, 2025-12-04T09:45:29.0846257Z "digest": "sha256:39cefc00ffedebc9098261c798408b87a20c95a88fccb110594077f48dadf760" 2025-12-04T09:45:29.0846566Z }, 2025-12-04T09:45:29.0846690Z { 2025-12-04T09:45:29.0846902Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-12-04T09:45:29.0847170Z "size": 8071, 2025-12-04T09:45:29.0847440Z "digest": "sha256:6ae51eb61a325b2c2995a5088c81aa20821b75be65b5aa722c7c40556b5d03ea" 2025-12-04T09:45:29.0847751Z }, 2025-12-04T09:45:29.0847873Z { 2025-12-04T09:45:29.0848084Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-12-04T09:45:29.0848355Z "size": 304, 2025-12-04T09:45:29.0848619Z "digest": "sha256:1fd5341e66dfc0c1ae23af014641a92a6fd02640c528fe6d4dc55921ed659a26" 2025-12-04T09:45:29.0848935Z }, 2025-12-04T09:45:29.0849059Z { 2025-12-04T09:45:29.0849269Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-12-04T09:45:29.0849552Z "size": 13364291, 2025-12-04T09:45:29.0849840Z "digest": "sha256:72a7c87e35e40ab796f90aee1b51add7902f0cdc44406d2505b6c6a1f55a8da6" 2025-12-04T09:45:29.0850153Z }, 2025-12-04T09:45:29.0850273Z { 2025-12-04T09:45:29.0850485Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-12-04T09:45:29.0850759Z "size": 108, 2025-12-04T09:45:29.0851030Z "digest": "sha256:ec36862ac98ebaac52ee1a8b1d162d45bd0e3bf59ae7e19c8f80ad3960b4c600" 2025-12-04T09:45:29.0851348Z }, 2025-12-04T09:45:29.0851472Z { 2025-12-04T09:45:29.0851682Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-12-04T09:45:29.0851959Z "size": 54145699, 2025-12-04T09:45:29.0852237Z "digest": "sha256:05ddbf246e8add0e293474dbf88bb028d5a295a25ac59e8648a18db644377773" 2025-12-04T09:45:29.0852547Z }, 2025-12-04T09:45:29.0852676Z { 2025-12-04T09:45:29.0852887Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-12-04T09:45:29.0853159Z "size": 32, 2025-12-04T09:45:29.0853422Z "digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1" 2025-12-04T09:45:29.0853814Z } 2025-12-04T09:45:29.0853939Z ] 2025-12-04T09:45:29.0854061Z } 2025-12-04T09:45:29.0854200Z + exit 0 2025-12-04T09:45:29.0876920Z ##[group]Run set -eux 2025-12-04T09:45:29.0877126Z set -eux 2025-12-04T09:45:29.0877413Z # It's ok if this steps fails, it would then be an anonymous user like what we used to have 2025-12-04T09:45:29.0878618Z aws secretsmanager get-secret-value --secret-id docker_hub_readonly_token | jq --raw-output '.SecretString' | jq -r .docker_hub_readonly_token | docker login --username pytorchbot --password-stdin || true 2025-12-04T09:45:29.0887273Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-12-04T09:45:29.0887553Z env: 2025-12-04T09:45:29.0887706Z GIT_DEFAULT_BRANCH: main 2025-12-04T09:45:29.0887885Z ##[endgroup] 2025-12-04T09:45:29.0919007Z + aws secretsmanager get-secret-value --secret-id docker_hub_readonly_token 2025-12-04T09:45:29.0919777Z + jq --raw-output .SecretString 2025-12-04T09:45:29.0921286Z + jq -r .docker_hub_readonly_token 2025-12-04T09:45:29.0922197Z + docker login --username pytorchbot --password-stdin 2025-12-04T09:45:29.6122186Z WARNING! Your password will be stored unencrypted in /home/ec2-user/.docker/config.json. 2025-12-04T09:45:29.6122755Z Configure a credential helper to remove this warning. See 2025-12-04T09:45:29.6123275Z https://docs.docker.com/engine/reference/commandline/login/#credentials-store 2025-12-04T09:45:29.6123624Z 2025-12-04T09:45:29.6123861Z Login Succeeded 2025-12-04T09:45:29.6203961Z ##[group]Run tag=${ECR_DOCKER_IMAGE##*:} 2025-12-04T09:45:29.6204284Z tag=${ECR_DOCKER_IMAGE##*:} 2025-12-04T09:45:29.6204591Z echo "docker pull ghcr.io/pytorch/ci-image:${tag/:/-}" 2025-12-04T09:45:29.6212444Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-12-04T09:45:29.6212747Z env: 2025-12-04T09:45:29.6212914Z GIT_DEFAULT_BRANCH: main 2025-12-04T09:45:29.6213516Z ECR_DOCKER_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/ci-image:pytorch-linux-jammy-cuda12.8-cudnn9-py3-gcc11-f0cd68561080d537ef3d3d6f81b25a6416ad600a 2025-12-04T09:45:29.6214154Z ##[endgroup] 2025-12-04T09:45:29.6242849Z docker pull ghcr.io/pytorch/ci-image:pytorch-linux-jammy-cuda12.8-cudnn9-py3-gcc11-f0cd68561080d537ef3d3d6f81b25a6416ad600a 2025-12-04T09:45:29.6280706Z ##[group]Run pytorch/test-infra/.github/actions/pull-docker-image@main 2025-12-04T09:45:29.6281055Z with: 2025-12-04T09:45:29.6281612Z docker-image: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/ci-image:pytorch-linux-jammy-cuda12.8-cudnn9-py3-gcc11-f0cd68561080d537ef3d3d6f81b25a6416ad600a 2025-12-04T09:45:29.6282279Z docker-registry: 308535385114.dkr.ecr.us-east-1.amazonaws.com 2025-12-04T09:45:29.6282553Z env: 2025-12-04T09:45:29.6282712Z GIT_DEFAULT_BRANCH: main 2025-12-04T09:45:29.6282904Z ##[endgroup] 2025-12-04T09:45:29.6296461Z ##[group]Run set -x 2025-12-04T09:45:29.6296656Z set -x 2025-12-04T09:45:29.6296824Z set +e 2025-12-04T09:45:29.6296983Z  2025-12-04T09:45:29.6297140Z login() { 2025-12-04T09:45:29.6297488Z  aws ecr get-login-password --region us-east-1 | docker login -u AWS --password-stdin "$1" 2025-12-04T09:45:29.6297860Z } 2025-12-04T09:45:29.6298011Z  2025-12-04T09:45:29.6298182Z retry () { 2025-12-04T09:45:29.6298369Z  $* || (sleep 1 && $*) || (sleep 2 && $*) 2025-12-04T09:45:29.6298599Z } 2025-12-04T09:45:29.6298752Z  2025-12-04T09:45:29.6298919Z retry login "${DOCKER_REGISTRY}" 2025-12-04T09:45:29.6299132Z  2025-12-04T09:45:29.6299479Z IMAGE_SIZE=$(docker manifest inspect "${DOCKER_IMAGE}" | jq '[.layers[].size, .config.size] | add / 1024 / 1024') 2025-12-04T09:45:29.6299955Z echo "Compressed size of image in MB: ${IMAGE_SIZE}" 2025-12-04T09:45:29.6300217Z  2025-12-04T09:45:29.6300371Z set -e 2025-12-04T09:45:29.6300776Z # ignore output since only exit code is used for conditional 2025-12-04T09:45:29.6301128Z # only pull docker image if it's not available locally 2025-12-04T09:45:29.6301507Z if ! docker inspect --type=image "${DOCKER_IMAGE}" >/dev/null 2>/dev/null; then 2025-12-04T09:45:29.6301862Z  retry docker pull "${DOCKER_IMAGE}" 2025-12-04T09:45:29.6302089Z fi 2025-12-04T09:45:29.6309114Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-12-04T09:45:29.6309409Z env: 2025-12-04T09:45:29.6309567Z GIT_DEFAULT_BRANCH: main 2025-12-04T09:45:29.6310145Z DOCKER_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/ci-image:pytorch-linux-jammy-cuda12.8-cudnn9-py3-gcc11-f0cd68561080d537ef3d3d6f81b25a6416ad600a 2025-12-04T09:45:29.6310821Z DOCKER_REGISTRY: 308535385114.dkr.ecr.us-east-1.amazonaws.com 2025-12-04T09:45:29.6311103Z ##[endgroup] 2025-12-04T09:45:29.6336653Z + set +e 2025-12-04T09:45:29.6336947Z + retry login 308535385114.dkr.ecr.us-east-1.amazonaws.com 2025-12-04T09:45:29.6337353Z + login 308535385114.dkr.ecr.us-east-1.amazonaws.com 2025-12-04T09:45:29.6340030Z + aws ecr get-login-password --region us-east-1 2025-12-04T09:45:29.6341156Z + docker login -u AWS --password-stdin 308535385114.dkr.ecr.us-east-1.amazonaws.com 2025-12-04T09:45:30.0932631Z WARNING! Your password will be stored unencrypted in /home/ec2-user/.docker/config.json. 2025-12-04T09:45:30.0933500Z Configure a credential helper to remove this warning. See 2025-12-04T09:45:30.0934292Z https://docs.docker.com/engine/reference/commandline/login/#credentials-store 2025-12-04T09:45:30.0934836Z 2025-12-04T09:45:30.0935297Z Login Succeeded 2025-12-04T09:45:30.0959205Z ++ docker manifest inspect 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/ci-image:pytorch-linux-jammy-cuda12.8-cudnn9-py3-gcc11-f0cd68561080d537ef3d3d6f81b25a6416ad600a 2025-12-04T09:45:30.0960102Z ++ jq '[.layers[].size, .config.size] | add / 1024 / 1024' 2025-12-04T09:45:30.2989783Z + IMAGE_SIZE=15091.581844329834 2025-12-04T09:45:30.2990335Z + echo 'Compressed size of image in MB: 15091.581844329834' 2025-12-04T09:45:30.2990743Z + set -e 2025-12-04T09:45:30.2990986Z Compressed size of image in MB: 15091.581844329834 2025-12-04T09:45:30.2992120Z + docker inspect --type=image 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/ci-image:pytorch-linux-jammy-cuda12.8-cudnn9-py3-gcc11-f0cd68561080d537ef3d3d6f81b25a6416ad600a 2025-12-04T09:45:30.3121857Z + retry docker pull 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/ci-image:pytorch-linux-jammy-cuda12.8-cudnn9-py3-gcc11-f0cd68561080d537ef3d3d6f81b25a6416ad600a 2025-12-04T09:45:30.3123252Z + docker pull 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/ci-image:pytorch-linux-jammy-cuda12.8-cudnn9-py3-gcc11-f0cd68561080d537ef3d3d6f81b25a6416ad600a 2025-12-04T09:45:30.5426475Z pytorch-linux-jammy-cuda12.8-cudnn9-py3-gcc11-f0cd68561080d537ef3d3d6f81b25a6416ad600a: Pulling from pytorch/ci-image 2025-12-04T09:45:30.5438579Z 63e5bc7682b8: Pulling fs layer 2025-12-04T09:45:30.5438848Z 0678d56345c9: Pulling fs layer 2025-12-04T09:45:30.5439060Z 45f5c9ddfce7: Pulling fs layer 2025-12-04T09:45:30.5439263Z 086b1df51ac1: Pulling fs layer 2025-12-04T09:45:30.5439460Z fe8a7b64bf98: Pulling fs layer 2025-12-04T09:45:30.5439647Z 7680723e9a57: Pulling fs layer 2025-12-04T09:45:30.5439840Z 9c5027aeeb4e: Pulling fs layer 2025-12-04T09:45:30.5440030Z 9a5652110360: Pulling fs layer 2025-12-04T09:45:30.5440414Z 375c4427e914: Pulling fs layer 2025-12-04T09:45:30.5440777Z a86faaa7dbdd: Pulling fs layer 2025-12-04T09:45:30.5441116Z fb7848686804: Pulling fs layer 2025-12-04T09:45:30.5441459Z 3541df015cdb: Pulling fs layer 2025-12-04T09:45:30.5441785Z 79dc80f426b2: Pulling fs layer 2025-12-04T09:45:30.5442102Z a13fcc1b90bb: Pulling fs layer 2025-12-04T09:45:30.5442444Z 4f4fb700ef54: Pulling fs layer 2025-12-04T09:45:30.5442795Z 549db4d6c618: Pulling fs layer 2025-12-04T09:45:30.5443132Z 5c63528cb580: Pulling fs layer 2025-12-04T09:45:30.5443467Z 75bd83b989a4: Pulling fs layer 2025-12-04T09:45:30.5444055Z de6e78970f51: Pulling fs layer 2025-12-04T09:45:30.5444400Z e13ed7c7e473: Pulling fs layer 2025-12-04T09:45:30.5444599Z 6e2949bcb741: Pulling fs layer 2025-12-04T09:45:30.5444793Z 14d69d9aaec7: Pulling fs layer 2025-12-04T09:45:30.5444993Z 5c02769dd8e5: Pulling fs layer 2025-12-04T09:45:30.5445186Z 35041ce524ac: Pulling fs layer 2025-12-04T09:45:30.5445388Z 2fa92dc5885e: Pulling fs layer 2025-12-04T09:45:30.5445589Z 2b85eafbd92a: Pulling fs layer 2025-12-04T09:45:30.5445777Z ff755a4ddad7: Pulling fs layer 2025-12-04T09:45:30.5445973Z 09eb41bdf42d: Pulling fs layer 2025-12-04T09:45:30.5446165Z 11ede4d59e93: Pulling fs layer 2025-12-04T09:45:30.5446349Z 1283cd8f801a: Pulling fs layer 2025-12-04T09:45:30.5446543Z 024fa855425f: Pulling fs layer 2025-12-04T09:45:30.5446722Z 549db4d6c618: Waiting 2025-12-04T09:45:30.5446902Z 303e6747a62e: Pulling fs layer 2025-12-04T09:45:30.5447095Z 3017cdf4838b: Pulling fs layer 2025-12-04T09:45:30.5447282Z 79dc80f426b2: Waiting 2025-12-04T09:45:30.5447459Z 6b6cd1c358e8: Pulling fs layer 2025-12-04T09:45:30.5447648Z a13fcc1b90bb: Waiting 2025-12-04T09:45:30.5447822Z b2dd04501124: Pulling fs layer 2025-12-04T09:45:30.5448147Z 4f4fb700ef54: Waiting 2025-12-04T09:45:30.5448384Z 55adc51fe589: Pulling fs layer 2025-12-04T09:45:30.5448607Z 7680723e9a57: Waiting 2025-12-04T09:45:30.5448880Z a43ca0e4b837: Pulling fs layer 2025-12-04T09:45:30.5449064Z a86faaa7dbdd: Waiting 2025-12-04T09:45:30.5449234Z b7212f17fd14: Pulling fs layer 2025-12-04T09:45:30.5449416Z 9c5027aeeb4e: Waiting 2025-12-04T09:45:30.5449597Z 083e42cac090: Pulling fs layer 2025-12-04T09:45:30.5449931Z fb7848686804: Waiting 2025-12-04T09:45:30.5450209Z 9a5652110360: Waiting 2025-12-04T09:45:30.5450504Z 0a00b784a4aa: Pulling fs layer 2025-12-04T09:45:30.5450734Z c6173c779f7b: Pulling fs layer 2025-12-04T09:45:30.5450927Z ed3d1e3387b9: Pulling fs layer 2025-12-04T09:45:30.5451108Z b29343478586: Pulling fs layer 2025-12-04T09:45:30.5451297Z c6f0520487fb: Pulling fs layer 2025-12-04T09:45:30.5451484Z 5c63528cb580: Waiting 2025-12-04T09:45:30.5451652Z 148171691cd4: Pulling fs layer 2025-12-04T09:45:30.5451854Z 2c666d30ed77: Pulling fs layer 2025-12-04T09:45:30.5452047Z 5d8d3a0a98e0: Pulling fs layer 2025-12-04T09:45:30.5452230Z b06bafce9e81: Pulling fs layer 2025-12-04T09:45:30.5452421Z 15e0d7e4590d: Pulling fs layer 2025-12-04T09:45:30.5452811Z 75bd83b989a4: Waiting 2025-12-04T09:45:30.5452996Z a514bd1add31: Pulling fs layer 2025-12-04T09:45:30.5453329Z 57b84ee60002: Pulling fs layer 2025-12-04T09:45:30.5453646Z 6b6cd1c358e8: Waiting 2025-12-04T09:45:30.5453915Z b2dd04501124: Waiting 2025-12-04T09:45:30.5454170Z 5c02769dd8e5: Waiting 2025-12-04T09:45:30.5454449Z 55adc51fe589: Waiting 2025-12-04T09:45:30.5454743Z 6e2949bcb741: Waiting 2025-12-04T09:45:30.5455038Z b8babeff6d81: Pulling fs layer 2025-12-04T09:45:30.5455363Z a43ca0e4b837: Waiting 2025-12-04T09:45:30.5455669Z 83779ddf6a85: Pulling fs layer 2025-12-04T09:45:30.5455990Z 14d69d9aaec7: Waiting 2025-12-04T09:45:30.5456272Z b7212f17fd14: Waiting 2025-12-04T09:45:30.5456566Z 083e42cac090: Waiting 2025-12-04T09:45:30.5456836Z 2b85eafbd92a: Waiting 2025-12-04T09:45:30.5457124Z 2fa92dc5885e: Waiting 2025-12-04T09:45:30.5457405Z 35041ce524ac: Waiting 2025-12-04T09:45:30.5457683Z 0a00b784a4aa: Waiting 2025-12-04T09:45:30.5457965Z ed3d1e3387b9: Waiting 2025-12-04T09:45:30.5458229Z a514bd1add31: Waiting 2025-12-04T09:45:30.5458505Z 8b7620c0d736: Pulling fs layer 2025-12-04T09:45:30.5458834Z b8babeff6d81: Waiting 2025-12-04T09:45:30.5459136Z 3bcfa090e4ef: Pulling fs layer 2025-12-04T09:45:30.5459463Z ff755a4ddad7: Waiting 2025-12-04T09:45:30.5459760Z eb0504ec4d92: Pulling fs layer 2025-12-04T09:45:30.5460087Z 09eb41bdf42d: Waiting 2025-12-04T09:45:30.5460367Z c6173c779f7b: Waiting 2025-12-04T09:45:30.5460649Z 11ede4d59e93: Waiting 2025-12-04T09:45:30.5460943Z 15d0fec09d7b: Pulling fs layer 2025-12-04T09:45:30.5461284Z cca81fcc62a9: Pulling fs layer 2025-12-04T09:45:30.5461597Z c6f0520487fb: Waiting 2025-12-04T09:45:30.5461859Z 148171691cd4: Waiting 2025-12-04T09:45:30.5462249Z 83779ddf6a85: Waiting 2025-12-04T09:45:30.5462540Z b0b8f9b5c6ab: Pulling fs layer 2025-12-04T09:45:30.5462862Z 8b7620c0d736: Waiting 2025-12-04T09:45:30.5463159Z 0606ca4d47a8: Pulling fs layer 2025-12-04T09:45:30.5463473Z 2c666d30ed77: Waiting 2025-12-04T09:45:30.5463883Z 2f80a4e1b3b9: Pulling fs layer 2025-12-04T09:45:30.5464214Z 303e6747a62e: Waiting 2025-12-04T09:45:30.5464500Z 35c916fb1bd0: Pulling fs layer 2025-12-04T09:45:30.5464834Z 3bcfa090e4ef: Waiting 2025-12-04T09:45:30.5465141Z 195537b7dafc: Pulling fs layer 2025-12-04T09:45:30.5465465Z 57b84ee60002: Waiting 2025-12-04T09:45:30.5465746Z eb0504ec4d92: Waiting 2025-12-04T09:45:30.5466043Z dc454fd3967e: Pulling fs layer 2025-12-04T09:45:30.5466370Z 5d8d3a0a98e0: Waiting 2025-12-04T09:45:30.5466668Z b0b8f9b5c6ab: Waiting 2025-12-04T09:45:30.5466963Z 701b34f115fa: Pulling fs layer 2025-12-04T09:45:30.5467291Z 0606ca4d47a8: Waiting 2025-12-04T09:45:30.5467584Z 15d0fec09d7b: Waiting 2025-12-04T09:45:30.5467899Z 39cefc00ffed: Pulling fs layer 2025-12-04T09:45:30.5468401Z 1283cd8f801a: Waiting 2025-12-04T09:45:30.5468698Z cca81fcc62a9: Waiting 2025-12-04T09:45:30.5469011Z 6ae51eb61a32: Pulling fs layer 2025-12-04T09:45:30.5469370Z 1fd5341e66df: Pulling fs layer 2025-12-04T09:45:30.5469558Z dc454fd3967e: Waiting 2025-12-04T09:45:30.5469718Z 701b34f115fa: Waiting 2025-12-04T09:45:30.5469892Z 72a7c87e35e4: Pulling fs layer 2025-12-04T09:45:30.5470076Z ec36862ac98e: Pulling fs layer 2025-12-04T09:45:30.5470260Z 2f80a4e1b3b9: Waiting 2025-12-04T09:45:30.5470438Z 05ddbf246e8a: Pulling fs layer 2025-12-04T09:45:30.5470613Z b06bafce9e81: Waiting 2025-12-04T09:45:30.5470786Z 3017cdf4838b: Waiting 2025-12-04T09:45:30.5470951Z 35c916fb1bd0: Waiting 2025-12-04T09:45:30.5471105Z b29343478586: Waiting 2025-12-04T09:45:30.5471267Z 39cefc00ffed: Waiting 2025-12-04T09:45:30.5471430Z 6ae51eb61a32: Waiting 2025-12-04T09:45:30.5471578Z 72a7c87e35e4: Waiting 2025-12-04T09:45:30.5471740Z 1fd5341e66df: Waiting 2025-12-04T09:45:30.5471897Z 195537b7dafc: Waiting 2025-12-04T09:45:30.5472060Z ec36862ac98e: Waiting 2025-12-04T09:45:30.5472214Z 05ddbf246e8a: Waiting 2025-12-04T09:45:30.5472377Z 3541df015cdb: Waiting 2025-12-04T09:45:30.5472535Z e13ed7c7e473: Waiting 2025-12-04T09:45:30.5472689Z 15e0d7e4590d: Waiting 2025-12-04T09:45:30.5472846Z 024fa855425f: Waiting 2025-12-04T09:45:30.5473003Z de6e78970f51: Waiting 2025-12-04T09:45:30.5473279Z 375c4427e914: Waiting 2025-12-04T09:45:30.6380259Z 0678d56345c9: Verifying Checksum 2025-12-04T09:45:30.6380699Z 0678d56345c9: Download complete 2025-12-04T09:45:30.7379401Z 086b1df51ac1: Verifying Checksum 2025-12-04T09:45:30.7379745Z 086b1df51ac1: Download complete 2025-12-04T09:45:30.8282565Z fe8a7b64bf98: Verifying Checksum 2025-12-04T09:45:30.8282892Z fe8a7b64bf98: Download complete 2025-12-04T09:45:30.8832548Z 63e5bc7682b8: Verifying Checksum 2025-12-04T09:45:30.8832847Z 63e5bc7682b8: Download complete 2025-12-04T09:45:30.9027364Z 7680723e9a57: Download complete 2025-12-04T09:45:30.9542986Z 9c5027aeeb4e: Verifying Checksum 2025-12-04T09:45:30.9543419Z 9c5027aeeb4e: Download complete 2025-12-04T09:45:30.9896058Z 9a5652110360: Verifying Checksum 2025-12-04T09:45:30.9896352Z 9a5652110360: Download complete 2025-12-04T09:45:31.0692750Z a86faaa7dbdd: Verifying Checksum 2025-12-04T09:45:31.0693116Z a86faaa7dbdd: Download complete 2025-12-04T09:45:31.1345673Z fb7848686804: Verifying Checksum 2025-12-04T09:45:31.1346127Z fb7848686804: Download complete 2025-12-04T09:45:31.2236556Z 3541df015cdb: Verifying Checksum 2025-12-04T09:45:31.2237058Z 3541df015cdb: Download complete 2025-12-04T09:45:31.2968941Z 79dc80f426b2: Verifying Checksum 2025-12-04T09:45:31.2969462Z 79dc80f426b2: Download complete 2025-12-04T09:45:31.7856037Z 63e5bc7682b8: Pull complete 2025-12-04T09:45:31.8099492Z 0678d56345c9: Pull complete 2025-12-04T09:45:32.1088700Z 375c4427e914: Verifying Checksum 2025-12-04T09:45:32.1089045Z 375c4427e914: Download complete 2025-12-04T09:45:32.1168141Z 4f4fb700ef54: Verifying Checksum 2025-12-04T09:45:32.1168447Z 4f4fb700ef54: Download complete 2025-12-04T09:45:32.2167873Z 549db4d6c618: Verifying Checksum 2025-12-04T09:45:32.2168194Z 549db4d6c618: Download complete 2025-12-04T09:45:32.2800918Z 5c63528cb580: Verifying Checksum 2025-12-04T09:45:32.2801187Z 5c63528cb580: Download complete 2025-12-04T09:45:32.3497628Z 75bd83b989a4: Verifying Checksum 2025-12-04T09:45:32.3498076Z 75bd83b989a4: Download complete 2025-12-04T09:45:32.4220093Z de6e78970f51: Verifying Checksum 2025-12-04T09:45:32.4220403Z de6e78970f51: Download complete 2025-12-04T09:45:32.5093499Z e13ed7c7e473: Verifying Checksum 2025-12-04T09:45:32.5093849Z e13ed7c7e473: Download complete 2025-12-04T09:45:32.5948133Z 6e2949bcb741: Verifying Checksum 2025-12-04T09:45:32.5948443Z 6e2949bcb741: Download complete 2025-12-04T09:45:32.6783034Z 14d69d9aaec7: Verifying Checksum 2025-12-04T09:45:32.6783362Z 14d69d9aaec7: Download complete 2025-12-04T09:45:32.7697391Z 5c02769dd8e5: Verifying Checksum 2025-12-04T09:45:32.7697708Z 5c02769dd8e5: Download complete 2025-12-04T09:45:33.7141458Z 45f5c9ddfce7: Verifying Checksum 2025-12-04T09:45:33.7141785Z 45f5c9ddfce7: Download complete 2025-12-04T09:45:33.7861603Z 2fa92dc5885e: Verifying Checksum 2025-12-04T09:45:33.7861970Z 2fa92dc5885e: Download complete 2025-12-04T09:45:34.1651384Z 2b85eafbd92a: Verifying Checksum 2025-12-04T09:45:34.1651837Z 2b85eafbd92a: Download complete 2025-12-04T09:45:34.2571464Z ff755a4ddad7: Verifying Checksum 2025-12-04T09:45:34.2571818Z ff755a4ddad7: Download complete 2025-12-04T09:45:34.3296488Z 09eb41bdf42d: Verifying Checksum 2025-12-04T09:45:34.3296845Z 09eb41bdf42d: Download complete 2025-12-04T09:45:38.9826120Z 11ede4d59e93: Verifying Checksum 2025-12-04T09:45:39.0349426Z 1283cd8f801a: Download complete 2025-12-04T09:45:39.1307484Z 024fa855425f: Verifying Checksum 2025-12-04T09:45:39.1307771Z 024fa855425f: Download complete 2025-12-04T09:45:39.2088113Z 303e6747a62e: Download complete 2025-12-04T09:45:39.2991245Z 3017cdf4838b: Verifying Checksum 2025-12-04T09:45:39.2991914Z 3017cdf4838b: Download complete 2025-12-04T09:45:39.5392766Z 6b6cd1c358e8: Verifying Checksum 2025-12-04T09:45:39.5393103Z 6b6cd1c358e8: Download complete 2025-12-04T09:45:39.6303675Z b2dd04501124: Verifying Checksum 2025-12-04T09:45:39.6304125Z b2dd04501124: Download complete 2025-12-04T09:45:39.6985107Z 55adc51fe589: Verifying Checksum 2025-12-04T09:45:39.6985563Z 55adc51fe589: Download complete 2025-12-04T09:45:39.7725203Z a43ca0e4b837: Verifying Checksum 2025-12-04T09:45:39.7725795Z a43ca0e4b837: Download complete 2025-12-04T09:45:39.8349548Z b7212f17fd14: Download complete 2025-12-04T09:45:39.9226226Z 083e42cac090: Verifying Checksum 2025-12-04T09:45:39.9226666Z 083e42cac090: Download complete 2025-12-04T09:45:40.0475654Z 0a00b784a4aa: Verifying Checksum 2025-12-04T09:45:40.0476182Z 0a00b784a4aa: Download complete 2025-12-04T09:45:40.1289815Z c6173c779f7b: Verifying Checksum 2025-12-04T09:45:40.1290244Z c6173c779f7b: Download complete 2025-12-04T09:45:40.8694676Z 45f5c9ddfce7: Pull complete 2025-12-04T09:45:40.8941911Z 086b1df51ac1: Pull complete 2025-12-04T09:45:40.9234134Z fe8a7b64bf98: Pull complete 2025-12-04T09:45:40.9469945Z 7680723e9a57: Pull complete 2025-12-04T09:45:40.9714355Z 9c5027aeeb4e: Pull complete 2025-12-04T09:45:40.9948436Z 9a5652110360: Pull complete 2025-12-04T09:45:41.6458408Z ed3d1e3387b9: Verifying Checksum 2025-12-04T09:45:41.6458843Z ed3d1e3387b9: Download complete 2025-12-04T09:45:41.7267449Z b29343478586: Verifying Checksum 2025-12-04T09:45:41.7267889Z b29343478586: Download complete 2025-12-04T09:45:42.9166953Z 375c4427e914: Pull complete 2025-12-04T09:45:43.1404829Z a86faaa7dbdd: Pull complete 2025-12-04T09:45:43.3078171Z fb7848686804: Pull complete 2025-12-04T09:45:43.4134722Z 3541df015cdb: Pull complete 2025-12-04T09:45:43.5088318Z 79dc80f426b2: Pull complete 2025-12-04T09:45:44.8931266Z c6f0520487fb: Download complete 2025-12-04T09:46:03.2090496Z a13fcc1b90bb: Verifying Checksum 2025-12-04T09:46:03.2090797Z a13fcc1b90bb: Download complete 2025-12-04T09:46:03.2897378Z 2c666d30ed77: Verifying Checksum 2025-12-04T09:46:03.2898054Z 2c666d30ed77: Download complete 2025-12-04T09:46:03.3747199Z 5d8d3a0a98e0: Verifying Checksum 2025-12-04T09:46:03.3747577Z 5d8d3a0a98e0: Download complete 2025-12-04T09:46:03.4654306Z b06bafce9e81: Verifying Checksum 2025-12-04T09:46:03.4654670Z b06bafce9e81: Download complete 2025-12-04T09:46:03.5276187Z 15e0d7e4590d: Verifying Checksum 2025-12-04T09:46:03.5276534Z 15e0d7e4590d: Download complete 2025-12-04T09:46:03.6142927Z a514bd1add31: Download complete 2025-12-04T09:46:03.7073323Z 57b84ee60002: Verifying Checksum 2025-12-04T09:46:03.7073751Z 57b84ee60002: Download complete 2025-12-04T09:46:03.8028970Z b8babeff6d81: Verifying Checksum 2025-12-04T09:46:03.8029323Z b8babeff6d81: Download complete 2025-12-04T09:46:03.8805813Z 83779ddf6a85: Verifying Checksum 2025-12-04T09:46:03.8806256Z 83779ddf6a85: Download complete 2025-12-04T09:46:03.9559766Z 8b7620c0d736: Download complete 2025-12-04T09:46:04.0430456Z 3bcfa090e4ef: Verifying Checksum 2025-12-04T09:46:04.0430745Z 3bcfa090e4ef: Download complete 2025-12-04T09:46:04.1379239Z eb0504ec4d92: Verifying Checksum 2025-12-04T09:46:04.1379528Z eb0504ec4d92: Download complete 2025-12-04T09:46:04.2306747Z 15d0fec09d7b: Download complete 2025-12-04T09:46:04.3235369Z cca81fcc62a9: Verifying Checksum 2025-12-04T09:46:04.3237504Z cca81fcc62a9: Download complete 2025-12-04T09:46:04.4104272Z b0b8f9b5c6ab: Verifying Checksum 2025-12-04T09:46:04.4104592Z b0b8f9b5c6ab: Download complete 2025-12-04T09:46:04.4749270Z 0606ca4d47a8: Verifying Checksum 2025-12-04T09:46:04.4749570Z 0606ca4d47a8: Download complete 2025-12-04T09:46:04.5524997Z 2f80a4e1b3b9: Verifying Checksum 2025-12-04T09:46:04.5525394Z 2f80a4e1b3b9: Download complete 2025-12-04T09:46:04.6281889Z 35c916fb1bd0: Verifying Checksum 2025-12-04T09:46:04.6282347Z 35c916fb1bd0: Download complete 2025-12-04T09:46:06.6712550Z 195537b7dafc: Verifying Checksum 2025-12-04T09:46:06.6712846Z 195537b7dafc: Download complete 2025-12-04T09:46:06.7875684Z dc454fd3967e: Download complete 2025-12-04T09:46:06.8778421Z 701b34f115fa: Verifying Checksum 2025-12-04T09:46:06.8778743Z 701b34f115fa: Download complete 2025-12-04T09:46:06.9622009Z 39cefc00ffed: Verifying Checksum 2025-12-04T09:46:06.9622311Z 39cefc00ffed: Download complete 2025-12-04T09:46:07.0572525Z 6ae51eb61a32: Verifying Checksum 2025-12-04T09:46:07.0572879Z 6ae51eb61a32: Download complete 2025-12-04T09:46:07.1440166Z 1fd5341e66df: Verifying Checksum 2025-12-04T09:46:07.1440544Z 1fd5341e66df: Download complete 2025-12-04T09:46:07.3359715Z 72a7c87e35e4: Verifying Checksum 2025-12-04T09:46:07.3360018Z 72a7c87e35e4: Download complete 2025-12-04T09:46:07.3993289Z ec36862ac98e: Verifying Checksum 2025-12-04T09:46:07.3993644Z ec36862ac98e: Download complete 2025-12-04T09:46:07.9991726Z 05ddbf246e8a: Verifying Checksum 2025-12-04T09:46:07.9992036Z 05ddbf246e8a: Download complete 2025-12-04T09:46:15.5294295Z 148171691cd4: Verifying Checksum 2025-12-04T09:46:15.5294631Z 148171691cd4: Download complete 2025-12-04T09:46:53.2854292Z 35041ce524ac: Verifying Checksum 2025-12-04T09:46:53.2854680Z 35041ce524ac: Download complete 2025-12-04T09:47:24.4607111Z a13fcc1b90bb: Pull complete 2025-12-04T09:47:24.6636600Z 4f4fb700ef54: Pull complete 2025-12-04T09:47:24.8733261Z 549db4d6c618: Pull complete 2025-12-04T09:47:25.1116622Z 5c63528cb580: Pull complete 2025-12-04T09:47:25.3247564Z 75bd83b989a4: Pull complete 2025-12-04T09:47:25.5955474Z de6e78970f51: Pull complete 2025-12-04T09:47:25.8262940Z e13ed7c7e473: Pull complete 2025-12-04T09:47:26.0565062Z 6e2949bcb741: Pull complete 2025-12-04T09:47:26.2817378Z 14d69d9aaec7: Pull complete 2025-12-04T09:47:26.4834784Z 5c02769dd8e5: Pull complete 2025-12-04T09:48:59.4669129Z 35041ce524ac: Pull complete 2025-12-04T09:48:59.6856318Z 2fa92dc5885e: Pull complete 2025-12-04T09:49:00.1991341Z 2b85eafbd92a: Pull complete 2025-12-04T09:49:00.2236441Z ff755a4ddad7: Pull complete 2025-12-04T09:49:00.2485292Z 09eb41bdf42d: Pull complete 2025-12-04T09:49:06.8219071Z 11ede4d59e93: Pull complete 2025-12-04T09:49:07.0328171Z 1283cd8f801a: Pull complete 2025-12-04T09:49:07.2609346Z 024fa855425f: Pull complete 2025-12-04T09:49:07.6960510Z 303e6747a62e: Pull complete 2025-12-04T09:49:07.9132847Z 3017cdf4838b: Pull complete 2025-12-04T09:49:08.2849566Z 6b6cd1c358e8: Pull complete 2025-12-04T09:49:08.5089401Z b2dd04501124: Pull complete 2025-12-04T09:49:08.7336484Z 55adc51fe589: Pull complete 2025-12-04T09:49:09.1787308Z a43ca0e4b837: Pull complete 2025-12-04T09:49:09.4016376Z b7212f17fd14: Pull complete 2025-12-04T09:49:09.6205358Z 083e42cac090: Pull complete 2025-12-04T09:49:10.0556940Z 0a00b784a4aa: Pull complete 2025-12-04T09:49:10.2699561Z c6173c779f7b: Pull complete 2025-12-04T09:49:13.0534816Z ed3d1e3387b9: Pull complete 2025-12-04T09:49:13.2711191Z b29343478586: Pull complete 2025-12-04T09:49:14.3315423Z c6f0520487fb: Pull complete 2025-12-04T09:49:58.3224996Z 148171691cd4: Pull complete 2025-12-04T09:49:58.3733727Z 2c666d30ed77: Pull complete 2025-12-04T09:49:58.5124792Z 5d8d3a0a98e0: Pull complete 2025-12-04T09:49:58.8097349Z b06bafce9e81: Pull complete 2025-12-04T09:49:59.1076607Z 15e0d7e4590d: Pull complete 2025-12-04T09:49:59.2812235Z a514bd1add31: Pull complete 2025-12-04T09:49:59.6322916Z 57b84ee60002: Pull complete 2025-12-04T09:49:59.9298778Z b8babeff6d81: Pull complete 2025-12-04T09:50:00.1379525Z 83779ddf6a85: Pull complete 2025-12-04T09:50:00.4541596Z 8b7620c0d736: Pull complete 2025-12-04T09:50:00.8218720Z 3bcfa090e4ef: Pull complete 2025-12-04T09:50:01.0270907Z eb0504ec4d92: Pull complete 2025-12-04T09:50:01.3801087Z 15d0fec09d7b: Pull complete 2025-12-04T09:50:01.5857532Z cca81fcc62a9: Pull complete 2025-12-04T09:50:01.9658748Z b0b8f9b5c6ab: Pull complete 2025-12-04T09:50:02.1532130Z 0606ca4d47a8: Pull complete 2025-12-04T09:50:02.5395544Z 2f80a4e1b3b9: Pull complete 2025-12-04T09:50:02.7545786Z 35c916fb1bd0: Pull complete 2025-12-04T09:50:08.0690012Z 195537b7dafc: Pull complete 2025-12-04T09:50:08.2924580Z dc454fd3967e: Pull complete 2025-12-04T09:50:08.5190166Z 701b34f115fa: Pull complete 2025-12-04T09:50:08.7039882Z 39cefc00ffed: Pull complete 2025-12-04T09:50:08.9321183Z 6ae51eb61a32: Pull complete 2025-12-04T09:50:09.1434062Z 1fd5341e66df: Pull complete 2025-12-04T09:50:10.5979747Z 72a7c87e35e4: Pull complete 2025-12-04T09:50:10.7967251Z ec36862ac98e: Pull complete 2025-12-04T09:50:11.9486673Z 05ddbf246e8a: Pull complete 2025-12-04T09:50:12.1016969Z Digest: sha256:ba21003510dba4bdeed83df81a56fa468e0ee1b612a9445ae1f402a280804f97 2025-12-04T09:50:12.1569121Z Status: Downloaded newer image for 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/ci-image:pytorch-linux-jammy-cuda12.8-cudnn9-py3-gcc11-f0cd68561080d537ef3d3d6f81b25a6416ad600a 2025-12-04T09:50:12.1806009Z 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/ci-image:pytorch-linux-jammy-cuda12.8-cudnn9-py3-gcc11-f0cd68561080d537ef3d3d6f81b25a6416ad600a 2025-12-04T09:50:12.1886503Z ##[group]Run echo "IN_CONTAINER_RUNNER=$(if [ -f /.inarc ] || [ -f /.incontainer ]; then echo true ; else echo false; fi)" >> "$GITHUB_OUTPUT" 2025-12-04T09:50:12.1887253Z echo "IN_CONTAINER_RUNNER=$(if [ -f /.inarc ] || [ -f /.incontainer ]; then echo true ; else echo false; fi)" >> "$GITHUB_OUTPUT" 2025-12-04T09:50:12.1897137Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-12-04T09:50:12.1897424Z env: 2025-12-04T09:50:12.1897584Z GIT_DEFAULT_BRANCH: main 2025-12-04T09:50:12.1897775Z ##[endgroup] 2025-12-04T09:50:12.2051070Z ##[group]Run pytorch/test-infra/.github/actions/setup-nvidia@main 2025-12-04T09:50:12.2051401Z with: 2025-12-04T09:50:12.2051585Z driver-version: 580.82.07 2025-12-04T09:50:12.2051769Z env: 2025-12-04T09:50:12.2051924Z GIT_DEFAULT_BRANCH: main 2025-12-04T09:50:12.2052111Z ##[endgroup] 2025-12-04T09:50:12.2135714Z ##[group]Run echo "IN_CONTAINER_RUNNER=$(if [ -f /.inarc ] || [ -f /.incontainer ]; then echo true ; else echo false; fi)" >> "$GITHUB_OUTPUT" 2025-12-04T09:50:12.2136374Z echo "IN_CONTAINER_RUNNER=$(if [ -f /.inarc ] || [ -f /.incontainer ]; then echo true ; else echo false; fi)" >> "$GITHUB_OUTPUT" 2025-12-04T09:50:12.2144031Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-12-04T09:50:12.2144586Z env: 2025-12-04T09:50:12.2144747Z GIT_DEFAULT_BRANCH: main 2025-12-04T09:50:12.2144931Z ##[endgroup] 2025-12-04T09:50:12.2221703Z ##[group]Run set -euo pipefail 2025-12-04T09:50:12.2221955Z set -euo pipefail 2025-12-04T09:50:12.2222176Z  2025-12-04T09:50:12.2222334Z has_gpu=false 2025-12-04T09:50:12.2222514Z devices="" 2025-12-04T09:50:12.2222684Z  2025-12-04T09:50:12.2222882Z if command -v nvidia-smi >/dev/null 2>&1; then 2025-12-04T09:50:12.2223216Z  if nvidia-smi -L >/tmp/nvidia_devices 2>/dev/null; then 2025-12-04T09:50:12.2223494Z  has_gpu=true 2025-12-04T09:50:12.2223805Z  devices=$(cat /tmp/nvidia_devices) 2025-12-04T09:50:12.2224025Z  fi 2025-12-04T09:50:12.2224181Z fi 2025-12-04T09:50:12.2224330Z  2025-12-04T09:50:12.2224488Z if [ "$has_gpu" = false ]; then 2025-12-04T09:50:12.2224771Z  if ls /dev/nvidia* >/tmp/nvidia_devices 2>/dev/null; then 2025-12-04T09:50:12.2225049Z  has_gpu=true 2025-12-04T09:50:12.2225278Z  devices=$(cat /tmp/nvidia_devices) 2025-12-04T09:50:12.2225523Z  fi 2025-12-04T09:50:12.2225673Z fi 2025-12-04T09:50:12.2225828Z  2025-12-04T09:50:12.2226064Z if [ "$has_gpu" = false ] && command -v lspci >/dev/null 2>&1; then 2025-12-04T09:50:12.2226434Z  if lspci | grep -i 'nvidia' >/tmp/nvidia_devices 2>/dev/null; then 2025-12-04T09:50:12.2226729Z  has_gpu=true 2025-12-04T09:50:12.2226939Z  devices=$(cat /tmp/nvidia_devices) 2025-12-04T09:50:12.2227155Z  fi 2025-12-04T09:50:12.2227309Z fi 2025-12-04T09:50:12.2227474Z  2025-12-04T09:50:12.2227691Z printf 'HAS_NVIDIA=%s\n' "$has_gpu" >> "$GITHUB_OUTPUT" 2025-12-04T09:50:12.2228074Z printf 'DETECTED_DEVICES<> "$GITHUB_OUTPUT" 2025-12-04T09:50:12.2235185Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-12-04T09:50:12.2235460Z env: 2025-12-04T09:50:12.2235613Z GIT_DEFAULT_BRANCH: main 2025-12-04T09:50:12.2235801Z ##[endgroup] 2025-12-04T09:50:13.8722386Z ##[group]Run if [ "${HAS_NVIDIA}" = "true" ]; then 2025-12-04T09:50:13.8722689Z if [ "${HAS_NVIDIA}" = "true" ]; then 2025-12-04T09:50:13.8722950Z  echo "HAS_NVIDIA_GPU=true" >> "${GITHUB_ENV}" 2025-12-04T09:50:13.8723375Z  echo "GPU_FLAG=--gpus all -e NVIDIA_DRIVER_CAPABILITIES=all" >> "${GITHUB_ENV}" 2025-12-04T09:50:13.8723697Z else 2025-12-04T09:50:13.8723905Z  echo "HAS_NVIDIA_GPU=false" >> "${GITHUB_ENV}" 2025-12-04T09:50:13.8724145Z fi 2025-12-04T09:50:13.8732799Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-12-04T09:50:13.8733071Z env: 2025-12-04T09:50:13.8733244Z GIT_DEFAULT_BRANCH: main 2025-12-04T09:50:13.8733434Z HAS_NVIDIA: true 2025-12-04T09:50:13.8733603Z ##[endgroup] 2025-12-04T09:50:13.8851095Z ##[group]Run nick-fields/retry@3e91a01664abd3c5cd539100d10d33b9c5b68482 2025-12-04T09:50:13.8851396Z with: 2025-12-04T09:50:13.8851547Z timeout_minutes: 10 2025-12-04T09:50:13.8851724Z max_attempts: 3 2025-12-04T09:50:13.8870848Z command: # Is it disgusting to have a full shell script here in this github action? Sure # But is it the best way to make it so that this action relies on nothing else? Absolutely set -eou pipefail DISTRIBUTION=$(. /etc/os-release;echo $ID$VERSION_ID) DRIVER_FN="NVIDIA-Linux-x86_64-${DRIVER_VERSION}.run" install_nvidia_docker2_amzn2() { ( set -x # Needed for yum-config-manager sudo yum install -y yum-utils if [[ "${DISTRIBUTION}" == "amzn2023" ]] ; then YUM_REPO_URL="https://nvidia.github.io/libnvidia-container/stable/rpm/nvidia-container-toolkit.repo" else # Amazon Linux 2 YUM_REPO_URL="https://nvidia.github.io/nvidia-docker/${DISTRIBUTION}/nvidia-docker.repo" fi sudo yum-config-manager --add-repo "${YUM_REPO_URL}" sudo yum install -y \ nvidia-container-toolkit-1.17.8 \ libnvidia-container-tools-1.17.8 \ libnvidia-container1-1.17.8 \ nvidia-container-toolkit-base-1.17.8 sudo systemctl restart docker ) } install_nvidia_docker2_ubuntu20() { ( set -x # Install nvidia-driver package if not installed status="$(dpkg-query -W --showformat='${db:Status-Status}' nvidia-docker2 2>&1)" if [ ! $? = 0 ] || [ ! "$status" = installed ]; then sudo apt-get install -y nvidia-container-toolkit-1.17.8 sudo systemctl restart docker fi ) } pre_install_nvidia_driver_amzn2() { ( # Purge any nvidia driver installed from RHEL repo sudo yum remove -y nvidia-driver-latest-dkms ) } install_nvidia_driver_common() { ( # Try to gather more information about the runner and its existing NVIDIA driver if any echo "Before installing NVIDIA driver" lspci lsmod modinfo nvidia || true HAS_NVIDIA_DRIVER=0 # Check if NVIDIA driver has already been installed if [ -x "$(command -v nvidia-smi)" ]; then set +e # The driver exists, check its version next. Also check only the first GPU if there are more than one of them # so that the same driver version is not print over multiple lines INSTALLED_DRIVER_VERSION=$(nvidia-smi --query-gpu=driver_version --format=csv,noheader --id=0) NVIDIA_SMI_STATUS=$? if [ "$NVIDIA_SMI_STATUS" -ne 0 ] && [ "$NVIDIA_SMI_STATUS" -ne 14 ]; then echo "Failed to get NVIDIA driver version ($INSTALLED_DRIVER_VERSION). Continuing" elif [ "$INSTALLED_DRIVER_VERSION" != "$DRIVER_VERSION" ]; then echo "NVIDIA driver ($INSTALLED_DRIVER_VERSION) has been installed, but we expect to have $DRIVER_VERSION instead. Continuing" # Turn off persistent mode so that the installation script can unload the kernel module sudo killall nvidia-persistenced || true else HAS_NVIDIA_DRIVER=1 echo "NVIDIA driver ($INSTALLED_DRIVER_VERSION) has already been installed. Skipping NVIDIA driver installation" fi set -e fi if [ "$HAS_NVIDIA_DRIVER" -eq 0 ]; then # CAUTION: this may need to be updated in future if [ "${DISTRIBUTION}" != ubuntu20.04 ]; then sudo yum groupinstall -y "Development Tools" # ensure our kernel install is the same as our underlying kernel, # groupinstall "Development Tools" has a habit of mismatching kernel headers sudo yum install -y "kernel-devel-uname-r == $(uname -r)" sudo modprobe backlight fi sudo curl -fsL -o /tmp/nvidia_driver "https://s3.amazonaws.com/ossci-linux/nvidia_driver/$DRIVER_FN" set +e sudo /bin/bash /tmp/nvidia_driver -s --no-drm NVIDIA_INSTALLATION_STATUS=$? RESET_GPU=0 if [ "$NVIDIA_INSTALLATION_STATUS" -ne 0 ]; then sudo cat /var/log/nvidia-installer.log # Fail to install NVIDIA driver, try to reset the GPU RESET_GPU=1 elif [ -x "$(command -v nvidia-smi)" ]; then # Check again if nvidia-smi works even if the driver installation completes successfully INSTALLED_DRIVER_VERSION=$(nvidia-smi --query-gpu=driver_version --format=csv,noheader --id=0) NVIDIA_SMI_STATUS=$? if [ "$NVIDIA_SMI_STATUS" -ne 0 ] && [ "$NVIDIA_SMI_STATUS" -ne 14 ]; then RESET_GPU=1 fi fi if [ "$RESET_GPU" -eq 1 ]; then NVIDIA_DEVICES=$(lspci -D | grep -i NVIDIA | cut -d' ' -f1) # The GPU can get stuck in a failure state if somehow the test crashs the GPU microcode. When this # happens, we'll try to reset all NVIDIA devices https://github.com/pytorch/pytorch/issues/88388 for PCI_ID in $NVIDIA_DEVICES; do DEVICE_ENABLED=$(cat /sys/bus/pci/devices/$PCI_ID/enable) echo "Reseting $PCI_ID (enabled state: $DEVICE_ENABLED)" # This requires sudo permission of course echo "1" | sudo tee /sys/bus/pci/devices/$PCI_ID/reset sleep 1 done fi sudo rm -fv /tmp/nvidia_driver set -e fi ) } post_install_nvidia_driver_common() { ( sudo modprobe nvidia || true echo "After installing NVIDIA driver" lspci lsmod modinfo nvidia || true ( set +e nvidia-smi # NB: Annoyingly, nvidia-smi command returns successfully with return code 0 even in # the case where the driver has already crashed as it still can get the driver version # and some basic information like the bus ID. However, the rest of the information # would be missing (ERR!), for example: # # +-----------------------------------------------------------------------------+ # | NVIDIA-SMI 525.89.02 Driver Version: 525.89.02 CUDA Version: 12.0 | # |-------------------------------+----------------------+----------------------+ # | GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC | # | Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. | # | | | MIG M. | # |===============================+======================+======================| # | 0 ERR! Off | 00000000:00:1E.0 Off | ERR! | # |ERR! ERR! ERR! ERR! / ERR! | 4184MiB / 23028MiB | ERR! Default | # | | | ERR! | # +-------------------------------+----------------------+----------------------+ # # +-----------------------------------------------------------------------------+ # | Processes: | # | GPU GI CI PID Type Process name GPU Memory | # | ID ID Usage | # |=============================================================================| # +-----------------------------------------------------------------------------+ # # This should be reported as a failure instead as it will guarantee to fail when # Docker tries to run with --gpus all # # So, the correct check here is to query one of the missing piece of info like # GPU name, so that the command can fail accordingly nvidia-smi --query-gpu=gpu_name --format=csv,noheader --id=0 NVIDIA_SMI_STATUS=$? # Allowable exit statuses for nvidia-smi, see: https://github.com/NVIDIA/gpu-operator/issues/285 if [ "$NVIDIA_SMI_STATUS" -eq 0 ] || [ "$NVIDIA_SMI_STATUS" -eq 14 ]; then echo "INFO: Ignoring allowed status ${NVIDIA_SMI_STATUS}" else echo "ERROR: nvidia-smi exited with unresolved status ${NVIDIA_SMI_STATUS}" exit ${NVIDIA_SMI_STATUS} fi set -e ) ) } install_nvidia_driver_amzn2() { ( set -x pre_install_nvidia_driver_amzn2 install_nvidia_driver_common post_install_nvidia_driver_common ) } install_nvidia_driver_ubuntu20() { ( set -x install_nvidia_driver_common post_install_nvidia_driver_common ) } echo "== Installing nvidia driver ${DRIVER_FN} ==" case "${DISTRIBUTION}" in amzn*) install_nvidia_driver_amzn2 ;; ubuntu20.04) install_nvidia_driver_ubuntu20 ;; *) echo "ERROR: Unknown distribution ${DISTRIBUTION}" exit 1 ;; esac # Install container toolkit based on distribution echo "== Installing nvidia container toolkit for ${DISTRIBUTION} ==" case "${DISTRIBUTION}" in amzn*) install_nvidia_docker2_amzn2 ;; ubuntu20.04) install_nvidia_docker2_ubuntu20 ;; *) echo "ERROR: Unknown distribution ${DISTRIBUTION}" exit 1 ;; esac # Fix https://github.com/NVIDIA/nvidia-docker/issues/1648 on runners with # more than one GPUs. This just needs to be run once. The command fails # on subsequent runs and complains that the mode is already on, but that's # ok sudo nvidia-persistenced || true # This should show persistence mode ON nvidia-smi # check if the container-toolkit is correctly installed and CUDA is available inside a container docker run --rm -t --gpus=all public.ecr.aws/docker/library/python:3.13 nvidia-smi 2025-12-04T09:50:13.8890261Z retry_wait_seconds: 10 2025-12-04T09:50:13.8890460Z polling_interval_seconds: 1 2025-12-04T09:50:13.8890654Z warning_on_retry: true 2025-12-04T09:50:13.8890848Z continue_on_error: false 2025-12-04T09:50:13.8891037Z env: 2025-12-04T09:50:13.8891183Z GIT_DEFAULT_BRANCH: main 2025-12-04T09:50:13.8891369Z HAS_NVIDIA_GPU: true 2025-12-04T09:50:13.8891602Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2025-12-04T09:50:13.8891858Z DRIVER_VERSION: 580.82.07 2025-12-04T09:50:13.8892041Z ##[endgroup] 2025-12-04T09:50:13.9984786Z == Installing nvidia driver NVIDIA-Linux-x86_64-580.82.07.run == 2025-12-04T09:50:13.9986007Z + pre_install_nvidia_driver_amzn2 2025-12-04T09:50:13.9987157Z + sudo yum remove -y nvidia-driver-latest-dkms 2025-12-04T09:50:14.6177740Z No match for argument: nvidia-driver-latest-dkms 2025-12-04T09:50:14.6178473Z No packages marked for removal. 2025-12-04T09:50:14.6235679Z Dependencies resolved. 2025-12-04T09:50:14.6244910Z Nothing to do. 2025-12-04T09:50:14.6245381Z Complete! 2025-12-04T09:50:14.6742411Z + install_nvidia_driver_common 2025-12-04T09:50:14.6745466Z + echo 'Before installing NVIDIA driver' 2025-12-04T09:50:14.6747061Z Before installing NVIDIA driver 2025-12-04T09:50:14.6749258Z + lspci 2025-12-04T09:50:14.7805409Z 00:00.0 Host bridge: Intel Corporation 440FX - 82441FX PMC [Natoma] 2025-12-04T09:50:14.7805930Z 00:01.0 ISA bridge: Intel Corporation 82371SB PIIX3 ISA [Natoma/Triton II] 2025-12-04T09:50:14.7806480Z 00:01.3 Non-VGA unclassified device: Intel Corporation 82371AB/EB/MB PIIX4 ACPI (rev 08) 2025-12-04T09:50:14.7806968Z 00:03.0 VGA compatible controller: Amazon.com, Inc. Device 1111 2025-12-04T09:50:14.7807431Z 00:04.0 Non-Volatile memory controller: Amazon.com, Inc. NVMe EBS Controller 2025-12-04T09:50:14.7807863Z 01:00.0 PCI bridge: Amazon.com, Inc. Device 0200 2025-12-04T09:50:14.7808190Z 02:00.0 PCI bridge: Amazon.com, Inc. Device 0200 2025-12-04T09:50:14.7808532Z 03:00.0 PCI bridge: Amazon.com, Inc. Device 0200 2025-12-04T09:50:14.7808838Z 03:00.1 PCI bridge: Amazon.com, Inc. Device 0200 2025-12-04T09:50:14.7809136Z 03:00.2 PCI bridge: Amazon.com, Inc. Device 0200 2025-12-04T09:50:14.7809695Z 03:00.3 PCI bridge: Amazon.com, Inc. Device 0200 2025-12-04T09:50:14.7810026Z 03:00.4 PCI bridge: Amazon.com, Inc. Device 0200 2025-12-04T09:50:14.7810363Z 03:00.5 PCI bridge: Amazon.com, Inc. Device 0200 2025-12-04T09:50:14.7810664Z 03:00.6 PCI bridge: Amazon.com, Inc. Device 0200 2025-12-04T09:50:14.7810967Z 03:00.7 PCI bridge: Amazon.com, Inc. Device 0200 2025-12-04T09:50:14.7811273Z 03:01.0 PCI bridge: Amazon.com, Inc. Device 0200 2025-12-04T09:50:14.7811570Z 03:01.1 PCI bridge: Amazon.com, Inc. Device 0200 2025-12-04T09:50:14.7811814Z 03:01.2 PCI bridge: Amazon.com, Inc. Device 0200 2025-12-04T09:50:14.7812057Z 03:01.3 PCI bridge: Amazon.com, Inc. Device 0200 2025-12-04T09:50:14.7812300Z 03:01.4 PCI bridge: Amazon.com, Inc. Device 0200 2025-12-04T09:50:14.7812696Z 03:01.5 PCI bridge: Amazon.com, Inc. Device 0200 2025-12-04T09:50:14.7812939Z 03:01.6 PCI bridge: Amazon.com, Inc. Device 0200 2025-12-04T09:50:14.7813193Z 03:01.7 PCI bridge: Amazon.com, Inc. Device 0200 2025-12-04T09:50:14.7813439Z 03:02.0 PCI bridge: Amazon.com, Inc. Device 0200 2025-12-04T09:50:14.7813695Z 03:02.1 PCI bridge: Amazon.com, Inc. Device 0200 2025-12-04T09:50:14.7813941Z 03:02.2 PCI bridge: Amazon.com, Inc. Device 0200 2025-12-04T09:50:14.7814179Z 03:02.3 PCI bridge: Amazon.com, Inc. Device 0200 2025-12-04T09:50:14.7814421Z 03:02.4 PCI bridge: Amazon.com, Inc. Device 0200 2025-12-04T09:50:14.7814665Z 03:02.5 PCI bridge: Amazon.com, Inc. Device 0200 2025-12-04T09:50:14.7814908Z 03:02.6 PCI bridge: Amazon.com, Inc. Device 0200 2025-12-04T09:50:14.7815145Z 03:02.7 PCI bridge: Amazon.com, Inc. Device 0200 2025-12-04T09:50:14.7815390Z 03:03.0 PCI bridge: Amazon.com, Inc. Device 0200 2025-12-04T09:50:14.7815631Z 03:03.1 PCI bridge: Amazon.com, Inc. Device 0200 2025-12-04T09:50:14.7815874Z 03:03.2 PCI bridge: Amazon.com, Inc. Device 0200 2025-12-04T09:50:14.7816125Z 03:03.3 PCI bridge: Amazon.com, Inc. Device 0200 2025-12-04T09:50:14.7816371Z 03:03.4 PCI bridge: Amazon.com, Inc. Device 0200 2025-12-04T09:50:14.7816623Z 03:03.5 PCI bridge: Amazon.com, Inc. Device 0200 2025-12-04T09:50:14.7816864Z 03:03.6 PCI bridge: Amazon.com, Inc. Device 0200 2025-12-04T09:50:14.7817101Z 03:03.7 PCI bridge: Amazon.com, Inc. Device 0200 2025-12-04T09:50:14.7817345Z 24:00.0 PCI bridge: Amazon.com, Inc. Device 0200 2025-12-04T09:50:14.7817593Z 25:00.0 PCI bridge: Amazon.com, Inc. Device 0200 2025-12-04T09:50:14.7817837Z 26:00.0 PCI bridge: Amazon.com, Inc. Device 0200 2025-12-04T09:50:14.7818076Z 26:00.1 PCI bridge: Amazon.com, Inc. Device 0200 2025-12-04T09:50:14.7818308Z 26:00.2 PCI bridge: Amazon.com, Inc. Device 0200 2025-12-04T09:50:14.7818547Z 26:00.3 PCI bridge: Amazon.com, Inc. Device 0200 2025-12-04T09:50:14.7818845Z 26:00.4 PCI bridge: Amazon.com, Inc. Device 0200 2025-12-04T09:50:14.7819136Z 26:00.5 PCI bridge: Amazon.com, Inc. Device 0200 2025-12-04T09:50:14.7819396Z 26:00.6 PCI bridge: Amazon.com, Inc. Device 0200 2025-12-04T09:50:14.7819639Z 26:00.7 PCI bridge: Amazon.com, Inc. Device 0200 2025-12-04T09:50:14.7819891Z 26:01.0 PCI bridge: Amazon.com, Inc. Device 0200 2025-12-04T09:50:14.7820213Z 27:00.0 Ethernet controller: Amazon.com, Inc. Elastic Network Adapter (ENA) 2025-12-04T09:50:14.7820541Z 30:00.0 PCI bridge: Amazon.com, Inc. Device 0200 2025-12-04T09:50:14.7820788Z 31:00.0 PCI bridge: Amazon.com, Inc. Device 0200 2025-12-04T09:50:14.7821057Z 32:00.0 PCI bridge: Amazon.com, Inc. Device 0200 2025-12-04T09:50:14.7821374Z 33:00.0 Non-Volatile memory controller: Amazon.com, Inc. NVMe SSD Controller 2025-12-04T09:50:14.7821700Z 34:00.0 PCI bridge: Amazon.com, Inc. Device 0200 2025-12-04T09:50:14.7821992Z 35:00.0 3D controller: NVIDIA Corporation AD104GL [L4] (rev a1) 2025-12-04T09:50:14.7822249Z + lsmod 2025-12-04T09:50:14.7854838Z Module Size Used by 2025-12-04T09:50:14.7855133Z nvidia_uvm 1925120 0 2025-12-04T09:50:14.7855386Z nvidia 14286848 1 nvidia_uvm 2025-12-04T09:50:14.7855663Z drm 602112 1 nvidia 2025-12-04T09:50:14.7855957Z drm_panel_orientation_quirks 32768 1 drm 2025-12-04T09:50:14.7856401Z backlight 24576 1 drm 2025-12-04T09:50:14.7856679Z i2c_core 110592 2 nvidia,drm 2025-12-04T09:50:14.7856949Z xt_conntrack 16384 1 2025-12-04T09:50:14.7857186Z nft_chain_nat 16384 3 2025-12-04T09:50:14.7857408Z xt_MASQUERADE 20480 1 2025-12-04T09:50:14.7857694Z nf_nat 57344 2 nft_chain_nat,xt_MASQUERADE 2025-12-04T09:50:14.7858026Z nf_conntrack_netlink 57344 0 2025-12-04T09:50:14.7858395Z nf_conntrack 184320 4 xt_conntrack,nf_nat,nf_conntrack_netlink,xt_MASQUERADE 2025-12-04T09:50:14.7858794Z nf_defrag_ipv6 24576 1 nf_conntrack 2025-12-04T09:50:14.7859079Z nf_defrag_ipv4 16384 1 nf_conntrack 2025-12-04T09:50:14.7859448Z xfrm_user 57344 1 2025-12-04T09:50:14.7859686Z xfrm_algo 16384 1 xfrm_user 2025-12-04T09:50:14.7859937Z xt_addrtype 16384 2 2025-12-04T09:50:14.7860162Z nft_compat 20480 4 2025-12-04T09:50:14.7860440Z nf_tables 311296 57 nft_compat,nft_chain_nat 2025-12-04T09:50:14.7860810Z nfnetlink 20480 4 nft_compat,nf_conntrack_netlink,nf_tables 2025-12-04T09:50:14.7861155Z br_netfilter 36864 0 2025-12-04T09:50:14.7861373Z bridge 323584 1 br_netfilter 2025-12-04T09:50:14.7861583Z stp 16384 1 bridge 2025-12-04T09:50:14.7861795Z llc 16384 2 bridge,stp 2025-12-04T09:50:14.7861999Z overlay 167936 0 2025-12-04T09:50:14.7862169Z tls 139264 0 2025-12-04T09:50:14.7862359Z nls_ascii 16384 1 2025-12-04T09:50:14.7862537Z nls_cp437 20480 1 2025-12-04T09:50:14.7862710Z vfat 24576 1 2025-12-04T09:50:14.7862892Z fat 86016 1 vfat 2025-12-04T09:50:14.7863101Z sunrpc 700416 1 2025-12-04T09:50:14.7863351Z i8042 45056 0 2025-12-04T09:50:14.7863631Z ena 184320 0 2025-12-04T09:50:14.7863822Z serio 28672 3 i8042 2025-12-04T09:50:14.7864025Z ghash_clmulni_intel 16384 0 2025-12-04T09:50:14.7864212Z button 24576 0 2025-12-04T09:50:14.7864392Z sch_fq_codel 20480 9 2025-12-04T09:50:14.7864572Z dm_mod 188416 0 2025-12-04T09:50:14.7864742Z fuse 184320 1 2025-12-04T09:50:14.7864915Z loop 36864 0 2025-12-04T09:50:14.7865092Z configfs 57344 1 2025-12-04T09:50:14.7865280Z dmi_sysfs 20480 0 2025-12-04T09:50:14.7865462Z crc32_pclmul 16384 0 2025-12-04T09:50:14.7865646Z crc32c_intel 24576 0 2025-12-04T09:50:14.7865828Z efivarfs 24576 1 2025-12-04T09:50:14.7866001Z + modinfo nvidia 2025-12-04T09:50:14.7876094Z filename: /lib/modules/6.1.150-174.273.amzn2023.x86_64/kernel/drivers/video/nvidia.ko 2025-12-04T09:50:14.7876530Z import_ns: DMA_BUF 2025-12-04T09:50:14.7876755Z alias: char-major-195-* 2025-12-04T09:50:14.7877011Z version: 580.82.07 2025-12-04T09:50:14.7877258Z supported: external 2025-12-04T09:50:14.7877492Z license: Dual MIT/GPL 2025-12-04T09:50:14.7877747Z firmware: nvidia/580.82.07/gsp_tu10x.bin 2025-12-04T09:50:14.7878435Z firmware: nvidia/580.82.07/gsp_ga10x.bin 2025-12-04T09:50:14.7878737Z srcversion: BA7240A71DCF7DC6FE88C1D 2025-12-04T09:50:14.7879035Z alias: of:N*T*Cnvidia,tegra264-displayC* 2025-12-04T09:50:14.7879375Z alias: of:N*T*Cnvidia,tegra264-display 2025-12-04T09:50:14.7879696Z alias: of:N*T*Cnvidia,tegra234-displayC* 2025-12-04T09:50:14.7880016Z alias: of:N*T*Cnvidia,tegra234-display 2025-12-04T09:50:14.7880324Z alias: pci:v000010DEd*sv*sd*bc06sc80i00* 2025-12-04T09:50:14.7880634Z alias: pci:v000010DEd*sv*sd*bc03sc02i00* 2025-12-04T09:50:14.7880922Z alias: pci:v000010DEd*sv*sd*bc03sc00i00* 2025-12-04T09:50:14.7881203Z depends: i2c-core,drm 2025-12-04T09:50:14.7881452Z retpoline: Y 2025-12-04T09:50:14.7881824Z name: nvidia 2025-12-04T09:50:14.7882104Z vermagic: 6.1.150-174.273.amzn2023.x86_64 SMP preempt mod_unload modversions 2025-12-04T09:50:14.7882473Z parm: NvSwitchRegDwords:NvSwitch regkey (charp) 2025-12-04T09:50:14.7882804Z parm: NvSwitchBlacklist:NvSwitchBlacklist=uuid[,uuid...] (charp) 2025-12-04T09:50:14.7883204Z parm: NVreg_ResmanDebugLevel:int 2025-12-04T09:50:14.7883435Z parm: NVreg_RmLogonRC:int 2025-12-04T09:50:14.7883654Z parm: NVreg_ModifyDeviceFiles:int 2025-12-04T09:50:14.7883876Z parm: NVreg_DeviceFileUID:int 2025-12-04T09:50:14.7884094Z parm: NVreg_DeviceFileGID:int 2025-12-04T09:50:14.7884311Z parm: NVreg_DeviceFileMode:int 2025-12-04T09:50:14.7884722Z parm: NVreg_InitializeSystemMemoryAllocations:int 2025-12-04T09:50:14.7884999Z parm: NVreg_UsePageAttributeTable:int 2025-12-04T09:50:14.7885249Z parm: NVreg_EnablePCIeGen3:int 2025-12-04T09:50:14.7885477Z parm: NVreg_EnableMSI:int 2025-12-04T09:50:14.7885694Z parm: NVreg_EnableStreamMemOPs:int 2025-12-04T09:50:14.7885956Z parm: NVreg_RestrictProfilingToAdminUsers:int 2025-12-04T09:50:14.7886248Z parm: NVreg_PreserveVideoMemoryAllocations:int 2025-12-04T09:50:14.7886516Z parm: NVreg_EnableS0ixPowerManagement:int 2025-12-04T09:50:14.7886814Z parm: NVreg_S0ixPowerManagementVideoMemoryThreshold:int 2025-12-04T09:50:14.7887120Z parm: NVreg_DynamicPowerManagement:int 2025-12-04T09:50:14.7887425Z parm: NVreg_DynamicPowerManagementVideoMemoryThreshold:int 2025-12-04T09:50:14.7887714Z parm: NVreg_EnableGpuFirmware:int 2025-12-04T09:50:14.7887968Z parm: NVreg_EnableGpuFirmwareLogs:int 2025-12-04T09:50:14.7888245Z parm: NVreg_OpenRmEnableUnsupportedGpus:int 2025-12-04T09:50:14.7888510Z parm: NVreg_EnableUserNUMAManagement:int 2025-12-04T09:50:14.7888758Z parm: NVreg_MemoryPoolSize:int 2025-12-04T09:50:14.7889004Z parm: NVreg_KMallocHeapMaxSize:int 2025-12-04T09:50:14.7889240Z parm: NVreg_VMallocHeapMaxSize:int 2025-12-04T09:50:14.7889474Z parm: NVreg_IgnoreMMIOCheck:int 2025-12-04T09:50:14.7889706Z parm: NVreg_NvLinkDisable:int 2025-12-04T09:50:14.7889964Z parm: NVreg_EnablePCIERelaxedOrderingMode:int 2025-12-04T09:50:14.7890220Z parm: NVreg_RegisterPCIDriver:int 2025-12-04T09:50:14.7890476Z parm: NVreg_RegisterPlatformDeviceDriver:int 2025-12-04T09:50:14.7890737Z parm: NVreg_EnableResizableBar:int 2025-12-04T09:50:14.7890973Z parm: NVreg_EnableDbgBreakpoint:int 2025-12-04T09:50:14.7891222Z parm: NVreg_EnableNonblockingOpen:int 2025-12-04T09:50:14.7891488Z parm: NVreg_CoherentGPUMemoryMode:charp 2025-12-04T09:50:14.7891727Z parm: NVreg_RegistryDwords:charp 2025-12-04T09:50:14.7891975Z parm: NVreg_RegistryDwordsPerDevice:charp 2025-12-04T09:50:14.7892215Z parm: NVreg_RmMsg:charp 2025-12-04T09:50:14.7892423Z parm: NVreg_GpuBlacklist:charp 2025-12-04T09:50:14.7892658Z parm: NVreg_TemporaryFilePath:charp 2025-12-04T09:50:14.7892903Z parm: NVreg_ExcludedGpus:charp 2025-12-04T09:50:14.7893127Z parm: NVreg_DmaRemapPeerMmio:int 2025-12-04T09:50:14.7893354Z parm: NVreg_RmNvlinkBandwidth:charp 2025-12-04T09:50:14.7893609Z parm: NVreg_RmNvlinkBandwidthLinkCount:int 2025-12-04T09:50:14.7893856Z parm: NVreg_ImexChannelCount:int 2025-12-04T09:50:14.7894082Z parm: NVreg_CreateImexChannel0:int 2025-12-04T09:50:14.7894330Z parm: NVreg_GrdmaPciTopoCheckOverride:int 2025-12-04T09:50:14.7894570Z parm: rm_firmware_active:charp 2025-12-04T09:50:14.7894787Z + HAS_NVIDIA_DRIVER=0 2025-12-04T09:50:14.7894980Z ++ command -v nvidia-smi 2025-12-04T09:50:14.7895171Z + '[' -x /usr/bin/nvidia-smi ']' 2025-12-04T09:50:14.7895356Z + set +e 2025-12-04T09:50:14.7895693Z ++ nvidia-smi --query-gpu=driver_version --format=csv,noheader --id=0 2025-12-04T09:50:16.4259519Z + INSTALLED_DRIVER_VERSION=580.82.07 2025-12-04T09:50:16.4259896Z + NVIDIA_SMI_STATUS=0 2025-12-04T09:50:16.4260159Z + '[' 0 -ne 0 ']' 2025-12-04T09:50:16.4260373Z + '[' 580.82.07 '!=' 580.82.07 ']' 2025-12-04T09:50:16.4260637Z + HAS_NVIDIA_DRIVER=1 2025-12-04T09:50:16.4261048Z + echo 'NVIDIA driver (580.82.07) has already been installed. Skipping NVIDIA driver installation' 2025-12-04T09:50:16.4261503Z + set -e 2025-12-04T09:50:16.4261685Z + '[' 1 -eq 0 ']' 2025-12-04T09:50:16.4262037Z NVIDIA driver (580.82.07) has already been installed. Skipping NVIDIA driver installation 2025-12-04T09:50:16.4262393Z + post_install_nvidia_driver_common 2025-12-04T09:50:16.4265656Z + sudo modprobe nvidia 2025-12-04T09:50:16.5874383Z + echo 'After installing NVIDIA driver' 2025-12-04T09:50:16.5874800Z + lspci 2025-12-04T09:50:16.5875085Z After installing NVIDIA driver 2025-12-04T09:50:16.6050298Z 00:00.0 Host bridge: Intel Corporation 440FX - 82441FX PMC [Natoma] 2025-12-04T09:50:16.6050860Z 00:01.0 ISA bridge: Intel Corporation 82371SB PIIX3 ISA [Natoma/Triton II] 2025-12-04T09:50:16.6051573Z 00:01.3 Non-VGA unclassified device: Intel Corporation 82371AB/EB/MB PIIX4 ACPI (rev 08) 2025-12-04T09:50:16.6052095Z 00:03.0 VGA compatible controller: Amazon.com, Inc. Device 1111 2025-12-04T09:50:16.6052562Z 00:04.0 Non-Volatile memory controller: Amazon.com, Inc. NVMe EBS Controller 2025-12-04T09:50:16.6052984Z 01:00.0 PCI bridge: Amazon.com, Inc. Device 0200 2025-12-04T09:50:16.6053306Z 02:00.0 PCI bridge: Amazon.com, Inc. Device 0200 2025-12-04T09:50:16.6053624Z 03:00.0 PCI bridge: Amazon.com, Inc. Device 0200 2025-12-04T09:50:16.6053952Z 03:00.1 PCI bridge: Amazon.com, Inc. Device 0200 2025-12-04T09:50:16.6054293Z 03:00.2 PCI bridge: Amazon.com, Inc. Device 0200 2025-12-04T09:50:16.6054722Z 03:00.3 PCI bridge: Amazon.com, Inc. Device 0200 2025-12-04T09:50:16.6055026Z 03:00.4 PCI bridge: Amazon.com, Inc. Device 0200 2025-12-04T09:50:16.6055327Z 03:00.5 PCI bridge: Amazon.com, Inc. Device 0200 2025-12-04T09:50:16.6055630Z 03:00.6 PCI bridge: Amazon.com, Inc. Device 0200 2025-12-04T09:50:16.6055940Z 03:00.7 PCI bridge: Amazon.com, Inc. Device 0200 2025-12-04T09:50:16.6056246Z 03:01.0 PCI bridge: Amazon.com, Inc. Device 0200 2025-12-04T09:50:16.6056542Z 03:01.1 PCI bridge: Amazon.com, Inc. Device 0200 2025-12-04T09:50:16.6056842Z 03:01.2 PCI bridge: Amazon.com, Inc. Device 0200 2025-12-04T09:50:16.6057147Z 03:01.3 PCI bridge: Amazon.com, Inc. Device 0200 2025-12-04T09:50:16.6057558Z 03:01.4 PCI bridge: Amazon.com, Inc. Device 0200 2025-12-04T09:50:16.6057862Z 03:01.5 PCI bridge: Amazon.com, Inc. Device 0200 2025-12-04T09:50:16.6058162Z 03:01.6 PCI bridge: Amazon.com, Inc. Device 0200 2025-12-04T09:50:16.6058464Z 03:01.7 PCI bridge: Amazon.com, Inc. Device 0200 2025-12-04T09:50:16.6058765Z 03:02.0 PCI bridge: Amazon.com, Inc. Device 0200 2025-12-04T09:50:16.6059083Z 03:02.1 PCI bridge: Amazon.com, Inc. Device 0200 2025-12-04T09:50:16.6059386Z 03:02.2 PCI bridge: Amazon.com, Inc. Device 0200 2025-12-04T09:50:16.6059688Z 03:02.3 PCI bridge: Amazon.com, Inc. Device 0200 2025-12-04T09:50:16.6059987Z 03:02.4 PCI bridge: Amazon.com, Inc. Device 0200 2025-12-04T09:50:16.6060289Z 03:02.5 PCI bridge: Amazon.com, Inc. Device 0200 2025-12-04T09:50:16.6060583Z 03:02.6 PCI bridge: Amazon.com, Inc. Device 0200 2025-12-04T09:50:16.6060887Z 03:02.7 PCI bridge: Amazon.com, Inc. Device 0200 2025-12-04T09:50:16.6061200Z 03:03.0 PCI bridge: Amazon.com, Inc. Device 0200 2025-12-04T09:50:16.6061499Z 03:03.1 PCI bridge: Amazon.com, Inc. Device 0200 2025-12-04T09:50:16.6061807Z 03:03.2 PCI bridge: Amazon.com, Inc. Device 0200 2025-12-04T09:50:16.6062051Z 03:03.3 PCI bridge: Amazon.com, Inc. Device 0200 2025-12-04T09:50:16.6062295Z 03:03.4 PCI bridge: Amazon.com, Inc. Device 0200 2025-12-04T09:50:16.6062552Z 03:03.5 PCI bridge: Amazon.com, Inc. Device 0200 2025-12-04T09:50:16.6062795Z 03:03.6 PCI bridge: Amazon.com, Inc. Device 0200 2025-12-04T09:50:16.6063041Z 03:03.7 PCI bridge: Amazon.com, Inc. Device 0200 2025-12-04T09:50:16.6063583Z 24:00.0 PCI bridge: Amazon.com, Inc. Device 0200 2025-12-04T09:50:16.6063882Z 25:00.0 PCI bridge: Amazon.com, Inc. Device 0200 2025-12-04T09:50:16.6064138Z 26:00.0 PCI bridge: Amazon.com, Inc. Device 0200 2025-12-04T09:50:16.6064395Z 26:00.1 PCI bridge: Amazon.com, Inc. Device 0200 2025-12-04T09:50:16.6064653Z 26:00.2 PCI bridge: Amazon.com, Inc. Device 0200 2025-12-04T09:50:16.6064899Z 26:00.3 PCI bridge: Amazon.com, Inc. Device 0200 2025-12-04T09:50:16.6065144Z 26:00.4 PCI bridge: Amazon.com, Inc. Device 0200 2025-12-04T09:50:16.6065384Z 26:00.5 PCI bridge: Amazon.com, Inc. Device 0200 2025-12-04T09:50:16.6065635Z 26:00.6 PCI bridge: Amazon.com, Inc. Device 0200 2025-12-04T09:50:16.6065884Z 26:00.7 PCI bridge: Amazon.com, Inc. Device 0200 2025-12-04T09:50:16.6066276Z 26:01.0 PCI bridge: Amazon.com, Inc. Device 0200 2025-12-04T09:50:16.6066600Z 27:00.0 Ethernet controller: Amazon.com, Inc. Elastic Network Adapter (ENA) 2025-12-04T09:50:16.6066929Z 30:00.0 PCI bridge: Amazon.com, Inc. Device 0200 2025-12-04T09:50:16.6067181Z 31:00.0 PCI bridge: Amazon.com, Inc. Device 0200 2025-12-04T09:50:16.6067422Z 32:00.0 PCI bridge: Amazon.com, Inc. Device 0200 2025-12-04T09:50:16.6067753Z 33:00.0 Non-Volatile memory controller: Amazon.com, Inc. NVMe SSD Controller 2025-12-04T09:50:16.6068083Z 34:00.0 PCI bridge: Amazon.com, Inc. Device 0200 2025-12-04T09:50:16.6068358Z 35:00.0 3D controller: NVIDIA Corporation AD104GL [L4] (rev a1) 2025-12-04T09:50:16.6068614Z + lsmod 2025-12-04T09:50:16.6089318Z Module Size Used by 2025-12-04T09:50:16.6089578Z nvidia_uvm 1925120 0 2025-12-04T09:50:16.6089797Z nvidia 14286848 1 nvidia_uvm 2025-12-04T09:50:16.6090032Z drm 602112 1 nvidia 2025-12-04T09:50:16.6090277Z drm_panel_orientation_quirks 32768 1 drm 2025-12-04T09:50:16.6090510Z backlight 24576 1 drm 2025-12-04T09:50:16.6090728Z i2c_core 110592 2 nvidia,drm 2025-12-04T09:50:16.6090946Z xt_conntrack 16384 1 2025-12-04T09:50:16.6091146Z nft_chain_nat 16384 3 2025-12-04T09:50:16.6091334Z xt_MASQUERADE 20480 1 2025-12-04T09:50:16.6091552Z nf_nat 57344 2 nft_chain_nat,xt_MASQUERADE 2025-12-04T09:50:16.6091852Z nf_conntrack_netlink 57344 0 2025-12-04T09:50:16.6092238Z nf_conntrack 184320 4 xt_conntrack,nf_nat,nf_conntrack_netlink,xt_MASQUERADE 2025-12-04T09:50:16.6093454Z nf_defrag_ipv6 24576 1 nf_conntrack 2025-12-04T09:50:16.6094594Z nf_defrag_ipv4 16384 1 nf_conntrack 2025-12-04T09:50:16.6095487Z xfrm_user 57344 1 2025-12-04T09:50:16.6096039Z xfrm_algo 16384 1 xfrm_user 2025-12-04T09:50:16.6096769Z xt_addrtype 16384 2 2025-12-04T09:50:16.6097554Z nft_compat 20480 4 2025-12-04T09:50:16.6098498Z nf_tables 311296 57 nft_compat,nft_chain_nat 2025-12-04T09:50:16.6099591Z nfnetlink 20480 4 nft_compat,nf_conntrack_netlink,nf_tables 2025-12-04T09:50:16.6100266Z br_netfilter 36864 0 2025-12-04T09:50:16.6100760Z bridge 323584 1 br_netfilter 2025-12-04T09:50:16.6101284Z stp 16384 1 bridge 2025-12-04T09:50:16.6101855Z llc 16384 2 bridge,stp 2025-12-04T09:50:16.6102434Z overlay 167936 0 2025-12-04T09:50:16.6102950Z tls 139264 0 2025-12-04T09:50:16.6103488Z nls_ascii 16384 1 2025-12-04T09:50:16.6104492Z nls_cp437 20480 1 2025-12-04T09:50:16.6105394Z vfat 24576 1 2025-12-04T09:50:16.6106303Z fat 86016 1 vfat 2025-12-04T09:50:16.6107250Z sunrpc 700416 1 2025-12-04T09:50:16.6107732Z i8042 45056 0 2025-12-04T09:50:16.6108107Z ena 184320 0 2025-12-04T09:50:16.6108294Z serio 28672 3 i8042 2025-12-04T09:50:16.6108499Z ghash_clmulni_intel 16384 0 2025-12-04T09:50:16.6108689Z button 24576 0 2025-12-04T09:50:16.6108873Z sch_fq_codel 20480 9 2025-12-04T09:50:16.6109235Z dm_mod 188416 0 2025-12-04T09:50:16.6109445Z fuse 184320 1 2025-12-04T09:50:16.6109634Z loop 36864 0 2025-12-04T09:50:16.6109805Z configfs 57344 1 2025-12-04T09:50:16.6109986Z dmi_sysfs 20480 0 2025-12-04T09:50:16.6110169Z crc32_pclmul 16384 0 2025-12-04T09:50:16.6110342Z crc32c_intel 24576 0 2025-12-04T09:50:16.6110533Z efivarfs 24576 1 2025-12-04T09:50:16.6110719Z + modinfo nvidia 2025-12-04T09:50:16.6116758Z filename: /lib/modules/6.1.150-174.273.amzn2023.x86_64/kernel/drivers/video/nvidia.ko 2025-12-04T09:50:16.6117119Z import_ns: DMA_BUF 2025-12-04T09:50:16.6117477Z alias: char-major-195-* 2025-12-04T09:50:16.6117678Z version: 580.82.07 2025-12-04T09:50:16.6117857Z supported: external 2025-12-04T09:50:16.6118053Z license: Dual MIT/GPL 2025-12-04T09:50:16.6118265Z firmware: nvidia/580.82.07/gsp_tu10x.bin 2025-12-04T09:50:16.6118518Z firmware: nvidia/580.82.07/gsp_ga10x.bin 2025-12-04T09:50:16.6118757Z srcversion: BA7240A71DCF7DC6FE88C1D 2025-12-04T09:50:16.6119008Z alias: of:N*T*Cnvidia,tegra264-displayC* 2025-12-04T09:50:16.6119267Z alias: of:N*T*Cnvidia,tegra264-display 2025-12-04T09:50:16.6119534Z alias: of:N*T*Cnvidia,tegra234-displayC* 2025-12-04T09:50:16.6119785Z alias: of:N*T*Cnvidia,tegra234-display 2025-12-04T09:50:16.6120037Z alias: pci:v000010DEd*sv*sd*bc06sc80i00* 2025-12-04T09:50:16.6120280Z alias: pci:v000010DEd*sv*sd*bc03sc02i00* 2025-12-04T09:50:16.6120533Z alias: pci:v000010DEd*sv*sd*bc03sc00i00* 2025-12-04T09:50:16.6120770Z depends: i2c-core,drm 2025-12-04T09:50:16.6121106Z retpoline: Y 2025-12-04T09:50:16.6121409Z name: nvidia 2025-12-04T09:50:16.6121879Z vermagic: 6.1.150-174.273.amzn2023.x86_64 SMP preempt mod_unload modversions 2025-12-04T09:50:16.6122340Z parm: NvSwitchRegDwords:NvSwitch regkey (charp) 2025-12-04T09:50:16.6122679Z parm: NvSwitchBlacklist:NvSwitchBlacklist=uuid[,uuid...] (charp) 2025-12-04T09:50:16.6122994Z parm: NVreg_ResmanDebugLevel:int 2025-12-04T09:50:16.6123223Z parm: NVreg_RmLogonRC:int 2025-12-04T09:50:16.6123438Z parm: NVreg_ModifyDeviceFiles:int 2025-12-04T09:50:16.6123666Z parm: NVreg_DeviceFileUID:int 2025-12-04T09:50:16.6123893Z parm: NVreg_DeviceFileGID:int 2025-12-04T09:50:16.6124105Z parm: NVreg_DeviceFileMode:int 2025-12-04T09:50:16.6124368Z parm: NVreg_InitializeSystemMemoryAllocations:int 2025-12-04T09:50:16.6124657Z parm: NVreg_UsePageAttributeTable:int 2025-12-04T09:50:16.6124899Z parm: NVreg_EnablePCIeGen3:int 2025-12-04T09:50:16.6125108Z parm: NVreg_EnableMSI:int 2025-12-04T09:50:16.6125334Z parm: NVreg_EnableStreamMemOPs:int 2025-12-04T09:50:16.6125597Z parm: NVreg_RestrictProfilingToAdminUsers:int 2025-12-04T09:50:16.6125881Z parm: NVreg_PreserveVideoMemoryAllocations:int 2025-12-04T09:50:16.6126159Z parm: NVreg_EnableS0ixPowerManagement:int 2025-12-04T09:50:16.6126460Z parm: NVreg_S0ixPowerManagementVideoMemoryThreshold:int 2025-12-04T09:50:16.6126749Z parm: NVreg_DynamicPowerManagement:int 2025-12-04T09:50:16.6127195Z parm: NVreg_DynamicPowerManagementVideoMemoryThreshold:int 2025-12-04T09:50:16.6127718Z parm: NVreg_EnableGpuFirmware:int 2025-12-04T09:50:16.6128151Z parm: NVreg_EnableGpuFirmwareLogs:int 2025-12-04T09:50:16.6128573Z parm: NVreg_OpenRmEnableUnsupportedGpus:int 2025-12-04T09:50:16.6128850Z parm: NVreg_EnableUserNUMAManagement:int 2025-12-04T09:50:16.6129103Z parm: NVreg_MemoryPoolSize:int 2025-12-04T09:50:16.6129328Z parm: NVreg_KMallocHeapMaxSize:int 2025-12-04T09:50:16.6129562Z parm: NVreg_VMallocHeapMaxSize:int 2025-12-04T09:50:16.6129805Z parm: NVreg_IgnoreMMIOCheck:int 2025-12-04T09:50:16.6130145Z parm: NVreg_NvLinkDisable:int 2025-12-04T09:50:16.6130407Z parm: NVreg_EnablePCIERelaxedOrderingMode:int 2025-12-04T09:50:16.6130669Z parm: NVreg_RegisterPCIDriver:int 2025-12-04T09:50:16.6130929Z parm: NVreg_RegisterPlatformDeviceDriver:int 2025-12-04T09:50:16.6131182Z parm: NVreg_EnableResizableBar:int 2025-12-04T09:50:16.6131425Z parm: NVreg_EnableDbgBreakpoint:int 2025-12-04T09:50:16.6131678Z parm: NVreg_EnableNonblockingOpen:int 2025-12-04T09:50:16.6131943Z parm: NVreg_CoherentGPUMemoryMode:charp 2025-12-04T09:50:16.6132203Z parm: NVreg_RegistryDwords:charp 2025-12-04T09:50:16.6132645Z parm: NVreg_RegistryDwordsPerDevice:charp 2025-12-04T09:50:16.6133205Z parm: NVreg_RmMsg:charp 2025-12-04T09:50:16.6133579Z parm: NVreg_GpuBlacklist:charp 2025-12-04T09:50:16.6133906Z parm: NVreg_TemporaryFilePath:charp 2025-12-04T09:50:16.6134140Z parm: NVreg_ExcludedGpus:charp 2025-12-04T09:50:16.6134383Z parm: NVreg_DmaRemapPeerMmio:int 2025-12-04T09:50:16.6134622Z parm: NVreg_RmNvlinkBandwidth:charp 2025-12-04T09:50:16.6134879Z parm: NVreg_RmNvlinkBandwidthLinkCount:int 2025-12-04T09:50:16.6135124Z parm: NVreg_ImexChannelCount:int 2025-12-04T09:50:16.6135358Z parm: NVreg_CreateImexChannel0:int 2025-12-04T09:50:16.6135611Z parm: NVreg_GrdmaPciTopoCheckOverride:int 2025-12-04T09:50:16.6135853Z parm: rm_firmware_active:charp 2025-12-04T09:50:16.6136054Z + set +e 2025-12-04T09:50:16.6136193Z + nvidia-smi 2025-12-04T09:50:18.0491482Z Thu Dec 4 09:50:18 2025 2025-12-04T09:50:18.0491897Z +-----------------------------------------------------------------------------------------+ 2025-12-04T09:50:18.0492431Z | NVIDIA-SMI 580.82.07 Driver Version: 580.82.07 CUDA Version: 13.0 | 2025-12-04T09:50:18.0492918Z +-----------------------------------------+------------------------+----------------------+ 2025-12-04T09:50:18.0493392Z | GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC | 2025-12-04T09:50:18.0493908Z | Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. | 2025-12-04T09:50:18.0494301Z | | | MIG M. | 2025-12-04T09:50:18.0494595Z |=========================================+========================+======================| 2025-12-04T09:50:18.0558771Z | 0 NVIDIA L4 Off | 00000000:35:00.0 Off | 0 | 2025-12-04T09:50:18.0559212Z | N/A 33C P0 29W / 72W | 0MiB / 23034MiB | 4% Default | 2025-12-04T09:50:18.0559580Z | | | N/A | 2025-12-04T09:50:18.0559945Z +-----------------------------------------+------------------------+----------------------+ 2025-12-04T09:50:18.0560218Z 2025-12-04T09:50:18.0560383Z +-----------------------------------------------------------------------------------------+ 2025-12-04T09:50:18.0560782Z | Processes: | 2025-12-04T09:50:18.0561181Z | GPU GI CI PID Type Process name GPU Memory | 2025-12-04T09:50:18.0561568Z | ID ID Usage | 2025-12-04T09:50:18.0561880Z |=========================================================================================| 2025-12-04T09:50:18.0563628Z | No running processes found | 2025-12-04T09:50:18.0563988Z +-----------------------------------------------------------------------------------------+ 2025-12-04T09:50:18.3851406Z + nvidia-smi --query-gpu=gpu_name --format=csv,noheader --id=0 2025-12-04T09:50:19.8427710Z NVIDIA L4 2025-12-04T09:50:20.0252703Z + NVIDIA_SMI_STATUS=0 2025-12-04T09:50:20.0253304Z + '[' 0 -eq 0 ']' 2025-12-04T09:50:20.0253571Z + echo 'INFO: Ignoring allowed status 0' 2025-12-04T09:50:20.0253860Z + set -e 2025-12-04T09:50:20.0254063Z INFO: Ignoring allowed status 0 2025-12-04T09:50:20.0261692Z == Installing nvidia container toolkit for amzn2023 == 2025-12-04T09:50:20.0265194Z + sudo yum install -y yum-utils 2025-12-04T09:50:20.4402731Z Last metadata expiration check: 0:08:03 ago on Thu Dec 4 09:42:17 2025. 2025-12-04T09:50:20.4629645Z Package dnf-utils-4.3.0-13.amzn2023.0.5.noarch is already installed. 2025-12-04T09:50:20.5035518Z Dependencies resolved. 2025-12-04T09:50:20.5275808Z Nothing to do. 2025-12-04T09:50:20.5276217Z Complete! 2025-12-04T09:50:20.5825716Z + [[ amzn2023 == \a\m\z\n\2\0\2\3 ]] 2025-12-04T09:50:20.5827431Z + YUM_REPO_URL=https://nvidia.github.io/libnvidia-container/stable/rpm/nvidia-container-toolkit.repo 2025-12-04T09:50:20.5829070Z + sudo yum-config-manager --add-repo https://nvidia.github.io/libnvidia-container/stable/rpm/nvidia-container-toolkit.repo 2025-12-04T09:50:20.9368103Z Adding repo from: https://nvidia.github.io/libnvidia-container/stable/rpm/nvidia-container-toolkit.repo 2025-12-04T09:50:20.9783066Z + sudo yum install -y nvidia-container-toolkit-1.17.8 libnvidia-container-tools-1.17.8 libnvidia-container1-1.17.8 nvidia-container-toolkit-base-1.17.8 2025-12-04T09:50:21.4875679Z nvidia-container-toolkit 25 kB/s | 833 B 00:00 2025-12-04T09:50:21.5539311Z Dependencies resolved. 2025-12-04T09:50:21.5767007Z ================================================================================ 2025-12-04T09:50:21.5767430Z Package Arch Version Repository Size 2025-12-04T09:50:21.5767848Z ================================================================================ 2025-12-04T09:50:21.5768241Z Downgrading: 2025-12-04T09:50:21.5768647Z libnvidia-container-tools x86_64 1.17.8-1 nvidia-container-toolkit 40 k 2025-12-04T09:50:21.5769183Z libnvidia-container1 x86_64 1.17.8-1 nvidia-container-toolkit 1.0 M 2025-12-04T09:50:21.5769720Z nvidia-container-toolkit x86_64 1.17.8-1 nvidia-container-toolkit 1.2 M 2025-12-04T09:50:21.5770255Z nvidia-container-toolkit-base x86_64 1.17.8-1 nvidia-container-toolkit 5.8 M 2025-12-04T09:50:21.5770580Z 2025-12-04T09:50:21.5770671Z Transaction Summary 2025-12-04T09:50:21.5770897Z ================================================================================ 2025-12-04T09:50:21.5771199Z Downgrade 4 Packages 2025-12-04T09:50:21.5771346Z 2025-12-04T09:50:21.5771455Z Total download size: 8.0 M 2025-12-04T09:50:21.5772283Z Downloading Packages: 2025-12-04T09:50:21.6564993Z (1/4): libnvidia-container-tools-1.17.8-1.x86_6 519 kB/s | 40 kB 00:00 2025-12-04T09:50:21.6794897Z (2/4): libnvidia-container1-1.17.8-1.x86_64.rpm 9.7 MB/s | 1.0 MB 00:00 2025-12-04T09:50:21.7101281Z (3/4): nvidia-container-toolkit-1.17.8-1.x86_64 9.4 MB/s | 1.2 MB 00:00 2025-12-04T09:50:21.8154943Z (4/4): nvidia-container-toolkit-base-1.17.8-1.x 36 MB/s | 5.8 MB 00:00 2025-12-04T09:50:21.8163796Z -------------------------------------------------------------------------------- 2025-12-04T09:50:21.8166827Z Total 34 MB/s | 8.0 MB 00:00 2025-12-04T09:50:21.8169524Z Running transaction check 2025-12-04T09:50:21.8290823Z Transaction check succeeded. 2025-12-04T09:50:21.8291467Z Running transaction test 2025-12-04T09:50:21.8730556Z Transaction test succeeded. 2025-12-04T09:50:21.8733539Z Running transaction 2025-12-04T09:50:22.5078009Z Preparing : 1/1 2025-12-04T09:50:22.6120085Z Downgrading : nvidia-container-toolkit-base-1.17.8-1.x86_64 1/8 2025-12-04T09:50:22.6361299Z Downgrading : libnvidia-container1-1.17.8-1.x86_64 2/8 2025-12-04T09:50:22.6856376Z Running scriptlet: libnvidia-container1-1.17.8-1.x86_64 2/8 2025-12-04T09:50:22.7875799Z Downgrading : libnvidia-container-tools-1.17.8-1.x86_64 3/8 2025-12-04T09:50:22.8014637Z Downgrading : nvidia-container-toolkit-1.17.8-1.x86_64 4/8 2025-12-04T09:50:22.8717784Z Running scriptlet: nvidia-container-toolkit-1.17.8-1.x86_64 4/8 2025-12-04T09:50:22.8772627Z Running scriptlet: nvidia-container-toolkit-1.18.1-1.x86_64 5/8 2025-12-04T09:50:22.8775001Z Cleanup : nvidia-container-toolkit-1.18.1-1.x86_64 5/8 2025-12-04T09:50:22.8996269Z Running scriptlet: nvidia-container-toolkit-1.18.1-1.x86_64 5/8 2025-12-04T09:50:22.9048119Z Running scriptlet: libnvidia-container-tools-1.18.1-1.x86_64 6/8 2025-12-04T09:50:22.9049033Z Cleanup : libnvidia-container-tools-1.18.1-1.x86_64 6/8 2025-12-04T09:50:22.9322388Z Running scriptlet: libnvidia-container-tools-1.18.1-1.x86_64 6/8 2025-12-04T09:50:22.9383946Z Running scriptlet: libnvidia-container1-1.18.1-1.x86_64 7/8 2025-12-04T09:50:22.9384823Z Cleanup : libnvidia-container1-1.18.1-1.x86_64 7/8 2025-12-04T09:50:22.9648583Z Running scriptlet: libnvidia-container1-1.18.1-1.x86_64 7/8 2025-12-04T09:50:22.9706936Z Running scriptlet: nvidia-container-toolkit-base-1.18.1-1.x86_64 8/8 2025-12-04T09:50:22.9707920Z Cleanup : nvidia-container-toolkit-base-1.18.1-1.x86_64 8/8 2025-12-04T09:50:22.9932775Z Running scriptlet: nvidia-container-toolkit-base-1.18.1-1.x86_64 8/8 2025-12-04T09:50:23.0371722Z Running scriptlet: nvidia-container-toolkit-1.17.8-1.x86_64 8/8 2025-12-04T09:51:08.7117849Z Running scriptlet: nvidia-container-toolkit-base-1.18.1-1.x86_64 8/8 2025-12-04T09:51:08.7122779Z Verifying : libnvidia-container-tools-1.17.8-1.x86_64 1/8 2025-12-04T09:51:08.7123376Z Verifying : libnvidia-container-tools-1.18.1-1.x86_64 2/8 2025-12-04T09:51:08.7123902Z Verifying : libnvidia-container1-1.17.8-1.x86_64 3/8 2025-12-04T09:51:08.7124400Z Verifying : libnvidia-container1-1.18.1-1.x86_64 4/8 2025-12-04T09:51:08.7124887Z Verifying : nvidia-container-toolkit-1.17.8-1.x86_64 5/8 2025-12-04T09:51:08.7125361Z Verifying : nvidia-container-toolkit-1.18.1-1.x86_64 6/8 2025-12-04T09:51:08.7125839Z Verifying : nvidia-container-toolkit-base-1.17.8-1.x86_64 7/8 2025-12-04T09:51:08.8542909Z Verifying : nvidia-container-toolkit-base-1.18.1-1.x86_64 8/8================================================================================ 2025-12-04T09:51:08.8543629Z WARNING: 2025-12-04T09:51:08.8543881Z A newer release of "Amazon Linux" is available. 2025-12-04T09:51:08.8544132Z 2025-12-04T09:51:08.8544224Z Available Versions: 2025-12-04T09:51:08.8544370Z 2025-12-04T09:51:08.8544483Z Version 2023.9.20250929: 2025-12-04T09:51:08.8544779Z Run the following command to upgrade to 2023.9.20250929: 2025-12-04T09:51:08.8545030Z 2025-12-04T09:51:08.8545157Z dnf upgrade --releasever=2023.9.20250929 2025-12-04T09:51:08.8545364Z 2025-12-04T09:51:08.8545453Z Release notes: 2025-12-04T09:51:08.8545877Z https://docs.aws.amazon.com/linux/al2023/release-notes/relnotes-2023.9.20250929.html 2025-12-04T09:51:08.8546251Z 2025-12-04T09:51:08.8546334Z Version 2023.9.20251014: 2025-12-04T09:51:08.8546649Z Run the following command to upgrade to 2023.9.20251014: 2025-12-04T09:51:08.8546893Z 2025-12-04T09:51:08.8547012Z dnf upgrade --releasever=2023.9.20251014 2025-12-04T09:51:08.8547210Z 2025-12-04T09:51:08.8547297Z Release notes: 2025-12-04T09:51:08.8547683Z https://docs.aws.amazon.com/linux/al2023/release-notes/relnotes-2023.9.20251014.html 2025-12-04T09:51:08.8548060Z 2025-12-04T09:51:08.8548145Z Version 2023.9.20251020: 2025-12-04T09:51:08.8548723Z Run the following command to upgrade to 2023.9.20251020: 2025-12-04T09:51:08.8548975Z 2025-12-04T09:51:08.8549064Z dnf upgrade --releasever=2023.9.20251020 2025-12-04T09:51:08.8549226Z 2025-12-04T09:51:08.8549290Z Release notes: 2025-12-04T09:51:08.8549599Z https://docs.aws.amazon.com/linux/al2023/release-notes/relnotes-2023.9.20251020.html 2025-12-04T09:51:08.8549880Z 2025-12-04T09:51:08.8549944Z Version 2023.9.20251027: 2025-12-04T09:51:08.8550175Z Run the following command to upgrade to 2023.9.20251027: 2025-12-04T09:51:08.8550368Z 2025-12-04T09:51:08.8550453Z dnf upgrade --releasever=2023.9.20251027 2025-12-04T09:51:08.8550607Z 2025-12-04T09:51:08.8550677Z Release notes: 2025-12-04T09:51:08.8551122Z https://docs.aws.amazon.com/linux/al2023/release-notes/relnotes-2023.9.20251027.html 2025-12-04T09:51:08.8551406Z 2025-12-04T09:51:08.8551473Z Version 2023.9.20251105: 2025-12-04T09:51:08.8551700Z Run the following command to upgrade to 2023.9.20251105: 2025-12-04T09:51:08.8551884Z 2025-12-04T09:51:08.8551976Z dnf upgrade --releasever=2023.9.20251105 2025-12-04T09:51:08.8552130Z 2025-12-04T09:51:08.8552193Z Release notes: 2025-12-04T09:51:08.8552492Z https://docs.aws.amazon.com/linux/al2023/release-notes/relnotes-2023.9.20251105.html 2025-12-04T09:51:08.8552763Z 2025-12-04T09:51:08.8552835Z Version 2023.9.20251110: 2025-12-04T09:51:08.8553054Z Run the following command to upgrade to 2023.9.20251110: 2025-12-04T09:51:08.8553244Z 2025-12-04T09:51:08.8553326Z dnf upgrade --releasever=2023.9.20251110 2025-12-04T09:51:08.8553484Z 2025-12-04T09:51:08.8553546Z Release notes: 2025-12-04T09:51:08.8553836Z https://docs.aws.amazon.com/linux/al2023/release-notes/relnotes-2023.9.20251110.html 2025-12-04T09:51:08.8554114Z 2025-12-04T09:51:08.8554179Z Version 2023.9.20251117: 2025-12-04T09:51:08.8554405Z Run the following command to upgrade to 2023.9.20251117: 2025-12-04T09:51:08.8554588Z 2025-12-04T09:51:08.8554691Z dnf upgrade --releasever=2023.9.20251117 2025-12-04T09:51:08.8554851Z 2025-12-04T09:51:08.8554919Z Release notes: 2025-12-04T09:51:08.8555206Z https://docs.aws.amazon.com/linux/al2023/release-notes/relnotes-2023.9.20251117.html 2025-12-04T09:51:08.8555485Z 2025-12-04T09:51:08.8555569Z ================================================================================ 2025-12-04T09:51:08.9002897Z 2025-12-04T09:51:08.9003099Z 2025-12-04T09:51:08.9003217Z Downgraded: 2025-12-04T09:51:08.9003583Z libnvidia-container-tools-1.17.8-1.x86_64 2025-12-04T09:51:08.9004120Z libnvidia-container1-1.17.8-1.x86_64 2025-12-04T09:51:08.9004652Z nvidia-container-toolkit-1.17.8-1.x86_64 2025-12-04T09:51:08.9005215Z nvidia-container-toolkit-base-1.17.8-1.x86_64 2025-12-04T09:51:08.9005542Z 2025-12-04T09:51:08.9005622Z Complete! 2025-12-04T09:51:08.9472850Z + sudo systemctl restart docker 2025-12-04T09:51:17.4374257Z Thu Dec 4 09:51:17 2025 2025-12-04T09:51:17.4374681Z +-----------------------------------------------------------------------------------------+ 2025-12-04T09:51:17.4375174Z | NVIDIA-SMI 580.82.07 Driver Version: 580.82.07 CUDA Version: 13.0 | 2025-12-04T09:51:17.4375642Z +-----------------------------------------+------------------------+----------------------+ 2025-12-04T09:51:17.4376260Z | GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC | 2025-12-04T09:51:17.4377073Z | Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. | 2025-12-04T09:51:17.4377660Z | | | MIG M. | 2025-12-04T09:51:17.4378237Z |=========================================+========================+======================| 2025-12-04T09:51:17.4450263Z | 0 NVIDIA L4 On | 00000000:35:00.0 Off | 0 | 2025-12-04T09:51:17.4450983Z | N/A 33C P0 29W / 72W | 0MiB / 23034MiB | 4% Default | 2025-12-04T09:51:17.4451363Z | | | N/A | 2025-12-04T09:51:17.4451733Z +-----------------------------------------+------------------------+----------------------+ 2025-12-04T09:51:17.4452007Z 2025-12-04T09:51:17.4452169Z +-----------------------------------------------------------------------------------------+ 2025-12-04T09:51:17.4452568Z | Processes: | 2025-12-04T09:51:17.4452976Z | GPU GI CI PID Type Process name GPU Memory | 2025-12-04T09:51:17.4453524Z | ID ID Usage | 2025-12-04T09:51:17.4453849Z |=========================================================================================| 2025-12-04T09:51:17.4454991Z | No running processes found | 2025-12-04T09:51:17.4455440Z +-----------------------------------------------------------------------------------------+ 2025-12-04T09:51:17.6073661Z Unable to find image 'public.ecr.aws/docker/library/python:3.13' locally 2025-12-04T09:51:17.8175340Z 3.13: Pulling from docker/library/python 2025-12-04T09:51:17.9414786Z 53c88f1dfeb7: Pulling fs layer 2025-12-04T09:51:17.9415130Z eae668646f44: Pulling fs layer 2025-12-04T09:51:17.9415391Z ff2e6e687b6c: Pulling fs layer 2025-12-04T09:51:17.9415665Z 7c40a3faff76: Pulling fs layer 2025-12-04T09:51:17.9415924Z 967a3b1c8fef: Pulling fs layer 2025-12-04T09:51:17.9416177Z a64e1a44f22a: Pulling fs layer 2025-12-04T09:51:17.9416469Z 52655f8a5bcc: Pulling fs layer 2025-12-04T09:51:17.9416707Z 967a3b1c8fef: Waiting 2025-12-04T09:51:17.9416934Z a64e1a44f22a: Waiting 2025-12-04T09:51:17.9417135Z 52655f8a5bcc: Waiting 2025-12-04T09:51:17.9417334Z 7c40a3faff76: Waiting 2025-12-04T09:51:18.0336906Z eae668646f44: Verifying Checksum 2025-12-04T09:51:18.0337282Z eae668646f44: Download complete 2025-12-04T09:51:18.1461145Z 53c88f1dfeb7: Verifying Checksum 2025-12-04T09:51:18.1461440Z 53c88f1dfeb7: Download complete 2025-12-04T09:51:18.1514214Z ff2e6e687b6c: Verifying Checksum 2025-12-04T09:51:18.1514483Z ff2e6e687b6c: Download complete 2025-12-04T09:51:18.2112834Z 967a3b1c8fef: Verifying Checksum 2025-12-04T09:51:18.2113144Z 967a3b1c8fef: Download complete 2025-12-04T09:51:18.2621976Z 52655f8a5bcc: Download complete 2025-12-04T09:51:18.2756853Z a64e1a44f22a: Verifying Checksum 2025-12-04T09:51:18.2757377Z a64e1a44f22a: Download complete 2025-12-04T09:51:18.7226809Z 7c40a3faff76: Verifying Checksum 2025-12-04T09:51:18.7227291Z 7c40a3faff76: Download complete 2025-12-04T09:51:19.4684651Z 53c88f1dfeb7: Pull complete 2025-12-04T09:51:20.0029471Z eae668646f44: Pull complete 2025-12-04T09:51:21.8235726Z ff2e6e687b6c: Pull complete 2025-12-04T09:51:26.9637027Z 7c40a3faff76: Pull complete 2025-12-04T09:51:27.3063350Z 967a3b1c8fef: Pull complete 2025-12-04T09:51:27.9803537Z a64e1a44f22a: Pull complete 2025-12-04T09:51:28.0043118Z 52655f8a5bcc: Pull complete 2025-12-04T09:51:28.0182605Z Digest: sha256:3f986299a7b8b44b0d8cf9bda2b22361ce5c3058ef5d7cb17fb7452506680ab0 2025-12-04T09:51:28.0225514Z Status: Downloaded newer image for public.ecr.aws/docker/library/python:3.13 2025-12-04T09:51:35.3025098Z Thu Dec 4 09:51:35 2025 2025-12-04T09:51:35.3025513Z +-----------------------------------------------------------------------------------------+ 2025-12-04T09:51:35.3026135Z | NVIDIA-SMI 580.82.07 Driver Version: 580.82.07 CUDA Version: 13.0 | 2025-12-04T09:51:35.3026732Z +-----------------------------------------+------------------------+----------------------+ 2025-12-04T09:51:35.3027397Z | GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC | 2025-12-04T09:51:35.3028293Z | Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. | 2025-12-04T09:51:35.3028633Z | | | MIG M. | 2025-12-04T09:51:35.3028881Z |=========================================+========================+======================| 2025-12-04T09:51:35.3144463Z | 0 NVIDIA L4 On | 00000000:35:00.0 Off | 0 | 2025-12-04T09:51:35.3144938Z | N/A 32C P8 12W / 72W | 0MiB / 23034MiB | 0% Default | 2025-12-04T09:51:35.3145345Z | | | N/A | 2025-12-04T09:51:35.3145757Z +-----------------------------------------+------------------------+----------------------+ 2025-12-04T09:51:35.3148093Z 2025-12-04T09:51:35.3148314Z +-----------------------------------------------------------------------------------------+ 2025-12-04T09:51:35.3148778Z | Processes: | 2025-12-04T09:51:35.3149250Z | GPU GI CI PID Type Process name GPU Memory | 2025-12-04T09:51:35.3149675Z | ID ID Usage | 2025-12-04T09:51:35.3150064Z |=========================================================================================| 2025-12-04T09:51:35.3153861Z | No running processes found | 2025-12-04T09:51:35.3154392Z +-----------------------------------------------------------------------------------------+ 2025-12-04T09:51:37.0297140Z Command completed after 1 attempt(s). 2025-12-04T09:51:37.0385128Z Prepare all required actions 2025-12-04T09:51:37.0409362Z ##[group]Run ./.github/actions/get-workflow-job-id 2025-12-04T09:51:37.0409608Z with: 2025-12-04T09:51:37.0410128Z github-token: *** 2025-12-04T09:51:37.0410315Z env: 2025-12-04T09:51:37.0410467Z GIT_DEFAULT_BRANCH: main 2025-12-04T09:51:37.0410672Z HAS_NVIDIA_GPU: true 2025-12-04T09:51:37.0410900Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2025-12-04T09:51:37.0411152Z ##[endgroup] 2025-12-04T09:51:37.0424225Z ##[group]Run set -eux 2025-12-04T09:51:37.0424415Z set -eux 2025-12-04T09:51:37.0424739Z python3 .github/scripts/get_workflow_job_id.py "${GITHUB_RUN_ID}" "${RUNNER_NAME}" 2025-12-04T09:51:37.0436902Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-12-04T09:51:37.0437174Z env: 2025-12-04T09:51:37.0437336Z GIT_DEFAULT_BRANCH: main 2025-12-04T09:51:37.0437538Z HAS_NVIDIA_GPU: true 2025-12-04T09:51:37.0437804Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2025-12-04T09:51:37.0438172Z GITHUB_TOKEN: *** 2025-12-04T09:51:37.0438349Z ##[endgroup] 2025-12-04T09:51:37.0472529Z + python3 .github/scripts/get_workflow_job_id.py 19922826259 i-02fefd73b1a8f16c5 2025-12-04T09:51:40.3780589Z Setting output job-id=57120265687 2025-12-04T09:51:40.3781403Z Setting output job-name=linux-jammy-cuda12.8-py3.10-gcc11-debug / test (default, 6, 7, linux.g6.4xlarge.experimental.nvidia.gpu, oncall:debug-build, mem_leak_check) 2025-12-04T09:51:40.3920774Z ##[group]Run python3 -m pip install psutil==5.9.8 dataclasses_json==0.6.7 nvidia-ml-py==11.525.84 2025-12-04T09:51:40.3922270Z python3 -m pip install psutil==5.9.8 dataclasses_json==0.6.7 nvidia-ml-py==11.525.84 2025-12-04T09:51:40.3924000Z python3 -m tools.stats.monitor --log-interval "$MONITOR_LOG_INTERVAL" --data-collect-interval "$MONITOR_DATA_COLLECT_INTERVAL" > usage_log.txt 2>&1 & 2025-12-04T09:51:40.3925432Z echo "monitor-script-pid=${!}" >> "${GITHUB_OUTPUT}" 2025-12-04T09:51:40.3935984Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-12-04T09:51:40.3936278Z env: 2025-12-04T09:51:40.3936436Z GIT_DEFAULT_BRANCH: main 2025-12-04T09:51:40.3936626Z HAS_NVIDIA_GPU: true 2025-12-04T09:51:40.3936855Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2025-12-04T09:51:40.3937116Z JOB_ID: 57120265687 2025-12-04T09:51:40.3937646Z JOB_NAME: linux-jammy-cuda12.8-py3.10-gcc11-debug / test (default, 6, 7, linux.g6.4xlarge.experimental.nvidia.gpu, oncall:debug-build, mem_leak_check) 2025-12-04T09:51:40.3938186Z WORKFLOW_NAME: periodic 2025-12-04T09:51:40.3938378Z WORKFLOW_RUN_ID: 19922826259 2025-12-04T09:51:40.3938588Z MONITOR_LOG_INTERVAL: 5 2025-12-04T09:51:40.3938797Z MONITOR_DATA_COLLECT_INTERVAL: 1 2025-12-04T09:51:40.3939002Z ##[endgroup] 2025-12-04T09:51:40.6708947Z Defaulting to user installation because normal site-packages is not writeable 2025-12-04T09:51:41.0062992Z Collecting psutil==5.9.8 2025-12-04T09:51:41.0224939Z Downloading psutil-5.9.8-cp36-abi3-manylinux_2_12_x86_64.manylinux2010_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl (288 kB) 2025-12-04T09:51:41.0895129Z Collecting dataclasses_json==0.6.7 2025-12-04T09:51:41.0928297Z Downloading dataclasses_json-0.6.7-py3-none-any.whl (28 kB) 2025-12-04T09:51:41.1212747Z Collecting nvidia-ml-py==11.525.84 2025-12-04T09:51:41.1240194Z Downloading nvidia_ml_py-11.525.84-py3-none-any.whl (34 kB) 2025-12-04T09:51:41.2346435Z Collecting marshmallow<4.0.0,>=3.18.0 2025-12-04T09:51:41.2373737Z Downloading marshmallow-3.26.1-py3-none-any.whl (50 kB) 2025-12-04T09:51:41.2601142Z Collecting typing-inspect<1,>=0.4.0 2025-12-04T09:51:41.2634192Z Downloading typing_inspect-0.9.0-py3-none-any.whl (8.8 kB) 2025-12-04T09:51:41.3159690Z Collecting packaging>=17.0 2025-12-04T09:51:41.3187523Z Downloading packaging-25.0-py3-none-any.whl (66 kB) 2025-12-04T09:51:41.3648388Z Collecting typing-extensions>=3.7.4 2025-12-04T09:51:41.3682262Z Downloading typing_extensions-4.15.0-py3-none-any.whl (44 kB) 2025-12-04T09:51:41.3872863Z Collecting mypy-extensions>=0.3.0 2025-12-04T09:51:41.3905560Z Downloading mypy_extensions-1.1.0-py3-none-any.whl (5.0 kB) 2025-12-04T09:51:41.4777066Z Installing collected packages: typing-extensions, packaging, mypy-extensions, typing-inspect, marshmallow, psutil, nvidia-ml-py, dataclasses-json 2025-12-04T09:51:41.7360520Z Successfully installed dataclasses-json-0.6.7 marshmallow-3.26.1 mypy-extensions-1.1.0 nvidia-ml-py-11.525.84 packaging-25.0 psutil-5.9.8 typing-extensions-4.15.0 typing-inspect-0.9.0 2025-12-04T09:51:41.8975513Z Prepare all required actions 2025-12-04T09:51:41.8975863Z Getting action download info 2025-12-04T09:51:42.0852163Z Download action repository 'seemethere/download-artifact-s3@v4' (SHA:1da556a7aa0a088e3153970611f6c432d58e80e6) 2025-12-04T09:51:42.3399795Z Download action repository 'actions/download-artifact@v4' (SHA:d3f86a106a0bac45b974a628896c90dbdf5c8093) 2025-12-04T09:51:42.7669853Z ##[group]Run ./.github/actions/download-build-artifacts 2025-12-04T09:51:42.7670125Z with: 2025-12-04T09:51:42.7670326Z name: linux-jammy-cuda12.8-py3.10-gcc11-debug 2025-12-04T09:51:42.7670593Z s3-bucket: gha-artifacts 2025-12-04T09:51:42.7670780Z env: 2025-12-04T09:51:42.7670925Z GIT_DEFAULT_BRANCH: main 2025-12-04T09:51:42.7671117Z HAS_NVIDIA_GPU: true 2025-12-04T09:51:42.7671347Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2025-12-04T09:51:42.7671594Z ##[endgroup] 2025-12-04T09:51:42.7696855Z ##[group]Run seemethere/download-artifact-s3@v4 2025-12-04T09:51:42.7697130Z with: 2025-12-04T09:51:42.7697324Z name: linux-jammy-cuda12.8-py3.10-gcc11-debug 2025-12-04T09:51:42.7697580Z s3-bucket: gha-artifacts 2025-12-04T09:51:42.7697781Z region: us-east-1 2025-12-04T09:51:42.7697948Z env: 2025-12-04T09:51:42.7698098Z GIT_DEFAULT_BRANCH: main 2025-12-04T09:51:42.7698295Z HAS_NVIDIA_GPU: true 2025-12-04T09:51:42.7698538Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2025-12-04T09:51:42.7698802Z ##[endgroup] 2025-12-04T09:51:43.1911068Z (node:60620) NOTE: We are formalizing our plans to enter AWS SDK for JavaScript (v2) into maintenance mode in 2023. 2025-12-04T09:51:43.1911560Z 2025-12-04T09:51:43.1911747Z Please migrate your code to use AWS SDK for JavaScript (v3). 2025-12-04T09:51:43.1912252Z For more information, check the migration guide at https://a.co/7PzMCcy 2025-12-04T09:51:43.1912756Z (Use `node --trace-warnings ...` to show where the warning was created) 2025-12-04T09:51:43.4722685Z Found 1 objects with prefix pytorch/pytorch/19922826259/linux-jammy-cuda12.8-py3.10-gcc11-debug/ 2025-12-04T09:51:43.4723413Z Starting download (1/1): /home/ec2-user/actions-runner/_work/pytorch/pytorch/artifacts.zip 2025-12-04T09:51:51.5863866Z Finished download (1/1): /home/ec2-user/actions-runner/_work/pytorch/pytorch/artifacts.zip 2025-12-04T09:51:51.5869261Z Artifact download has finished successfully 2025-12-04T09:51:51.6149942Z ##[group]Run unzip -o artifacts.zip 2025-12-04T09:51:51.6150207Z unzip -o artifacts.zip 2025-12-04T09:51:51.6158662Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-12-04T09:51:51.6159141Z env: 2025-12-04T09:51:51.6159306Z GIT_DEFAULT_BRANCH: main 2025-12-04T09:51:51.6159494Z HAS_NVIDIA_GPU: true 2025-12-04T09:51:51.6159728Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2025-12-04T09:51:51.6159989Z ##[endgroup] 2025-12-04T09:51:51.6235625Z Archive: artifacts.zip 2025-12-04T09:51:51.6236812Z creating: dist/ 2025-12-04T09:51:53.4446647Z inflating: dist/torch-2.10.0a0+gitffd9b0f-cp310-cp310-linux_x86_64.whl 2025-12-04T09:51:53.4562979Z inflating: dist/.ninja_log 2025-12-04T09:51:53.4563734Z creating: build/custom_test_artifacts/ 2025-12-04T09:51:53.4564155Z creating: build/custom_test_artifacts/custom-op-build/ 2025-12-04T09:51:53.4564651Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/ 2025-12-04T09:51:53.4565273Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/pkgRedirects/ 2025-12-04T09:51:53.4572808Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeConfigureLog.yaml 2025-12-04T09:51:53.4573458Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.31.6/ 2025-12-04T09:51:53.4574055Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.31.6/CMakeSystem.cmake 2025-12-04T09:51:53.4574699Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.31.6/CompilerIdC/ 2025-12-04T09:51:53.4575598Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.31.6/CompilerIdC/tmp/ 2025-12-04T09:51:53.4578312Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.31.6/CompilerIdC/CMakeCCompilerId.c 2025-12-04T09:51:53.4579687Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.31.6/CompilerIdC/a.out 2025-12-04T09:51:53.4580669Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.31.6/CMakeCCompiler.cmake 2025-12-04T09:51:53.4581342Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.31.6/CompilerIdCXX/ 2025-12-04T09:51:53.4581990Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.31.6/CompilerIdCXX/tmp/ 2025-12-04T09:51:53.4584572Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.31.6/CompilerIdCXX/CMakeCXXCompilerId.cpp 2025-12-04T09:51:53.4586015Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.31.6/CompilerIdCXX/a.out 2025-12-04T09:51:53.4587121Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.31.6/CMakeCXXCompiler.cmake 2025-12-04T09:51:53.4588910Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.31.6/CMakeDetermineCompilerABI_C.bin 2025-12-04T09:51:53.4590813Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.31.6/CMakeDetermineCompilerABI_CXX.bin 2025-12-04T09:51:53.4591394Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.31.6/CompilerIdCUDA/ 2025-12-04T09:51:53.4591895Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.31.6/CompilerIdCUDA/tmp/ 2025-12-04T09:51:53.4643857Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.31.6/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp4.ii 2025-12-04T09:51:53.4696773Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.31.6/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.cpp 2025-12-04T09:51:53.4697731Z extracting: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.31.6/CompilerIdCUDA/tmp/CMakeCUDACompilerId.module_id 2025-12-04T09:51:53.4754113Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.31.6/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp1.ii 2025-12-04T09:51:53.4755102Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.31.6/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.c 2025-12-04T09:51:53.4756068Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.31.6/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.gpu 2025-12-04T09:51:53.4757158Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.31.6/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.stub.c 2025-12-04T09:51:53.4758388Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.31.6/CompilerIdCUDA/tmp/CMakeCUDACompilerId.ptx 2025-12-04T09:51:53.4759321Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.31.6/CompilerIdCUDA/tmp/CMakeCUDACompilerId.sm_52.cubin 2025-12-04T09:51:53.4760362Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.31.6/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin 2025-12-04T09:51:53.4761385Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.31.6/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin.c 2025-12-04T09:51:53.4762791Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.31.6/CompilerIdCUDA/tmp/CMakeCUDACompilerId.o 2025-12-04T09:51:53.4763575Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.31.6/CompilerIdCUDA/tmp/a_dlink.sm_52.cubin 2025-12-04T09:51:53.4764405Z extracting: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.31.6/CompilerIdCUDA/tmp/a_dlink.reg.c 2025-12-04T09:51:53.4765330Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.31.6/CompilerIdCUDA/tmp/a_dlink.fatbin 2025-12-04T09:51:53.4766359Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.31.6/CompilerIdCUDA/tmp/a_dlink.fatbin.c 2025-12-04T09:51:53.4767634Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.31.6/CompilerIdCUDA/tmp/a_dlink.o 2025-12-04T09:51:53.4770096Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.31.6/CompilerIdCUDA/CMakeCUDACompilerId.cu 2025-12-04T09:51:53.4834677Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.31.6/CompilerIdCUDA/a.out 2025-12-04T09:51:53.4835841Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.31.6/CMakeCUDACompiler.cmake 2025-12-04T09:51:53.4900450Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.31.6/CMakeDetermineCompilerABI_CUDA.bin 2025-12-04T09:51:53.4901265Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeScratch/ 2025-12-04T09:51:53.4901815Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeTmp/ 2025-12-04T09:51:53.4902377Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/cmake.check_cache 2025-12-04T09:51:53.4902959Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/ 2025-12-04T09:51:53.4903703Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/compiler_depend.ts 2025-12-04T09:51:53.4904594Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/compiler_depend.make 2025-12-04T09:51:53.4905365Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/depend.make 2025-12-04T09:51:53.4906271Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/link.txt 2025-12-04T09:51:53.4907121Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/cmake_clean.cmake 2025-12-04T09:51:53.4908208Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/build.make 2025-12-04T09:51:53.4909012Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/DependInfo.cmake 2025-12-04T09:51:53.4909875Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/flags.make 2025-12-04T09:51:53.4910994Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/progress.make 2025-12-04T09:51:53.4929625Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/op.cpp.o.d 2025-12-04T09:51:53.5104488Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/op.cpp.o 2025-12-04T09:51:53.5105156Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/ 2025-12-04T09:51:53.5105946Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/compiler_depend.ts 2025-12-04T09:51:53.5106980Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/compiler_depend.make 2025-12-04T09:51:53.5107716Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/depend.make 2025-12-04T09:51:53.5108481Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/link.txt 2025-12-04T09:51:53.5109317Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/cmake_clean.cmake 2025-12-04T09:51:53.5110500Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/build.make 2025-12-04T09:51:53.5111283Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/DependInfo.cmake 2025-12-04T09:51:53.5112112Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/flags.make 2025-12-04T09:51:53.5113114Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/progress.make 2025-12-04T09:51:53.5131804Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/test_custom_ops.cpp.o.d 2025-12-04T09:51:53.5203597Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/test_custom_ops.cpp.o 2025-12-04T09:51:53.5204634Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeDirectoryInformation.cmake 2025-12-04T09:51:53.5205353Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/TargetDirectories.txt 2025-12-04T09:51:53.5206070Z extracting: build/custom_test_artifacts/custom-op-build/CMakeFiles/progress.marks 2025-12-04T09:51:53.5207224Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/Makefile2 2025-12-04T09:51:53.5208982Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/Makefile.cmake 2025-12-04T09:51:53.5209552Z inflating: build/custom_test_artifacts/custom-op-build/detect_cuda_version.cc 2025-12-04T09:51:53.5212338Z inflating: build/custom_test_artifacts/custom-op-build/CMakeCache.txt 2025-12-04T09:51:53.5213333Z inflating: build/custom_test_artifacts/custom-op-build/Makefile 2025-12-04T09:51:53.5214202Z inflating: build/custom_test_artifacts/custom-op-build/cmake_install.cmake 2025-12-04T09:51:53.5364219Z inflating: build/custom_test_artifacts/custom-op-build/libcustom_ops.so 2025-12-04T09:51:53.5415245Z inflating: build/custom_test_artifacts/custom-op-build/test_custom_ops 2025-12-04T09:51:53.5415734Z creating: build/custom_test_artifacts/jit-hook-build/ 2025-12-04T09:51:53.5416167Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/ 2025-12-04T09:51:53.5416691Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/pkgRedirects/ 2025-12-04T09:51:53.5424102Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeConfigureLog.yaml 2025-12-04T09:51:53.5424692Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.31.6/ 2025-12-04T09:51:53.5425460Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.31.6/CMakeSystem.cmake 2025-12-04T09:51:53.5426089Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.31.6/CompilerIdC/ 2025-12-04T09:51:53.5426678Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.31.6/CompilerIdC/tmp/ 2025-12-04T09:51:53.5429371Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.31.6/CompilerIdC/CMakeCCompilerId.c 2025-12-04T09:51:53.5430758Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.31.6/CompilerIdC/a.out 2025-12-04T09:51:53.5431732Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.31.6/CMakeCCompiler.cmake 2025-12-04T09:51:53.5432359Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.31.6/CompilerIdCXX/ 2025-12-04T09:51:53.5432955Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.31.6/CompilerIdCXX/tmp/ 2025-12-04T09:51:53.5435592Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.31.6/CompilerIdCXX/CMakeCXXCompilerId.cpp 2025-12-04T09:51:53.5437194Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.31.6/CompilerIdCXX/a.out 2025-12-04T09:51:53.5438141Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.31.6/CMakeCXXCompiler.cmake 2025-12-04T09:51:53.5439905Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.31.6/CMakeDetermineCompilerABI_C.bin 2025-12-04T09:51:53.5441715Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.31.6/CMakeDetermineCompilerABI_CXX.bin 2025-12-04T09:51:53.5442270Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.31.6/CompilerIdCUDA/ 2025-12-04T09:51:53.5442765Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.31.6/CompilerIdCUDA/tmp/ 2025-12-04T09:51:53.5495100Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.31.6/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp4.ii 2025-12-04T09:51:53.5547663Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.31.6/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.cpp 2025-12-04T09:51:53.5548613Z extracting: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.31.6/CompilerIdCUDA/tmp/CMakeCUDACompilerId.module_id 2025-12-04T09:51:53.5605164Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.31.6/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp1.ii 2025-12-04T09:51:53.5606108Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.31.6/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.c 2025-12-04T09:51:53.5607267Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.31.6/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.gpu 2025-12-04T09:51:53.5608204Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.31.6/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.stub.c 2025-12-04T09:51:53.5609096Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.31.6/CompilerIdCUDA/tmp/CMakeCUDACompilerId.ptx 2025-12-04T09:51:53.5610196Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.31.6/CompilerIdCUDA/tmp/CMakeCUDACompilerId.sm_52.cubin 2025-12-04T09:51:53.5611148Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.31.6/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin 2025-12-04T09:51:53.5612229Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.31.6/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin.c 2025-12-04T09:51:53.5613583Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.31.6/CompilerIdCUDA/tmp/CMakeCUDACompilerId.o 2025-12-04T09:51:53.5614384Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.31.6/CompilerIdCUDA/tmp/a_dlink.sm_52.cubin 2025-12-04T09:51:53.5615195Z extracting: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.31.6/CompilerIdCUDA/tmp/a_dlink.reg.c 2025-12-04T09:51:53.5616110Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.31.6/CompilerIdCUDA/tmp/a_dlink.fatbin 2025-12-04T09:51:53.5617154Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.31.6/CompilerIdCUDA/tmp/a_dlink.fatbin.c 2025-12-04T09:51:53.5618233Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.31.6/CompilerIdCUDA/tmp/a_dlink.o 2025-12-04T09:51:53.5620844Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.31.6/CompilerIdCUDA/CMakeCUDACompilerId.cu 2025-12-04T09:51:53.5685505Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.31.6/CompilerIdCUDA/a.out 2025-12-04T09:51:53.5686369Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.31.6/CMakeCUDACompiler.cmake 2025-12-04T09:51:53.5751845Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.31.6/CMakeDetermineCompilerABI_CUDA.bin 2025-12-04T09:51:53.5752548Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeScratch/ 2025-12-04T09:51:53.5753302Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeTmp/ 2025-12-04T09:51:53.5753850Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/cmake.check_cache 2025-12-04T09:51:53.5754420Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/ 2025-12-04T09:51:53.5755260Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/compiler_depend.ts 2025-12-04T09:51:53.5756028Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/compiler_depend.make 2025-12-04T09:51:53.5756948Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/depend.make 2025-12-04T09:51:53.5757673Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/link.txt 2025-12-04T09:51:53.5758653Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/cmake_clean.cmake 2025-12-04T09:51:53.5759744Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/build.make 2025-12-04T09:51:53.5760531Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/DependInfo.cmake 2025-12-04T09:51:53.5761379Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/flags.make 2025-12-04T09:51:53.5762718Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/progress.make 2025-12-04T09:51:53.5781544Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/test_jit_hooks.cpp.o.d 2025-12-04T09:51:53.5837632Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/test_jit_hooks.cpp.o 2025-12-04T09:51:53.5838508Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeDirectoryInformation.cmake 2025-12-04T09:51:53.5839345Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/TargetDirectories.txt 2025-12-04T09:51:53.5840046Z extracting: build/custom_test_artifacts/jit-hook-build/CMakeFiles/progress.marks 2025-12-04T09:51:53.5841092Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/Makefile2 2025-12-04T09:51:53.5842928Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/Makefile.cmake 2025-12-04T09:51:53.5843437Z inflating: build/custom_test_artifacts/jit-hook-build/detect_cuda_version.cc 2025-12-04T09:51:53.5846286Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeCache.txt 2025-12-04T09:51:53.5847200Z inflating: build/custom_test_artifacts/jit-hook-build/Makefile 2025-12-04T09:51:53.5848079Z inflating: build/custom_test_artifacts/jit-hook-build/cmake_install.cmake 2025-12-04T09:51:53.5886680Z inflating: build/custom_test_artifacts/jit-hook-build/test_jit_hooks 2025-12-04T09:51:53.5887149Z creating: build/custom_test_artifacts/custom-backend-build/ 2025-12-04T09:51:53.5887612Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/ 2025-12-04T09:51:53.5888258Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/pkgRedirects/ 2025-12-04T09:51:53.5895747Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeConfigureLog.yaml 2025-12-04T09:51:53.5896391Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.31.6/ 2025-12-04T09:51:53.5897022Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.31.6/CMakeSystem.cmake 2025-12-04T09:51:53.5897683Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.31.6/CompilerIdC/ 2025-12-04T09:51:53.5898347Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.31.6/CompilerIdC/tmp/ 2025-12-04T09:51:53.5900913Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.31.6/CompilerIdC/CMakeCCompilerId.c 2025-12-04T09:51:53.5902235Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.31.6/CompilerIdC/a.out 2025-12-04T09:51:53.5903194Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.31.6/CMakeCCompiler.cmake 2025-12-04T09:51:53.5903835Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.31.6/CompilerIdCXX/ 2025-12-04T09:51:53.5904851Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.31.6/CompilerIdCXX/tmp/ 2025-12-04T09:51:53.5907492Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.31.6/CompilerIdCXX/CMakeCXXCompilerId.cpp 2025-12-04T09:51:53.5908999Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.31.6/CompilerIdCXX/a.out 2025-12-04T09:51:53.5910071Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.31.6/CMakeCXXCompiler.cmake 2025-12-04T09:51:53.5911813Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.31.6/CMakeDetermineCompilerABI_C.bin 2025-12-04T09:51:53.5913650Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.31.6/CMakeDetermineCompilerABI_CXX.bin 2025-12-04T09:51:53.5914272Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.31.6/CompilerIdCUDA/ 2025-12-04T09:51:53.5914814Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.31.6/CompilerIdCUDA/tmp/ 2025-12-04T09:51:53.5967207Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.31.6/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp4.ii 2025-12-04T09:51:53.6019687Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.31.6/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.cpp 2025-12-04T09:51:53.6020674Z extracting: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.31.6/CompilerIdCUDA/tmp/CMakeCUDACompilerId.module_id 2025-12-04T09:51:53.6076716Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.31.6/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp1.ii 2025-12-04T09:51:53.6077705Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.31.6/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.c 2025-12-04T09:51:53.6079173Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.31.6/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.gpu 2025-12-04T09:51:53.6080192Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.31.6/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.stub.c 2025-12-04T09:51:53.6081161Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.31.6/CompilerIdCUDA/tmp/CMakeCUDACompilerId.ptx 2025-12-04T09:51:53.6082116Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.31.6/CompilerIdCUDA/tmp/CMakeCUDACompilerId.sm_52.cubin 2025-12-04T09:51:53.6083077Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.31.6/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin 2025-12-04T09:51:53.6084109Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.31.6/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin.c 2025-12-04T09:51:53.6085407Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.31.6/CompilerIdCUDA/tmp/CMakeCUDACompilerId.o 2025-12-04T09:51:53.6086247Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.31.6/CompilerIdCUDA/tmp/a_dlink.sm_52.cubin 2025-12-04T09:51:53.6087078Z extracting: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.31.6/CompilerIdCUDA/tmp/a_dlink.reg.c 2025-12-04T09:51:53.6087990Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.31.6/CompilerIdCUDA/tmp/a_dlink.fatbin 2025-12-04T09:51:53.6089065Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.31.6/CompilerIdCUDA/tmp/a_dlink.fatbin.c 2025-12-04T09:51:53.6090038Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.31.6/CompilerIdCUDA/tmp/a_dlink.o 2025-12-04T09:51:53.6092697Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.31.6/CompilerIdCUDA/CMakeCUDACompilerId.cu 2025-12-04T09:51:53.6157067Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.31.6/CompilerIdCUDA/a.out 2025-12-04T09:51:53.6157870Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.31.6/CMakeCUDACompiler.cmake 2025-12-04T09:51:53.6222809Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.31.6/CMakeDetermineCompilerABI_CUDA.bin 2025-12-04T09:51:53.6223638Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeScratch/ 2025-12-04T09:51:53.6224226Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeTmp/ 2025-12-04T09:51:53.6225127Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/cmake.check_cache 2025-12-04T09:51:53.6225800Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/ 2025-12-04T09:51:53.6226533Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/compiler_depend.ts 2025-12-04T09:51:53.6227386Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/compiler_depend.make 2025-12-04T09:51:53.6228265Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/depend.make 2025-12-04T09:51:53.6229184Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/link.txt 2025-12-04T09:51:53.6230142Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/cmake_clean.cmake 2025-12-04T09:51:53.6231153Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/build.make 2025-12-04T09:51:53.6231970Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/DependInfo.cmake 2025-12-04T09:51:53.6232764Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/flags.make 2025-12-04T09:51:53.6233786Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/progress.make 2025-12-04T09:51:53.6238161Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/custom_backend.cpp.o.d 2025-12-04T09:51:53.6343278Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/custom_backend.cpp.o 2025-12-04T09:51:53.6344103Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/ 2025-12-04T09:51:53.6344857Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/compiler_depend.ts 2025-12-04T09:51:53.6345702Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/compiler_depend.make 2025-12-04T09:51:53.6346545Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/depend.make 2025-12-04T09:51:53.6347394Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/link.txt 2025-12-04T09:51:53.6348303Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/cmake_clean.cmake 2025-12-04T09:51:53.6349413Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/build.make 2025-12-04T09:51:53.6350328Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/DependInfo.cmake 2025-12-04T09:51:53.6351143Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/flags.make 2025-12-04T09:51:53.6352207Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/progress.make 2025-12-04T09:51:53.6370935Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/test_custom_backend.cpp.o.d 2025-12-04T09:51:53.6419783Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/test_custom_backend.cpp.o 2025-12-04T09:51:53.6420891Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeDirectoryInformation.cmake 2025-12-04T09:51:53.6421625Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/TargetDirectories.txt 2025-12-04T09:51:53.6422419Z extracting: build/custom_test_artifacts/custom-backend-build/CMakeFiles/progress.marks 2025-12-04T09:51:53.6423461Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/Makefile2 2025-12-04T09:51:53.6425297Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/Makefile.cmake 2025-12-04T09:51:53.6425805Z inflating: build/custom_test_artifacts/custom-backend-build/detect_cuda_version.cc 2025-12-04T09:51:53.6428560Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeCache.txt 2025-12-04T09:51:53.6429523Z inflating: build/custom_test_artifacts/custom-backend-build/Makefile 2025-12-04T09:51:53.6430398Z inflating: build/custom_test_artifacts/custom-backend-build/cmake_install.cmake 2025-12-04T09:51:53.6519469Z inflating: build/custom_test_artifacts/custom-backend-build/libcustom_backend.so 2025-12-04T09:51:53.6555321Z inflating: build/custom_test_artifacts/custom-backend-build/test_custom_backend 2025-12-04T09:51:53.6555743Z creating: build/lib/ 2025-12-04T09:51:53.6652630Z inflating: build/lib/libprotobuf-lite.a 2025-12-04T09:51:53.7094895Z inflating: build/lib/libprotobuf.a 2025-12-04T09:51:53.7529441Z inflating: build/lib/libprotoc.a 2025-12-04T09:51:53.7539489Z inflating: build/lib/libpthreadpool.a 2025-12-04T09:51:53.7547018Z inflating: build/lib/libcpuinfo.a 2025-12-04T09:51:53.7554095Z inflating: build/lib/libcpuinfo_internals.a 2025-12-04T09:51:53.7555019Z inflating: build/lib/libclog.a 2025-12-04T09:51:53.7573139Z inflating: build/lib/libpytorch_qnnpack.a 2025-12-04T09:51:53.7575492Z inflating: build/lib/libnnpack_reference_layers.a 2025-12-04T09:51:53.7591945Z inflating: build/lib/libnnpack.a 2025-12-04T09:51:53.7841043Z inflating: build/lib/libmicrokernels-prod.a 2025-12-04T09:51:53.8954535Z inflating: build/lib/libmicrokernels-all.a 2025-12-04T09:51:53.9015417Z inflating: build/lib/libgtest.a 2025-12-04T09:51:53.9030668Z inflating: build/lib/libgmock.a 2025-12-04T09:51:53.9031525Z inflating: build/lib/libgtest_main.a 2025-12-04T09:51:53.9032502Z inflating: build/lib/libgmock_main.a 2025-12-04T09:51:53.9128834Z inflating: build/lib/libXNNPACK.a 2025-12-04T09:51:53.9199675Z inflating: build/lib/libbenchmark.a 2025-12-04T09:51:53.9200462Z inflating: build/lib/libbenchmark_main.a 2025-12-04T09:51:53.9207858Z inflating: build/lib/libittnotify.a 2025-12-04T09:51:53.9208745Z inflating: build/lib/libjitprofiling.a 2025-12-04T09:51:53.9273960Z inflating: build/lib/libasmjit.a 2025-12-04T09:51:54.0467029Z inflating: build/lib/libfbgemm.a 2025-12-04T09:51:54.0496499Z inflating: build/lib/libtensorpipe_uv.a 2025-12-04T09:51:54.1015820Z inflating: build/lib/libtensorpipe.a 2025-12-04T09:51:54.1245424Z inflating: build/lib/libtensorpipe_cuda.a 2025-12-04T09:51:54.1362292Z inflating: build/lib/libgloo.a 2025-12-04T09:51:54.1419789Z inflating: build/lib/libonnx_proto.a 2025-12-04T09:51:54.1808992Z inflating: build/lib/libgloo_cuda.a 2025-12-04T09:51:54.2458991Z inflating: build/lib/libonnx.a 2025-12-04T09:51:55.1437449Z inflating: build/lib/libdnnl.a 2025-12-04T09:51:55.1455180Z inflating: build/lib/libfmt.a 2025-12-04T09:51:55.1885273Z inflating: build/lib/libkineto.a 2025-12-04T09:51:55.1989662Z inflating: build/lib/libc10.so 2025-12-04T09:51:55.2034029Z inflating: build/lib/libc10_cuda.so 2025-12-04T09:51:55.2035944Z inflating: build/lib/libcaffe2_nvrtc.so 2025-12-04T09:51:55.2037530Z inflating: build/lib/libtorch_global_deps.so 2025-12-04T09:51:57.9203485Z inflating: build/lib/libtorch_cpu.so 2025-12-04T09:51:57.9932446Z inflating: build/lib/libtorch_nvshmem.so 2025-12-04T09:52:00.5366554Z inflating: build/lib/libtorch_cuda.so 2025-12-04T09:52:00.5367768Z inflating: build/lib/libtorch.so 2025-12-04T09:52:00.5413221Z inflating: build/lib/libtorch_cuda_linalg.so 2025-12-04T09:52:00.5478837Z inflating: build/lib/libtorchbind_test.so 2025-12-04T09:52:00.5497493Z inflating: build/lib/libjitbackend_test.so 2025-12-04T09:52:00.5520725Z inflating: build/lib/libbackend_with_compiler.so 2025-12-04T09:52:00.5545719Z inflating: build/lib/libaoti_custom_ops.so 2025-12-04T09:52:00.5549864Z inflating: build/lib/libc10d_cuda_test.so 2025-12-04T09:52:00.5553932Z inflating: build/lib/libshm.so 2025-12-04T09:52:00.7681170Z inflating: build/lib/libtorch_python.so 2025-12-04T09:52:00.7715673Z inflating: build/lib/libnnapi_backend.so 2025-12-04T09:52:00.7715982Z creating: build/bin/ 2025-12-04T09:52:00.8131123Z inflating: build/bin/protoc-3.13.0.0 2025-12-04T09:52:00.8546154Z inflating: build/bin/protoc 2025-12-04T09:52:00.8597652Z inflating: build/bin/c10_AllocatorConfig_test 2025-12-04T09:52:00.8645741Z inflating: build/bin/c10_CompileTimeFunctionPointer_test 2025-12-04T09:52:00.8695962Z inflating: build/bin/c10_DeviceGuard_test 2025-12-04T09:52:00.8745455Z inflating: build/bin/c10_Device_test 2025-12-04T09:52:00.8802927Z inflating: build/bin/c10_DispatchKeySet_test 2025-12-04T09:52:00.8856155Z inflating: build/bin/c10_Scalar_test 2025-12-04T09:52:00.8903365Z inflating: build/bin/c10_StreamGuard_test 2025-12-04T09:52:00.8958849Z inflating: build/bin/c10_SymInt_test 2025-12-04T09:52:00.9013253Z inflating: build/bin/c10_InlineStreamGuard_test 2025-12-04T09:52:00.9080797Z inflating: build/bin/c10_cow_test 2025-12-04T09:52:00.9134416Z inflating: build/bin/c10_SizesAndStrides_test 2025-12-04T09:52:00.9188032Z inflating: build/bin/c10_InlineDeviceGuard_test 2025-12-04T09:52:00.9238840Z inflating: build/bin/c10_Bitset_test 2025-12-04T09:52:00.9286847Z inflating: build/bin/c10_ArrayRef_test 2025-12-04T09:52:00.9334046Z inflating: build/bin/c10_ConstexprCrc_test 2025-12-04T09:52:00.9382236Z inflating: build/bin/c10_DeadlockDetection_test 2025-12-04T09:52:00.9435755Z inflating: build/bin/c10_LeftRight_test 2025-12-04T09:52:00.9485191Z inflating: build/bin/c10_Half_test 2025-12-04T09:52:00.9535820Z inflating: build/bin/c10_IntrusiveList_test 2025-12-04T09:52:00.9587073Z inflating: build/bin/c10_NetworkFlow_test 2025-12-04T09:52:00.9641466Z inflating: build/bin/c10_Enumerate_test 2025-12-04T09:52:00.9689911Z inflating: build/bin/c10_Synchronized_test 2025-12-04T09:52:00.9737727Z inflating: build/bin/c10_Semaphore_test 2025-12-04T09:52:00.9791210Z inflating: build/bin/c10_ThreadLocal_test 2025-12-04T09:52:00.9841000Z inflating: build/bin/c10_accumulate_test 2025-12-04T09:52:00.9890872Z inflating: build/bin/c10_TypeIndex_test 2025-12-04T09:52:00.9944162Z inflating: build/bin/c10_bfloat16_test 2025-12-04T09:52:00.9992796Z inflating: build/bin/c10_bit_cast_test 2025-12-04T09:52:01.0040526Z inflating: build/bin/c10_error_test 2025-12-04T09:52:01.0094771Z inflating: build/bin/c10_complex_math_test 2025-12-04T09:52:01.0145350Z inflating: build/bin/c10_exception_test 2025-12-04T09:52:01.0198358Z inflating: build/bin/c10_complex_test 2025-12-04T09:52:01.0246875Z inflating: build/bin/c10_flags_test 2025-12-04T09:52:01.0295488Z inflating: build/bin/c10_generic_math_test 2025-12-04T09:52:01.0344527Z inflating: build/bin/c10_irange_test 2025-12-04T09:52:01.0496765Z inflating: build/bin/c10_intrusive_ptr_test 2025-12-04T09:52:01.0547995Z inflating: build/bin/c10_lazy_test 2025-12-04T09:52:01.0602710Z inflating: build/bin/c10_logging_test 2025-12-04T09:52:01.0651589Z inflating: build/bin/c10_nofatal_test 2025-12-04T09:52:01.0722453Z inflating: build/bin/c10_optional_test 2025-12-04T09:52:01.0781360Z inflating: build/bin/c10_ordered_preserving_dict_test 2025-12-04T09:52:01.0832429Z inflating: build/bin/c10_registry_test 2025-12-04T09:52:01.0886396Z inflating: build/bin/c10_string_util_test 2025-12-04T09:52:01.1030014Z inflating: build/bin/c10_small_vector_test 2025-12-04T09:52:01.1079928Z inflating: build/bin/c10_ssize_test 2025-12-04T09:52:01.1128195Z inflating: build/bin/c10_tempfile_test 2025-12-04T09:52:01.1172538Z inflating: build/bin/c10_intrusive_ptr_benchmark 2025-12-04T09:52:01.1219874Z inflating: build/bin/c10_string_view_test 2025-12-04T09:52:01.1273758Z inflating: build/bin/c10_typeid_test 2025-12-04T09:52:01.1330632Z inflating: build/bin/c10_cuda_CUDAAssertionsTest_from_2_processes 2025-12-04T09:52:01.1387015Z inflating: build/bin/c10_cuda_CUDAAssertionsTest_1_var_test 2025-12-04T09:52:01.1443306Z inflating: build/bin/c10_cuda_CUDAAssertionsTest_catches_thread_and_block_and_device 2025-12-04T09:52:01.1490749Z inflating: build/bin/c10_cuda_CUDATest 2025-12-04T09:52:01.1547336Z inflating: build/bin/c10_cuda_CUDAAssertionsTest_catches_stream 2025-12-04T09:52:01.1604139Z inflating: build/bin/c10_cuda_CUDAAssertionsTest_multiple_writes_from_multiple_blocks 2025-12-04T09:52:01.1662926Z inflating: build/bin/c10_cuda_CUDAAssertionsTest_multiple_writes_from_blocks_and_threads 2025-12-04T09:52:01.1719415Z inflating: build/bin/c10_cuda_CUDAAssertionsTest_multiple_writes_from_same_block 2025-12-04T09:52:01.2278637Z inflating: build/bin/vec_test_all_types_DEFAULT 2025-12-04T09:52:01.2851286Z inflating: build/bin/vec_test_all_types_AVX512 2025-12-04T09:52:01.3432037Z inflating: build/bin/vec_test_all_types_AVX2 2025-12-04T09:52:01.3479728Z inflating: build/bin/test_vec_half_DEFAULT 2025-12-04T09:52:01.3570593Z inflating: build/bin/test_aoti_abi_check 2025-12-04T09:52:01.3619052Z inflating: build/bin/test_vec_half_AVX512 2025-12-04T09:52:01.3667275Z inflating: build/bin/test_vec_half_AVX2 2025-12-04T09:52:01.3717851Z inflating: build/bin/BackoffTest 2025-12-04T09:52:01.3769654Z inflating: build/bin/FileStoreTest 2025-12-04T09:52:01.3824445Z inflating: build/bin/TCPStoreTest 2025-12-04T09:52:01.3876370Z inflating: build/bin/HashStoreTest 2025-12-04T09:52:01.3946916Z inflating: build/bin/Dict_test 2025-12-04T09:52:01.3997501Z inflating: build/bin/Dimname_test 2025-12-04T09:52:01.4059989Z inflating: build/bin/MaybeOwned_test 2025-12-04T09:52:01.4115498Z inflating: build/bin/NamedTensor_test 2025-12-04T09:52:01.4172512Z inflating: build/bin/apply_utils_test 2025-12-04T09:52:01.4228746Z inflating: build/bin/atest 2025-12-04T09:52:01.4290469Z inflating: build/bin/basic 2025-12-04T09:52:01.4341645Z inflating: build/bin/broadcast_test 2025-12-04T09:52:01.4391790Z inflating: build/bin/cpu_allocator_test 2025-12-04T09:52:01.4447255Z inflating: build/bin/cpu_generator_test 2025-12-04T09:52:01.4499538Z inflating: build/bin/cpu_profiling_allocator_test 2025-12-04T09:52:01.4586787Z inflating: build/bin/cpu_rng_test 2025-12-04T09:52:01.4636010Z inflating: build/bin/dlconvertor_test 2025-12-04T09:52:01.4692948Z inflating: build/bin/extension_backend_test 2025-12-04T09:52:01.4746136Z inflating: build/bin/half_test 2025-12-04T09:52:01.4840073Z inflating: build/bin/ivalue_test 2025-12-04T09:52:01.4888419Z inflating: build/bin/lazy_tensor_test 2025-12-04T09:52:01.4940717Z inflating: build/bin/math_kernel_test 2025-12-04T09:52:01.4993559Z inflating: build/bin/memory_format_test 2025-12-04T09:52:01.5045913Z inflating: build/bin/memory_overlapping_test 2025-12-04T09:52:01.5097597Z inflating: build/bin/mobile_memory_cleanup 2025-12-04T09:52:01.5151975Z inflating: build/bin/native_test 2025-12-04T09:52:01.5201155Z inflating: build/bin/operator_name_test 2025-12-04T09:52:01.5250116Z inflating: build/bin/operators_test 2025-12-04T09:52:01.5300896Z inflating: build/bin/packedtensoraccessor_test 2025-12-04T09:52:01.5365635Z inflating: build/bin/pow_test 2025-12-04T09:52:01.5421077Z inflating: build/bin/quantized_test 2025-12-04T09:52:01.5469292Z inflating: build/bin/reduce_ops_test 2025-12-04T09:52:01.5518517Z inflating: build/bin/reportMemoryUsage_test 2025-12-04T09:52:01.5573287Z inflating: build/bin/scalar_tensor_test 2025-12-04T09:52:01.5629584Z inflating: build/bin/scalar_test 2025-12-04T09:52:01.5680620Z inflating: build/bin/StorageUtils_test 2025-12-04T09:52:01.5731831Z inflating: build/bin/stride_properties_test 2025-12-04T09:52:01.5805602Z inflating: build/bin/tensor_iterator_test 2025-12-04T09:52:01.5858316Z inflating: build/bin/type_ptr_test 2025-12-04T09:52:01.5907473Z inflating: build/bin/thread_init_test 2025-12-04T09:52:01.5961195Z inflating: build/bin/test_parallel 2025-12-04T09:52:01.6017788Z inflating: build/bin/type_test 2025-12-04T09:52:01.6068724Z inflating: build/bin/undefined_tensor_test 2025-12-04T09:52:01.6116670Z inflating: build/bin/verify_api_visibility 2025-12-04T09:52:01.6184892Z inflating: build/bin/legacy_vmap_test 2025-12-04T09:52:01.6234567Z inflating: build/bin/weakref_test 2025-12-04T09:52:01.6284223Z inflating: build/bin/wrapdim_test 2025-12-04T09:52:01.6335027Z inflating: build/bin/xla_tensor_test 2025-12-04T09:52:01.6392281Z inflating: build/bin/IListRef_test 2025-12-04T09:52:01.6491513Z inflating: build/bin/List_test 2025-12-04T09:52:01.6556070Z inflating: build/bin/KernelFunction_test 2025-12-04T09:52:01.6669458Z inflating: build/bin/kernel_function_legacy_test 2025-12-04T09:52:01.6761549Z inflating: build/bin/kernel_function_test 2025-12-04T09:52:01.6880585Z inflating: build/bin/kernel_lambda_legacy_test 2025-12-04T09:52:01.6978551Z inflating: build/bin/kernel_lambda_test 2025-12-04T09:52:01.7037315Z inflating: build/bin/kernel_stackbased_test 2025-12-04T09:52:01.7128996Z inflating: build/bin/make_boxed_from_unboxed_functor_test 2025-12-04T09:52:01.7178046Z inflating: build/bin/CppSignature_test 2025-12-04T09:52:01.7232196Z inflating: build/bin/backend_fallback_test 2025-12-04T09:52:01.7279807Z inflating: build/bin/op_allowlist_test 2025-12-04T09:52:01.7568428Z inflating: build/bin/op_registration_test 2025-12-04T09:52:01.7631878Z inflating: build/bin/inline_container_test 2025-12-04T09:52:01.7684638Z inflating: build/bin/cuda_allocator_test 2025-12-04T09:52:01.7735827Z inflating: build/bin/cuda_apply_test 2025-12-04T09:52:01.7805627Z inflating: build/bin/cuda_atomic_ops_test 2025-12-04T09:52:01.7860757Z inflating: build/bin/cuda_caching_host_allocator_test 2025-12-04T09:52:01.7956262Z inflating: build/bin/cuda_complex_math_test 2025-12-04T09:52:01.8025201Z inflating: build/bin/cuda_complex_test 2025-12-04T09:52:01.8092094Z inflating: build/bin/cuda_cub_test 2025-12-04T09:52:01.8143129Z inflating: build/bin/cuda_cublas_handle_pool_test 2025-12-04T09:52:01.8191275Z inflating: build/bin/cuda_device_test 2025-12-04T09:52:01.8275176Z inflating: build/bin/cuda_distributions_test 2025-12-04T09:52:01.8325418Z inflating: build/bin/cuda_dlconvertor_test 2025-12-04T09:52:01.8377243Z inflating: build/bin/cuda_event_test 2025-12-04T09:52:01.8425439Z inflating: build/bin/cuda_exchange_device_test 2025-12-04T09:52:01.8500138Z inflating: build/bin/cuda_generator_test 2025-12-04T09:52:01.8562658Z inflating: build/bin/cuda_half_test 2025-12-04T09:52:01.8620344Z inflating: build/bin/cuda_integer_divider_test 2025-12-04T09:52:01.8683526Z inflating: build/bin/cuda_optional_test 2025-12-04T09:52:01.8752881Z inflating: build/bin/cuda_packedtensoraccessor_test 2025-12-04T09:52:01.8803555Z inflating: build/bin/cuda_reportMemoryUsage_test 2025-12-04T09:52:01.8852200Z inflating: build/bin/cuda_allocatorTraceTracker_test 2025-12-04T09:52:01.8911202Z inflating: build/bin/cuda_stream_test 2025-12-04T09:52:01.8975121Z inflating: build/bin/cuda_vectorized_test 2025-12-04T09:52:01.9023506Z inflating: build/bin/cuda_cudnn_test 2025-12-04T09:52:01.9338266Z inflating: build/bin/test_lazy 2025-12-04T09:52:02.0354824Z inflating: build/bin/test_jit 2025-12-04T09:52:02.0419953Z inflating: build/bin/ProcessGroupGlooTest 2025-12-04T09:52:02.0482199Z inflating: build/bin/ProcessGroupNCCLTest 2025-12-04T09:52:02.0536359Z inflating: build/bin/ProcessGroupGlooAsyncTest 2025-12-04T09:52:02.0596144Z inflating: build/bin/ProcessGroupNCCLErrorsTest 2025-12-04T09:52:02.0609915Z inflating: build/bin/ProcessGroupMPITest 2025-12-04T09:52:02.0614033Z inflating: build/bin/example_allreduce 2025-12-04T09:52:02.0668166Z inflating: build/bin/test_dist_autograd 2025-12-04T09:52:02.0734608Z inflating: build/bin/test_cpp_rpc 2025-12-04T09:52:02.1770039Z inflating: build/bin/test_api 2025-12-04T09:52:02.1772549Z inflating: build/bin/parallel_benchmark 2025-12-04T09:52:02.1776166Z inflating: build/bin/torch_shm_manager 2025-12-04T09:52:02.1776609Z creating: .additional_ci_files/ 2025-12-04T09:52:02.1833214Z inflating: .additional_ci_files/test-times.json 2025-12-04T09:52:02.2036964Z inflating: .additional_ci_files/test-class-times.json 2025-12-04T09:52:02.2072494Z ##[group]Run rm artifacts.zip 2025-12-04T09:52:02.2072723Z rm artifacts.zip 2025-12-04T09:52:02.2084107Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-12-04T09:52:02.2084481Z env: 2025-12-04T09:52:02.2084674Z GIT_DEFAULT_BRANCH: main 2025-12-04T09:52:02.2085102Z HAS_NVIDIA_GPU: true 2025-12-04T09:52:02.2085405Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2025-12-04T09:52:02.2085727Z ##[endgroup] 2025-12-04T09:52:02.3197535Z ##[group]Run df -H 2025-12-04T09:52:02.3197723Z df -H 2025-12-04T09:52:02.3204839Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-12-04T09:52:02.3205121Z env: 2025-12-04T09:52:02.3205277Z GIT_DEFAULT_BRANCH: main 2025-12-04T09:52:02.3205525Z HAS_NVIDIA_GPU: true 2025-12-04T09:52:02.3205786Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2025-12-04T09:52:02.3206052Z ##[endgroup] 2025-12-04T09:52:02.3256971Z Filesystem Size Used Avail Use% Mounted on 2025-12-04T09:52:02.3257327Z devtmpfs 4.2M 0 4.2M 0% /dev 2025-12-04T09:52:02.3257647Z tmpfs 33G 0 33G 0% /dev/shm 2025-12-04T09:52:02.3257958Z tmpfs 13G 779k 13G 1% /run 2025-12-04T09:52:02.3258247Z /dev/nvme0n1p1 161G 55G 107G 34% / 2025-12-04T09:52:02.3258560Z tmpfs 33G 17k 33G 1% /tmp 2025-12-04T09:52:02.3258873Z /dev/nvme0n1p128 11M 1.4M 9.2M 13% /boot/efi 2025-12-04T09:52:02.3259197Z tmpfs 6.5G 0 6.5G 0% /run/user/0 2025-12-04T09:52:02.3288477Z Prepare all required actions 2025-12-04T09:52:02.3289230Z Getting action download info 2025-12-04T09:52:02.4967156Z ##[group]Run ./.github/actions/download-td-artifacts 2025-12-04T09:52:02.4967429Z with: 2025-12-04T09:52:02.4967582Z env: 2025-12-04T09:52:02.4967734Z GIT_DEFAULT_BRANCH: main 2025-12-04T09:52:02.4967944Z HAS_NVIDIA_GPU: true 2025-12-04T09:52:02.4968197Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2025-12-04T09:52:02.4968470Z ##[endgroup] 2025-12-04T09:52:02.5005748Z ##[group]Run seemethere/download-artifact-s3@v4 2025-12-04T09:52:02.5005989Z with: 2025-12-04T09:52:02.5006134Z name: td_results 2025-12-04T09:52:02.5006320Z s3-bucket: gha-artifacts 2025-12-04T09:52:02.5006509Z region: us-east-1 2025-12-04T09:52:02.5006663Z env: 2025-12-04T09:52:02.5006812Z GIT_DEFAULT_BRANCH: main 2025-12-04T09:52:02.5006999Z HAS_NVIDIA_GPU: true 2025-12-04T09:52:02.5007235Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2025-12-04T09:52:02.5007535Z ##[endgroup] 2025-12-04T09:52:02.9375862Z (node:60641) NOTE: We are formalizing our plans to enter AWS SDK for JavaScript (v2) into maintenance mode in 2023. 2025-12-04T09:52:02.9376517Z 2025-12-04T09:52:02.9376787Z Please migrate your code to use AWS SDK for JavaScript (v3). 2025-12-04T09:52:02.9377499Z For more information, check the migration guide at https://a.co/7PzMCcy 2025-12-04T09:52:02.9378970Z (Use `node --trace-warnings ...` to show where the warning was created) 2025-12-04T09:52:03.0381937Z Found 1 objects with prefix pytorch/pytorch/19922826259/td_results/ 2025-12-04T09:52:03.0382748Z Starting download (1/1): /home/ec2-user/actions-runner/_work/pytorch/pytorch/td_results.json 2025-12-04T09:52:03.0913446Z Finished download (1/1): /home/ec2-user/actions-runner/_work/pytorch/pytorch/td_results.json 2025-12-04T09:52:03.0919126Z Artifact download has finished successfully 2025-12-04T09:52:03.1174493Z ##[group]Run mkdir -p .additional_ci_files 2025-12-04T09:52:03.1174946Z mkdir -p .additional_ci_files 2025-12-04T09:52:03.1175341Z mv td_results.json .additional_ci_files/td_results.json || true 2025-12-04T09:52:03.1184533Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-12-04T09:52:03.1184816Z env: 2025-12-04T09:52:03.1184987Z GIT_DEFAULT_BRANCH: main 2025-12-04T09:52:03.1185191Z HAS_NVIDIA_GPU: true 2025-12-04T09:52:03.1185440Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2025-12-04T09:52:03.1185704Z ##[endgroup] 2025-12-04T09:52:03.1304461Z ##[group]Run .github/scripts/parse_ref.py 2025-12-04T09:52:03.1304857Z .github/scripts/parse_ref.py 2025-12-04T09:52:03.1312001Z shell: /usr/bin/bash -e {0} 2025-12-04T09:52:03.1312200Z env: 2025-12-04T09:52:03.1312355Z GIT_DEFAULT_BRANCH: main 2025-12-04T09:52:03.1312547Z HAS_NVIDIA_GPU: true 2025-12-04T09:52:03.1312772Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2025-12-04T09:52:03.1313031Z ##[endgroup] 2025-12-04T09:52:03.1835000Z Setting output branch=main 2025-12-04T09:52:03.1931298Z Prepare all required actions 2025-12-04T09:52:03.1931628Z Getting action download info 2025-12-04T09:52:03.3369957Z ##[group]Run ./.github/actions/filter-test-configs 2025-12-04T09:52:03.3370203Z with: 2025-12-04T09:52:03.3370524Z github-token: *** 2025-12-04T09:52:03.3376465Z test-matrix: {"include": [{"config": "default", "shard": 1, "num_shards": 7, "runner": "linux.g6.4xlarge.experimental.nvidia.gpu", "owners": ["oncall:debug-build"], "mem_leak_check": "mem_leak_check"}, {"config": "default", "shard": 1, "num_shards": 7, "runner": "linux.g6.4xlarge.experimental.nvidia.gpu", "owners": ["oncall:debug-build"], "rerun_disabled_tests": "rerun_disabled_tests"}, {"config": "default", "shard": 2, "num_shards": 7, "runner": "linux.g6.4xlarge.experimental.nvidia.gpu", "owners": ["oncall:debug-build"], "mem_leak_check": "mem_leak_check"}, {"config": "default", "shard": 2, "num_shards": 7, "runner": "linux.g6.4xlarge.experimental.nvidia.gpu", "owners": ["oncall:debug-build"], "rerun_disabled_tests": "rerun_disabled_tests"}, {"config": "default", "shard": 3, "num_shards": 7, "runner": "linux.g6.4xlarge.experimental.nvidia.gpu", "owners": ["oncall:debug-build"], "mem_leak_check": "mem_leak_check"}, {"config": "default", "shard": 3, "num_shards": 7, "runner": "linux.g6.4xlarge.experimental.nvidia.gpu", "owners": ["oncall:debug-build"], "rerun_disabled_tests": "rerun_disabled_tests"}, {"config": "default", "shard": 4, "num_shards": 7, "runner": "linux.g6.4xlarge.experimental.nvidia.gpu", "owners": ["oncall:debug-build"], "mem_leak_check": "mem_leak_check"}, {"config": "default", "shard": 4, "num_shards": 7, "runner": "linux.g6.4xlarge.experimental.nvidia.gpu", "owners": ["oncall:debug-build"], "rerun_disabled_tests": "rerun_disabled_tests"}, {"config": "default", "shard": 5, "num_shards": 7, "runner": "linux.g6.4xlarge.experimental.nvidia.gpu", "owners": ["oncall:debug-build"], "mem_leak_check": "mem_leak_check"}, {"config": "default", "shard": 5, "num_shards": 7, "runner": "linux.g6.4xlarge.experimental.nvidia.gpu", "owners": ["oncall:debug-build"], "rerun_disabled_tests": "rerun_disabled_tests"}, {"config": "default", "shard": 6, "num_shards": 7, "runner": "linux.g6.4xlarge.experimental.nvidia.gpu", "owners": ["oncall:debug-build"], "mem_leak_check": "mem_leak_check"}, {"config": "default", "shard": 6, "num_shards": 7, "runner": "linux.g6.4xlarge.experimental.nvidia.gpu", "owners": ["oncall:debug-build"], "rerun_disabled_tests": "rerun_disabled_tests"}, {"config": "default", "shard": 7, "num_shards": 7, "runner": "linux.g6.4xlarge.experimental.nvidia.gpu", "owners": ["oncall:debug-build"], "mem_leak_check": "mem_leak_check"}, {"config": "default", "shard": 7, "num_shards": 7, "runner": "linux.g6.4xlarge.experimental.nvidia.gpu", "owners": ["oncall:debug-build"], "rerun_disabled_tests": "rerun_disabled_tests"}]} 2025-12-04T09:52:03.3383467Z job-name: linux-jammy-cuda12.8-py3.10-gcc11-debug / test (default, 6, 7, linux.g6.4xlarge.experimental.nvidia.gpu, oncall:debug-build, mem_leak_check) 2025-12-04T09:52:03.3384062Z env: 2025-12-04T09:52:03.3384226Z GIT_DEFAULT_BRANCH: main 2025-12-04T09:52:03.3384423Z HAS_NVIDIA_GPU: true 2025-12-04T09:52:03.3384656Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2025-12-04T09:52:03.3384919Z ##[endgroup] 2025-12-04T09:52:03.3472934Z ##[group]Run nick-fields/retry@v3.0.0 2025-12-04T09:52:03.3473156Z with: 2025-12-04T09:52:03.3473308Z shell: bash 2025-12-04T09:52:03.3473475Z timeout_minutes: 10 2025-12-04T09:52:03.3473658Z max_attempts: 5 2025-12-04T09:52:03.3473823Z retry_wait_seconds: 30 2025-12-04T09:52:03.3474404Z command: set -eux # PyYAML 6.0 doesn't work with MacOS x86 anymore # This must run on Python-3.7 (AmazonLinux2) so can't use request=3.32.2 python3 -m pip install requests==2.27.1 pyyaml==6.0.2 2025-12-04T09:52:03.3474985Z polling_interval_seconds: 1 2025-12-04T09:52:03.3475190Z warning_on_retry: true 2025-12-04T09:52:03.3475373Z continue_on_error: false 2025-12-04T09:52:03.3475555Z env: 2025-12-04T09:52:03.3475899Z GIT_DEFAULT_BRANCH: main 2025-12-04T09:52:03.3476104Z HAS_NVIDIA_GPU: true 2025-12-04T09:52:03.3476345Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2025-12-04T09:52:03.3476739Z GITHUB_TOKEN: *** 2025-12-04T09:52:03.3476916Z ##[endgroup] 2025-12-04T09:52:03.4792497Z + python3 -m pip install requests==2.27.1 pyyaml==6.0.2 2025-12-04T09:52:03.6957673Z Defaulting to user installation because normal site-packages is not writeable 2025-12-04T09:52:03.8385855Z Collecting requests==2.27.1 2025-12-04T09:52:03.8531382Z Downloading requests-2.27.1-py2.py3-none-any.whl (63 kB) 2025-12-04T09:52:04.0582790Z Collecting pyyaml==6.0.2 2025-12-04T09:52:04.0612921Z Downloading PyYAML-6.0.2-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (737 kB) 2025-12-04T09:52:04.1079143Z Requirement already satisfied: urllib3<1.27,>=1.21.1 in /usr/lib/python3.9/site-packages (from requests==2.27.1) (1.25.10) 2025-12-04T09:52:04.4783918Z Collecting charset-normalizer~=2.0.0 2025-12-04T09:52:04.4814870Z Downloading charset_normalizer-2.0.12-py3-none-any.whl (39 kB) 2025-12-04T09:52:04.5745146Z Collecting certifi>=2017.4.17 2025-12-04T09:52:04.5781121Z Downloading certifi-2025.11.12-py3-none-any.whl (159 kB) 2025-12-04T09:52:04.6055494Z Requirement already satisfied: idna<4,>=2.5 in /usr/lib/python3.9/site-packages (from requests==2.27.1) (2.10) 2025-12-04T09:52:04.6791562Z Installing collected packages: charset-normalizer, certifi, requests, pyyaml 2025-12-04T09:52:04.9614874Z Successfully installed certifi-2025.11.12 charset-normalizer-2.0.12 pyyaml-6.0.2 requests-2.27.1 2025-12-04T09:52:05.4563891Z Command completed after 1 attempt(s). 2025-12-04T09:52:05.4641991Z ##[group]Run set -x 2025-12-04T09:52:05.4642181Z set -x 2025-12-04T09:52:05.4642350Z  2025-12-04T09:52:05.4642631Z # Use relative path here as this could be checked out anywhere, not necessarily 2025-12-04T09:52:05.4642977Z # in runner workspace 2025-12-04T09:52:05.4643266Z python3 "${GITHUB_ACTION_PATH}/../../scripts/parse_ref.py" 2025-12-04T09:52:05.4651381Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-12-04T09:52:05.4651672Z env: 2025-12-04T09:52:05.4651830Z GIT_DEFAULT_BRANCH: main 2025-12-04T09:52:05.4652050Z HAS_NVIDIA_GPU: true 2025-12-04T09:52:05.4652299Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2025-12-04T09:52:05.4652762Z ##[endgroup] 2025-12-04T09:52:05.4682962Z + python3 /home/ec2-user/actions-runner/_work/pytorch/pytorch/./.github/actions/filter-test-configs/../../scripts/parse_ref.py 2025-12-04T09:52:05.4853469Z Setting output branch=main 2025-12-04T09:52:05.4924067Z ##[group]Run echo "Workflow: ${GITHUB_WORKFLOW}" 2025-12-04T09:52:05.4924440Z echo "Workflow: ${GITHUB_WORKFLOW}" 2025-12-04T09:52:05.4924753Z echo "Job name: ${JOB_NAME}" 2025-12-04T09:52:05.4925030Z  2025-12-04T09:52:05.4925376Z # Use relative path here as this could be checked out anywhere, not necessarily 2025-12-04T09:52:05.4925825Z # in runner workspace 2025-12-04T09:52:05.4926213Z python3 "${GITHUB_ACTION_PATH}/../../scripts/filter_test_configs.py" \ 2025-12-04T09:52:05.4926643Z  --workflow "${GITHUB_WORKFLOW}" \ 2025-12-04T09:52:05.4926924Z  --job-name "${JOB_NAME}" \ 2025-12-04T09:52:05.4933158Z  --test-matrix "{"include": [{"config": "default", "shard": 1, "num_shards": 7, "runner": "linux.g6.4xlarge.experimental.nvidia.gpu", "owners": ["oncall:debug-build"], "mem_leak_check": "mem_leak_check"}, {"config": "default", "shard": 1, "num_shards": 7, "runner": "linux.g6.4xlarge.experimental.nvidia.gpu", "owners": ["oncall:debug-build"], "rerun_disabled_tests": "rerun_disabled_tests"}, {"config": "default", "shard": 2, "num_shards": 7, "runner": "linux.g6.4xlarge.experimental.nvidia.gpu", "owners": ["oncall:debug-build"], "mem_leak_check": "mem_leak_check"}, {"config": "default", "shard": 2, "num_shards": 7, "runner": "linux.g6.4xlarge.experimental.nvidia.gpu", "owners": ["oncall:debug-build"], "rerun_disabled_tests": "rerun_disabled_tests"}, {"config": "default", "shard": 3, "num_shards": 7, "runner": "linux.g6.4xlarge.experimental.nvidia.gpu", "owners": ["oncall:debug-build"], "mem_leak_check": "mem_leak_check"}, {"config": "default", "shard": 3, "num_shards": 7, "runner": "linux.g6.4xlarge.experimental.nvidia.gpu", "owners": ["oncall:debug-build"], "rerun_disabled_tests": "rerun_disabled_tests"}, {"config": "default", "shard": 4, "num_shards": 7, "runner": "linux.g6.4xlarge.experimental.nvidia.gpu", "owners": ["oncall:debug-build"], "mem_leak_check": "mem_leak_check"}, {"config": "default", "shard": 4, "num_shards": 7, "runner": "linux.g6.4xlarge.experimental.nvidia.gpu", "owners": ["oncall:debug-build"], "rerun_disabled_tests": "rerun_disabled_tests"}, {"config": "default", "shard": 5, "num_shards": 7, "runner": "linux.g6.4xlarge.experimental.nvidia.gpu", "owners": ["oncall:debug-build"], "mem_leak_check": "mem_leak_check"}, {"config": "default", "shard": 5, "num_shards": 7, "runner": "linux.g6.4xlarge.experimental.nvidia.gpu", "owners": ["oncall:debug-build"], "rerun_disabled_tests": "rerun_disabled_tests"}, {"config": "default", "shard": 6, "num_shards": 7, "runner": "linux.g6.4xlarge.experimental.nvidia.gpu", "owners": ["oncall:debug-build"], "mem_leak_check": "mem_leak_check"}, {"config": "default", "shard": 6, "num_shards": 7, "runner": "linux.g6.4xlarge.experimental.nvidia.gpu", "owners": ["oncall:debug-build"], "rerun_disabled_tests": "rerun_disabled_tests"}, {"config": "default", "shard": 7, "num_shards": 7, "runner": "linux.g6.4xlarge.experimental.nvidia.gpu", "owners": ["oncall:debug-build"], "mem_leak_check": "mem_leak_check"}, {"config": "default", "shard": 7, "num_shards": 7, "runner": "linux.g6.4xlarge.experimental.nvidia.gpu", "owners": ["oncall:debug-build"], "rerun_disabled_tests": "rerun_disabled_tests"}]}" \ 2025-12-04T09:52:05.4939372Z  --selected-test-configs "" \ 2025-12-04T09:52:05.4939626Z  --pr-number "${PR_NUMBER}" \ 2025-12-04T09:52:05.4939856Z  --tag "${TAG}" \ 2025-12-04T09:52:05.4940087Z  --event-name "${EVENT_NAME}" \ 2025-12-04T09:52:05.4940323Z  --schedule "${SCHEDULE}" \ 2025-12-04T09:52:05.4940544Z  --branch "${HEAD_BRANCH}" 2025-12-04T09:52:05.4947764Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-12-04T09:52:05.4948050Z env: 2025-12-04T09:52:05.4948371Z GIT_DEFAULT_BRANCH: main 2025-12-04T09:52:05.4948564Z HAS_NVIDIA_GPU: true 2025-12-04T09:52:05.4948795Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2025-12-04T09:52:05.4949333Z GITHUB_TOKEN: *** 2025-12-04T09:52:05.4949838Z JOB_NAME: linux-jammy-cuda12.8-py3.10-gcc11-debug / test (default, 6, 7, linux.g6.4xlarge.experimental.nvidia.gpu, oncall:debug-build, mem_leak_check) 2025-12-04T09:52:05.4950357Z PR_NUMBER: 2025-12-04T09:52:05.4950519Z TAG: 2025-12-04T09:52:05.4950670Z EVENT_NAME: schedule 2025-12-04T09:52:05.4950845Z SCHEDULE: 29 8 * * * 2025-12-04T09:52:05.4951014Z HEAD_BRANCH: main 2025-12-04T09:52:05.4951190Z ##[endgroup] 2025-12-04T09:52:05.4976823Z Workflow: periodic 2025-12-04T09:52:05.4977682Z Job name: linux-jammy-cuda12.8-py3.10-gcc11-debug / test (default, 6, 7, linux.g6.4xlarge.experimental.nvidia.gpu, oncall:debug-build, mem_leak_check) 2025-12-04T09:52:05.6750415Z Setting output keep-going=True 2025-12-04T09:52:05.6750981Z Setting output ci-verbose-test-logs=False 2025-12-04T09:52:05.6751541Z Setting output ci-test-showlocals=False 2025-12-04T09:52:05.6751960Z Setting output ci-no-test-timeout=False 2025-12-04T09:52:05.6752247Z Setting output ci-no-td=False 2025-12-04T09:52:05.6752524Z Setting output ci-td-distributed=False 2025-12-04T09:52:05.6752818Z Setting output is-unstable=False 2025-12-04T09:52:05.6753084Z Setting output reenabled-issues= 2025-12-04T09:52:05.6767352Z Setting output test-matrix={"include": [{"config": "default", "shard": 1, "num_shards": 7, "runner": "linux.g6.4xlarge.experimental.nvidia.gpu", "owners": ["oncall:debug-build"], "mem_leak_check": "mem_leak_check"}, {"config": "default", "shard": 1, "num_shards": 7, "runner": "linux.g6.4xlarge.experimental.nvidia.gpu", "owners": ["oncall:debug-build"], "mem_leak_check": "mem_leak_check", "rerun_disabled_tests": "rerun_disabled_tests"}, {"config": "default", "shard": 1, "num_shards": 7, "runner": "linux.g6.4xlarge.experimental.nvidia.gpu", "owners": ["oncall:debug-build"], "rerun_disabled_tests": "rerun_disabled_tests", "mem_leak_check": "mem_leak_check"}, {"config": "default", "shard": 1, "num_shards": 7, "runner": "linux.g6.4xlarge.experimental.nvidia.gpu", "owners": ["oncall:debug-build"], "rerun_disabled_tests": "rerun_disabled_tests"}, {"config": "default", "shard": 2, "num_shards": 7, "runner": "linux.g6.4xlarge.experimental.nvidia.gpu", "owners": ["oncall:debug-build"], "mem_leak_check": "mem_leak_check"}, {"config": "default", "shard": 2, "num_shards": 7, "runner": "linux.g6.4xlarge.experimental.nvidia.gpu", "owners": ["oncall:debug-build"], "mem_leak_check": "mem_leak_check", "rerun_disabled_tests": "rerun_disabled_tests"}, {"config": "default", "shard": 2, "num_shards": 7, "runner": "linux.g6.4xlarge.experimental.nvidia.gpu", "owners": ["oncall:debug-build"], "rerun_disabled_tests": "rerun_disabled_tests", "mem_leak_check": "mem_leak_check"}, {"config": "default", "shard": 2, "num_shards": 7, "runner": "linux.g6.4xlarge.experimental.nvidia.gpu", "owners": ["oncall:debug-build"], "rerun_disabled_tests": "rerun_disabled_tests"}, {"config": "default", "shard": 3, "num_shards": 7, "runner": "linux.g6.4xlarge.experimental.nvidia.gpu", "owners": ["oncall:debug-build"], "mem_leak_check": "mem_leak_check"}, {"config": "default", "shard": 3, "num_shards": 7, "runner": "linux.g6.4xlarge.experimental.nvidia.gpu", "owners": ["oncall:debug-build"], "mem_leak_check": "mem_leak_check", "rerun_disabled_tests": "rerun_disabled_tests"}, {"config": "default", "shard": 3, "num_shards": 7, "runner": "linux.g6.4xlarge.experimental.nvidia.gpu", "owners": ["oncall:debug-build"], "rerun_disabled_tests": "rerun_disabled_tests", "mem_leak_check": "mem_leak_check"}, {"config": "default", "shard": 3, "num_shards": 7, "runner": "linux.g6.4xlarge.experimental.nvidia.gpu", "owners": ["oncall:debug-build"], "rerun_disabled_tests": "rerun_disabled_tests"}, {"config": "default", "shard": 4, "num_shards": 7, "runner": "linux.g6.4xlarge.experimental.nvidia.gpu", "owners": ["oncall:debug-build"], "mem_leak_check": "mem_leak_check"}, {"config": "default", "shard": 4, "num_shards": 7, "runner": "linux.g6.4xlarge.experimental.nvidia.gpu", "owners": ["oncall:debug-build"], "mem_leak_check": "mem_leak_check", "rerun_disabled_tests": "rerun_disabled_tests"}, {"config": "default", "shard": 4, "num_shards": 7, "runner": "linux.g6.4xlarge.experimental.nvidia.gpu", "owners": ["oncall:debug-build"], "rerun_disabled_tests": "rerun_disabled_tests", "mem_leak_check": "mem_leak_check"}, {"config": "default", "shard": 4, "num_shards": 7, "runner": "linux.g6.4xlarge.experimental.nvidia.gpu", "owners": ["oncall:debug-build"], "rerun_disabled_tests": "rerun_disabled_tests"}, {"config": "default", "shard": 5, "num_shards": 7, "runner": "linux.g6.4xlarge.experimental.nvidia.gpu", "owners": ["oncall:debug-build"], "mem_leak_check": "mem_leak_check"}, {"config": "default", "shard": 5, "num_shards": 7, "runner": "linux.g6.4xlarge.experimental.nvidia.gpu", "owners": ["oncall:debug-build"], "mem_leak_check": "mem_leak_check", "rerun_disabled_tests": "rerun_disabled_tests"}, {"config": "default", "shard": 5, "num_shards": 7, "runner": "linux.g6.4xlarge.experimental.nvidia.gpu", "owners": ["oncall:debug-build"], "rerun_disabled_tests": "rerun_disabled_tests", "mem_leak_check": "mem_leak_check"}, {"config": "default", "shard": 5, "num_shards": 7, "runner": "linux.g6.4xlarge.experimental.nvidia.gpu", "owners": ["oncall:debug-build"], "rerun_disabled_tests": "rerun_disabled_tests"}, {"config": "default", "shard": 6, "num_shards": 7, "runner": "linux.g6.4xlarge.experimental.nvidia.gpu", "owners": ["oncall:debug-build"], "mem_leak_check": "mem_leak_check"}, {"config": "default", "shard": 6, "num_shards": 7, "runner": "linux.g6.4xlarge.experimental.nvidia.gpu", "owners": ["oncall:debug-build"], "mem_leak_check": "mem_leak_check", "rerun_disabled_tests": "rerun_disabled_tests"}, {"config": "default", "shard": 6, "num_shards": 7, "runner": "linux.g6.4xlarge.experimental.nvidia.gpu", "owners": ["oncall:debug-build"], "rerun_disabled_tests": "rerun_disabled_tests", "mem_leak_check": "mem_leak_check"}, {"config": "default", "shard": 6, "num_shards": 7, "runner": "linux.g6.4xlarge.experimental.nvidia.gpu", "owners": ["oncall:debug-build"], "rerun_disabled_tests": "rerun_disabled_tests"}, {"config": "default", "shard": 7, "num_shards": 7, "runner": "linux.g6.4xlarge.experimental.nvidia.gpu", "owners": ["oncall:debug-build"], "mem_leak_check": "mem_leak_check"}, {"config": "default", "shard": 7, "num_shards": 7, "runner": "linux.g6.4xlarge.experimental.nvidia.gpu", "owners": ["oncall:debug-build"], "mem_leak_check": "mem_leak_check", "rerun_disabled_tests": "rerun_disabled_tests"}, {"config": "default", "shard": 7, "num_shards": 7, "runner": "linux.g6.4xlarge.experimental.nvidia.gpu", "owners": ["oncall:debug-build"], "rerun_disabled_tests": "rerun_disabled_tests", "mem_leak_check": "mem_leak_check"}, {"config": "default", "shard": 7, "num_shards": 7, "runner": "linux.g6.4xlarge.experimental.nvidia.gpu", "owners": ["oncall:debug-build"], "rerun_disabled_tests": "rerun_disabled_tests"}]} 2025-12-04T09:52:05.6780878Z Setting output is-test-matrix-empty=False 2025-12-04T09:52:05.6897935Z ##[group]Run echo "Filtered matrix:" 2025-12-04T09:52:05.6898224Z echo "Filtered matrix:" 2025-12-04T09:52:05.6911290Z echo "{"include": [{"config": "default", "shard": 1, "num_shards": 7, "runner": "linux.g6.4xlarge.experimental.nvidia.gpu", "owners": ["oncall:debug-build"], "mem_leak_check": "mem_leak_check"}, {"config": "default", "shard": 1, "num_shards": 7, "runner": "linux.g6.4xlarge.experimental.nvidia.gpu", "owners": ["oncall:debug-build"], "mem_leak_check": "mem_leak_check", "rerun_disabled_tests": "rerun_disabled_tests"}, {"config": "default", "shard": 1, "num_shards": 7, "runner": "linux.g6.4xlarge.experimental.nvidia.gpu", "owners": ["oncall:debug-build"], "rerun_disabled_tests": "rerun_disabled_tests", "mem_leak_check": "mem_leak_check"}, {"config": "default", "shard": 1, "num_shards": 7, "runner": "linux.g6.4xlarge.experimental.nvidia.gpu", "owners": ["oncall:debug-build"], "rerun_disabled_tests": "rerun_disabled_tests"}, {"config": "default", "shard": 2, "num_shards": 7, "runner": "linux.g6.4xlarge.experimental.nvidia.gpu", "owners": ["oncall:debug-build"], "mem_leak_check": "mem_leak_check"}, {"config": "default", "shard": 2, "num_shards": 7, "runner": "linux.g6.4xlarge.experimental.nvidia.gpu", "owners": ["oncall:debug-build"], "mem_leak_check": "mem_leak_check", "rerun_disabled_tests": "rerun_disabled_tests"}, {"config": "default", "shard": 2, "num_shards": 7, "runner": "linux.g6.4xlarge.experimental.nvidia.gpu", "owners": ["oncall:debug-build"], "rerun_disabled_tests": "rerun_disabled_tests", "mem_leak_check": "mem_leak_check"}, {"config": "default", "shard": 2, "num_shards": 7, "runner": "linux.g6.4xlarge.experimental.nvidia.gpu", "owners": ["oncall:debug-build"], "rerun_disabled_tests": "rerun_disabled_tests"}, {"config": "default", "shard": 3, "num_shards": 7, "runner": "linux.g6.4xlarge.experimental.nvidia.gpu", "owners": ["oncall:debug-build"], "mem_leak_check": "mem_leak_check"}, {"config": "default", "shard": 3, "num_shards": 7, "runner": "linux.g6.4xlarge.experimental.nvidia.gpu", "owners": ["oncall:debug-build"], "mem_leak_check": "mem_leak_check", "rerun_disabled_tests": "rerun_disabled_tests"}, {"config": "default", "shard": 3, "num_shards": 7, "runner": "linux.g6.4xlarge.experimental.nvidia.gpu", "owners": ["oncall:debug-build"], "rerun_disabled_tests": "rerun_disabled_tests", "mem_leak_check": "mem_leak_check"}, {"config": "default", "shard": 3, "num_shards": 7, "runner": "linux.g6.4xlarge.experimental.nvidia.gpu", "owners": ["oncall:debug-build"], "rerun_disabled_tests": "rerun_disabled_tests"}, {"config": "default", "shard": 4, "num_shards": 7, "runner": "linux.g6.4xlarge.experimental.nvidia.gpu", "owners": ["oncall:debug-build"], "mem_leak_check": "mem_leak_check"}, {"config": "default", "shard": 4, "num_shards": 7, "runner": "linux.g6.4xlarge.experimental.nvidia.gpu", "owners": ["oncall:debug-build"], "mem_leak_check": "mem_leak_check", "rerun_disabled_tests": "rerun_disabled_tests"}, {"config": "default", "shard": 4, "num_shards": 7, "runner": "linux.g6.4xlarge.experimental.nvidia.gpu", "owners": ["oncall:debug-build"], "rerun_disabled_tests": "rerun_disabled_tests", "mem_leak_check": "mem_leak_check"}, {"config": "default", "shard": 4, "num_shards": 7, "runner": "linux.g6.4xlarge.experimental.nvidia.gpu", "owners": ["oncall:debug-build"], "rerun_disabled_tests": "rerun_disabled_tests"}, {"config": "default", "shard": 5, "num_shards": 7, "runner": "linux.g6.4xlarge.experimental.nvidia.gpu", "owners": ["oncall:debug-build"], "mem_leak_check": "mem_leak_check"}, {"config": "default", "shard": 5, "num_shards": 7, "runner": "linux.g6.4xlarge.experimental.nvidia.gpu", "owners": ["oncall:debug-build"], "mem_leak_check": "mem_leak_check", "rerun_disabled_tests": "rerun_disabled_tests"}, {"config": "default", "shard": 5, "num_shards": 7, "runner": "linux.g6.4xlarge.experimental.nvidia.gpu", "owners": ["oncall:debug-build"], "rerun_disabled_tests": "rerun_disabled_tests", "mem_leak_check": "mem_leak_check"}, {"config": "default", "shard": 5, "num_shards": 7, "runner": "linux.g6.4xlarge.experimental.nvidia.gpu", "owners": ["oncall:debug-build"], "rerun_disabled_tests": "rerun_disabled_tests"}, {"config": "default", "shard": 6, "num_shards": 7, "runner": "linux.g6.4xlarge.experimental.nvidia.gpu", "owners": ["oncall:debug-build"], "mem_leak_check": "mem_leak_check"}, {"config": "default", "shard": 6, "num_shards": 7, "runner": "linux.g6.4xlarge.experimental.nvidia.gpu", "owners": ["oncall:debug-build"], "mem_leak_check": "mem_leak_check", "rerun_disabled_tests": "rerun_disabled_tests"}, {"config": "default", "shard": 6, "num_shards": 7, "runner": "linux.g6.4xlarge.experimental.nvidia.gpu", "owners": ["oncall:debug-build"], "rerun_disabled_tests": "rerun_disabled_tests", "mem_leak_check": "mem_leak_check"}, {"config": "default", "shard": 6, "num_shards": 7, "runner": "linux.g6.4xlarge.experimental.nvidia.gpu", "owners": ["oncall:debug-build"], "rerun_disabled_tests": "rerun_disabled_tests"}, {"config": "default", "shard": 7, "num_shards": 7, "runner": "linux.g6.4xlarge.experimental.nvidia.gpu", "owners": ["oncall:debug-build"], "mem_leak_check": "mem_leak_check"}, {"config": "default", "shard": 7, "num_shards": 7, "runner": "linux.g6.4xlarge.experimental.nvidia.gpu", "owners": ["oncall:debug-build"], "mem_leak_check": "mem_leak_check", "rerun_disabled_tests": "rerun_disabled_tests"}, {"config": "default", "shard": 7, "num_shards": 7, "runner": "linux.g6.4xlarge.experimental.nvidia.gpu", "owners": ["oncall:debug-build"], "rerun_disabled_tests": "rerun_disabled_tests", "mem_leak_check": "mem_leak_check"}, {"config": "default", "shard": 7, "num_shards": 7, "runner": "linux.g6.4xlarge.experimental.nvidia.gpu", "owners": ["oncall:debug-build"], "rerun_disabled_tests": "rerun_disabled_tests"}]}" 2025-12-04T09:52:05.6924481Z  2025-12-04T09:52:05.6924647Z echo 2025-12-04T09:52:05.6924849Z echo "Is the current job unstable? False" 2025-12-04T09:52:05.6925111Z  2025-12-04T09:52:05.6925259Z echo 2025-12-04T09:52:05.6925449Z echo "Is keep-going label set? True" 2025-12-04T09:52:05.6925680Z  2025-12-04T09:52:05.6925828Z echo 2025-12-04T09:52:05.6926003Z echo "Reenabled issues? " 2025-12-04T09:52:05.6933762Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-12-04T09:52:05.6934053Z env: 2025-12-04T09:52:05.6934219Z GIT_DEFAULT_BRANCH: main 2025-12-04T09:52:05.6934420Z HAS_NVIDIA_GPU: true 2025-12-04T09:52:05.6934651Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2025-12-04T09:52:05.6935006Z ##[endgroup] 2025-12-04T09:52:05.6961871Z Filtered matrix: 2025-12-04T09:52:05.6980711Z {include: [{config: default, shard: 1, num_shards: 7, runner: linux.g6.4xlarge.experimental.nvidia.gpu, owners: [oncall:debug-build], mem_leak_check: mem_leak_check}, {config: default, shard: 1, num_shards: 7, runner: linux.g6.4xlarge.experimental.nvidia.gpu, owners: [oncall:debug-build], mem_leak_check: mem_leak_check, rerun_disabled_tests: rerun_disabled_tests}, {config: default, shard: 1, num_shards: 7, runner: linux.g6.4xlarge.experimental.nvidia.gpu, owners: [oncall:debug-build], rerun_disabled_tests: rerun_disabled_tests, mem_leak_check: mem_leak_check}, {config: default, shard: 1, num_shards: 7, runner: linux.g6.4xlarge.experimental.nvidia.gpu, owners: [oncall:debug-build], rerun_disabled_tests: rerun_disabled_tests}, {config: default, shard: 2, num_shards: 7, runner: linux.g6.4xlarge.experimental.nvidia.gpu, owners: [oncall:debug-build], mem_leak_check: mem_leak_check}, {config: default, shard: 2, num_shards: 7, runner: linux.g6.4xlarge.experimental.nvidia.gpu, owners: [oncall:debug-build], mem_leak_check: mem_leak_check, rerun_disabled_tests: rerun_disabled_tests}, {config: default, shard: 2, num_shards: 7, runner: linux.g6.4xlarge.experimental.nvidia.gpu, owners: [oncall:debug-build], rerun_disabled_tests: rerun_disabled_tests, mem_leak_check: mem_leak_check}, {config: default, shard: 2, num_shards: 7, runner: linux.g6.4xlarge.experimental.nvidia.gpu, owners: [oncall:debug-build], rerun_disabled_tests: rerun_disabled_tests}, {config: default, shard: 3, num_shards: 7, runner: linux.g6.4xlarge.experimental.nvidia.gpu, owners: [oncall:debug-build], mem_leak_check: mem_leak_check}, {config: default, shard: 3, num_shards: 7, runner: linux.g6.4xlarge.experimental.nvidia.gpu, owners: [oncall:debug-build], mem_leak_check: mem_leak_check, rerun_disabled_tests: rerun_disabled_tests}, {config: default, shard: 3, num_shards: 7, runner: linux.g6.4xlarge.experimental.nvidia.gpu, owners: [oncall:debug-build], rerun_disabled_tests: rerun_disabled_tests, mem_leak_check: mem_leak_check}, {config: default, shard: 3, num_shards: 7, runner: linux.g6.4xlarge.experimental.nvidia.gpu, owners: [oncall:debug-build], rerun_disabled_tests: rerun_disabled_tests}, {config: default, shard: 4, num_shards: 7, runner: linux.g6.4xlarge.experimental.nvidia.gpu, owners: [oncall:debug-build], mem_leak_check: mem_leak_check}, {config: default, shard: 4, num_shards: 7, runner: linux.g6.4xlarge.experimental.nvidia.gpu, owners: [oncall:debug-build], mem_leak_check: mem_leak_check, rerun_disabled_tests: rerun_disabled_tests}, {config: default, shard: 4, num_shards: 7, runner: linux.g6.4xlarge.experimental.nvidia.gpu, owners: [oncall:debug-build], rerun_disabled_tests: rerun_disabled_tests, mem_leak_check: mem_leak_check}, {config: default, shard: 4, num_shards: 7, runner: linux.g6.4xlarge.experimental.nvidia.gpu, owners: [oncall:debug-build], rerun_disabled_tests: rerun_disabled_tests}, {config: default, shard: 5, num_shards: 7, runner: linux.g6.4xlarge.experimental.nvidia.gpu, owners: [oncall:debug-build], mem_leak_check: mem_leak_check}, {config: default, shard: 5, num_shards: 7, runner: linux.g6.4xlarge.experimental.nvidia.gpu, owners: [oncall:debug-build], mem_leak_check: mem_leak_check, rerun_disabled_tests: rerun_disabled_tests}, {config: default, shard: 5, num_shards: 7, runner: linux.g6.4xlarge.experimental.nvidia.gpu, owners: [oncall:debug-build], rerun_disabled_tests: rerun_disabled_tests, mem_leak_check: mem_leak_check}, {config: default, shard: 5, num_shards: 7, runner: linux.g6.4xlarge.experimental.nvidia.gpu, owners: [oncall:debug-build], rerun_disabled_tests: rerun_disabled_tests}, {config: default, shard: 6, num_shards: 7, runner: linux.g6.4xlarge.experimental.nvidia.gpu, owners: [oncall:debug-build], mem_leak_check: mem_leak_check}, {config: default, shard: 6, num_shards: 7, runner: linux.g6.4xlarge.experimental.nvidia.gpu, owners: [oncall:debug-build], mem_leak_check: mem_leak_check, rerun_disabled_tests: rerun_disabled_tests}, {config: default, shard: 6, num_shards: 7, runner: linux.g6.4xlarge.experimental.nvidia.gpu, owners: [oncall:debug-build], rerun_disabled_tests: rerun_disabled_tests, mem_leak_check: mem_leak_check}, {config: default, shard: 6, num_shards: 7, runner: linux.g6.4xlarge.experimental.nvidia.gpu, owners: [oncall:debug-build], rerun_disabled_tests: rerun_disabled_tests}, {config: default, shard: 7, num_shards: 7, runner: linux.g6.4xlarge.experimental.nvidia.gpu, owners: [oncall:debug-build], mem_leak_check: mem_leak_check}, {config: default, shard: 7, num_shards: 7, runner: linux.g6.4xlarge.experimental.nvidia.gpu, owners: [oncall:debug-build], mem_leak_check: mem_leak_check, rerun_disabled_tests: rerun_disabled_tests}, {config: default, shard: 7, num_shards: 7, runner: linux.g6.4xlarge.experimental.nvidia.gpu, owners: [oncall:debug-build], rerun_disabled_tests: rerun_disabled_tests, mem_leak_check: mem_leak_check}, {config: default, shard: 7, num_shards: 7, runner: linux.g6.4xlarge.experimental.nvidia.gpu, owners: [oncall:debug-build], rerun_disabled_tests: rerun_disabled_tests}]} 2025-12-04T09:52:05.6993736Z 2025-12-04T09:52:05.6993832Z Is the current job unstable? False 2025-12-04T09:52:05.6993985Z 2025-12-04T09:52:05.6994065Z Is keep-going label set? True 2025-12-04T09:52:05.6994199Z 2025-12-04T09:52:05.6994271Z Reenabled issues? 2025-12-04T09:52:05.7046957Z ##[group]Run echo "timeout=$((JOB_TIMEOUT-30))" >> "${GITHUB_OUTPUT}" 2025-12-04T09:52:05.7047402Z echo "timeout=$((JOB_TIMEOUT-30))" >> "${GITHUB_OUTPUT}" 2025-12-04T09:52:05.7054438Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-12-04T09:52:05.7054722Z env: 2025-12-04T09:52:05.7054888Z GIT_DEFAULT_BRANCH: main 2025-12-04T09:52:05.7055086Z HAS_NVIDIA_GPU: true 2025-12-04T09:52:05.7055312Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2025-12-04T09:52:05.7055573Z JOB_TIMEOUT: 600 2025-12-04T09:52:05.7055738Z ##[endgroup] 2025-12-04T09:52:05.7130256Z ##[group]Run env | grep '^GITHUB' >> "/tmp/github_env_${GITHUB_RUN_ID}" 2025-12-04T09:52:05.7130663Z env | grep '^GITHUB' >> "/tmp/github_env_${GITHUB_RUN_ID}" 2025-12-04T09:52:05.7130991Z env | grep '^CI' >> "/tmp/github_env_${GITHUB_RUN_ID}" 2025-12-04T09:52:05.7138014Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-12-04T09:52:05.7138296Z env: 2025-12-04T09:52:05.7138458Z GIT_DEFAULT_BRANCH: main 2025-12-04T09:52:05.7138786Z HAS_NVIDIA_GPU: true 2025-12-04T09:52:05.7139012Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2025-12-04T09:52:05.7139268Z ##[endgroup] 2025-12-04T09:52:05.7265075Z ##[group]Run set -x 2025-12-04T09:52:05.7265349Z set -x 2025-12-04T09:52:05.7265521Z  2025-12-04T09:52:05.7265717Z if [[ $TEST_CONFIG == 'multigpu' ]]; then 2025-12-04T09:52:05.7266009Z  TEST_COMMAND=.ci/pytorch/multigpu-test.sh 2025-12-04T09:52:05.7266306Z elif [[ $BUILD_ENVIRONMENT == *onnx* ]]; then 2025-12-04T09:52:05.7266579Z  TEST_COMMAND=.ci/onnx/test.sh 2025-12-04T09:52:05.7266804Z else 2025-12-04T09:52:05.7267001Z  TEST_COMMAND=.ci/pytorch/test.sh 2025-12-04T09:52:05.7267230Z fi 2025-12-04T09:52:05.7267378Z  2025-12-04T09:52:05.7267576Z # Leaving 1GB for the runner and other things 2025-12-04T09:52:05.7268013Z TOTAL_AVAILABLE_MEMORY_IN_GB=$(awk '/MemTotal/ { printf "%.3f \n", $2/1024/1024 - 1 }' /proc/meminfo) 2025-12-04T09:52:05.7268653Z # https://docs.docker.com/engine/containers/resource_constraints/#--memory-swap-details, the 3GB swap 2025-12-04T09:52:05.7269162Z # comes from https://github.com/pytorch/test-infra/pull/6058 2025-12-04T09:52:05.7269549Z TOTAL_MEMORY_WITH_SWAP=$(("${TOTAL_AVAILABLE_MEMORY_IN_GB%.*}" + 3)) 2025-12-04T09:52:05.7269848Z  2025-12-04T09:52:05.7270037Z if [[ ${BUILD_ENVIRONMENT} == *"s390x"* ]]; then 2025-12-04T09:52:05.7270283Z  SHM_OPTS= 2025-12-04T09:52:05.7270462Z  JENKINS_USER= 2025-12-04T09:52:05.7270710Z  # ensure that docker container cleanly exits in 12 hours 2025-12-04T09:52:05.7271047Z  # if for some reason cleanup action doesn't stop container 2025-12-04T09:52:05.7271329Z  # when job is cancelled 2025-12-04T09:52:05.7271547Z  DOCKER_SHELL_CMD="sleep 12h" 2025-12-04T09:52:05.7271784Z  USED_IMAGE="${DOCKER_IMAGE_S390X}" 2025-12-04T09:52:05.7272020Z else 2025-12-04T09:52:05.7272206Z  SHM_OPTS="--shm-size=${SHM_SIZE}" 2025-12-04T09:52:05.7272441Z  JENKINS_USER="--user jenkins" 2025-12-04T09:52:05.7272663Z  DOCKER_SHELL_CMD= 2025-12-04T09:52:05.7272870Z  USED_IMAGE="${DOCKER_IMAGE}" 2025-12-04T09:52:05.7273074Z fi 2025-12-04T09:52:05.7273223Z  2025-12-04T09:52:05.7273462Z # detached container should get cleaned up by teardown_ec2_linux 2025-12-04T09:52:05.7273846Z # TODO: Stop building test binaries as part of the build phase 2025-12-04T09:52:05.7274278Z # Used for GPU_FLAG, SHM_OPTS, JENKINS_USER and DOCKER_SHELL_CMD since that doesn't play nice 2025-12-04T09:52:05.7274651Z # shellcheck disable=SC2086,SC2090 2025-12-04T09:52:05.7274891Z container_name=$(docker run \ 2025-12-04T09:52:05.7275110Z  ${GPU_FLAG:-} \ 2025-12-04T09:52:05.7275331Z  ${SCCACHE_SERVER_PORT_DOCKER_FLAG:-} \ 2025-12-04T09:52:05.7275586Z  -e BUILD_ENVIRONMENT \ 2025-12-04T09:52:05.7275800Z  -e PR_NUMBER \ 2025-12-04T09:52:05.7275995Z  -e GITHUB_ACTIONS \ 2025-12-04T09:52:05.7276357Z  -e GITHUB_REPOSITORY \ 2025-12-04T09:52:05.7276573Z  -e GITHUB_WORKFLOW \ 2025-12-04T09:52:05.7276778Z  -e GITHUB_JOB \ 2025-12-04T09:52:05.7276973Z  -e GITHUB_RUN_ID \ 2025-12-04T09:52:05.7277187Z  -e GITHUB_RUN_NUMBER \ 2025-12-04T09:52:05.7277392Z  -e GITHUB_RUN_ATTEMPT \ 2025-12-04T09:52:05.7277601Z  -e JOB_ID \ 2025-12-04T09:52:05.7277780Z  -e JOB_NAME \ 2025-12-04T09:52:05.7278253Z  -e BASE_SHA \ 2025-12-04T09:52:05.7278429Z  -e BRANCH \ 2025-12-04T09:52:05.7278605Z  -e SHA1 \ 2025-12-04T09:52:05.7278785Z  -e AWS_DEFAULT_REGION \ 2025-12-04T09:52:05.7278988Z  -e IN_WHEEL_TEST \ 2025-12-04T09:52:05.7279360Z  -e SHARD_NUMBER \ 2025-12-04T09:52:05.7279558Z  -e TEST_CONFIG \ 2025-12-04T09:52:05.7279749Z  -e NUM_TEST_SHARDS \ 2025-12-04T09:52:05.7280089Z  -e REENABLED_ISSUES \ 2025-12-04T09:52:05.7280332Z  -e CONTINUE_THROUGH_ERROR \ 2025-12-04T09:52:05.7280558Z  -e VERBOSE_TEST_LOGS \ 2025-12-04T09:52:05.7280771Z  -e TEST_SHOWLOCALS \ 2025-12-04T09:52:05.7280978Z  -e NO_TEST_TIMEOUT \ 2025-12-04T09:52:05.7281172Z  -e NO_TD \ 2025-12-04T09:52:05.7281347Z  -e TD_DISTRIBUTED \ 2025-12-04T09:52:05.7281550Z  -e PR_LABELS \ 2025-12-04T09:52:05.7281763Z  -e MAX_JOBS="$(nproc --ignore=2)" \ 2025-12-04T09:52:05.7281995Z  -e SCCACHE_BUCKET \ 2025-12-04T09:52:05.7282196Z  -e SCCACHE_REGION \ 2025-12-04T09:52:05.7282388Z  -e XLA_CUDA \ 2025-12-04T09:52:05.7282587Z  -e XLA_CLANG_CACHE_S3_BUCKET_NAME \ 2025-12-04T09:52:05.7282842Z  -e PYTORCH_TEST_CUDA_MEM_LEAK_CHECK \ 2025-12-04T09:52:05.7283108Z  -e PYTORCH_TEST_RERUN_DISABLED_TESTS \ 2025-12-04T09:52:05.7283384Z  -e SKIP_SCCACHE_INITIALIZATION=1 \ 2025-12-04T09:52:05.7283627Z  -e HUGGING_FACE_HUB_TOKEN \ 2025-12-04T09:52:05.7283863Z  -e VLLM_TEST_HUGGING_FACE_TOKEN \ 2025-12-04T09:52:05.7284106Z  -e SCRIBE_GRAPHQL_ACCESS_TOKEN \ 2025-12-04T09:52:05.7284323Z  -e DASHBOARD_TAG \ 2025-12-04T09:52:05.7284526Z  -e ARTIFACTS_FILE_SUFFIX \ 2025-12-04T09:52:05.7284787Z  --memory="${TOTAL_AVAILABLE_MEMORY_IN_GB%.*}g" \ 2025-12-04T09:52:05.7285095Z  --memory-swap="${TOTAL_MEMORY_WITH_SWAP}g" \ 2025-12-04T09:52:05.7285385Z  --env-file="/tmp/github_env_${GITHUB_RUN_ID}" \ 2025-12-04T09:52:05.7285663Z  --security-opt seccomp=unconfined \ 2025-12-04T09:52:05.7285909Z  --cap-add=SYS_PTRACE \ 2025-12-04T09:52:05.7286114Z  --ipc=host \ 2025-12-04T09:52:05.7286301Z  ${SHM_OPTS} \ 2025-12-04T09:52:05.7286482Z  --tty \ 2025-12-04T09:52:05.7286657Z  --detach \ 2025-12-04T09:52:05.7286853Z  --name="${container_name}" \ 2025-12-04T09:52:05.7287079Z  ${JENKINS_USER} \ 2025-12-04T09:52:05.7287329Z  -v "${GITHUB_WORKSPACE}:/var/lib/jenkins/workspace" \ 2025-12-04T09:52:05.7287612Z  -w /var/lib/jenkins/workspace \ 2025-12-04T09:52:05.7287843Z  "${USED_IMAGE}" \ 2025-12-04T09:52:05.7288037Z  ${DOCKER_SHELL_CMD} 2025-12-04T09:52:05.7288220Z ) 2025-12-04T09:52:05.7288458Z echo "DOCKER_CONTAINER_ID=${container_name}" >> "${GITHUB_ENV}" 2025-12-04T09:52:05.7288751Z  2025-12-04T09:52:05.7288937Z if [[ ${BUILD_ENVIRONMENT} == *"s390x"* ]]; then 2025-12-04T09:52:05.7289368Z  docker exec -t "${container_name}" sh -c "python3 -m pip install -r .ci/docker/requirements-ci.txt" 2025-12-04T09:52:05.7289738Z fi 2025-12-04T09:52:05.7289894Z  2025-12-04T09:52:05.7290247Z docker exec -t "${container_name}" sh -c "python3 -m pip install $(echo dist/*.whl)[opt-einsum] && ${TEST_COMMAND}" 2025-12-04T09:52:05.7297449Z shell: /usr/bin/bash -e {0} 2025-12-04T09:52:05.7297655Z env: 2025-12-04T09:52:05.7297813Z GIT_DEFAULT_BRANCH: main 2025-12-04T09:52:05.7298018Z HAS_NVIDIA_GPU: true 2025-12-04T09:52:05.7298257Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2025-12-04T09:52:05.7298588Z BUILD_ENVIRONMENT: linux-jammy-cuda12.8-py3.10-gcc11-debug 2025-12-04T09:52:05.7298875Z PR_NUMBER: 2025-12-04T09:52:05.7299060Z GITHUB_REPOSITORY: pytorch/pytorch 2025-12-04T09:52:05.7299286Z GITHUB_WORKFLOW: periodic 2025-12-04T09:52:05.7299483Z GITHUB_JOB: test 2025-12-04T09:52:05.7299653Z GITHUB_RUN_ID: 19922826259 2025-12-04T09:52:05.7299845Z GITHUB_RUN_NUMBER: 19107 2025-12-04T09:52:05.7300026Z GITHUB_RUN_ATTEMPT: 1 2025-12-04T09:52:05.7300206Z JOB_ID: 57120265687 2025-12-04T09:52:05.7300786Z JOB_NAME: linux-jammy-cuda12.8-py3.10-gcc11-debug / test (default, 6, 7, linux.g6.4xlarge.experimental.nvidia.gpu, oncall:debug-build, mem_leak_check) 2025-12-04T09:52:05.7301400Z BRANCH: main 2025-12-04T09:52:05.7301599Z SHA1: ffd9b0fb4355e97af82fc42cf185c3ffa0fc0a32 2025-12-04T09:52:05.7301873Z BASE_SHA: ffd9b0fb4355e97af82fc42cf185c3ffa0fc0a32 2025-12-04T09:52:05.7302110Z TEST_CONFIG: default 2025-12-04T09:52:05.7302275Z SHARD_NUMBER: 6 2025-12-04T09:52:05.7302441Z NUM_TEST_SHARDS: 7 2025-12-04T09:52:05.7302606Z EXTRA_FLAGS: 2025-12-04T09:52:05.7302774Z OP_BENCHMARK_TESTS: 2025-12-04T09:52:05.7302952Z REENABLED_ISSUES: 2025-12-04T09:52:05.7303124Z CONTINUE_THROUGH_ERROR: True 2025-12-04T09:52:05.7303323Z VERBOSE_TEST_LOGS: False 2025-12-04T09:52:05.7303513Z TEST_SHOWLOCALS: False 2025-12-04T09:52:05.7303784Z NO_TEST_TIMEOUT: False 2025-12-04T09:52:05.7303951Z NO_TD: False 2025-12-04T09:52:05.7304112Z TD_DISTRIBUTED: False 2025-12-04T09:52:05.7304458Z SCCACHE_BUCKET: ossci-compiler-cache-circleci-v2 2025-12-04T09:52:05.7304707Z SCCACHE_REGION: us-east-1 2025-12-04T09:52:05.7304895Z SHM_SIZE: 2g 2025-12-04T09:52:05.7305446Z DOCKER_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/ci-image:pytorch-linux-jammy-cuda12.8-cudnn9-py3-gcc11-f0cd68561080d537ef3d3d6f81b25a6416ad600a 2025-12-04T09:52:05.7306433Z DOCKER_IMAGE_S390X: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/ci-image:pytorch-linux-jammy-cuda12.8-cudnn9-py3-gcc11-f0cd68561080d537ef3d3d6f81b25a6416ad600a 2025-12-04T09:52:05.7307034Z XLA_CUDA: 2025-12-04T09:52:05.7307289Z XLA_CLANG_CACHE_S3_BUCKET_NAME: ossci-compiler-clang-cache-circleci-xla 2025-12-04T09:52:05.7307612Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK: 1 2025-12-04T09:52:05.7307851Z PYTORCH_TEST_RERUN_DISABLED_TESTS: 0 2025-12-04T09:52:05.7308056Z DASHBOARD_TAG: 2025-12-04T09:52:05.7308379Z VLLM_TEST_HUGGING_FACE_TOKEN: *** 2025-12-04T09:52:05.7308700Z HUGGING_FACE_HUB_TOKEN: *** 2025-12-04T09:52:05.7308998Z SCRIBE_GRAPHQL_ACCESS_TOKEN: *** 2025-12-04T09:52:05.7309374Z ARTIFACTS_FILE_SUFFIX: test-default-6-7-linux.g6.4xlarge.experimental.nvidia.gpu_57120265687 2025-12-04T09:52:05.7309758Z ##[endgroup] 2025-12-04T09:52:05.7335291Z + [[ default == \m\u\l\t\i\g\p\u ]] 2025-12-04T09:52:05.7335600Z + [[ linux-jammy-cuda12.8-py3.10-gcc11-debug == *onnx* ]] 2025-12-04T09:52:05.7335877Z + TEST_COMMAND=.ci/pytorch/test.sh 2025-12-04T09:52:05.7338501Z ++ awk '/MemTotal/ { printf "%.3f \n", $2/1024/1024 - 1 }' /proc/meminfo 2025-12-04T09:52:05.7360696Z + TOTAL_AVAILABLE_MEMORY_IN_GB='59.453 ' 2025-12-04T09:52:05.7360985Z + TOTAL_MEMORY_WITH_SWAP=62 2025-12-04T09:52:05.7361310Z + [[ linux-jammy-cuda12.8-py3.10-gcc11-debug == *\s\3\9\0\x* ]] 2025-12-04T09:52:05.7361678Z + SHM_OPTS=--shm-size=2g 2025-12-04T09:52:05.7361921Z + JENKINS_USER='--user jenkins' 2025-12-04T09:52:05.7374324Z + DOCKER_SHELL_CMD= 2025-12-04T09:52:05.7374915Z + USED_IMAGE=308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/ci-image:pytorch-linux-jammy-cuda12.8-cudnn9-py3-gcc11-f0cd68561080d537ef3d3d6f81b25a6416ad600a 2025-12-04T09:52:05.7375540Z +++ nproc --ignore=2 2025-12-04T09:52:05.7624758Z ++ docker run --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all -e BUILD_ENVIRONMENT -e PR_NUMBER -e GITHUB_ACTIONS -e GITHUB_REPOSITORY -e GITHUB_WORKFLOW -e GITHUB_JOB -e GITHUB_RUN_ID -e GITHUB_RUN_NUMBER -e GITHUB_RUN_ATTEMPT -e JOB_ID -e JOB_NAME -e BASE_SHA -e BRANCH -e SHA1 -e AWS_DEFAULT_REGION -e IN_WHEEL_TEST -e SHARD_NUMBER -e TEST_CONFIG -e NUM_TEST_SHARDS -e REENABLED_ISSUES -e CONTINUE_THROUGH_ERROR -e VERBOSE_TEST_LOGS -e TEST_SHOWLOCALS -e NO_TEST_TIMEOUT -e NO_TD -e TD_DISTRIBUTED -e PR_LABELS -e MAX_JOBS=14 -e SCCACHE_BUCKET -e SCCACHE_REGION -e XLA_CUDA -e XLA_CLANG_CACHE_S3_BUCKET_NAME -e PYTORCH_TEST_CUDA_MEM_LEAK_CHECK -e PYTORCH_TEST_RERUN_DISABLED_TESTS -e SKIP_SCCACHE_INITIALIZATION=1 -e HUGGING_FACE_HUB_TOKEN -e VLLM_TEST_HUGGING_FACE_TOKEN -e SCRIBE_GRAPHQL_ACCESS_TOKEN -e DASHBOARD_TAG -e ARTIFACTS_FILE_SUFFIX --memory=59g --memory-swap=62g --env-file=/tmp/github_env_19922826259 --security-opt seccomp=unconfined --cap-add=SYS_PTRACE --ipc=host --shm-size=2g --tty --detach --name= --user jenkins -v /home/ec2-user/actions-runner/_work/pytorch/pytorch:/var/lib/jenkins/workspace -w /var/lib/jenkins/workspace 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/ci-image:pytorch-linux-jammy-cuda12.8-cudnn9-py3-gcc11-f0cd68561080d537ef3d3d6f81b25a6416ad600a 2025-12-04T09:52:15.9152645Z + container_name=657e31a8be774492437c4338a4b954650ffa0203d69f0088e2d19ede1e69df31 2025-12-04T09:52:15.9153362Z + echo DOCKER_CONTAINER_ID=657e31a8be774492437c4338a4b954650ffa0203d69f0088e2d19ede1e69df31 2025-12-04T09:52:15.9154230Z + [[ linux-jammy-cuda12.8-py3.10-gcc11-debug == *\s\3\9\0\x* ]] 2025-12-04T09:52:15.9158438Z ++ echo dist/torch-2.10.0a0+gitffd9b0f-cp310-cp310-linux_x86_64.whl 2025-12-04T09:52:15.9161099Z + docker exec -t 657e31a8be774492437c4338a4b954650ffa0203d69f0088e2d19ede1e69df31 sh -c 'python3 -m pip install dist/torch-2.10.0a0+gitffd9b0f-cp310-cp310-linux_x86_64.whl[opt-einsum] && .ci/pytorch/test.sh' 2025-12-04T09:52:16.3599319Z Processing ./dist/torch-2.10.0a0+gitffd9b0f-cp310-cp310-linux_x86_64.whl (from torch==2.10.0a0+gitffd9b0f) 2025-12-04T09:52:16.6708309Z Requirement already satisfied: filelock in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.10.0a0+gitffd9b0f->torch==2.10.0a0+gitffd9b0f) (3.18.0) 2025-12-04T09:52:16.6711555Z Requirement already satisfied: typing-extensions>=4.10.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.10.0a0+gitffd9b0f->torch==2.10.0a0+gitffd9b0f) (4.12.2) 2025-12-04T09:52:16.6715428Z Requirement already satisfied: sympy>=1.13.3 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.10.0a0+gitffd9b0f->torch==2.10.0a0+gitffd9b0f) (1.13.3) 2025-12-04T09:52:16.6719642Z Requirement already satisfied: networkx>=2.5.1 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.10.0a0+gitffd9b0f->torch==2.10.0a0+gitffd9b0f) (2.8.8) 2025-12-04T09:52:16.6723049Z Requirement already satisfied: jinja2 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.10.0a0+gitffd9b0f->torch==2.10.0a0+gitffd9b0f) (3.1.6) 2025-12-04T09:52:16.6727327Z Requirement already satisfied: fsspec>=0.8.5 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.10.0a0+gitffd9b0f->torch==2.10.0a0+gitffd9b0f) (2025.10.0) 2025-12-04T09:52:16.6739906Z Requirement already satisfied: opt-einsum>=3.3 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.10.0a0+gitffd9b0f->torch==2.10.0a0+gitffd9b0f) (3.3.0) 2025-12-04T09:52:16.7072935Z Requirement already satisfied: numpy>=1.7 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from opt-einsum>=3.3->torch==2.10.0a0+gitffd9b0f->torch==2.10.0a0+gitffd9b0f) (1.22.4) 2025-12-04T09:52:16.7091187Z Requirement already satisfied: mpmath<1.4,>=1.1.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from sympy>=1.13.3->torch==2.10.0a0+gitffd9b0f->torch==2.10.0a0+gitffd9b0f) (1.3.0) 2025-12-04T09:52:16.7143750Z Requirement already satisfied: MarkupSafe>=2.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from jinja2->torch==2.10.0a0+gitffd9b0f->torch==2.10.0a0+gitffd9b0f) (3.0.3) 2025-12-04T09:52:17.0463682Z Installing collected packages: torch 2025-12-04T09:52:27.5187331Z Successfully installed torch-2.10.0a0+gitffd9b0f 2025-12-04T09:52:27.5797384Z + export TERM=vt100 2025-12-04T09:52:27.5797740Z + TERM=vt100 2025-12-04T09:52:27.5799676Z ++ dirname .ci/pytorch/test.sh 2025-12-04T09:52:27.5809351Z + source .ci/pytorch/common.sh 2025-12-04T09:52:27.5812822Z +++ dirname .ci/pytorch/common.sh 2025-12-04T09:52:27.5820259Z ++ source .ci/pytorch/common_utils.sh 2025-12-04T09:52:27.5821349Z +++ declare -f -t trap_add 2025-12-04T09:52:27.5825712Z ++ set -ex -o pipefail 2025-12-04T09:52:27.5826020Z ++ [[ linux-jammy-cuda12.8-py3.10-gcc11-debug == *rocm* ]] 2025-12-04T09:52:27.5826306Z ++ BUILD_TEST_LIBTORCH=0 2025-12-04T09:52:27.5829193Z ++ dirname .ci/pytorch/test.sh 2025-12-04T09:52:27.5836772Z + source .ci/pytorch/common-build.sh 2025-12-04T09:52:27.5838610Z ++ [[ linux-jammy-cuda12.8-py3.10-gcc11-debug != *win-* ]] 2025-12-04T09:52:27.5845001Z ++++ dirname .ci/pytorch/common-build.sh 2025-12-04T09:52:27.5852937Z +++ cd .ci/pytorch 2025-12-04T09:52:27.5853166Z +++ pwd -P 2025-12-04T09:52:27.5855681Z ++ script_dir=/var/lib/jenkins/workspace/.ci/pytorch 2025-12-04T09:52:27.5856066Z ++ [[ linux-jammy-cuda12.8-py3.10-gcc11-debug == *-pch* ]] 2025-12-04T09:52:27.5856332Z ++ which sccache 2025-12-04T09:52:27.5878244Z ++ [[ -z ossci-compiler-cache-circleci-v2 ]] 2025-12-04T09:52:27.5878579Z ++ sccache --stop-server 2025-12-04T09:52:27.5906503Z ++ true 2025-12-04T09:52:27.5906769Z ++ rm -f /var/lib/jenkins/sccache_error.log 2025-12-04T09:52:27.5917812Z ++ trap_add sccache_epilogue EXIT 2025-12-04T09:52:27.5918129Z ++ trap_add_cmd=sccache_epilogue 2025-12-04T09:52:27.5918378Z ++ shift 2025-12-04T09:52:27.5918568Z ++ for trap_add_name in "$@" 2025-12-04T09:52:27.5925532Z ++++ trap -p EXIT 2025-12-04T09:52:27.5929007Z +++ eval 'extract_trap_cmd ' 2025-12-04T09:52:27.5929498Z ++++ extract_trap_cmd 2025-12-04T09:52:27.5929783Z ++++ printf '%s\n' '' 2025-12-04T09:52:27.5931883Z +++ printf '%s\n' sccache_epilogue 2025-12-04T09:52:27.5933438Z ++ trap -- ' 2025-12-04T09:52:27.5933720Z sccache_epilogue' EXIT 2025-12-04T09:52:27.5934115Z ++ [[ -n 1 ]] 2025-12-04T09:52:27.5934496Z ++ echo 'Skipping sccache server initialization, setting environment variables' 2025-12-04T09:52:27.5935033Z Skipping sccache server initialization, setting environment variables 2025-12-04T09:52:27.5935367Z ++ export SCCACHE_IDLE_TIMEOUT=0 2025-12-04T09:52:27.5935578Z ++ SCCACHE_IDLE_TIMEOUT=0 2025-12-04T09:52:27.5935836Z ++ export SCCACHE_ERROR_LOG=/var/lib/jenkins/sccache_error.log 2025-12-04T09:52:27.5936151Z ++ SCCACHE_ERROR_LOG=/var/lib/jenkins/sccache_error.log 2025-12-04T09:52:27.5943390Z ++ export RUST_LOG=sccache::server=error 2025-12-04T09:52:27.5943760Z ++ RUST_LOG=sccache::server=error 2025-12-04T09:52:27.5943984Z ++ sccache --zero-stats 2025-12-04T09:52:28.0243187Z Statistics zeroed. 2025-12-04T09:52:28.0250287Z ++ which ccache 2025-12-04T09:52:28.0275414Z + [[ linux-jammy-cuda12.8-py3.10-gcc11-debug != *rocm* ]] 2025-12-04T09:52:28.0276020Z + [[ linux-jammy-cuda12.8-py3.10-gcc11-debug != *s390x* ]] 2025-12-04T09:52:28.0276390Z + [[ -d /var/lib/jenkins/workspace ]] 2025-12-04T09:52:28.0278966Z ++ stat -c %u /var/lib/jenkins/workspace 2025-12-04T09:52:28.0295400Z + WORKSPACE_ORIGINAL_OWNER_ID=1000 2025-12-04T09:52:28.0295836Z + trap_add cleanup_workspace EXIT 2025-12-04T09:52:28.0296118Z + trap_add_cmd=cleanup_workspace 2025-12-04T09:52:28.0296376Z + shift 2025-12-04T09:52:28.0296574Z + for trap_add_name in "$@" 2025-12-04T09:52:28.0303139Z +++ trap -p EXIT 2025-12-04T09:52:28.0306453Z ++ eval 'extract_trap_cmd trap -- '\'' 2025-12-04T09:52:28.0306841Z sccache_epilogue'\'' EXIT' 2025-12-04T09:52:28.0307049Z +++ extract_trap_cmd trap -- ' 2025-12-04T09:52:28.0307255Z sccache_epilogue' EXIT 2025-12-04T09:52:28.0307454Z +++ printf '%s\n' ' 2025-12-04T09:52:28.0307636Z sccache_epilogue' 2025-12-04T09:52:28.0307818Z ++ printf '%s\n' cleanup_workspace 2025-12-04T09:52:28.0309647Z + trap -- ' 2025-12-04T09:52:28.0309821Z sccache_epilogue 2025-12-04T09:52:28.0309992Z cleanup_workspace' EXIT 2025-12-04T09:52:28.0310237Z + sudo chown -R jenkins /var/lib/jenkins/workspace 2025-12-04T09:52:28.9995151Z + git config --global --add safe.directory /var/lib/jenkins/workspace 2025-12-04T09:52:29.0016799Z + [[ linux-jammy-cuda12.8-py3.10-gcc11-debug == *cuda* ]] 2025-12-04T09:52:29.0019943Z ++ python -c 'import os;import numba.cuda; print(os.path.dirname(numba.cuda.__file__))' 2025-12-04T09:52:29.4247505Z + NUMBA_CUDA_DIR=/opt/conda/envs/py_3.10/lib/python3.10/site-packages/numba/cuda 2025-12-04T09:52:29.4248120Z + '[' -n /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numba/cuda ']' 2025-12-04T09:52:29.4253251Z +++ realpath .ci/pytorch/test.sh 2025-12-04T09:52:29.4264832Z ++ dirname /var/lib/jenkins/workspace/.ci/pytorch/test.sh 2025-12-04T09:52:29.4450524Z + NUMBA_PATCH=/var/lib/jenkins/workspace/.ci/pytorch/numba-cuda-13.patch 2025-12-04T09:52:29.4451749Z + pushd /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numba/cuda 2025-12-04T09:52:29.4452630Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numba/cuda ~/workspace 2025-12-04T09:52:29.4453239Z + patch -p4 2025-12-04T09:52:29.4466208Z patching file cudadrv/driver.py 2025-12-04T09:52:29.4466702Z Hunk #1 succeeded at 357 (offset -8 lines). 2025-12-04T09:52:29.4529444Z + popd 2025-12-04T09:52:29.4529763Z ~/workspace 2025-12-04T09:52:29.4530346Z + echo 'Environment variables:' 2025-12-04T09:52:29.4530787Z Environment variables: 2025-12-04T09:52:29.4531050Z + env 2025-12-04T09:52:29.4539761Z GITHUB_WORKSPACE=/home/ec2-user/actions-runner/_work/pytorch/pytorch 2025-12-04T09:52:29.4540118Z CONTINUE_THROUGH_ERROR=True 2025-12-04T09:52:29.4540560Z BUILD_ENVIRONMENT=linux-jammy-cuda12.8-py3.10-gcc11-debug 2025-12-04T09:52:29.4541181Z VLLM_TEST_HUGGING_FACE_TOKEN=*** 2025-12-04T09:52:29.4541573Z HOSTNAME=657e31a8be77 2025-12-04T09:52:29.4542322Z GITHUB_PATH=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/add_path_147361cc-110b-4b46-b198-55270b1a05c5 2025-12-04T09:52:29.4542812Z GITHUB_ACTION=__run_3 2025-12-04T09:52:29.4543012Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 2025-12-04T09:52:29.4543239Z GITHUB_RUN_NUMBER=19107 2025-12-04T09:52:29.4543413Z TEST_CONFIG=default 2025-12-04T09:52:29.4543700Z GITHUB_REPOSITORY_OWNER_ID=21003710 2025-12-04T09:52:29.4543944Z TORCH_NVCC_FLAGS=-Xfatbin -compress-all 2025-12-04T09:52:29.4544165Z SCCACHE_IDLE_TIMEOUT=0 2025-12-04T09:52:29.4544455Z SCRIBE_GRAPHQL_ACCESS_TOKEN=*** 2025-12-04T09:52:29.4544675Z GITHUB_TRIGGERING_ACTOR=huydhn 2025-12-04T09:52:29.4544882Z GITHUB_REF_TYPE=branch 2025-12-04T09:52:29.4545093Z BASE_SHA=ffd9b0fb4355e97af82fc42cf185c3ffa0fc0a32 2025-12-04T09:52:29.4545340Z XLA_CUDA= 2025-12-04T09:52:29.4545638Z NCCL_LIB_DIR=/usr/local/cuda/lib64/ 2025-12-04T09:52:29.4546132Z HUGGING_FACE_HUB_TOKEN=*** 2025-12-04T09:52:29.4546748Z *** 2025-12-04T09:52:29.4547018Z GITHUB_REPOSITORY_ID=65600975 2025-12-04T09:52:29.4547495Z GITHUB_ACTIONS=true 2025-12-04T09:52:29.4547840Z NVIDIA_DRIVER_CAPABILITIES=all 2025-12-04T09:52:29.4548155Z SCCACHE_ERROR_LOG=/var/lib/jenkins/sccache_error.log 2025-12-04T09:52:29.4548629Z SHA1=ffd9b0fb4355e97af82fc42cf185c3ffa0fc0a32 2025-12-04T09:52:29.4549011Z GITHUB_SHA=ffd9b0fb4355e97af82fc42cf185c3ffa0fc0a32 2025-12-04T09:52:29.4549440Z GITHUB_WORKFLOW_REF=pytorch/pytorch/.github/workflows/periodic.yml@refs/heads/main 2025-12-04T09:52:29.4550034Z UCC_HOME=/usr 2025-12-04T09:52:29.4550290Z VERBOSE_TEST_LOGS=False 2025-12-04T09:52:29.4550546Z GITHUB_REF=refs/heads/main 2025-12-04T09:52:29.4550877Z SHARD_NUMBER=6 2025-12-04T09:52:29.4551134Z GITHUB_REF_PROTECTED=true 2025-12-04T09:52:29.4551393Z HOME=/var/lib/jenkins 2025-12-04T09:52:29.4551732Z GITHUB_API_URL=https://api.github.com 2025-12-04T09:52:29.4552076Z PYTORCH_TEST_RERUN_DISABLED_TESTS=0 2025-12-04T09:52:29.4552402Z UCX_COMMIT=7836b165abdbe468a2f607e7254011c07d788152 2025-12-04T09:52:29.4552838Z USE_SYSTEM_NCCL=1 2025-12-04T09:52:29.4553089Z NUM_TEST_SHARDS=7 2025-12-04T09:52:29.4553319Z UCX_HOME=/usr 2025-12-04T09:52:29.4553899Z GITHUB_STATE=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/save_state_147361cc-110b-4b46-b198-55270b1a05c5 2025-12-04T09:52:29.4554723Z JOB_NAME=linux-jammy-cuda12.8-py3.10-gcc11-debug / test (default, 6, 7, linux.g6.4xlarge.experimental.nvidia.gpu, oncall:debug-build, mem_leak_check) 2025-12-04T09:52:29.4555545Z GITHUB_ENV=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/set_env_147361cc-110b-4b46-b198-55270b1a05c5 2025-12-04T09:52:29.4557007Z GITHUB_EVENT_PATH=/home/ec2-user/actions-runner/_work/_temp/_github_workflow/event.json 2025-12-04T09:52:29.4557489Z GITHUB_EVENT_NAME=schedule 2025-12-04T09:52:29.4557749Z DASHBOARD_TAG= 2025-12-04T09:52:29.4558086Z GITHUB_RUN_ID=19922826259 2025-12-04T09:52:29.4558361Z INSTALLED_OPENBLAS= 2025-12-04T09:52:29.4558848Z GITHUB_STEP_SUMMARY=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/step_summary_147361cc-110b-4b46-b198-55270b1a05c5 2025-12-04T09:52:29.4559708Z GITHUB_ACTOR=huydhn 2025-12-04T09:52:29.4559986Z PR_NUMBER= 2025-12-04T09:52:29.4560301Z DESIRED_CUDA=12.8.1 2025-12-04T09:52:29.4560651Z GITHUB_RUN_ATTEMPT=1 2025-12-04T09:52:29.4560945Z ANACONDA_PYTHON_VERSION=3.10 2025-12-04T09:52:29.4561326Z GITHUB_GRAPHQL_URL=https://api.github.com/graphql 2025-12-04T09:52:29.4561659Z TERM=vt100 2025-12-04T09:52:29.4561895Z INSTALLED_VISION=yes 2025-12-04T09:52:29.4562193Z BRANCH=main 2025-12-04T09:52:29.4562437Z SCCACHE_REGION=us-east-1 2025-12-04T09:52:29.4562718Z OPENSSL_ROOT_DIR=/opt/openssl 2025-12-04T09:52:29.4563071Z BUILD_AOT_INDUCTOR_TEST= 2025-12-04T09:52:29.4563341Z CUDA_PATH=/usr/local/cuda 2025-12-04T09:52:29.4563802Z GITHUB_ACTION_PATH=/home/ec2-user/actions-runner/_work/pytorch/pytorch/./.github/actions/setup-linux 2025-12-04T09:52:29.4564349Z GITHUB_SERVER_URL=https://github.com 2025-12-04T09:52:29.4564688Z UCC_COMMIT=430e241bf5d38cbc73fc7a6b89155397232e3f96 2025-12-04T09:52:29.4565023Z REENABLED_ISSUES= 2025-12-04T09:52:29.4565315Z DOCS= 2025-12-04T09:52:29.4565567Z SHLVL=1 2025-12-04T09:52:29.4565765Z MAX_JOBS=14 2025-12-04T09:52:29.4566076Z GITHUB_ACTOR_ID=475357 2025-12-04T09:52:29.4566422Z GITHUB_WORKFLOW_SHA=ffd9b0fb4355e97af82fc42cf185c3ffa0fc0a32 2025-12-04T09:52:29.4566774Z GITHUB_REF_NAME=main 2025-12-04T09:52:29.4567182Z XLA_CLANG_CACHE_S3_BUCKET_NAME=ossci-compiler-clang-cache-circleci-xla 2025-12-04T09:52:29.4567580Z GITHUB_JOB=test 2025-12-04T09:52:29.4567799Z NO_TEST_TIMEOUT=False 2025-12-04T09:52:29.4568113Z TD_DISTRIBUTED=False 2025-12-04T09:52:29.4568411Z GITHUB_REPOSITORY=pytorch/pytorch 2025-12-04T09:52:29.4568675Z GITHUB_RETENTION_DAYS=90 2025-12-04T09:52:29.4569008Z OPENSSL_DIR=/opt/openssl 2025-12-04T09:52:29.4569292Z GITHUB_ACTION_REPOSITORY= 2025-12-04T09:52:29.4569946Z PATH=/opt/cache/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/opt/conda/envs/py_3.10/bin:/opt/conda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin 2025-12-04T09:52:29.4570599Z GITHUB_BASE_REF= 2025-12-04T09:52:29.4570859Z INSTALLED_ACL= 2025-12-04T09:52:29.4571313Z ARTIFACTS_FILE_SUFFIX=test-default-6-7-linux.g6.4xlarge.experimental.nvidia.gpu_57120265687 2025-12-04T09:52:29.4571767Z CI=true 2025-12-04T09:52:29.4572028Z GITHUB_REPOSITORY_OWNER=pytorch 2025-12-04T09:52:29.4572362Z RUST_LOG=sccache::server=error 2025-12-04T09:52:29.4572683Z JOB_ID=57120265687 2025-12-04T09:52:29.4572894Z GITHUB_HEAD_REF= 2025-12-04T09:52:29.4573166Z GITHUB_ACTION_REF= 2025-12-04T09:52:29.4573501Z SCCACHE_BUCKET=ossci-compiler-cache-circleci-v2 2025-12-04T09:52:29.4573805Z TEST_SHOWLOCALS=False 2025-12-04T09:52:29.4574090Z GITHUB_WORKFLOW=periodic 2025-12-04T09:52:29.4574425Z DEBIAN_FRONTEND=noninteractive 2025-12-04T09:52:29.4574932Z GITHUB_OUTPUT=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/set_output_147361cc-110b-4b46-b198-55270b1a05c5 2025-12-04T09:52:29.4575497Z NO_TD=False 2025-12-04T09:52:29.4575805Z SKIP_SCCACHE_INITIALIZATION=1 2025-12-04T09:52:29.4576126Z NCCL_INCLUDE_DIR=/usr/local/cuda/include/ 2025-12-04T09:52:29.4576394Z _=/usr/bin/env 2025-12-04T09:52:29.4576789Z OLDPWD=/opt/conda/envs/py_3.10/lib/python3.10/site-packages/numba/cuda 2025-12-04T09:52:29.4577268Z ++ python -c 'import site; print(site.getsitepackages()[0])' 2025-12-04T09:52:29.4667646Z + TORCH_INSTALL_DIR=/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch 2025-12-04T09:52:29.4668614Z + TORCH_BIN_DIR=/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/bin 2025-12-04T09:52:29.4669400Z + TORCH_LIB_DIR=/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib 2025-12-04T09:52:29.4670239Z + TORCH_TEST_DIR=/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/test 2025-12-04T09:52:29.4670739Z + BUILD_DIR=build 2025-12-04T09:52:29.4671208Z + BUILD_RENAMED_DIR=build_renamed 2025-12-04T09:52:29.4671658Z + BUILD_BIN_DIR=build/bin 2025-12-04T09:52:29.4672100Z + SHARD_NUMBER=6 2025-12-04T09:52:29.4672425Z + NUM_TEST_SHARDS=7 2025-12-04T09:52:29.4672823Z + export TORCH_SERIALIZATION_DEBUG=1 2025-12-04T09:52:29.4673457Z + TORCH_SERIALIZATION_DEBUG=1 2025-12-04T09:52:29.4673766Z + export VALGRIND=ON 2025-12-04T09:52:29.4674303Z + VALGRIND=ON 2025-12-04T09:52:29.4674713Z + [[ linux-jammy-cuda12.8-py3.10-gcc11-debug == *clang9* ]] 2025-12-04T09:52:29.4675172Z + [[ linux-jammy-cuda12.8-py3.10-gcc11-debug == *xpu* ]] 2025-12-04T09:52:29.4675676Z + detect_cuda_arch 2025-12-04T09:52:29.4676108Z + [[ linux-jammy-cuda12.8-py3.10-gcc11-debug == *cuda* ]] 2025-12-04T09:52:29.4676522Z + command -v nvidia-smi 2025-12-04T09:52:29.4676906Z /usr/bin/nvidia-smi 2025-12-04T09:52:29.4679104Z ++ nvidia-smi --query-gpu=compute_cap --format=csv 2025-12-04T09:52:29.4680271Z ++ tail -n 1 2025-12-04T09:52:29.4928787Z + TORCH_CUDA_ARCH_LIST=8.9 2025-12-04T09:52:29.4929252Z + export TORCH_CUDA_ARCH_LIST 2025-12-04T09:52:29.4929721Z + [[ linux-jammy-cuda12.8-py3.10-gcc11-debug == *s390x* ]] 2025-12-04T09:52:29.4930199Z + [[ 0 == \1 ]] 2025-12-04T09:52:29.4930536Z + [[ True == \1 ]] 2025-12-04T09:52:29.4930967Z + [[ linux-jammy-cuda12.8-py3.10-gcc11-debug != *bazel* ]] 2025-12-04T09:52:29.4933416Z ++ realpath build/custom_test_artifacts 2025-12-04T09:52:29.5061628Z + CUSTOM_TEST_ARTIFACT_BUILD_DIR=/var/lib/jenkins/workspace/build/custom_test_artifacts 2025-12-04T09:52:29.5062232Z + [[ -n '' ]] 2025-12-04T09:52:29.5062708Z + echo 'Environment variables' 2025-12-04T09:52:29.5063072Z Environment variables 2025-12-04T09:52:29.5063392Z + env 2025-12-04T09:52:29.5221498Z GITHUB_WORKSPACE=/home/ec2-user/actions-runner/_work/pytorch/pytorch 2025-12-04T09:52:29.5222736Z CONTINUE_THROUGH_ERROR=True 2025-12-04T09:52:29.5223841Z BUILD_ENVIRONMENT=linux-jammy-cuda12.8-py3.10-gcc11-debug 2025-12-04T09:52:29.5224706Z VLLM_TEST_HUGGING_FACE_TOKEN=*** 2025-12-04T09:52:29.5225343Z HOSTNAME=657e31a8be77 2025-12-04T09:52:29.5225944Z GITHUB_PATH=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/add_path_147361cc-110b-4b46-b198-55270b1a05c5 2025-12-04T09:52:29.5226488Z GITHUB_ACTION=__run_3 2025-12-04T09:52:29.5226842Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 2025-12-04T09:52:29.5227148Z GITHUB_RUN_NUMBER=19107 2025-12-04T09:52:29.5227416Z TEST_CONFIG=default 2025-12-04T09:52:29.5227741Z GITHUB_REPOSITORY_OWNER_ID=21003710 2025-12-04T09:52:29.5228072Z TORCH_NVCC_FLAGS=-Xfatbin -compress-all 2025-12-04T09:52:29.5228386Z SCCACHE_IDLE_TIMEOUT=0 2025-12-04T09:52:29.5228835Z SCRIBE_GRAPHQL_ACCESS_TOKEN=*** 2025-12-04T09:52:29.5229129Z GITHUB_TRIGGERING_ACTOR=huydhn 2025-12-04T09:52:29.5229403Z GITHUB_REF_TYPE=branch 2025-12-04T09:52:29.5229748Z TORCH_CUDA_ARCH_LIST=8.9 2025-12-04T09:52:29.5230158Z BASE_SHA=ffd9b0fb4355e97af82fc42cf185c3ffa0fc0a32 2025-12-04T09:52:29.5230665Z XLA_CUDA= 2025-12-04T09:52:29.5231213Z NCCL_LIB_DIR=/usr/local/cuda/lib64/ 2025-12-04T09:52:29.5231963Z HUGGING_FACE_HUB_TOKEN=*** 2025-12-04T09:52:29.5232456Z *** 2025-12-04T09:52:29.5233020Z GITHUB_REPOSITORY_ID=65600975 2025-12-04T09:52:29.5233346Z GITHUB_ACTIONS=true 2025-12-04T09:52:29.5233573Z NVIDIA_DRIVER_CAPABILITIES=all 2025-12-04T09:52:29.5234027Z SCCACHE_ERROR_LOG=/var/lib/jenkins/sccache_error.log 2025-12-04T09:52:29.5234407Z SHA1=ffd9b0fb4355e97af82fc42cf185c3ffa0fc0a32 2025-12-04T09:52:29.5234737Z GITHUB_SHA=ffd9b0fb4355e97af82fc42cf185c3ffa0fc0a32 2025-12-04T09:52:29.5235127Z GITHUB_WORKFLOW_REF=pytorch/pytorch/.github/workflows/periodic.yml@refs/heads/main 2025-12-04T09:52:29.5235462Z UCC_HOME=/usr 2025-12-04T09:52:29.5235637Z TORCH_SERIALIZATION_DEBUG=1 2025-12-04T09:52:29.5235836Z VERBOSE_TEST_LOGS=False 2025-12-04T09:52:29.5236026Z GITHUB_REF=refs/heads/main 2025-12-04T09:52:29.5236216Z SHARD_NUMBER=6 2025-12-04T09:52:29.5236388Z GITHUB_REF_PROTECTED=true 2025-12-04T09:52:29.5236573Z HOME=/var/lib/jenkins 2025-12-04T09:52:29.5236774Z GITHUB_API_URL=https://api.github.com 2025-12-04T09:52:29.5237021Z PYTORCH_TEST_RERUN_DISABLED_TESTS=0 2025-12-04T09:52:29.5237272Z UCX_COMMIT=7836b165abdbe468a2f607e7254011c07d788152 2025-12-04T09:52:29.5237507Z USE_SYSTEM_NCCL=1 2025-12-04T09:52:29.5237666Z NUM_TEST_SHARDS=7 2025-12-04T09:52:29.5237819Z UCX_HOME=/usr 2025-12-04T09:52:29.5238436Z GITHUB_STATE=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/save_state_147361cc-110b-4b46-b198-55270b1a05c5 2025-12-04T09:52:29.5239324Z JOB_NAME=linux-jammy-cuda12.8-py3.10-gcc11-debug / test (default, 6, 7, linux.g6.4xlarge.experimental.nvidia.gpu, oncall:debug-build, mem_leak_check) 2025-12-04T09:52:29.5240055Z GITHUB_ENV=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/set_env_147361cc-110b-4b46-b198-55270b1a05c5 2025-12-04T09:52:29.5240612Z GITHUB_EVENT_PATH=/home/ec2-user/actions-runner/_work/_temp/_github_workflow/event.json 2025-12-04T09:52:29.5240951Z GITHUB_EVENT_NAME=schedule 2025-12-04T09:52:29.5241147Z DASHBOARD_TAG= 2025-12-04T09:52:29.5241316Z GITHUB_RUN_ID=19922826259 2025-12-04T09:52:29.5241498Z INSTALLED_OPENBLAS= 2025-12-04T09:52:29.5241920Z GITHUB_STEP_SUMMARY=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/step_summary_147361cc-110b-4b46-b198-55270b1a05c5 2025-12-04T09:52:29.5242391Z GITHUB_ACTOR=huydhn 2025-12-04T09:52:29.5242552Z PR_NUMBER= 2025-12-04T09:52:29.5242703Z DESIRED_CUDA=12.8.1 2025-12-04T09:52:29.5242867Z GITHUB_RUN_ATTEMPT=1 2025-12-04T09:52:29.5243040Z VALGRIND=ON 2025-12-04T09:52:29.5243197Z ANACONDA_PYTHON_VERSION=3.10 2025-12-04T09:52:29.5243451Z GITHUB_GRAPHQL_URL=https://api.github.com/graphql 2025-12-04T09:52:29.5243698Z TERM=vt100 2025-12-04T09:52:29.5243842Z INSTALLED_VISION=yes 2025-12-04T09:52:29.5244009Z BRANCH=main 2025-12-04T09:52:29.5244170Z SCCACHE_REGION=us-east-1 2025-12-04T09:52:29.5244367Z OPENSSL_ROOT_DIR=/opt/openssl 2025-12-04T09:52:29.5244572Z BUILD_AOT_INDUCTOR_TEST= 2025-12-04T09:52:29.5244756Z CUDA_PATH=/usr/local/cuda 2025-12-04T09:52:29.5245122Z GITHUB_ACTION_PATH=/home/ec2-user/actions-runner/_work/pytorch/pytorch/./.github/actions/setup-linux 2025-12-04T09:52:29.5245536Z GITHUB_SERVER_URL=https://github.com 2025-12-04T09:52:29.5245789Z UCC_COMMIT=430e241bf5d38cbc73fc7a6b89155397232e3f96 2025-12-04T09:52:29.5246021Z REENABLED_ISSUES= 2025-12-04T09:52:29.5246186Z DOCS= 2025-12-04T09:52:29.5246331Z SHLVL=1 2025-12-04T09:52:29.5246490Z MAX_JOBS=14 2025-12-04T09:52:29.5246637Z GITHUB_ACTOR_ID=475357 2025-12-04T09:52:29.5246886Z GITHUB_WORKFLOW_SHA=ffd9b0fb4355e97af82fc42cf185c3ffa0fc0a32 2025-12-04T09:52:29.5247162Z GITHUB_REF_NAME=main 2025-12-04T09:52:29.5247427Z XLA_CLANG_CACHE_S3_BUCKET_NAME=ossci-compiler-clang-cache-circleci-xla 2025-12-04T09:52:29.5247728Z GITHUB_JOB=test 2025-12-04T09:52:29.5247906Z NO_TEST_TIMEOUT=False 2025-12-04T09:52:29.5248076Z TD_DISTRIBUTED=False 2025-12-04T09:52:29.5248265Z GITHUB_REPOSITORY=pytorch/pytorch 2025-12-04T09:52:29.5248484Z GITHUB_RETENTION_DAYS=90 2025-12-04T09:52:29.5248668Z OPENSSL_DIR=/opt/openssl 2025-12-04T09:52:29.5248859Z GITHUB_ACTION_REPOSITORY= 2025-12-04T09:52:29.5249408Z PATH=/opt/cache/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/opt/conda/envs/py_3.10/bin:/opt/conda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin 2025-12-04T09:52:29.5249965Z GITHUB_BASE_REF= 2025-12-04T09:52:29.5250122Z INSTALLED_ACL= 2025-12-04T09:52:29.5250459Z ARTIFACTS_FILE_SUFFIX=test-default-6-7-linux.g6.4xlarge.experimental.nvidia.gpu_57120265687 2025-12-04T09:52:29.5250828Z CI=true 2025-12-04T09:52:29.5250982Z GITHUB_REPOSITORY_OWNER=pytorch 2025-12-04T09:52:29.5251227Z RUST_LOG=sccache::server=error 2025-12-04T09:52:29.5251420Z JOB_ID=57120265687 2025-12-04T09:52:29.5251572Z GITHUB_HEAD_REF= 2025-12-04T09:52:29.5251731Z GITHUB_ACTION_REF= 2025-12-04T09:52:29.5251933Z SCCACHE_BUCKET=ossci-compiler-cache-circleci-v2 2025-12-04T09:52:29.5252173Z TEST_SHOWLOCALS=False 2025-12-04T09:52:29.5252351Z GITHUB_WORKFLOW=periodic 2025-12-04T09:52:29.5252544Z DEBIAN_FRONTEND=noninteractive 2025-12-04T09:52:29.5252979Z GITHUB_OUTPUT=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/set_output_147361cc-110b-4b46-b198-55270b1a05c5 2025-12-04T09:52:29.5253413Z NO_TD=False 2025-12-04T09:52:29.5253575Z SKIP_SCCACHE_INITIALIZATION=1 2025-12-04T09:52:29.5253789Z NCCL_INCLUDE_DIR=/usr/local/cuda/include/ 2025-12-04T09:52:29.5254195Z OLDPWD=/opt/conda/envs/py_3.10/lib/python3.10/site-packages/numba/cuda 2025-12-04T09:52:29.5254498Z _=/usr/bin/env 2025-12-04T09:52:29.5254663Z + echo 'Testing pytorch' 2025-12-04T09:52:29.5254930Z Testing pytorch 2025-12-04T09:52:29.5255108Z + export LANG=C.UTF-8 2025-12-04T09:52:29.5255272Z + LANG=C.UTF-8 2025-12-04T09:52:29.5255419Z + PR_NUMBER= 2025-12-04T09:52:29.5255581Z + [[ default == \d\e\f\a\u\l\t ]] 2025-12-04T09:52:29.5255793Z + export CUDA_VISIBLE_DEVICES=0 2025-12-04T09:52:29.5255984Z + CUDA_VISIBLE_DEVICES=0 2025-12-04T09:52:29.5256174Z + export HIP_VISIBLE_DEVICES=0 2025-12-04T09:52:29.5256372Z + HIP_VISIBLE_DEVICES=0 2025-12-04T09:52:29.5256558Z + [[ default == \d\i\s\t\r\i\b\u\t\e\d ]] 2025-12-04T09:52:29.5256778Z + [[ default == \s\l\o\w ]] 2025-12-04T09:52:29.5257050Z + [[ linux-jammy-cuda12.8-py3.10-gcc11-debug == *slow-gradcheck* ]] 2025-12-04T09:52:29.5257396Z + [[ linux-jammy-cuda12.8-py3.10-gcc11-debug == *cuda* ]] 2025-12-04T09:52:29.5257674Z + export PYTORCH_TESTING_DEVICE_ONLY_FOR=cuda 2025-12-04T09:52:29.5257926Z + PYTORCH_TESTING_DEVICE_ONLY_FOR=cuda 2025-12-04T09:52:29.5258149Z + [[ default == *crossref* ]] 2025-12-04T09:52:29.5258389Z + [[ linux-jammy-cuda12.8-py3.10-gcc11-debug == *rocm* ]] 2025-12-04T09:52:29.5258689Z + [[ linux-jammy-cuda12.8-py3.10-gcc11-debug == *xpu* ]] 2025-12-04T09:52:29.5258997Z + [[ linux-jammy-cuda12.8-py3.10-gcc11-debug != *-bazel-* ]] 2025-12-04T09:52:29.5259273Z + pip_install ninja==1.10.2 2025-12-04T09:52:29.5259534Z + pip_install_pkg='python3 -m pip install --progress-bar off' 2025-12-04T09:52:29.5259859Z + python3 -m pip install --progress-bar off ninja==1.10.2 2025-12-04T09:52:30.0986608Z Collecting ninja==1.10.2 2025-12-04T09:52:30.1201463Z Downloading ninja-1.10.2-py2.py3-none-manylinux_2_5_x86_64.manylinux1_x86_64.whl.metadata (5.0 kB) 2025-12-04T09:52:30.1570576Z Downloading ninja-1.10.2-py2.py3-none-manylinux_2_5_x86_64.manylinux1_x86_64.whl (108 kB) 2025-12-04T09:52:30.5166570Z Installing collected packages: ninja 2025-12-04T09:52:30.5166981Z Attempting uninstall: ninja 2025-12-04T09:52:30.5173397Z Found existing installation: ninja 1.11.1.4 2025-12-04T09:52:30.5195779Z Uninstalling ninja-1.11.1.4: 2025-12-04T09:52:30.5302854Z Successfully uninstalled ninja-1.11.1.4 2025-12-04T09:52:30.5952161Z Successfully installed ninja-1.10.2 2025-12-04T09:52:30.6385680Z + export PATH=/var/lib/jenkins/.local/bin:/opt/cache/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/opt/conda/envs/py_3.10/bin:/opt/conda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin 2025-12-04T09:52:30.6387135Z + PATH=/var/lib/jenkins/.local/bin:/opt/cache/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/opt/conda/envs/py_3.10/bin:/opt/conda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin 2025-12-04T09:52:30.6388033Z + [[ linux-jammy-cuda12.8-py3.10-gcc11-debug == *aarch64* ]] 2025-12-04T09:52:30.6388603Z + [[ linux-jammy-cuda12.8-py3.10-gcc11-debug == *asan* ]] 2025-12-04T09:52:30.6389293Z + [[ linux-jammy-cuda12.8-py3.10-gcc11-debug == *-debug* ]] 2025-12-04T09:52:30.6390322Z + echo 'We are in debug mode: linux-jammy-cuda12.8-py3.10-gcc11-debug. Expect the python assertion to fail' 2025-12-04T09:52:30.6391051Z We are in debug mode: linux-jammy-cuda12.8-py3.10-gcc11-debug. Expect the python assertion to fail 2025-12-04T09:52:30.6391514Z + cd test 2025-12-04T09:52:30.6391872Z + get_exit_code python -c 'import torch; torch._C._crash_if_debug_asserts_fail(424242)' 2025-12-04T09:52:30.6392277Z + set +e 2025-12-04T09:52:30.6392573Z + python -c 'import torch; torch._C._crash_if_debug_asserts_fail(424242)' 2025-12-04T09:52:31.9356941Z Traceback (most recent call last): 2025-12-04T09:52:31.9357512Z File "", line 1, in 2025-12-04T09:52:31.9358357Z RuntimeError: THPUtils_unpackInt(arg) != 424242 INTERNAL ASSERT FAILED at "/var/lib/jenkins/workspace/torch/csrc/Module.cpp":325, please report a bug to PyTorch. Expect anything but 424242 as an input for debug builds 2025-12-04T09:52:32.1705190Z + retcode=1 2025-12-04T09:52:32.1705835Z + set -e 2025-12-04T09:52:32.1706016Z + return 1 2025-12-04T09:52:32.1707860Z + [[ default == \n\o\g\p\u\_\N\O\_\A\V\X\2 ]] 2025-12-04T09:52:32.1708430Z + [[ default == \n\o\g\p\u\_\A\V\X\5\1\2 ]] 2025-12-04T09:52:32.1708781Z + [[ default == \l\e\g\a\c\y\_\n\v\i\d\i\a\_\d\r\i\v\e\r ]] 2025-12-04T09:52:32.1714624Z + DYNAMO_BENCHMARK_FLAGS=() 2025-12-04T09:52:32.1715491Z + [[ default == *pr_time_benchmarks* ]] 2025-12-04T09:52:32.1715921Z + [[ default == *dynamo_eager* ]] 2025-12-04T09:52:32.1716180Z + [[ default == *aot_eager* ]] 2025-12-04T09:52:32.1716428Z + [[ default == *aot_inductor* ]] 2025-12-04T09:52:32.1716698Z + [[ default == *max_autotune_inductor* ]] 2025-12-04T09:52:32.1716966Z + [[ default == *inductor* ]] 2025-12-04T09:52:32.1717215Z + [[ default == *dynamic* ]] 2025-12-04T09:52:32.1717452Z + [[ default == *cpu* ]] 2025-12-04T09:52:32.1717679Z + [[ default == *xpu* ]] 2025-12-04T09:52:32.1717929Z + DYNAMO_BENCHMARK_FLAGS+=(--device cuda) 2025-12-04T09:52:32.1843994Z + [[ linux-jammy-cuda12.8-py3.10-gcc11-debug == *libtorch* ]] 2025-12-04T09:52:32.1844445Z + [[ linux-jammy-cuda12.8-py3.10-gcc11-debug == *-bazel-* ]] 2025-12-04T09:52:32.1846967Z + cd test 2025-12-04T09:52:32.1848019Z + python -c 'import torch; print(torch.__config__.show())' 2025-12-04T09:52:33.7312216Z PyTorch built with: 2025-12-04T09:52:33.7312501Z - GCC 11.4 2025-12-04T09:52:33.7312705Z - C++ Version: 201703 2025-12-04T09:52:33.7313223Z - Intel(R) oneAPI Math Kernel Library Version 2024.2-Product Build 20240605 for Intel(R) 64 architecture applications 2025-12-04T09:52:33.7313902Z - Intel(R) MKL-DNN v3.7.1 (Git Hash 8d263e693366ef8db40acc569cc7d8edf644556d) 2025-12-04T09:52:33.7314297Z - OpenMP 201511 (a.k.a. OpenMP 4.5) 2025-12-04T09:52:33.7314606Z - LAPACK is enabled (usually provided by MKL) 2025-12-04T09:52:33.7314915Z - NNPACK is enabled 2025-12-04T09:52:33.7315150Z - CPU capability usage: AVX2 2025-12-04T09:52:33.7315388Z - CUDA Runtime 12.8 2025-12-04T09:52:33.7315710Z - NVCC architecture flags: -gencode;arch=compute_89,code=sm_89 2025-12-04T09:52:33.7316044Z - CuDNN 91.0.2 (built against CUDA 12.9) 2025-12-04T09:52:33.7319425Z - Build settings: BLAS_INFO=mkl, BUILD_TYPE=RelWithAssert, COMMIT_SHA=ffd9b0fb4355e97af82fc42cf185c3ffa0fc0a32, CUDA_VERSION=12.8, CUDNN_VERSION=9.10.2, CXX_COMPILER=/opt/cache/bin/c++, CXX_FLAGS= -fvisibility-inlines-hidden -DUSE_PTHREADPOOL -DUSE_KINETO -DLIBKINETO_NOROCTRACER -DLIBKINETO_NOXPUPTI=ON -DUSE_FBGEMM -DUSE_PYTORCH_QNNPACK -DUSE_XNNPACK -DSYMBOLICATE_MOBILE_DEBUG_HANDLE -O2 -fPIC -DC10_NODEPRECATED -Wall -Wextra -Werror=return-type -Werror=non-virtual-dtor -Werror=range-loop-construct -Werror=bool-operation -Wnarrowing -Wno-missing-field-initializers -Wno-unknown-pragmas -Wno-unused-parameter -Wno-strict-overflow -Wno-strict-aliasing -Wno-stringop-overflow -Wsuggest-override -Wno-psabi -Wno-error=old-style-cast -faligned-new -Werror -Wno-maybe-uninitialized -fno-math-errno -fno-trapping-math -Werror=format -Wno-stringop-overflow, FORCE_FALLBACK_CUDA_MPI=1, LAPACK_INFO=mkl, PERF_WITH_AVX=1, PERF_WITH_AVX2=1, TORCH_VERSION=2.10.0, USE_CUDA=ON, USE_CUDNN=ON, USE_CUSPARSELT=ON, USE_GFLAGS=OFF, USE_GLOG=OFF, USE_GLOO=ON, USE_MKL=ON, USE_MKLDNN=ON, USE_MPI=ON, USE_NCCL=ON, USE_NNPACK=ON, USE_OPENMP=ON, USE_ROCM=OFF, USE_ROCM_KERNEL_ASSERT=OFF, USE_XCCL=OFF, USE_XPU=OFF, 2025-12-04T09:52:33.7322831Z 2025-12-04T09:52:34.0079344Z + cd test 2025-12-04T09:52:34.0079729Z + python -c 'import torch; print(torch.__config__.parallel_info())' 2025-12-04T09:52:35.2556899Z ATen/Parallel: 2025-12-04T09:52:35.2557206Z at::get_num_threads() : 8 2025-12-04T09:52:35.2557486Z at::get_num_interop_threads() : 8 2025-12-04T09:52:35.2557766Z OpenMP 201511 (a.k.a. OpenMP 4.5) 2025-12-04T09:52:35.2558031Z omp_get_max_threads() : 8 2025-12-04T09:52:35.2558535Z Intel(R) oneAPI Math Kernel Library Version 2024.2-Product Build 20240605 for Intel(R) 64 architecture applications 2025-12-04T09:52:35.2559062Z mkl_get_max_threads() : 8 2025-12-04T09:52:35.2559409Z Intel(R) MKL-DNN v3.7.1 (Git Hash 8d263e693366ef8db40acc569cc7d8edf644556d) 2025-12-04T09:52:35.2560216Z std::thread::hardware_concurrency() : 16 2025-12-04T09:52:35.2560508Z Environment variables: 2025-12-04T09:52:35.2560923Z OMP_NUM_THREADS : [not set] 2025-12-04T09:52:35.2561165Z MKL_NUM_THREADS : [not set] 2025-12-04T09:52:35.2561410Z ATen parallel backend: OpenMP 2025-12-04T09:52:35.2561572Z 2025-12-04T09:52:35.4954799Z + [[ default == *numpy_2* ]] 2025-12-04T09:52:35.4955312Z + [[ linux-jammy-cuda12.8-py3.10-gcc11-debug == *aarch64* ]] 2025-12-04T09:52:35.4955723Z + [[ default == *backward* ]] 2025-12-04T09:52:35.4956010Z + [[ default == *libtorch_agnostic_targetting* ]] 2025-12-04T09:52:35.4956309Z + [[ default == *xla* ]] 2025-12-04T09:52:35.4956537Z + [[ default == *vllm* ]] 2025-12-04T09:52:35.4956763Z + [[ default == *executorch* ]] 2025-12-04T09:52:35.4956971Z + [[ default == \j\i\t\_\l\e\g\a\c\y ]] 2025-12-04T09:52:35.4957194Z + [[ default == \q\u\a\n\t\i\z\a\t\i\o\n ]] 2025-12-04T09:52:35.4957479Z + [[ linux-jammy-cuda12.8-py3.10-gcc11-debug == *libtorch* ]] 2025-12-04T09:52:35.4957783Z + [[ default == distributed ]] 2025-12-04T09:52:35.4957986Z + [[ default == *operator_benchmark* ]] 2025-12-04T09:52:35.4958222Z + [[ default == *operator_microbenchmark* ]] 2025-12-04T09:52:35.4958465Z + [[ default == *attention_microbenchmark* ]] 2025-12-04T09:52:35.4958695Z + [[ default == *inductor_distributed* ]] 2025-12-04T09:52:35.4958915Z + [[ default == *inductor-halide* ]] 2025-12-04T09:52:35.4959129Z + [[ default == *inductor-pallas* ]] 2025-12-04T09:52:35.4959349Z + [[ default == *inductor-triton-cpu* ]] 2025-12-04T09:52:35.4959679Z + [[ default == *inductor-micro-benchmark* ]] 2025-12-04T09:52:35.4959941Z + [[ default == *aoti_cross_compile_for_windows* ]] 2025-12-04T09:52:35.4960183Z + [[ default == *huggingface* ]] 2025-12-04T09:52:35.4960392Z + [[ default == *timm* ]] 2025-12-04T09:52:35.4960568Z + [[ default == cachebench ]] 2025-12-04T09:52:35.4960764Z + [[ default == verify_cachebench ]] 2025-12-04T09:52:35.4960969Z + [[ default == *torchbench* ]] 2025-12-04T09:52:35.4961175Z + [[ default == *inductor_cpp_wrapper* ]] 2025-12-04T09:52:35.4961397Z + [[ default == *inductor_core* ]] 2025-12-04T09:52:35.4961605Z + [[ default == *inductor* ]] 2025-12-04T09:52:35.4961798Z + [[ default == *einops* ]] 2025-12-04T09:52:35.4961991Z + [[ default == *dynamo_core* ]] 2025-12-04T09:52:35.4962198Z + [[ default == *dynamo_wrapped* ]] 2025-12-04T09:52:35.4962455Z + [[ linux-jammy-cuda12.8-py3.10-gcc11-debug == *rocm* ]] 2025-12-04T09:52:35.4962707Z + [[ 6 == 1 ]] 2025-12-04T09:52:35.4962860Z + [[ 6 == 2 ]] 2025-12-04T09:52:35.4963016Z + [[ 6 -gt 2 ]] 2025-12-04T09:52:35.4963180Z + install_torchvision 2025-12-04T09:52:35.4963367Z + local orig_preload 2025-12-04T09:52:35.4963539Z + local commit 2025-12-04T09:52:35.4963697Z ++ get_pinned_commit vision 2025-12-04T09:52:35.4963909Z ++ cat .github/ci_commit_pins/vision.txt 2025-12-04T09:52:35.4975960Z + commit=617079d944b0e72632311c30ae2bbdf1168b901e 2025-12-04T09:52:35.4976234Z + orig_preload= 2025-12-04T09:52:35.4976566Z + '[' -n '' ']' 2025-12-04T09:52:35.4976969Z + [[ linux-jammy-cuda12.8-py3.10-gcc11-debug == *cuda* ]] 2025-12-04T09:52:35.4977507Z + export FORCE_CUDA=1 2025-12-04T09:52:35.4978103Z + FORCE_CUDA=1 2025-12-04T09:52:35.4978380Z + export WITH_CUDA=1 2025-12-04T09:52:35.4978584Z + WITH_CUDA=1 2025-12-04T09:52:35.4979097Z + pip_build_and_install git+https://github.com/pytorch/vision.git@617079d944b0e72632311c30ae2bbdf1168b901e dist/vision 2025-12-04T09:52:35.4979882Z + local build_target=git+https://github.com/pytorch/vision.git@617079d944b0e72632311c30ae2bbdf1168b901e 2025-12-04T09:52:35.4980388Z + local wheel_dir=dist/vision 2025-12-04T09:52:35.4980623Z + local found_whl=0 2025-12-04T09:52:35.4980847Z + for file in "${wheel_dir}"/*.whl 2025-12-04T09:52:35.4981112Z + [[ -f dist/vision/*.whl ]] 2025-12-04T09:52:35.4981334Z + '[' 0 == 0 ']' 2025-12-04T09:52:35.4981936Z + python3 -m pip wheel --no-build-isolation --no-deps -w dist/vision git+https://github.com/pytorch/vision.git@617079d944b0e72632311c30ae2bbdf1168b901e 2025-12-04T09:52:35.7872688Z Collecting git+https://github.com/pytorch/vision.git@617079d944b0e72632311c30ae2bbdf1168b901e 2025-12-04T09:52:35.7877598Z Cloning https://github.com/pytorch/vision.git (to revision 617079d944b0e72632311c30ae2bbdf1168b901e) to /tmp/pip-req-build-m8luzow2 2025-12-04T09:52:35.8075949Z Running command git clone --filter=blob:none --quiet https://github.com/pytorch/vision.git /tmp/pip-req-build-m8luzow2 2025-12-04T09:52:37.2560207Z Running command git rev-parse -q --verify 'sha^617079d944b0e72632311c30ae2bbdf1168b901e' 2025-12-04T09:52:37.2585078Z Running command git fetch -q https://github.com/pytorch/vision.git 617079d944b0e72632311c30ae2bbdf1168b901e 2025-12-04T09:52:37.3608673Z Resolved https://github.com/pytorch/vision.git to commit 617079d944b0e72632311c30ae2bbdf1168b901e 2025-12-04T09:52:39.2636699Z Preparing metadata (pyproject.toml) ... [?25l- \ | done 2025-12-04T09:52:39.2669290Z [?25hBuilding wheels for collected packages: torchvision 2025-12-04T09:53:51.3941031Z Building wheel for torchvision (pyproject.toml) ... [?25l- \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | done 2025-12-04T09:53:51.3970212Z [?25h Created wheel for torchvision: filename=torchvision-0.25.0a0+617079d-cp310-cp310-linux_x86_64.whl size=1786196 sha256=9252e64a4083251b04ee592f840c428dcabd4335c4c395648aaecfa7cb375d38 2025-12-04T09:53:51.3971232Z Stored in directory: /var/lib/jenkins/.cache/pip/wheels/12/b2/29/1f82685c5b5173629e1f36a9b93989ce92ce563e5fb91d27ac 2025-12-04T09:53:51.4005421Z Successfully built torchvision 2025-12-04T09:53:51.4947193Z + for file in "${wheel_dir}"/*.whl 2025-12-04T09:53:51.4947746Z + pip_install_whl dist/vision/torchvision-0.25.0a0+617079d-cp310-cp310-linux_x86_64.whl 2025-12-04T09:53:51.4948356Z + args=('dist/vision/torchvision-0.25.0a0+617079d-cp310-cp310-linux_x86_64.whl') 2025-12-04T09:53:51.4948832Z + local args 2025-12-04T09:53:51.4949202Z + [[ dist/vision/torchvision-0.25.0a0+617079d-cp310-cp310-linux_x86_64.whl == *\ * ]] 2025-12-04T09:53:51.4949640Z + for path in "${args[@]}" 2025-12-04T09:53:51.4950071Z + echo 'Installing dist/vision/torchvision-0.25.0a0+617079d-cp310-cp310-linux_x86_64.whl' 2025-12-04T09:53:51.4950639Z Installing dist/vision/torchvision-0.25.0a0+617079d-cp310-cp310-linux_x86_64.whl 2025-12-04T09:53:51.4951204Z + python3 -mpip install --no-index --no-deps dist/vision/torchvision-0.25.0a0+617079d-cp310-cp310-linux_x86_64.whl 2025-12-04T09:53:51.7934421Z Processing ./dist/vision/torchvision-0.25.0a0+617079d-cp310-cp310-linux_x86_64.whl 2025-12-04T09:53:51.8015427Z Installing collected packages: torchvision 2025-12-04T09:53:52.2212425Z Successfully installed torchvision-0.25.0a0+617079d 2025-12-04T09:53:52.2491836Z + '[' -n '' ']' 2025-12-04T09:53:52.2492109Z + test_python_shard 6 2025-12-04T09:53:52.2492335Z + [[ -z 7 ]] 2025-12-04T09:53:52.2493043Z + python test/run_test.py --exclude-jit-executor --exclude-distributed-tests --exclude-quantization-tests --shard 6 7 --verbose --upload-artifacts-while-running 2025-12-04T09:53:56.6376451Z Downloading https://ossci-metrics.s3.amazonaws.com/disabled-tests-condensed.json to /var/lib/jenkins/workspace/test/.pytorch-disabled-tests.json 2025-12-04T09:53:56.6910452Z Ignoring disabled issues: [''] 2025-12-04T09:53:56.6987742Z Found test times from artifacts 2025-12-04T09:53:56.7303662Z Found test times from artifacts 2025-12-04T09:53:56.7313041Z Running all tests 2025-12-04T09:53:56.7889350Z Running parallel tests on 1 processes 2025-12-04T09:53:56.7893419Z Name: tests to run (est. time: 183.53min) 2025-12-04T09:53:56.7893850Z Serial tests (73): 2025-12-04T09:53:56.7894092Z inductor/test_torchinductor 1/2 2025-12-04T09:53:56.7894426Z inductor/test_torchinductor_dynamic_shapes 1/4 2025-12-04T09:53:56.7894774Z inductor/test_kernel_benchmark 1/1 2025-12-04T09:53:56.7895070Z inductor/test_torchinductor_opinfo 4/14 2025-12-04T09:53:56.7895770Z inductor/test_torchinductor_opinfo 11/14 2025-12-04T09:53:56.7896068Z dynamo/test_repros 1/1 2025-12-04T09:53:56.7896546Z inductor/test_cuda_select_algorithm 1/1 2025-12-04T09:53:56.7896866Z inductor/test_compile_subprocess 1/2 2025-12-04T09:53:56.7897155Z inductor/test_deterministic 2/3 2025-12-04T09:53:56.7897454Z dynamo/test_fx_graph_runnable 1/1 2025-12-04T09:53:56.7897885Z inductor/test_scatter_optimization 1/1 2025-12-04T09:53:56.7898176Z inductor/test_padding 1/1 2025-12-04T09:53:56.7898426Z dynamo/test_callback 1/1 2025-12-04T09:53:56.7898718Z inductor/test_cudagraph_trees_expandable_segments 1/1 2025-12-04T09:53:56.7899036Z test_autograd 1/1 2025-12-04T09:53:56.7899252Z test_dataloader 2/2 2025-12-04T09:53:56.7899461Z test_decomp 5/17 2025-12-04T09:53:56.7899670Z test_decomp 12/17 2025-12-04T09:53:56.7899888Z test_ops_fwd_gradients 2/2 2025-12-04T09:53:56.7900117Z test_ops_jit 2/2 2025-12-04T09:53:56.7900345Z test_nestedtensor 3/3 2025-12-04T09:53:56.7900583Z test_ops 7/9 2025-12-04T09:53:56.7900916Z functorch/test_ops 2/8 2025-12-04T09:53:56.7901163Z functorch/test_vmap 1/1 2025-12-04T09:53:56.7901396Z inductor/test_smoke 1/1 2025-12-04T09:53:56.7901637Z dynamo/test_after_aot 1/1 2025-12-04T09:53:56.7901884Z inductor/test_snode_runtime 1/1 2025-12-04T09:53:56.7902145Z inductor/test_minifier 1/1 2025-12-04T09:53:56.7902388Z inductor/test_compiled_autograd 1/2 2025-12-04T09:53:56.7902637Z inductor/test_torchinductor_strided_blocks 1/1 2025-12-04T09:53:56.7902888Z inductor/test_pad_mm 1/1 2025-12-04T09:53:56.7903089Z inductor/test_triton_syntax 1/1 2025-12-04T09:53:56.7903314Z inductor/test_triton_extension_backend 1/1 2025-12-04T09:53:56.7903659Z test_sparse_semi_structured 1/1 2025-12-04T09:53:56.7903874Z inductor/test_gpu_cpp_wrapper 2/2 2025-12-04T09:53:56.7904090Z inductor/test_control_deps 1/1 2025-12-04T09:53:56.7904300Z inductor/test_benchmarking 1/1 2025-12-04T09:53:56.7904512Z inductor/test_best_config 1/1 2025-12-04T09:53:56.7904712Z export/test_tools 1/1 2025-12-04T09:53:56.7904916Z inductor/test_compiled_optimizers 1/3 2025-12-04T09:53:56.7905141Z inductor/test_control_flow 5/5 2025-12-04T09:53:56.7905345Z inductor/test_pallas 1/1 2025-12-04T09:53:56.7905536Z test_per_overload_api 1/1 2025-12-04T09:53:56.7905730Z test_multiprocessing 1/1 2025-12-04T09:53:56.7905924Z functorch/test_control_flow 2/3 2025-12-04T09:53:56.7906125Z optim/test_swa_utils 1/1 2025-12-04T09:53:56.7906312Z test_tensorboard 1/1 2025-12-04T09:53:56.7906519Z torch_np/numpy_tests/core/test_indexing 1/1 2025-12-04T09:53:56.7906739Z test_futures 1/1 2025-12-04T09:53:56.7906915Z test_tensor_creation_ops 1/1 2025-12-04T09:53:56.7907133Z torch_np/numpy_tests/linalg/test_linalg 1/1 2025-12-04T09:53:56.7907374Z torch_np/numpy_tests/core/test_dtype 1/1 2025-12-04T09:53:56.7907600Z lazy/test_debug_util 1/1 2025-12-04T09:53:56.7907805Z nn/test_load_state_dict 1/1 2025-12-04T09:53:56.7907995Z test_shape_ops 1/1 2025-12-04T09:53:56.7908189Z profiler/test_memory_profiler 1/1 2025-12-04T09:53:56.7908407Z test_indexing 1/1 2025-12-04T09:53:56.7908609Z torch_np/numpy_tests/core/test_getlimits 1/1 2025-12-04T09:53:56.7908847Z torch_np/test_ndarray_methods 1/1 2025-12-04T09:53:56.7909056Z test_view_ops 1/1 2025-12-04T09:53:56.7909228Z test_type_info 1/1 2025-12-04T09:53:56.7909407Z functorch/test_aotdispatch 1/1 2025-12-04T09:53:56.7909614Z test_native_mha 1/1 2025-12-04T09:53:56.7909826Z torch_np/numpy_tests/core/test_numerictypes 1/1 2025-12-04T09:53:56.7910067Z test_function_schema 1/1 2025-12-04T09:53:56.7910252Z nn/test_init 1/1 2025-12-04T09:53:56.7910447Z torch_np/test_scalars_0D_arrays 1/1 2025-12-04T09:53:56.7910662Z test_cuda_primary_ctx 1/1 2025-12-04T09:53:56.7910894Z profiler/test_profiler_tree 1/1 2025-12-04T09:53:56.7911094Z test_dlpack 1/1 2025-12-04T09:53:56.7911366Z lazy/test_reuse_ir 1/1 2025-12-04T09:53:56.7911581Z test_functional_autograd_benchmark 1/1 2025-12-04T09:53:56.7911901Z distributions/test_distributions 1/1 2025-12-04T09:53:56.7912139Z Parallel tests (0): 2025-12-04T09:53:56.7912327Z Name: excluded (est. time: 0.0min) 2025-12-04T09:53:56.7912527Z Serial tests (0): 2025-12-04T09:53:56.7912708Z Parallel tests (0): 2025-12-04T09:53:56.7913020Z Running inductor/test_torchinductor 1/2 ... [2025-12-04 09:53:56.789722][884.731941021] 2025-12-04T09:53:56.7913367Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2025-12-04T09:53:56.7914135Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'inductor/test_torchinductor.py', '--shard-id=1', '--num-shards=2', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-12-04 09:53:56.790043] 2025-12-04T09:59:09.5444593Z 2025-12-04T09:59:09.5445770Z inductor/test_torchinductor 1/2 was successful, full logs can be found in artifacts with path test/test-reports/inductor.test_torchinductor_1.2_e5d5d93df97dd444_.log 2025-12-04T09:59:09.5635394Z Running 475 items in this shard: test/inductor/test_torchinductor.py::SweepInputsGPUTest::test_cuda_broadcast1_broadcast2, test/inductor/test_torchinductor.py::SweepInputsGPUTest::test_cuda_broadcast1_strided, test/inductor/test_torchinductor.py::SweepInputsGPUTest::test_cuda_broadcast1_transposed, test/inductor/test_torchinductor.py::SweepInputsGPUTest::test_cuda_broadcast2_broadcast2, test/inductor/test_torchinductor.py::SweepInputsGPUTest::test_cuda_broadcast2_double, test/inductor/test_torchinductor.py::SweepInputsGPUTest::test_cuda_broadcast2_int, test/inductor/test_torchinductor.py::SweepInputsGPUTest::test_cuda_broadcast2_transposed, test/inductor/test_torchinductor.py::SweepInputsGPUTest::test_cuda_broadcast3_dense, test/inductor/test_torchinductor.py::SweepInputsGPUTest::test_cuda_broadcast3_int, test/inductor/test_torchinductor.py::SweepInputsGPUTest::test_cuda_dense_broadcast1, test/inductor/test_torchinductor.py::SweepInputsGPUTest::test_cuda_dense_broadcast3, test/inductor/test_torchinductor.py::SweepInputsGPUTest::test_cuda_dense_dense, test/inductor/test_torchinductor.py::SweepInputsGPUTest::test_cuda_dense_double, test/inductor/test_torchinductor.py::SweepInputsGPUTest::test_cuda_dense_strided, test/inductor/test_torchinductor.py::SweepInputsGPUTest::test_cuda_double_broadcast2, test/inductor/test_torchinductor.py::SweepInputsGPUTest::test_cuda_double_double, test/inductor/test_torchinductor.py::SweepInputsGPUTest::test_cuda_double_int, test/inductor/test_torchinductor.py::SweepInputsGPUTest::test_cuda_double_strided, test/inductor/test_torchinductor.py::SweepInputsGPUTest::test_cuda_double_transposed, test/inductor/test_torchinductor.py::SweepInputsGPUTest::test_cuda_int_broadcast3, test/inductor/test_torchinductor.py::SweepInputsGPUTest::test_cuda_int_strided, test/inductor/test_torchinductor.py::SweepInputsGPUTest::test_cuda_int_transposed, test/inductor/test_torchinductor.py::SweepInputsGPUTest::test_cuda_strided_broadcast1, test/inductor/test_torchinductor.py::SweepInputsGPUTest::test_cuda_strided_broadcast3, test/inductor/test_torchinductor.py::SweepInputsGPUTest::test_cuda_strided_strided, test/inductor/test_torchinductor.py::SweepInputsGPUTest::test_cuda_strided_transposed, test/inductor/test_torchinductor.py::SweepInputsGPUTest::test_cuda_transposed_broadcast2, test/inductor/test_torchinductor.py::SweepInputsGPUTest::test_cuda_transposed_double, test/inductor/test_torchinductor.py::SweepInputsGPUTest::test_cuda_transposed_int, test/inductor/test_torchinductor.py::GPUTests::test_AllenaiLongformerBase_repro_cuda, test/inductor/test_torchinductor.py::GPUTests::test__dyn_quant_pack_4bit_weight_bf16_cuda, test/inductor/test_torchinductor.py::GPUTests::test__unsafe_masked_index_put_accumulate_cuda, test/inductor/test_torchinductor.py::GPUTests::test_abs_cuda, test/inductor/test_torchinductor.py::GPUTests::test_adaptive_avg_pool1d_argmax_cuda, test/inductor/test_torchinductor.py::GPUTests::test_adaptive_avg_pool2d1_cuda, test/inductor/test_torchinductor.py::GPUTests::test_adaptive_avg_pool_with_output_size_0_cuda, test/inductor/test_torchinductor.py::GPUTests::test_adaptive_max_pool2d1_cuda, test/inductor/test_torchinductor.py::GPUTests::test_adaptive_max_pool2d3_cuda, test/inductor/test_torchinductor.py::GPUTests::test_adaptive_pool_errors_with_long_cuda, test/inductor/test_torchinductor.py::GPUTests::test_add_complex10_cuda, test/inductor/test_torchinductor.py::GPUTests::test_add_complex4_cuda, test/inductor/test_torchinductor.py::GPUTests::test_add_complex5_cuda, test/inductor/test_torchinductor.py::GPUTests::test_add_complex6_cuda, test/inductor/test_torchinductor.py::GPUTests::test_add_complex8_cuda, test/inductor/test_torchinductor.py::GPUTests::test_add_complex9_cuda, test/inductor/test_torchinductor.py::GPUTests::test_add_const_float_cuda, test/inductor/test_torchinductor.py::GPUTests::test_add_inplace_permuted_cuda, test/inductor/test_torchinductor.py::GPUTests::test_addmm_cuda, test/inductor/test_torchinductor.py::GPUTests::test_alexnet_prefix_cuda, test/inductor/test_torchinductor.py::GPUTests::test_allow_reuse_active_if_under_peak_cuda, test/inductor/test_torchinductor.py::GPUTests::test_angle_cuda, test/inductor/test_torchinductor.py::GPUTests::test_any_cuda, test/inductor/test_torchinductor.py::GPUTests::test_aoti_eager_with_persistent_cache_cuda, test/inductor/test_torchinductor.py::GPUTests::test_aoti_eager_with_scalar_cuda, test/inductor/test_torchinductor.py::GPUTests::test_arange1_cuda, test/inductor/test_torchinductor.py::GPUTests::test_arange2_cuda, test/inductor/test_torchinductor.py::GPUTests::test_arange6_cuda, test/inductor/test_torchinductor.py::GPUTests::test_argmax_argmin1_cuda, test/inductor/test_torchinductor.py::GPUTests::test_argmax_argmin2_cuda, test/inductor/test_torchinductor.py::GPUTests::test_as_strided_cuda, test/inductor/test_torchinductor.py::GPUTests::test_as_strided_on_views_cuda, test/inductor/test_torchinductor.py::GPUTests::test_assert_alignment_op_name_fail_cuda, test/inductor/test_torchinductor.py::GPUTests::test_assert_size_stride_op_name_fail_cuda, test/inductor/test_torchinductor.py::GPUTests::test_assert_size_stride_op_name_pass_cuda, test/inductor/test_torchinductor.py::GPUTests::test_avg_pool2d4_cuda, test/inductor/test_torchinductor.py::GPUTests::test_avg_pool2d6_cuda, test/inductor/test_torchinductor.py::GPUTests::test_avg_pool2d7_cuda, test/inductor/test_torchinductor.py::GPUTests::test_avg_pool2d8_cuda, test/inductor/test_torchinductor.py::GPUTests::test_avg_pool3d_backward4_cuda, test/inductor/test_torchinductor.py::GPUTests::test_batch_norm_2d_2_cuda, test/inductor/test_torchinductor.py::GPUTests::test_batch_norm_2d_cuda, test/inductor/test_torchinductor.py::GPUTests::test_bitwise2_cuda, test/inductor/test_torchinductor.py::GPUTests::test_bmm2_cuda, test/inductor/test_torchinductor.py::GPUTests::test_both_scalars_cuda, test/inductor/test_torchinductor.py::GPUTests::test_bucketize_add_autotune_cuda, test/inductor/test_torchinductor.py::GPUTests::test_bucketize_computed_offsets_cuda, test/inductor/test_torchinductor.py::GPUTests::test_bucketize_default_kwargs_cuda, test/inductor/test_torchinductor.py::GPUTests::test_bucketize_int_int32_int16_cuda, test/inductor/test_torchinductor.py::GPUTests::test_bucketize_int_int32_int32_cuda, test/inductor/test_torchinductor.py::GPUTests::test_bucketize_int_int32_uint8_cuda, test/inductor/test_torchinductor.py::GPUTests::test_bucketize_int_int64_int16_cuda, test/inductor/test_torchinductor.py::GPUTests::test_bucketize_int_int64_int32_cuda, test/inductor/test_torchinductor.py::GPUTests::test_bucketize_int_int64_int64_cuda, test/inductor/test_torchinductor.py::GPUTests::test_bucketize_int_int64_int8_cuda, test/inductor/test_torchinductor.py::GPUTests::test_bucketize_int_int64_uint8_cuda, test/inductor/test_torchinductor.py::GPUTests::test_bucketize_int_int8_int8_cuda, test/inductor/test_torchinductor.py::GPUTests::test_bucketize_int_int8_uint8_cuda, test/inductor/test_torchinductor.py::GPUTests::test_bucketize_int_uint8_int32_cuda, test/inductor/test_torchinductor.py::GPUTests::test_bucketize_int_uint8_int64_cuda, test/inductor/test_torchinductor.py::GPUTests::test_bucketize_int_uint8_int8_cuda, test/inductor/test_torchinductor.py::GPUTests::test_builtins_round_float_ndigits_neg_cuda, test/inductor/test_torchinductor.py::GPUTests::test_builtins_round_int_ndigits_pos_cuda, test/inductor/test_torchinductor.py::GPUTests::test_cat_cuda, test/inductor/test_torchinductor.py::GPUTests::test_cat_empty_cuda, test/inductor/test_torchinductor.py::GPUTests::test_cat_empty_index_cuda, test/inductor/test_torchinductor.py::GPUTests::test_cat_single_empty_cuda, test/inductor/test_torchinductor.py::GPUTests::test_cat_unbacked_empty_1d_cuda, test/inductor/test_torchinductor.py::GPUTests::test_cauchy_cuda, test/inductor/test_torchinductor.py::GPUTests::test_chunk_recompiles_cuda, test/inductor/test_torchinductor.py::GPUTests::test_clamp_type_promotion_cuda, test/inductor/test_torchinductor.py::GPUTests::test_clamp_type_promotion_non_tensor_cuda, test/inductor/test_torchinductor.py::GPUTests::test_clone_cuda, test/inductor/test_torchinductor.py::GPUTests::test_compar_cuda, test/inductor/test_torchinductor.py::GPUTests::test_complex_fallback_cuda, test/inductor/test_torchinductor.py::GPUTests::test_complex_from_real_imag_cuda, test/inductor/test_torchinductor.py::GPUTests::test_complex_memory_overlap_cuda, test/inductor/test_torchinductor.py::GPUTests::test_computed_buffer_inlining_cuda, test/inductor/test_torchinductor.py::GPUTests::test_concat_add_inplace_cuda, test/inductor/test_torchinductor.py::GPUTests::test_config_option_dont_assume_alignment_cudagraphs_cuda, test/inductor/test_torchinductor.py::GPUTests::test_config_option_dont_assume_alignment_recompiles_cuda, test/inductor/test_torchinductor.py::GPUTests::test_consecutive_split_cumprod_cuda, test/inductor/test_torchinductor.py::GPUTests::test_const_int32_to_float_cuda, test/inductor/test_torchinductor.py::GPUTests::test_constant_pad_2d_strides_nonpositive_cuda, test/inductor/test_torchinductor.py::GPUTests::test_constant_pad_3d_cuda, test/inductor/test_torchinductor.py::GPUTests::test_constant_pad_fill_dtype_cuda, test/inductor/test_torchinductor.py::GPUTests::test_constant_pad_nd_inplace_cuda, test/inductor/test_torchinductor.py::GPUTests::test_conv2d_backward_channels_last_cuda, test/inductor/test_torchinductor.py::GPUTests::test_conv2d_channels_last_cuda, test/inductor/test_torchinductor.py::GPUTests::test_conv3d_channels_last_use_block_ptr_False_cuda, test/inductor/test_torchinductor.py::GPUTests::test_conv3d_cuda, test/inductor/test_torchinductor.py::GPUTests::test_conv_shape_check_cuda, test/inductor/test_torchinductor.py::GPUTests::test_convolution1_cuda, test/inductor/test_torchinductor.py::GPUTests::test_convolution3_cuda, test/inductor/test_torchinductor.py::GPUTests::test_copy_non_blocking_is_pinned_use_cat_False_cuda, test/inductor/test_torchinductor.py::GPUTests::test_copy_with_scalar_src_cuda, test/inductor/test_torchinductor.py::GPUTests::test_cpu_scalar_with_cpu_scalar_cuda, test/inductor/test_torchinductor.py::GPUTests::test_cpu_scalar_with_cpu_tensor_cuda, test/inductor/test_torchinductor.py::GPUTests::test_cpu_scalar_with_gpu_tensor_cpp_cuda, test/inductor/test_torchinductor.py::GPUTests::test_cpu_tensor_with_cpu_tensor_cuda, test/inductor/test_torchinductor.py::GPUTests::test_cudnn_rnn_cuda, test/inductor/test_torchinductor.py::GPUTests::test_cummin_cuda, test/inductor/test_torchinductor.py::GPUTests::test_cumprod_zero_dim_cuda, test/inductor/test_torchinductor.py::GPUTests::test_cumsum_inf_cuda, test/inductor/test_torchinductor.py::GPUTests::test_cumsum_pattern_matcher_issue_cuda, test/inductor/test_torchinductor.py::GPUTests::test_custom_op_3_cuda, test/inductor/test_torchinductor.py::GPUTests::test_custom_op_unbacked_symints_cuda, test/inductor/test_torchinductor.py::GPUTests::test_data_type_propogation_cuda, test/inductor/test_torchinductor.py::GPUTests::test_dense_mask_index_cuda, test/inductor/test_torchinductor.py::GPUTests::test_deterministic_codegen_on_graph_break_cuda, test/inductor/test_torchinductor.py::GPUTests::test_deterministic_codegen_with_suffix_cuda, test/inductor/test_torchinductor.py::GPUTests::test_device_assert_cuda, test/inductor/test_torchinductor.py::GPUTests::test_div1_cuda, test/inductor/test_torchinductor.py::GPUTests::test_div2_cuda, test/inductor/test_torchinductor.py::GPUTests::test_div3_cuda, test/inductor/test_torchinductor.py::GPUTests::test_div4_cuda, test/inductor/test_torchinductor.py::GPUTests::test_div7_cuda, test/inductor/test_torchinductor.py::GPUTests::test_div_by_zero_cuda, test/inductor/test_torchinductor.py::GPUTests::test_div_precision_cuda, test/inductor/test_torchinductor.py::GPUTests::test_div_presicion_accuracy_cuda, test/inductor/test_torchinductor.py::GPUTests::test_dont_constant_fold_cuda, test/inductor/test_torchinductor.py::GPUTests::test_dropout3_cuda, test/inductor/test_torchinductor.py::GPUTests::test_dropout_cuda, test/inductor/test_torchinductor.py::GPUTests::test_dropout_trivial_1_cuda, test/inductor/test_torchinductor.py::GPUTests::test_dtype_mismatch_issue_cuda, test/inductor/test_torchinductor.py::GPUTests::test_dtypeview_bfloat16_float32_cuda, test/inductor/test_torchinductor.py::GPUTests::test_dtypeview_bfloat16_float64_cuda, test/inductor/test_torchinductor.py::GPUTests::test_dtypeview_bfloat16_int16_cuda, test/inductor/test_torchinductor.py::GPUTests::test_dtypeview_bfloat16_int32_cuda, test/inductor/test_torchinductor.py::GPUTests::test_dtypeview_bfloat16_int64_cuda, test/inductor/test_torchinductor.py::GPUTests::test_dtypeview_bfloat16_int8_cuda, test/inductor/test_torchinductor.py::GPUTests::test_dtypeview_float16_bfloat16_cuda, test/inductor/test_torchinductor.py::GPUTests::test_dtypeview_float16_float32_cuda, test/inductor/test_torchinductor.py::GPUTests::test_dtypeview_float16_int16_cuda, test/inductor/test_torchinductor.py::GPUTests::test_dtypeview_float16_int8_cuda, test/inductor/test_torchinductor.py::GPUTests::test_dtypeview_float32_bfloat16_cuda, test/inductor/test_torchinductor.py::GPUTests::test_dtypeview_float32_float64_cuda, test/inductor/test_torchinductor.py::GPUTests::test_dtypeview_float32_int16_cuda, test/inductor/test_torchinductor.py::GPUTests::test_dtypeview_float32_int64_cuda, test/inductor/test_torchinductor.py::GPUTests::test_dtypeview_float64_bfloat16_cuda, test/inductor/test_torchinductor.py::GPUTests::test_dtypeview_float64_float32_cuda, test/inductor/test_torchinductor.py::GPUTests::test_dtypeview_float64_float64_cuda, test/inductor/test_torchinductor.py::GPUTests::test_dtypeview_float64_int32_cuda, test/inductor/test_torchinductor.py::GPUTests::test_dtypeview_float64_int64_cuda, test/inductor/test_torchinductor.py::GPUTests::test_dtypeview_float64_int8_cuda, test/inductor/test_torchinductor.py::GPUTests::test_dtypeview_float64_uint8_cuda, test/inductor/test_torchinductor.py::GPUTests::test_dtypeview_fusion_cuda, test/inductor/test_torchinductor.py::GPUTests::test_dtypeview_int16_bfloat16_cuda, test/inductor/test_torchinductor.py::GPUTests::test_dtypeview_int16_float64_cuda, test/inductor/test_torchinductor.py::GPUTests::test_dtypeview_int16_int16_cuda, test/inductor/test_torchinductor.py::GPUTests::test_dtypeview_int16_uint8_cuda, test/inductor/test_torchinductor.py::GPUTests::test_dtypeview_int32_bfloat16_cuda, test/inductor/test_torchinductor.py::GPUTests::test_dtypeview_int32_uint8_cuda, test/inductor/test_torchinductor.py::GPUTests::test_dtypeview_int64_bfloat16_cuda, test/inductor/test_torchinductor.py::GPUTests::test_dtypeview_int64_float16_cuda, test/inductor/test_torchinductor.py::GPUTests::test_dtypeview_int64_int16_cuda, test/inductor/test_torchinductor.py::GPUTests::test_dtypeview_int8_float16_cuda, test/inductor/test_torchinductor.py::GPUTests::test_dtypeview_int8_int8_cuda, test/inductor/test_torchinductor.py::GPUTests::test_dtypeview_int8_uint8_cuda, test/inductor/test_torchinductor.py::GPUTests::test_dtypeview_uint8_int16_cuda, test/inductor/test_torchinductor.py::GPUTests::test_dtypeview_uint8_int64_cuda, test/inductor/test_torchinductor.py::GPUTests::test_dtypeview_uint8_int8_cuda, test/inductor/test_torchinductor.py::GPUTests::test_dtypeview_uint8_uint8_cuda, test/inductor/test_torchinductor.py::GPUTests::test_elu_cuda, test/inductor/test_torchinductor.py::GPUTests::test_embedding_bag_byte_unpack_cuda, test/inductor/test_torchinductor.py::GPUTests::test_embedding_bag_cuda, test/inductor/test_torchinductor.py::GPUTests::test_embedding_sparse_cuda, test/inductor/test_torchinductor.py::GPUTests::test_empty_strided_cuda, test/inductor/test_torchinductor.py::GPUTests::test_exp2_cuda, test/inductor/test_torchinductor.py::GPUTests::test_exp_cuda, test/inductor/test_torchinductor.py::GPUTests::test_expand_as_cuda, test/inductor/test_torchinductor.py::GPUTests::test_expand_cuda, test/inductor/test_torchinductor.py::GPUTests::test_expanded_reduction_cuda, test/inductor/test_torchinductor.py::GPUTests::test_fallback_mutable_op_basic_cuda, test/inductor/test_torchinductor.py::GPUTests::test_fallback_mutable_op_list_tensor_cuda, test/inductor/test_torchinductor.py::GPUTests::test_fallback_mutable_op_with_return_cuda, test/inductor/test_torchinductor.py::GPUTests::test_fft_real_input_real_output_cuda, test/inductor/test_torchinductor.py::GPUTests::test_fill1_cuda, test/inductor/test_torchinductor.py::GPUTests::test_float16_to_int16_cuda, test/inductor/test_torchinductor.py::GPUTests::test_float32_to_int32_cuda, test/inductor/test_torchinductor.py::GPUTests::test_float_index_expression_type_promotion_cuda, test/inductor/test_torchinductor.py::GPUTests::test_floordiv_cuda, test/inductor/test_torchinductor.py::GPUTests::test_fmod_cuda, test/inductor/test_torchinductor.py::GPUTests::test_fmod_zero_dim_cuda, test/inductor/test_torchinductor.py::GPUTests::test_full_boolean_cuda, test/inductor/test_torchinductor.py::GPUTests::test_full_like_cuda, test/inductor/test_torchinductor.py::GPUTests::test_full_like_sliced_cuda, test/inductor/test_torchinductor.py::GPUTests::test_full_truncation_cuda, test/inductor/test_torchinductor.py::GPUTests::test_functionalize_rng_wrappers_cuda, test/inductor/test_torchinductor.py::GPUTests::test_fuse_large_params_cuda, test/inductor/test_torchinductor.py::GPUTests::test_fusing_write_into_disjoint_read_cuda, test/inductor/test_torchinductor.py::GPUTests::test_gather1_cuda, test/inductor/test_torchinductor.py::GPUTests::test_gather3_cuda, test/inductor/test_torchinductor.py::GPUTests::test_gelu_cuda, test/inductor/test_torchinductor.py::GPUTests::test_generate_rand_fp8_cuda, test/inductor/test_torchinductor.py::GPUTests::test_generated_code_has_alignment_assert_cuda, test/inductor/test_torchinductor.py::GPUTests::test_generated_code_has_size_stride_assert_cuda, test/inductor/test_torchinductor.py::GPUTests::test_glu_cuda, test/inductor/test_torchinductor.py::GPUTests::test_gpu_scalar_with_gpu_tensor_cuda, test/inductor/test_torchinductor.py::GPUTests::test_graph_partition_arange2_cuda, test/inductor/test_torchinductor.py::GPUTests::test_graph_partition_argmax_cuda, test/inductor/test_torchinductor.py::GPUTests::test_graph_partition_both_scalars_cuda, test/inductor/test_torchinductor.py::GPUTests::test_graph_partition_constant_tensor1_cuda, test/inductor/test_torchinductor.py::GPUTests::test_graph_partition_mutation_real_name_cuda, test/inductor/test_torchinductor.py::GPUTests::test_graph_partition_pad_dynamic_cuda, test/inductor/test_torchinductor.py::GPUTests::test_graph_partition_refcount_cuda, test/inductor/test_torchinductor.py::GPUTests::test_graph_partition_scalar_inputs_cuda, test/inductor/test_torchinductor.py::GPUTests::test_grid_sampler_2d_cuda, test/inductor/test_torchinductor.py::GPUTests::test_grid_sampler_expand_preserves_view_cuda, test/inductor/test_torchinductor.py::GPUTests::test_hardtanh_cuda, test/inductor/test_torchinductor.py::GPUTests::test_horizonal_fusion1_cuda, test/inductor/test_torchinductor.py::GPUTests::test_index2_cuda, test/inductor/test_torchinductor.py::GPUTests::test_index3_cuda, test/inductor/test_torchinductor.py::GPUTests::test_index_float_zero_cuda, test/inductor/test_torchinductor.py::GPUTests::test_index_propagation_cuda, test/inductor/test_torchinductor.py::GPUTests::test_index_propagation_flip_cuda, test/inductor/test_torchinductor.py::GPUTests::test_index_propagation_floordiv_cuda, test/inductor/test_torchinductor.py::GPUTests::test_index_propagation_remainder_cuda, test/inductor/test_torchinductor.py::GPUTests::test_index_put3_cuda, test/inductor/test_torchinductor.py::GPUTests::test_index_put_as_masked_fill_cuda, test/inductor/test_torchinductor.py::GPUTests::test_index_put_failed_reinplace_cuda, test/inductor/test_torchinductor.py::GPUTests::test_index_put_fallback1_cuda, test/inductor/test_torchinductor.py::GPUTests::test_index_put_fallback2_cuda, test/inductor/test_torchinductor.py::GPUTests::test_index_tensor_cuda, test/inductor/test_torchinductor.py::GPUTests::test_indirect_load_broadcast_cuda, test/inductor/test_torchinductor.py::GPUTests::test_inductor_assert_cuda, test/inductor/test_torchinductor.py::GPUTests::test_inductor_layout_optimization_input_mutations_cuda, test/inductor/test_torchinductor.py::GPUTests::test_inductor_triton_bucketize_respects_masking_cuda, test/inductor/test_torchinductor.py::GPUTests::test_inplace_mixed_dtype_ops_cuda, test/inductor/test_torchinductor.py::GPUTests::test_inplace_resize_as_cuda, test/inductor/test_torchinductor.py::GPUTests::test_inplace_where_pointwise_cuda, test/inductor/test_torchinductor.py::GPUTests::test_input_mutation2_cuda, test/inductor/test_torchinductor.py::GPUTests::test_int8_weight_only_quant_cuda, test/inductor/test_torchinductor.py::GPUTests::test_int_input_dynamic_shapes_cuda, test/inductor/test_torchinductor.py::GPUTests::test_invalid_operand_issue1_cuda, test/inductor/test_torchinductor.py::GPUTests::test_isin_tensor_scalar_cuda, test/inductor/test_torchinductor.py::GPUTests::test_isinf2_cuda, test/inductor/test_torchinductor.py::GPUTests::test_large_block_sizes_cuda, test/inductor/test_torchinductor.py::GPUTests::test_large_grid_use_block_ptr_False_cuda, test/inductor/test_torchinductor.py::GPUTests::test_large_offset_pointwise_cuda, test/inductor/test_torchinductor.py::GPUTests::test_layer_norm_cuda, test/inductor/test_torchinductor.py::GPUTests::test_lerp_cuda, test/inductor/test_torchinductor.py::GPUTests::test_like_channels_last_cuda, test/inductor/test_torchinductor.py::GPUTests::test_like_rands2_cuda, test/inductor/test_torchinductor.py::GPUTests::test_like_rands_cuda, test/inductor/test_torchinductor.py::GPUTests::test_linear_dynamic_maxautotune_cuda, test/inductor/test_torchinductor.py::GPUTests::test_linear_float64_cuda, test/inductor/test_torchinductor.py::GPUTests::test_linspace3_cuda, test/inductor/test_torchinductor.py::GPUTests::test_list_clearing_cuda, test/inductor/test_torchinductor.py::GPUTests::test_lite_dynamic_shape_assertion_cuda, test/inductor/test_torchinductor.py::GPUTests::test_lite_mode_not_decompose_cuda, test/inductor/test_torchinductor.py::GPUTests::test_lite_regional_compile_invoke_subgraph_cuda, test/inductor/test_torchinductor.py::GPUTests::test_log1p_cuda, test/inductor/test_torchinductor.py::GPUTests::test_log2_cuda, test/inductor/test_torchinductor.py::GPUTests::test_logsumexp_cuda, test/inductor/test_torchinductor.py::GPUTests::test_long_tensor_cuda, test/inductor/test_torchinductor.py::GPUTests::test_low_memory_max_pool_dilation_1_dim_2_cuda, test/inductor/test_torchinductor.py::GPUTests::test_low_memory_max_pool_dilation_1_dim_3_cuda, test/inductor/test_torchinductor.py::GPUTests::test_mark_dynamic_with_hint_override_cuda, test/inductor/test_torchinductor.py::GPUTests::test_mark_unbacked_with_hint_override_cuda, test/inductor/test_torchinductor.py::GPUTests::test_masked_fill_cuda, test/inductor/test_torchinductor.py::GPUTests::test_masked_fill_promotion_cuda, test/inductor/test_torchinductor.py::GPUTests::test_max_pool2d1_cuda, test/inductor/test_torchinductor.py::GPUTests::test_max_pool2d2_cuda, test/inductor/test_torchinductor.py::GPUTests::test_max_pool2d4_cuda, test/inductor/test_torchinductor.py::GPUTests::test_max_pool2d6_dilation_1_cuda, test/inductor/test_torchinductor.py::GPUTests::test_max_pool2d6_dilation_2_cuda, test/inductor/test_torchinductor.py::GPUTests::test_max_pool2d7_cuda, test/inductor/test_torchinductor.py::GPUTests::test_max_pool2d8_cuda, test/inductor/test_torchinductor.py::GPUTests::test_max_pool2d_with_indices_backward5_cuda, test/inductor/test_torchinductor.py::GPUTests::test_max_pool2d_with_indices_backward_cuda, test/inductor/test_torchinductor.py::GPUTests::test_mean_cuda, test/inductor/test_torchinductor.py::GPUTests::test_min_max_reduction_cuda, test/inductor/test_torchinductor.py::GPUTests::test_misaligned_address_issue1_cuda, test/inductor/test_torchinductor.py::GPUTests::test_move_arange_cuda, test/inductor/test_torchinductor.py::GPUTests::test_multi_device_cuda, test/inductor/test_torchinductor.py::GPUTests::test_multi_threading_cuda, test/inductor/test_torchinductor.py::GPUTests::test_multilayer_any_cuda, test/inductor/test_torchinductor.py::GPUTests::test_multilayer_prime_size_cuda, test/inductor/test_torchinductor.py::GPUTests::test_multilayer_sum_low_prec_cuda, test/inductor/test_torchinductor.py::GPUTests::test_mutations_loop_fusion_cuda, test/inductor/test_torchinductor.py::GPUTests::test_nan_assert_inside_triton_kernel_cuda, test/inductor/test_torchinductor.py::GPUTests::test_nan_sort_stable_True_descending_False_cuda, test/inductor/test_torchinductor.py::GPUTests::test_nan_sort_stable_True_descending_True_cuda, test/inductor/test_torchinductor.py::GPUTests::test_narrow_cuda, test/inductor/test_torchinductor.py::GPUTests::test_needs_contiguous_strides_cuda, test/inductor/test_torchinductor.py::GPUTests::test_new_empty_cuda, test/inductor/test_torchinductor.py::GPUTests::test_new_empty_strided_cuda, test/inductor/test_torchinductor.py::GPUTests::test_new_ones_cuda, test/inductor/test_torchinductor.py::GPUTests::test_one_hot_cuda, test/inductor/test_torchinductor.py::GPUTests::test_pad_cast_cuda, test/inductor/test_torchinductor.py::GPUTests::test_pad_single_cuda, test/inductor/test_torchinductor.py::GPUTests::test_pad_view_cuda, test/inductor/test_torchinductor.py::GPUTests::test_philox_rand_cuda, test/inductor/test_torchinductor.py::GPUTests::test_pixel_shuffle_channels_last_cuda, test/inductor/test_torchinductor.py::GPUTests::test_pointwise_airy_ai_cuda, test/inductor/test_torchinductor.py::GPUTests::test_pointwise_bessel_y0_cuda, test/inductor/test_torchinductor.py::GPUTests::test_pointwise_bessel_y1_cuda, test/inductor/test_torchinductor.py::GPUTests::test_pointwise_chebyshev_polynomial_t_cuda, test/inductor/test_torchinductor.py::GPUTests::test_pointwise_erf_cuda, test/inductor/test_torchinductor.py::GPUTests::test_pointwise_erfcx_cuda, test/inductor/test_torchinductor.py::GPUTests::test_pointwise_erfinv_cuda, test/inductor/test_torchinductor.py::GPUTests::test_pointwise_exp2_cuda, test/inductor/test_torchinductor.py::GPUTests::test_pointwise_gammaln_cuda, test/inductor/test_torchinductor.py::GPUTests::test_pointwise_hermite_polynomial_h_cuda, test/inductor/test_torchinductor.py::GPUTests::test_pointwise_i1_cuda, test/inductor/test_torchinductor.py::GPUTests::test_pointwise_laguerre_polynomial_l_cuda, test/inductor/test_torchinductor.py::GPUTests::test_pointwise_legendre_polynomial_p_cuda, test/inductor/test_torchinductor.py::GPUTests::test_pointwise_log1p_cuda, test/inductor/test_torchinductor.py::GPUTests::test_pointwise_log_ndtr_cuda, test/inductor/test_torchinductor.py::GPUTests::test_pointwise_modified_bessel_i0_cuda, test/inductor/test_torchinductor.py::GPUTests::test_pointwise_modified_bessel_i1_cuda, test/inductor/test_torchinductor.py::GPUTests::test_pointwise_modified_bessel_k1_cuda, test/inductor/test_torchinductor.py::GPUTests::test_pointwise_multigammaln_cuda, test/inductor/test_torchinductor.py::GPUTests::test_pointwise_ndtri_cuda, test/inductor/test_torchinductor.py::GPUTests::test_pointwise_psi_cuda, test/inductor/test_torchinductor.py::GPUTests::test_pointwise_round_cuda, test/inductor/test_torchinductor.py::GPUTests::test_pointwise_scaled_modified_bessel_k0_cuda, test/inductor/test_torchinductor.py::GPUTests::test_pointwise_shifted_chebyshev_polynomial_v_cuda, test/inductor/test_torchinductor.py::GPUTests::test_pointwise_shifted_chebyshev_polynomial_w_cuda, test/inductor/test_torchinductor.py::GPUTests::test_pointwise_spherical_bessel_j0_cuda, test/inductor/test_torchinductor.py::GPUTests::test_pointwise_zeta_cuda, test/inductor/test_torchinductor.py::GPUTests::test_pow_by_natural_log2_dynamic_shapes_cuda, test/inductor/test_torchinductor.py::GPUTests::test_prod_cuda, test/inductor/test_torchinductor.py::GPUTests::test_rand_like_deterministic_cuda, test/inductor/test_torchinductor.py::GPUTests::test_randint_kernel_count_cuda, test/inductor/test_torchinductor.py::GPUTests::test_randn_with_dtype_and_device_cuda, test/inductor/test_torchinductor.py::GPUTests::test_reduction4_cuda, test/inductor/test_torchinductor.py::GPUTests::test_reflection_pad2d_cuda, test/inductor/test_torchinductor.py::GPUTests::test_remainder_cuda, test/inductor/test_torchinductor.py::GPUTests::test_remove_noop_clone_cuda, test/inductor/test_torchinductor.py::GPUTests::test_remove_noop_copy_cuda, test/inductor/test_torchinductor.py::GPUTests::test_remove_noop_slice1_cuda, test/inductor/test_torchinductor.py::GPUTests::test_repeat_as_strided_cuda, test/inductor/test_torchinductor.py::GPUTests::test_repeat_cuda, test/inductor/test_torchinductor.py::GPUTests::test_repeat_interleave_Tensor_decomp_int32_nd_2_cuda, test/inductor/test_torchinductor.py::GPUTests::test_repeat_interleave_Tensor_decomp_int64_nd_1_cuda, test/inductor/test_torchinductor.py::GPUTests::test_repeat_interleave_Tensor_decomp_int64_nd_2_cuda, test/inductor/test_torchinductor.py::GPUTests::test_repeat_interleave_cuda, test/inductor/test_torchinductor.py::GPUTests::test_repeat_interleave_decomposition_has_clamp_cuda, test/inductor/test_torchinductor.py::GPUTests::test_roi_align_cuda, test/inductor/test_torchinductor.py::GPUTests::test_round_correctness_cuda, test/inductor/test_torchinductor.py::GPUTests::test_round_cuda, test/inductor/test_torchinductor.py::GPUTests::test_rsqrt_dynamic_shapes_cuda, test/inductor/test_torchinductor.py::GPUTests::test_scalar_cpu_tensor_arg_cuda, test/inductor/test_torchinductor.py::GPUTests::test_scalar_input_cuda, test/inductor/test_torchinductor.py::GPUTests::test_scaled_dot_product_efficient_attention_cuda, test/inductor/test_torchinductor.py::GPUTests::test_scatter6_cuda, test/inductor/test_torchinductor.py::GPUTests::test_scatter_add2_cuda, test/inductor/test_torchinductor.py::GPUTests::test_scatter_bf16_cuda, test/inductor/test_torchinductor.py::GPUTests::test_scatter_reduce2_cuda, test/inductor/test_torchinductor.py::GPUTests::test_scatter_reduce3_cuda, test/inductor/test_torchinductor.py::GPUTests::test_scheduler_vertical_fusion1_cuda, test/inductor/test_torchinductor.py::GPUTests::test_sdpa_prefer_nd_tiling_False_use_block_ptr_False_cuda, test/inductor/test_torchinductor.py::GPUTests::test_sdpa_prefer_nd_tiling_False_use_block_ptr_True_cuda, test/inductor/test_torchinductor.py::GPUTests::test_sdpa_prefer_nd_tiling_True_use_block_ptr_True_cuda, test/inductor/test_torchinductor.py::GPUTests::test_sdpa_unaligned_mask_cuda, test/inductor/test_torchinductor.py::GPUTests::test_searchsorted_broadcast_cuda, test/inductor/test_torchinductor.py::GPUTests::test_setitem_with_int_parameter_cuda, test/inductor/test_torchinductor.py::GPUTests::test_sgn_cuda, test/inductor/test_torchinductor.py::GPUTests::test_shape_prop_torch_ones_cuda, test/inductor/test_torchinductor.py::GPUTests::test_sign_dtype_cuda, test/inductor/test_torchinductor.py::GPUTests::test_simplify_loops_cuda, test/inductor/test_torchinductor.py::GPUTests::test_single_elem_indirect_cuda, test/inductor/test_torchinductor.py::GPUTests::test_size_asserts_for_multi_output_fallback_cuda, test/inductor/test_torchinductor.py::GPUTests::test_sizehint_issue1_cuda, test/inductor/test_torchinductor.py::GPUTests::test_slice2_cuda, test/inductor/test_torchinductor.py::GPUTests::test_slice_scatter3_cuda, test/inductor/test_torchinductor.py::GPUTests::test_slice_scatter_cuda, test/inductor/test_torchinductor.py::GPUTests::test_slice_view_with_graph_break_cuda, test/inductor/test_torchinductor.py::GPUTests::test_softmax_cuda, test/inductor/test_torchinductor.py::GPUTests::test_softmax_one_kernel_persist_cuda, test/inductor/test_torchinductor.py::GPUTests::test_sort_cuda, test/inductor/test_torchinductor.py::GPUTests::test_sort_stable_cuda, test/inductor/test_torchinductor.py::GPUTests::test_split_cumprod_cuda, test/inductor/test_torchinductor.py::GPUTests::test_split_failed_cuda, test/inductor/test_torchinductor.py::GPUTests::test_split_reduction_with_int64_size_cuda, test/inductor/test_torchinductor.py::GPUTests::test_split_with_list_cuda, test/inductor/test_torchinductor.py::GPUTests::test_split_with_sizes_with_unbacked_symints_cuda, test/inductor/test_torchinductor.py::GPUTests::test_split_with_unbacked_symints_cuda, test/inductor/test_torchinductor.py::GPUTests::test_sqrt_dynamic_shapes_cuda, test/inductor/test_torchinductor.py::GPUTests::test_squeeze1_cuda, test/inductor/test_torchinductor.py::GPUTests::test_stack_cuda, test/inductor/test_torchinductor.py::GPUTests::test_strided_inputs_cuda, test/inductor/test_torchinductor.py::GPUTests::test_sum4_cuda, test/inductor/test_torchinductor.py::GPUTests::test_sum_dtype_cuda, test/inductor/test_torchinductor.py::GPUTests::test_sum_int_cuda, test/inductor/test_torchinductor.py::GPUTests::test_sum_keepdims_cuda, test/inductor/test_torchinductor.py::GPUTests::test_tan_cuda, test/inductor/test_torchinductor.py::GPUTests::test_tanh_cuda, test/inductor/test_torchinductor.py::GPUTests::test_tensor1_cuda, test/inductor/test_torchinductor.py::GPUTests::test_tensor3_cuda, test/inductor/test_torchinductor.py::GPUTests::test_tensor_index_put_slice_cuda, test/inductor/test_torchinductor.py::GPUTests::test_tmp_not_defined_issue3_cuda, test/inductor/test_torchinductor.py::GPUTests::test_topk_cuda, test/inductor/test_torchinductor.py::GPUTests::test_torch_device_split_cuda, test/inductor/test_torchinductor.py::GPUTests::test_transpose_add_cuda, test/inductor/test_torchinductor.py::GPUTests::test_transpose_cuda, test/inductor/test_torchinductor.py::GPUTests::test_triton_kernel_bool_param_cuda, test/inductor/test_torchinductor.py::GPUTests::test_uint_cuda, test/inductor/test_torchinductor.py::GPUTests::test_unbacked_float_item_cuda, test/inductor/test_torchinductor.py::GPUTests::test_unroll_small_reduction_cuda, test/inductor/test_torchinductor.py::GPUTests::test_unspec_inputs_float16_cuda, test/inductor/test_torchinductor.py::GPUTests::test_unspec_inputs_int16_cuda, test/inductor/test_torchinductor.py::GPUTests::test_unspec_inputs_int64_cuda, test/inductor/test_torchinductor.py::GPUTests::test_unspec_inputs_int8_cuda, test/inductor/test_torchinductor.py::GPUTests::test_unsqueeze_cuda, test/inductor/test_torchinductor.py::GPUTests::test_unsqueeze_inplace_cuda, test/inductor/test_torchinductor.py::GPUTests::test_upsample_bicubic2d_cuda, test/inductor/test_torchinductor.py::GPUTests::test_upsample_cat_conv_cuda, test/inductor/test_torchinductor.py::GPUTests::test_upsample_nearest2d_backward_cuda, test/inductor/test_torchinductor.py::GPUTests::test_var_mean_div_by_cuda, test/inductor/test_torchinductor.py::GPUTests::test_vdd_clamp_cuda, test/inductor/test_torchinductor.py::GPUTests::test_vertical_fusion1_cuda, test/inductor/test_torchinductor.py::GPUTests::test_view_as_real_cuda, test/inductor/test_torchinductor.py::GPUTests::test_views1_cuda, test/inductor/test_torchinductor.py::GPUTests::test_views2_cuda, test/inductor/test_torchinductor.py::GPUTests::test_views3_cuda, test/inductor/test_torchinductor.py::GPUTests::test_weight_norm_conv2d_cuda, test/inductor/test_torchinductor.py::GPUTests::test_where_broadcast_cuda, test/inductor/test_torchinductor.py::GPUTests::test_where_with_logical_op_cuda, test/inductor/test_torchinductor.py::GPUTests::test_zeros_cuda, test/inductor/test_torchinductor.py::TritonCodeGenTests::test_bandwidth_profiler, test/inductor/test_torchinductor.py::TritonCodeGenTests::test_bf16_atomic_add, test/inductor/test_torchinductor.py::TritonCodeGenTests::test_codegen_config_option_dont_assume_alignment, test/inductor/test_torchinductor.py::TritonCodeGenTests::test_comment_graph_fragment, test/inductor/test_torchinductor.py::TritonCodeGenTests::test_computed_indirect_mask, test/inductor/test_torchinductor.py::TritonCodeGenTests::test_constant_folding_deallocation, test/inductor/test_torchinductor.py::TritonCodeGenTests::test_divisible_by_16_covers_numel_args, test/inductor/test_torchinductor.py::TritonCodeGenTests::test_donated_buffer_inplace_gpt, test/inductor/test_torchinductor.py::TritonCodeGenTests::test_evict_last_non_coalesced_loads, test/inductor/test_torchinductor.py::TritonCodeGenTests::test_indirect_device_assert, test/inductor/test_torchinductor.py::TritonCodeGenTests::test_inductor_detach_view_backend_aot_eager, test/inductor/test_torchinductor.py::TritonCodeGenTests::test_kernel_names_descriptive, test/inductor/test_torchinductor.py::TritonCodeGenTests::test_layer_norm_inplaces_after_matmul, test/inductor/test_torchinductor.py::TritonCodeGenTests::test_non_blocking_copy_codegen, test/inductor/test_torchinductor.py::TritonCodeGenTests::test_numpy_autograd, test/inductor/test_torchinductor.py::TritonCodeGenTests::test_optimize_indexing_dtype, test/inductor/test_torchinductor.py::TritonCodeGenTests::test_red_followed_by_transposed_pointwise, test/inductor/test_torchinductor.py::TritonCodeGenTests::test_respect_scaled_grouped_mm_layout_tag, test/inductor/test_torchinductor.py::TritonCodeGenTests::test_rope_fusion, test/inductor/test_torchinductor.py::TritonCodeGenTests::test_sdpa_inference_mode_aot_compile, test/inductor/test_torchinductor.py::TritonCodeGenTests::test_skip_l1_cache, test/inductor/test_torchinductor.py::TritonCodeGenTests::test_split_op_with_sym, test/inductor/test_torchinductor.py::TritonCodeGenTests::test_triton_attrs_dict_constexpr_signature, test/inductor/test_torchinductor.py::NanCheckerTest::test_nan_checker_fail 2025-12-04T09:59:09.5820238Z 2025-12-04T09:59:09.5820656Z Finished inductor/test_torchinductor 1/2 ... [2025-12-04 09:59:09.544203][1197.486418959], took 5.21min 2025-12-04T09:59:09.5822106Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/inductor.test_torchinductor/inductor.test_torchinductor-410bad42b3bb466c.xml 2025-12-04T09:59:10.0415813Z Uploading artifacts took 0.19 seconds 2025-12-04T09:59:10.0418519Z Running inductor/test_torchinductor_dynamic_shapes 1/4 ... [2025-12-04 09:59:10.041631][1197.983846102] 2025-12-04T09:59:10.0419211Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2025-12-04T09:59:10.0452229Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'inductor/test_torchinductor_dynamic_shapes.py', '--shard-id=1', '--num-shards=4', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-12-04 09:59:10.044928] 2025-12-04T10:06:52.9401888Z 2025-12-04T10:06:52.9403275Z inductor/test_torchinductor_dynamic_shapes 1/4 was successful, full logs can be found in artifacts with path test/test-reports/inductor.test_torchinductor_dynamic_shapes_1.4_f457e39532c32c3d_.log 2025-12-04T10:06:52.9564594Z Running 441 items in this shard: test/inductor/test_torchinductor_dynamic_shapes.py::DynamicShapesCpuTests::test__dyn_quant_pack_4bit_weight_fp32_dynamic_shapes_cpu, test/inductor/test_torchinductor_dynamic_shapes.py::DynamicShapesCpuTests::test__unsafe_masked_index_dynamic_shapes_cpu, test/inductor/test_torchinductor_dynamic_shapes.py::DynamicShapesCpuTests::test_adaptive_avg_pool1d_argmax_dynamic_shapes_cpu, test/inductor/test_torchinductor_dynamic_shapes.py::DynamicShapesCpuTests::test_adaptive_avg_pool2d_low_prec_dynamic_shapes_cpu, test/inductor/test_torchinductor_dynamic_shapes.py::DynamicShapesCpuTests::test_adaptive_avg_pool_with_output_size_0_dynamic_shapes_cpu, test/inductor/test_torchinductor_dynamic_shapes.py::DynamicShapesCpuTests::test_adaptive_pool_errors_with_long_dynamic_shapes_cpu, test/inductor/test_torchinductor_dynamic_shapes.py::DynamicShapesCpuTests::test_add_complex4_dynamic_shapes_cpu, test/inductor/test_torchinductor_dynamic_shapes.py::DynamicShapesCpuTests::test_add_complex_dynamic_shapes_cpu, test/inductor/test_torchinductor_dynamic_shapes.py::DynamicShapesCpuTests::test_add_complex_strided_fallback_dynamic_shapes_cpu, test/inductor/test_torchinductor_dynamic_shapes.py::DynamicShapesCpuTests::test_add_const_float_dynamic_shapes_cpu, test/inductor/test_torchinductor_dynamic_shapes.py::DynamicShapesCpuTests::test_add_const_int_dynamic_shapes_cpu, test/inductor/test_torchinductor_dynamic_shapes.py::DynamicShapesCpuTests::test_addmm_dynamic_shapes_cpu, test/inductor/test_torchinductor_dynamic_shapes.py::DynamicShapesCpuTests::test_angle_dynamic_shapes_cpu, test/inductor/test_torchinductor_dynamic_shapes.py::DynamicShapesCpuTests::test_any_dynamic_shapes_cpu, test/inductor/test_torchinductor_dynamic_shapes.py::DynamicShapesCpuTests::test_arange5_dynamic_shapes_cpu, test/inductor/test_torchinductor_dynamic_shapes.py::DynamicShapesCpuTests::test_arange6_dynamic_shapes_cpu, test/inductor/test_torchinductor_dynamic_shapes.py::DynamicShapesCpuTests::test_argmax_argmin1_dynamic_shapes_cpu, test/inductor/test_torchinductor_dynamic_shapes.py::DynamicShapesCpuTests::test_argmax_argmin_with_nan_dynamic_shapes_cpu, test/inductor/test_torchinductor_dynamic_shapes.py::DynamicShapesCpuTests::test_avg_pool2d_backward2_dynamic_shapes_cpu, test/inductor/test_torchinductor_dynamic_shapes.py::DynamicShapesCpuTests::test_avg_pool3d_backward4_dynamic_shapes_cpu, test/inductor/test_torchinductor_dynamic_shapes.py::DynamicShapesCpuTests::test_batch_norm_2d_dynamic_shapes_cpu, test/inductor/test_torchinductor_dynamic_shapes.py::DynamicShapesCpuTests::test_bitwise2_dynamic_shapes_cpu, test/inductor/test_torchinductor_dynamic_shapes.py::DynamicShapesCpuTests::test_bmm2_dynamic_shapes_cpu, test/inductor/test_torchinductor_dynamic_shapes.py::DynamicShapesCpuTests::test_bucketize_int_int16_int16_dynamic_shapes_cpu, test/inductor/test_torchinductor_dynamic_shapes.py::DynamicShapesCpuTests::test_bucketize_int_int16_int64_dynamic_shapes_cpu, test/inductor/test_torchinductor_dynamic_shapes.py::DynamicShapesCpuTests::test_bucketize_int_int64_int8_dynamic_shapes_cpu, test/inductor/test_torchinductor_dynamic_shapes.py::DynamicShapesCpuTests::test_bucketize_int_int8_int16_dynamic_shapes_cpu, test/inductor/test_torchinductor_dynamic_shapes.py::DynamicShapesCpuTests::test_bucketize_int_int8_int32_dynamic_shapes_cpu, test/inductor/test_torchinductor_dynamic_shapes.py::DynamicShapesCpuTests::test_bucketize_int_int8_int8_dynamic_shapes_cpu, test/inductor/test_torchinductor_dynamic_shapes.py::DynamicShapesCpuTests::test_bucketize_int_int8_uint8_dynamic_shapes_cpu, test/inductor/test_torchinductor_dynamic_shapes.py::DynamicShapesCpuTests::test_bucketize_int_uint8_int8_dynamic_shapes_cpu, test/inductor/test_torchinductor_dynamic_shapes.py::DynamicShapesCpuTests::test_buffer_copied_in_graph_with_different_shapes_dynamic_shapes_cpu, test/inductor/test_torchinductor_dynamic_shapes.py::DynamicShapesCpuTests::test_builtins_round_dynamic_shapes_cpu, test/inductor/test_torchinductor_dynamic_shapes.py::DynamicShapesCpuTests::test_builtins_round_int_ndigits_pos_dynamic_shapes_cpu, test/inductor/test_torchinductor_dynamic_shapes.py::DynamicShapesCpuTests::test_builtins_round_int_ndigits_zero_dynamic_shapes_cpu, test/inductor/test_torchinductor_dynamic_shapes.py::DynamicShapesCpuTests::test_cat_empty_index_dynamic_shapes_cpu, test/inductor/test_torchinductor_dynamic_shapes.py::DynamicShapesCpuTests::test_cat_negative_dim_dynamic_shapes_cpu, test/inductor/test_torchinductor_dynamic_shapes.py::DynamicShapesCpuTests::test_chunk_recompiles_dynamic_shapes_cpu, test/inductor/test_torchinductor_dynamic_shapes.py::DynamicShapesCpuTests::test_complex_memory_overlap_dynamic_shapes_cpu, test/inductor/test_torchinductor_dynamic_shapes.py::DynamicShapesCpuTests::test_constant_pad_fill_dtype_dynamic_shapes_cpu, test/inductor/test_torchinductor_dynamic_shapes.py::DynamicShapesCpuTests::test_constant_pad_float64_dynamic_shapes_cpu, test/inductor/test_torchinductor_dynamic_shapes.py::DynamicShapesCpuTests::test_conv_bn_fuse_dynamic_shapes_cpu, test/inductor/test_torchinductor_dynamic_shapes.py::DynamicShapesCpuTests::test_conv_shape_check_dynamic_shapes_cpu, test/inductor/test_torchinductor_dynamic_shapes.py::DynamicShapesCpuTests::test_conv_with_as_strided_dynamic_shapes_cpu, test/inductor/test_torchinductor_dynamic_shapes.py::DynamicShapesCpuTests::test_cpu_scalar_with_cpu_tensor_dynamic_shapes_cpu, test/inductor/test_torchinductor_dynamic_shapes.py::DynamicShapesCpuTests::test_cpu_tensor_with_gpu_tensor_dynamic_shapes_cpu, test/inductor/test_torchinductor_dynamic_shapes.py::DynamicShapesCpuTests::test_cudnn_rnn_dynamic_shapes_cpu, test/inductor/test_torchinductor_dynamic_shapes.py::DynamicShapesCpuTests::test_cummin_dynamic_shapes_cpu, test/inductor/test_torchinductor_dynamic_shapes.py::DynamicShapesCpuTests::test_custom_op_default_layout_constraint_dynamic_shapes_cpu, test/inductor/test_torchinductor_dynamic_shapes.py::DynamicShapesCpuTests::test_custom_scan_op_dynamic_shapes_cpu, test/inductor/test_torchinductor_dynamic_shapes.py::DynamicShapesCpuTests::test_dense_mask_index_dynamic_shapes_cpu, test/inductor/test_torchinductor_dynamic_shapes.py::DynamicShapesCpuTests::test_deterministic_codegen_with_suffix_dynamic_shapes_cpu, test/inductor/test_torchinductor_dynamic_shapes.py::DynamicShapesCpuTests::test_device_assert_dynamic_shapes_cpu, test/inductor/test_torchinductor_dynamic_shapes.py::DynamicShapesCpuTests::test_diagonal_copy_dynamic_shapes_cpu, test/inductor/test_torchinductor_dynamic_shapes.py::DynamicShapesCpuTests::test_dist_dynamic_shapes_cpu, test/inductor/test_torchinductor_dynamic_shapes.py::DynamicShapesCpuTests::test_div1_dynamic_shapes_cpu, test/inductor/test_torchinductor_dynamic_shapes.py::DynamicShapesCpuTests::test_div7_dynamic_shapes_cpu, test/inductor/test_torchinductor_dynamic_shapes.py::DynamicShapesCpuTests::test_div9_dynamic_shapes_cpu, test/inductor/test_torchinductor_dynamic_shapes.py::DynamicShapesCpuTests::test_div_softmax_symfloat_dynamic_shapes_cpu, test/inductor/test_torchinductor_dynamic_shapes.py::DynamicShapesCpuTests::test_div_zero_dim_dynamic_shapes_cpu, test/inductor/test_torchinductor_dynamic_shapes.py::DynamicShapesCpuTests::test_dtypeview_bfloat16_int8_dynamic_shapes_cpu, test/inductor/test_torchinductor_dynamic_shapes.py::DynamicShapesCpuTests::test_dtypeview_float16_float16_dynamic_shapes_cpu, test/inductor/test_torchinductor_dynamic_shapes.py::DynamicShapesCpuTests::test_dtypeview_float16_float32_dynamic_shapes_cpu, test/inductor/test_torchinductor_dynamic_shapes.py::DynamicShapesCpuTests::test_dtypeview_float16_int16_dynamic_shapes_cpu, test/inductor/test_torchinductor_dynamic_shapes.py::DynamicShapesCpuTests::test_dtypeview_float16_int32_dynamic_shapes_cpu, test/inductor/test_torchinductor_dynamic_shapes.py::DynamicShapesCpuTests::test_dtypeview_float16_int64_dynamic_shapes_cpu, test/inductor/test_torchinductor_dynamic_shapes.py::DynamicShapesCpuTests::test_dtypeview_float16_int8_dynamic_shapes_cpu, test/inductor/test_torchinductor_dynamic_shapes.py::DynamicShapesCpuTests::test_dtypeview_float32_float16_dynamic_shapes_cpu, test/inductor/test_torchinductor_dynamic_shapes.py::DynamicShapesCpuTests::test_dtypeview_float64_float64_dynamic_shapes_cpu, test/inductor/test_torchinductor_dynamic_shapes.py::DynamicShapesCpuTests::test_dtypeview_float64_int64_dynamic_shapes_cpu, test/inductor/test_torchinductor_dynamic_shapes.py::DynamicShapesCpuTests::test_dtypeview_float64_uint8_dynamic_shapes_cpu, test/inductor/test_torchinductor_dynamic_shapes.py::DynamicShapesCpuTests::test_dtypeview_int16_bfloat16_dynamic_shapes_cpu, test/inductor/test_torchinductor_dynamic_shapes.py::DynamicShapesCpuTests::test_dtypeview_int16_float16_dynamic_shapes_cpu, test/inductor/test_torchinductor_dynamic_shapes.py::DynamicShapesCpuTests::test_dtypeview_int16_int16_dynamic_shapes_cpu, test/inductor/test_torchinductor_dynamic_shapes.py::DynamicShapesCpuTests::test_dtypeview_int16_int8_dynamic_shapes_cpu, test/inductor/test_torchinductor_dynamic_shapes.py::DynamicShapesCpuTests::test_dtypeview_int32_int8_dynamic_shapes_cpu, test/inductor/test_torchinductor_dynamic_shapes.py::DynamicShapesCpuTests::test_dtypeview_int64_int32_dynamic_shapes_cpu, test/inductor/test_torchinductor_dynamic_shapes.py::DynamicShapesCpuTests::test_dtypeview_int64_uint8_dynamic_shapes_cpu, test/inductor/test_torchinductor_dynamic_shapes.py::DynamicShapesCpuTests::test_dtypeview_int8_float16_dynamic_shapes_cpu, test/inductor/test_torchinductor_dynamic_shapes.py::DynamicShapesCpuTests::test_dtypeview_int8_float64_dynamic_shapes_cpu, test/inductor/test_torchinductor_dynamic_shapes.py::DynamicShapesCpuTests::test_dtypeview_int8_int32_dynamic_shapes_cpu, test/inductor/test_torchinductor_dynamic_shapes.py::DynamicShapesCpuTests::test_dtypeview_uint8_float32_dynamic_shapes_cpu, test/inductor/test_torchinductor_dynamic_shapes.py::DynamicShapesCpuTests::test_dtypeview_uint8_uint8_dynamic_shapes_cpu, test/inductor/test_torchinductor_dynamic_shapes.py::DynamicShapesCpuTests::test_exp_dynamic_shapes_cpu, test/inductor/test_torchinductor_dynamic_shapes.py::DynamicShapesCpuTests::test_fallback_mutable_op_basic_dynamic_shapes_cpu, test/inductor/test_torchinductor_dynamic_shapes.py::DynamicShapesCpuTests::test_fallback_mutable_op_list_dynamic_shapes_cpu, test/inductor/test_torchinductor_dynamic_shapes.py::DynamicShapesCpuTests::test_float32_to_int32_dynamic_shapes_cpu, test/inductor/test_torchinductor_dynamic_shapes.py::DynamicShapesCpuTests::test_float_index_expression_type_promotion_dynamic_shapes_cpu, test/inductor/test_torchinductor_dynamic_shapes.py::DynamicShapesCpuTests::test_fmin_fmax_dynamic_shapes_cpu, test/inductor/test_torchinductor_dynamic_shapes.py::DynamicShapesCpuTests::test_fmod_zero_dim_dynamic_shapes_cpu, test/inductor/test_torchinductor_dynamic_shapes.py::DynamicShapesCpuTests::test_fractional_max_pool2d1_dynamic_shapes_cpu, test/inductor/test_torchinductor_dynamic_shapes.py::DynamicShapesCpuTests::test_fractional_max_pool2d2_dynamic_shapes_cpu, test/inductor/test_torchinductor_dynamic_shapes.py::DynamicShapesCpuTests::test_full_like_dynamic_shapes_cpu, test/inductor/test_torchinductor_dynamic_shapes.py::DynamicShapesCpuTests::test_gather2_dynamic_shapes_cpu, test/inductor/test_torchinductor_dynamic_shapes.py::DynamicShapesCpuTests::test_gelu_dynamic_shapes_cpu, test/inductor/test_torchinductor_dynamic_shapes.py::DynamicShapesCpuTests::test_graph_partition_constant_tensor1_dynamic_shapes_cpu, test/inductor/test_torchinductor_dynamic_shapes.py::DynamicShapesCpuTests::test_graph_partition_misaligned_input_dynamic_shapes_cpu, test/inductor/test_torchinductor_dynamic_shapes.py::DynamicShapesCpuTests::test_graph_partition_no_inputs_dynamic_shapes_cpu, test/inductor/test_torchinductor_dynamic_shapes.py::DynamicShapesCpuTests::test_graph_partition_unbacked_symint_as_output_dynamic_shapes_cpu, test/inductor/test_torchinductor_dynamic_shapes.py::DynamicShapesCpuTests::test_grid_sampler_expand_preserves_view_dynamic_shapes_cpu, test/inductor/test_torchinductor_dynamic_shapes.py::DynamicShapesCpuTests::test_hardsigmoid_dynamic_shapes_cpu, test/inductor/test_torchinductor_dynamic_shapes.py::DynamicShapesCpuTests::test_horizonal_fusion2_dynamic_shapes_cpu, test/inductor/test_torchinductor_dynamic_shapes.py::DynamicShapesCpuTests::test_index2_dynamic_shapes_cpu, test/inductor/test_torchinductor_dynamic_shapes.py::DynamicShapesCpuTests::test_index_propagation_device_assert_masked_dynamic_shapes_cpu, test/inductor/test_torchinductor_dynamic_shapes.py::DynamicShapesCpuTests::test_index_propagation_dynamic_shapes_cpu, test/inductor/test_torchinductor_dynamic_shapes.py::DynamicShapesCpuTests::test_index_propagation_flip_dynamic_shapes_cpu, test/inductor/test_torchinductor_dynamic_shapes.py::DynamicShapesCpuTests::test_index_propagation_nested_indirect_indexing_dynamic_shapes_cpu, test/inductor/test_torchinductor_dynamic_shapes.py::DynamicShapesCpuTests::test_index_put_deterministic_fallback_dynamic_shapes_cpu, test/inductor/test_torchinductor_dynamic_shapes.py::DynamicShapesCpuTests::test_index_put_index_dynamic_shapes_cpu, test/inductor/test_torchinductor_dynamic_shapes.py::DynamicShapesCpuTests::test_index_remainder_dynamic_shapes_cpu, test/inductor/test_torchinductor_dynamic_shapes.py::DynamicShapesCpuTests::test_index_select_dynamic_shapes_cpu, test/inductor/test_torchinductor_dynamic_shapes.py::DynamicShapesCpuTests::test_index_tensor_dynamic_shapes_cpu, test/inductor/test_torchinductor_dynamic_shapes.py::DynamicShapesCpuTests::test_input_mutation4_dynamic_shapes_cpu, test/inductor/test_torchinductor_dynamic_shapes.py::DynamicShapesCpuTests::test_int_input_dynamic_shapes_dynamic_shapes_cpu, test/inductor/test_torchinductor_dynamic_shapes.py::DynamicShapesCpuTests::test_invalid_operand_issue1_dynamic_shapes_cpu, test/inductor/test_torchinductor_dynamic_shapes.py::DynamicShapesCpuTests::test_issue102546_dynamic_shapes_cpu, test/inductor/test_torchinductor_dynamic_shapes.py::DynamicShapesCpuTests::test_large_block_sizes_dynamic_shapes_cpu, test/inductor/test_torchinductor_dynamic_shapes.py::DynamicShapesCpuTests::test_large_offset_pointwise_dynamic_shapes_cpu, test/inductor/test_torchinductor_dynamic_shapes.py::DynamicShapesCpuTests::test_layer_norm_dynamic_shapes_cpu, test/inductor/test_torchinductor_dynamic_shapes.py::DynamicShapesCpuTests::test_leaky_relu_dynamic_shapes_cpu, test/inductor/test_torchinductor_dynamic_shapes.py::DynamicShapesCpuTests::test_like_rands_sliced_dynamic_shapes_cpu, test/inductor/test_torchinductor_dynamic_shapes.py::DynamicShapesCpuTests::test_linalg_eig_stride_consistency_dynamic_shapes_cpu, test/inductor/test_torchinductor_dynamic_shapes.py::DynamicShapesCpuTests::test_linear_float64_dynamic_shapes_cpu, test/inductor/test_torchinductor_dynamic_shapes.py::DynamicShapesCpuTests::test_linspace1_dynamic_shapes_cpu, test/inductor/test_torchinductor_dynamic_shapes.py::DynamicShapesCpuTests::test_linspace2_dynamic_shapes_cpu, test/inductor/test_torchinductor_dynamic_shapes.py::DynamicShapesCpuTests::test_lite_mode_fallback_dynamic_shapes_cpu, test/inductor/test_torchinductor_dynamic_shapes.py::DynamicShapesCpuTests::test_lite_triton_kernel_wrapper_functional_dynamic_shapes_cpu, test/inductor/test_torchinductor_dynamic_shapes.py::DynamicShapesCpuTests::test_log_softmax_dynamic_shapes_cpu, test/inductor/test_torchinductor_dynamic_shapes.py::DynamicShapesCpuTests::test_logsumexp_dynamic_shapes_cpu, test/inductor/test_torchinductor_dynamic_shapes.py::DynamicShapesCpuTests::test_low_memory_max_pool_dilation_2_dim_3_dynamic_shapes_cpu, test/inductor/test_torchinductor_dynamic_shapes.py::DynamicShapesCpuTests::test_masked_scatter_dynamic_shapes_cpu, test/inductor/test_torchinductor_dynamic_shapes.py::DynamicShapesCpuTests::test_max_pool2d1_dynamic_shapes_cpu, test/inductor/test_torchinductor_dynamic_shapes.py::DynamicShapesCpuTests::test_max_pool2d4_dynamic_shapes_cpu, test/inductor/test_torchinductor_dynamic_shapes.py::DynamicShapesCpuTests::test_max_pool2d6_dilation_2_dynamic_shapes_cpu, test/inductor/test_torchinductor_dynamic_shapes.py::DynamicShapesCpuTests::test_max_pool2d_with_indices_backward2_dynamic_shapes_cpu, test/inductor/test_torchinductor_dynamic_shapes.py::DynamicShapesCpuTests::test_max_pool2d_with_indices_backward3_dynamic_shapes_cpu, test/inductor/test_torchinductor_dynamic_shapes.py::DynamicShapesCpuTests::test_max_pool2d_with_indices_backward4_dynamic_shapes_cpu, test/inductor/test_torchinductor_dynamic_shapes.py::DynamicShapesCpuTests::test_max_pool2d_with_indices_backward6_dynamic_shapes_cpu, test/inductor/test_torchinductor_dynamic_shapes.py::DynamicShapesCpuTests::test_min_max_reduction_dynamic_shapes_cpu, test/inductor/test_torchinductor_dynamic_shapes.py::DynamicShapesCpuTests::test_misaligned_address_issue1_dynamic_shapes_cpu, test/inductor/test_torchinductor_dynamic_shapes.py::DynamicShapesCpuTests::test_mixed_mm2_dynamic_shapes_cpu, test/inductor/test_torchinductor_dynamic_shapes.py::DynamicShapesCpuTests::test_mixed_mm3_dynamic_shapes_cpu, test/inductor/test_torchinductor_dynamic_shapes.py::DynamicShapesCpuTests::test_mm_mixed_dtype_dynamic_shapes_cpu, test/inductor/test_torchinductor_dynamic_shapes.py::DynamicShapesCpuTests::test_mul_softmax_symfloat_dynamic_shapes_cpu, test/inductor/test_torchinductor_dynamic_shapes.py::DynamicShapesCpuTests::test_multi_gpu_device_dynamic_shapes_cpu, test/inductor/test_torchinductor_dynamic_shapes.py::DynamicShapesCpuTests::test_mutable_custom_op_fixed_layout_dynamic_shapes_cpu, test/inductor/test_torchinductor_dynamic_shapes.py::DynamicShapesCpuTests::test_nan_sort_stable_False_descending_True_dynamic_shapes_cpu, test/inductor/test_torchinductor_dynamic_shapes.py::DynamicShapesCpuTests::test_nan_sort_stable_True_descending_True_dynamic_shapes_cpu, test/inductor/test_torchinductor_dynamic_shapes.py::DynamicShapesCpuTests::test_neg_index_dynamic_shapes_cpu, test/inductor/test_torchinductor_dynamic_shapes.py::DynamicShapesCpuTests::test_nll_loss_forward_dynamic_shapes_cpu, test/inductor/test_torchinductor_dynamic_shapes.py::DynamicShapesCpuTests::test_pad_single_dynamic_shapes_cpu, test/inductor/test_torchinductor_dynamic_shapes.py::DynamicShapesCpuTests::test_pattern_matcher_multi_user_dynamic_shapes_cpu, test/inductor/test_torchinductor_dynamic_shapes.py::DynamicShapesCpuTests::test_permute2_dynamic_shapes_cpu, test/inductor/test_torchinductor_dynamic_shapes.py::DynamicShapesCpuTests::test_pointwise_bessel_j0_dynamic_shapes_cpu, test/inductor/test_torchinductor_dynamic_shapes.py::DynamicShapesCpuTests::test_pointwise_erfc_dynamic_shapes_cpu, test/inductor/test_torchinductor_dynamic_shapes.py::DynamicShapesCpuTests::test_pointwise_erfcx_dynamic_shapes_cpu, test/inductor/test_torchinductor_dynamic_shapes.py::DynamicShapesCpuTests::test_pointwise_erfinv_dynamic_shapes_cpu, test/inductor/test_torchinductor_dynamic_shapes.py::DynamicShapesCpuTests::test_pointwise_gammainc_dynamic_shapes_cpu, test/inductor/test_torchinductor_dynamic_shapes.py::DynamicShapesCpuTests::test_pointwise_hermite_polynomial_h_dynamic_shapes_cpu, test/inductor/test_torchinductor_dynamic_shapes.py::DynamicShapesCpuTests::test_pointwise_hermite_polynomial_he_dynamic_shapes_cpu, test/inductor/test_torchinductor_dynamic_shapes.py::DynamicShapesCpuTests::test_pointwise_laguerre_polynomial_l_dynamic_shapes_cpu, test/inductor/test_torchinductor_dynamic_shapes.py::DynamicShapesCpuTests::test_pointwise_legendre_polynomial_p_dynamic_shapes_cpu, test/inductor/test_torchinductor_dynamic_shapes.py::DynamicShapesCpuTests::test_pointwise_log1p_dynamic_shapes_cpu, test/inductor/test_torchinductor_dynamic_shapes.py::DynamicShapesCpuTests::test_pointwise_log_ndtr_dynamic_shapes_cpu, test/inductor/test_torchinductor_dynamic_shapes.py::DynamicShapesCpuTests::test_pointwise_logit_dynamic_shapes_cpu, test/inductor/test_torchinductor_dynamic_shapes.py::DynamicShapesCpuTests::test_pointwise_modified_bessel_i0_dynamic_shapes_cpu, test/inductor/test_torchinductor_dynamic_shapes.py::DynamicShapesCpuTests::test_pointwise_modified_bessel_i1_dynamic_shapes_cpu, test/inductor/test_torchinductor_dynamic_shapes.py::DynamicShapesCpuTests::test_pointwise_multigammaln_dynamic_shapes_cpu, test/inductor/test_torchinductor_dynamic_shapes.py::DynamicShapesCpuTests::test_pointwise_ndtri_dynamic_shapes_cpu, test/inductor/test_torchinductor_dynamic_shapes.py::DynamicShapesCpuTests::test_pointwise_sinc_dynamic_shapes_cpu, test/inductor/test_torchinductor_dynamic_shapes.py::DynamicShapesCpuTests::test_polar_dynamic_shapes_cpu, test/inductor/test_torchinductor_dynamic_shapes.py::DynamicShapesCpuTests::test_pow_int_dynamic_shapes_cpu, test/inductor/test_torchinductor_dynamic_shapes.py::DynamicShapesCpuTests::test_rand_like_deterministic_dynamic_shapes_cpu, test/inductor/test_torchinductor_dynamic_shapes.py::DynamicShapesCpuTests::test_randn_like_empty_dynamic_shapes_cpu, test/inductor/test_torchinductor_dynamic_shapes.py::DynamicShapesCpuTests::test_randn_with_dtype_and_device_dynamic_shapes_cpu, test/inductor/test_torchinductor_dynamic_shapes.py::DynamicShapesCpuTests::test_reduction4_dynamic_shapes_cpu, test/inductor/test_torchinductor_dynamic_shapes.py::DynamicShapesCpuTests::test_remainder_dynamic_shapes_cpu, test/inductor/test_torchinductor_dynamic_shapes.py::DynamicShapesCpuTests::test_remove_no_ops_dynamic_shapes_cpu, test/inductor/test_torchinductor_dynamic_shapes.py::DynamicShapesCpuTests::test_remove_noop_view_default_dynamic_shapes_cpu, test/inductor/test_torchinductor_dynamic_shapes.py::DynamicShapesCpuTests::test_repeat_interleave_2_dynamic_shapes_cpu, test/inductor/test_torchinductor_dynamic_shapes.py::DynamicShapesCpuTests::test_repeat_interleave_Tensor_decomp_int32_nd_1_dynamic_shapes_cpu, test/inductor/test_torchinductor_dynamic_shapes.py::DynamicShapesCpuTests::test_replication_pad_errors_with_bool_dynamic_shapes_cpu, test/inductor/test_torchinductor_dynamic_shapes.py::DynamicShapesCpuTests::test_resize_as_dynamic_shapes_cpu, test/inductor/test_torchinductor_dynamic_shapes.py::DynamicShapesCpuTests::test_reuse_buffers_with_aliasing_dynamic_shapes_cpu, test/inductor/test_torchinductor_dynamic_shapes.py::DynamicShapesCpuTests::test_round_correctness_dynamic_shapes_cpu, test/inductor/test_torchinductor_dynamic_shapes.py::DynamicShapesCpuTests::test_round_dynamic_shapes_cpu, test/inductor/test_torchinductor_dynamic_shapes.py::DynamicShapesCpuTests::test_scalar_output_dynamic_shapes_cpu, test/inductor/test_torchinductor_dynamic_shapes.py::DynamicShapesCpuTests::test_scaled_dot_product_attention_dynamic_shapes_cpu, test/inductor/test_torchinductor_dynamic_shapes.py::DynamicShapesCpuTests::test_scaled_dot_product_efficient_attention_dynamic_shapes_cpu, test/inductor/test_torchinductor_dynamic_shapes.py::DynamicShapesCpuTests::test_scatter_add2_dynamic_shapes_cpu, test/inductor/test_torchinductor_dynamic_shapes.py::DynamicShapesCpuTests::test_scatter_reduce1_dynamic_shapes_cpu, test/inductor/test_torchinductor_dynamic_shapes.py::DynamicShapesCpuTests::test_scatter_reduce3_dynamic_shapes_cpu, test/inductor/test_torchinductor_dynamic_shapes.py::DynamicShapesCpuTests::test_sdpa_prefer_nd_tiling_True_use_block_ptr_False_dynamic_shapes_cpu, test/inductor/test_torchinductor_dynamic_shapes.py::DynamicShapesCpuTests::test_should_pad_bench_for_bmm_dynamic_shapes_cpu, test/inductor/test_torchinductor_dynamic_shapes.py::DynamicShapesCpuTests::test_sign_dtype_dynamic_shapes_cpu, test/inductor/test_torchinductor_dynamic_shapes.py::DynamicShapesCpuTests::test_sin_dynamic_shapes_cpu, test/inductor/test_torchinductor_dynamic_shapes.py::DynamicShapesCpuTests::test_size_asserts_for_multi_output_fallback_dynamic_shapes_cpu, test/inductor/test_torchinductor_dynamic_shapes.py::DynamicShapesCpuTests::test_slice4_dynamic_shapes_cpu, test/inductor/test_torchinductor_dynamic_shapes.py::DynamicShapesCpuTests::test_slice_scatter3_dynamic_shapes_cpu, test/inductor/test_torchinductor_dynamic_shapes.py::DynamicShapesCpuTests::test_slice_scatter5_dynamic_shapes_cpu, test/inductor/test_torchinductor_dynamic_shapes.py::DynamicShapesCpuTests::test_slice_scatter_dynamic_shapes_cpu, test/inductor/test_torchinductor_dynamic_shapes.py::DynamicShapesCpuTests::test_softmax_backward_data_dynamic_shapes_cpu, test/inductor/test_torchinductor_dynamic_shapes.py::DynamicShapesCpuTests::test_squeeze_varargs_dynamic_shapes_cpu, test/inductor/test_torchinductor_dynamic_shapes.py::DynamicShapesCpuTests::test_stack_dynamic_shapes_cpu, test/inductor/test_torchinductor_dynamic_shapes.py::DynamicShapesCpuTests::test_std_dynamic_shapes_cpu, test/inductor/test_torchinductor_dynamic_shapes.py::DynamicShapesCpuTests::test_sum1_dynamic_shapes_cpu, test/inductor/test_torchinductor_dynamic_shapes.py::DynamicShapesCpuTests::test_tanh_dynamic_shapes_cpu, test/inductor/test_torchinductor_dynamic_shapes.py::DynamicShapesCpuTests::test_tensor2_dynamic_shapes_cpu, test/inductor/test_torchinductor_dynamic_shapes.py::DynamicShapesCpuTests::test_transpose_add_dynamic_shapes_cpu, test/inductor/test_torchinductor_dynamic_shapes.py::DynamicShapesCpuTests::test_unbacked_floordiv_simplify_dynamic_shapes_cpu, test/inductor/test_torchinductor_dynamic_shapes.py::DynamicShapesCpuTests::test_unspec_inputs_float32_dynamic_shapes_cpu, test/inductor/test_torchinductor_dynamic_shapes.py::DynamicShapesCpuTests::test_unspec_inputs_int8_dynamic_shapes_cpu, test/inductor/test_torchinductor_dynamic_shapes.py::DynamicShapesCpuTests::test_unsqueeze_dynamic_shapes_cpu, test/inductor/test_torchinductor_dynamic_shapes.py::DynamicShapesCpuTests::test_unsqueeze_inplace_dynamic_shapes_cpu, test/inductor/test_torchinductor_dynamic_shapes.py::DynamicShapesCpuTests::test_upsample_bilinear2d_a_dynamic_shapes_cpu, test/inductor/test_torchinductor_dynamic_shapes.py::DynamicShapesCpuTests::test_upsample_cat_conv_dynamic_shapes_cpu, test/inductor/test_torchinductor_dynamic_shapes.py::DynamicShapesCpuTests::test_upsample_nearest2d_backward_dynamic_shapes_cpu, test/inductor/test_torchinductor_dynamic_shapes.py::DynamicShapesCpuTests::test_var_mean_tile_reduction_False_dynamic_shapes_cpu, test/inductor/test_torchinductor_dynamic_shapes.py::DynamicShapesCpuTests::test_vdd_clamp_dynamic_shapes_cpu, test/inductor/test_torchinductor_dynamic_shapes.py::DynamicShapesCpuTests::test_vectorized_ops_masked_dynamic_shapes_cpu, test/inductor/test_torchinductor_dynamic_shapes.py::DynamicShapesCpuTests::test_view_uint8_through_differing_bitwidths_dynamic_shapes_cpu, test/inductor/test_torchinductor_dynamic_shapes.py::DynamicShapesCpuTests::test_views2_dynamic_shapes_cpu, test/inductor/test_torchinductor_dynamic_shapes.py::DynamicShapesCpuTests::test_xblock_divides_xnumel_dynamic_shapes_cpu, test/inductor/test_torchinductor_dynamic_shapes.py::DynamicShapesCpuTests::test_zero_dim_reductions_dynamic_shapes_cpu, test/inductor/test_torchinductor_dynamic_shapes.py::DynamicShapesGPUTests::test__dyn_quant_matmul_4bit_bf16_input_dynamic_shapes_cuda, test/inductor/test_torchinductor_dynamic_shapes.py::DynamicShapesGPUTests::test__dyn_quant_matmul_4bit_fp32_input_dynamic_shapes_cuda, test/inductor/test_torchinductor_dynamic_shapes.py::DynamicShapesGPUTests::test__dyn_quant_pack_4bit_weight_fp32_dynamic_shapes_cuda, test/inductor/test_torchinductor_dynamic_shapes.py::DynamicShapesGPUTests::test__unsafe_masked_index_dynamic_shapes_cuda, test/inductor/test_torchinductor_dynamic_shapes.py::DynamicShapesGPUTests::test_adaptive_avg_pool1d_argmax_dynamic_shapes_cuda, test/inductor/test_torchinductor_dynamic_shapes.py::DynamicShapesGPUTests::test_adaptive_avg_pool2d1_dynamic_shapes_cuda, test/inductor/test_torchinductor_dynamic_shapes.py::DynamicShapesGPUTests::test_adaptive_avg_pool_errors_with_long_dynamic_shapes_cuda, test/inductor/test_torchinductor_dynamic_shapes.py::DynamicShapesGPUTests::test_add_complex4_dynamic_shapes_cuda, test/inductor/test_torchinductor_dynamic_shapes.py::DynamicShapesGPUTests::test_add_complex_dynamic_shapes_cuda, test/inductor/test_torchinductor_dynamic_shapes.py::DynamicShapesGPUTests::test_add_const_int_dynamic_shapes_cuda, test/inductor/test_torchinductor_dynamic_shapes.py::DynamicShapesGPUTests::test_allow_reuse_active_if_under_peak_dynamic_shapes_cuda, test/inductor/test_torchinductor_dynamic_shapes.py::DynamicShapesGPUTests::test_allow_reuse_disable_if_exceed_peak_dynamic_shapes_cuda, test/inductor/test_torchinductor_dynamic_shapes.py::DynamicShapesGPUTests::test_angle_dynamic_shapes_cuda, test/inductor/test_torchinductor_dynamic_shapes.py::DynamicShapesGPUTests::test_aoti_eager_support_out_dynamic_shapes_cuda, test/inductor/test_torchinductor_dynamic_shapes.py::DynamicShapesGPUTests::test_aoti_eager_with_persistent_cache_dynamic_shapes_cuda, test/inductor/test_torchinductor_dynamic_shapes.py::DynamicShapesGPUTests::test_arange3_dynamic_shapes_cuda, test/inductor/test_torchinductor_dynamic_shapes.py::DynamicShapesGPUTests::test_arange4_dynamic_shapes_cuda, test/inductor/test_torchinductor_dynamic_shapes.py::DynamicShapesGPUTests::test_arange5_dynamic_shapes_cuda, test/inductor/test_torchinductor_dynamic_shapes.py::DynamicShapesGPUTests::test_argmax_argmin1_dynamic_shapes_cuda, test/inductor/test_torchinductor_dynamic_shapes.py::DynamicShapesGPUTests::test_argmax_min_int32_dynamic_shapes_cuda, test/inductor/test_torchinductor_dynamic_shapes.py::DynamicShapesGPUTests::test_as_strided_on_views_dynamic_shapes_cuda, test/inductor/test_torchinductor_dynamic_shapes.py::DynamicShapesGPUTests::test_assert_alignment_op_name_pass_dynamic_shapes_cuda, test/inductor/test_torchinductor_dynamic_shapes.py::DynamicShapesGPUTests::test_avg_pool2d6_dynamic_shapes_cuda, test/inductor/test_torchinductor_dynamic_shapes.py::DynamicShapesGPUTests::test_avg_pool2d_backward4_dynamic_shapes_cuda, test/inductor/test_torchinductor_dynamic_shapes.py::DynamicShapesGPUTests::test_avg_pool3d_backward2_dynamic_shapes_cuda, test/inductor/test_torchinductor_dynamic_shapes.py::DynamicShapesGPUTests::test_bitwise2_dynamic_shapes_cuda, test/inductor/test_torchinductor_dynamic_shapes.py::DynamicShapesGPUTests::test_bool_dynamic_shapes_cuda, test/inductor/test_torchinductor_dynamic_shapes.py::DynamicShapesGPUTests::test_bucketize_int_int16_int32_dynamic_shapes_cuda, test/inductor/test_torchinductor_dynamic_shapes.py::DynamicShapesGPUTests::test_bucketize_int_int16_int64_dynamic_shapes_cuda, test/inductor/test_torchinductor_dynamic_shapes.py::DynamicShapesGPUTests::test_bucketize_int_int16_uint8_dynamic_shapes_cuda, test/inductor/test_torchinductor_dynamic_shapes.py::DynamicShapesGPUTests::test_bucketize_int_int64_int8_dynamic_shapes_cuda, test/inductor/test_torchinductor_dynamic_shapes.py::DynamicShapesGPUTests::test_bucketize_int_int64_uint8_dynamic_shapes_cuda, test/inductor/test_torchinductor_dynamic_shapes.py::DynamicShapesGPUTests::test_bucketize_int_int8_int16_dynamic_shapes_cuda, test/inductor/test_torchinductor_dynamic_shapes.py::DynamicShapesGPUTests::test_bucketize_int_uint8_uint8_dynamic_shapes_cuda, test/inductor/test_torchinductor_dynamic_shapes.py::DynamicShapesGPUTests::test_buffer_copied_in_graph_dynamic_shapes_cuda, test/inductor/test_torchinductor_dynamic_shapes.py::DynamicShapesGPUTests::test_buffer_copied_in_graph_with_different_shapes_dynamic_shapes_cuda, test/inductor/test_torchinductor_dynamic_shapes.py::DynamicShapesGPUTests::test_cat_dynamic_shapes_cuda, test/inductor/test_torchinductor_dynamic_shapes.py::DynamicShapesGPUTests::test_cat_negative_dim_dynamic_shapes_cuda, test/inductor/test_torchinductor_dynamic_shapes.py::DynamicShapesGPUTests::test_cat_uint8_dynamic_shapes_cuda, test/inductor/test_torchinductor_dynamic_shapes.py::DynamicShapesGPUTests::test_cat_unbacked_2d_dynamic_shapes_cuda, test/inductor/test_torchinductor_dynamic_shapes.py::DynamicShapesGPUTests::test_cat_unbacked_legacy_empty_dynamic_shapes_cuda, test/inductor/test_torchinductor_dynamic_shapes.py::DynamicShapesGPUTests::test_cauchy_dynamic_shapes_cuda, test/inductor/test_torchinductor_dynamic_shapes.py::DynamicShapesGPUTests::test_chunk_recompiles_dynamic_shapes_cuda, test/inductor/test_torchinductor_dynamic_shapes.py::DynamicShapesGPUTests::test_constant_pad_2d_strides_nonpositive_dynamic_shapes_cuda, test/inductor/test_torchinductor_dynamic_shapes.py::DynamicShapesGPUTests::test_conv_backward_dynamic_shapes_cuda, test/inductor/test_torchinductor_dynamic_shapes.py::DynamicShapesGPUTests::test_convolution2_dynamic_shapes_cuda, test/inductor/test_torchinductor_dynamic_shapes.py::DynamicShapesGPUTests::test_convolution3_dynamic_shapes_cuda, test/inductor/test_torchinductor_dynamic_shapes.py::DynamicShapesGPUTests::test_convolution4_dynamic_shapes_cuda, test/inductor/test_torchinductor_dynamic_shapes.py::DynamicShapesGPUTests::test_copy_non_blocking_is_pinned_use_cat_True_dynamic_shapes_cuda, test/inductor/test_torchinductor_dynamic_shapes.py::DynamicShapesGPUTests::test_cpu_scalar_with_cpu_scalar_dynamic_shapes_cuda, test/inductor/test_torchinductor_dynamic_shapes.py::DynamicShapesGPUTests::test_cpu_scalar_with_gpu_tensor_cpp_dynamic_shapes_cuda, test/inductor/test_torchinductor_dynamic_shapes.py::DynamicShapesGPUTests::test_cpu_scalar_with_gpu_tensor_dynamic_shapes_cuda, test/inductor/test_torchinductor_dynamic_shapes.py::DynamicShapesGPUTests::test_cummin_dynamic_shapes_cuda, test/inductor/test_torchinductor_dynamic_shapes.py::DynamicShapesGPUTests::test_cumprod_zero_dim_dynamic_shapes_cuda, test/inductor/test_torchinductor_dynamic_shapes.py::DynamicShapesGPUTests::test_cumsum_dynamic_shapes_cuda, test/inductor/test_torchinductor_dynamic_shapes.py::DynamicShapesGPUTests::test_cumsum_no_mask_dynamic_shapes_cuda, test/inductor/test_torchinductor_dynamic_shapes.py::DynamicShapesGPUTests::test_custom_op_unbacked_symints_dynamic_shapes_cuda, test/inductor/test_torchinductor_dynamic_shapes.py::DynamicShapesGPUTests::test_custom_scan_op_compiled_dynamic_shapes_cuda, test/inductor/test_torchinductor_dynamic_shapes.py::DynamicShapesGPUTests::test_custom_scan_would_split_dynamic_shapes_cuda, test/inductor/test_torchinductor_dynamic_shapes.py::DynamicShapesGPUTests::test_dist_bf16_dynamic_shapes_cuda, test/inductor/test_torchinductor_dynamic_shapes.py::DynamicShapesGPUTests::test_div1_dynamic_shapes_cuda, test/inductor/test_torchinductor_dynamic_shapes.py::DynamicShapesGPUTests::test_div5_dynamic_shapes_cuda, test/inductor/test_torchinductor_dynamic_shapes.py::DynamicShapesGPUTests::test_div_prim_dynamic_shapes_cuda, test/inductor/test_torchinductor_dynamic_shapes.py::DynamicShapesGPUTests::test_div_zero_dim_dynamic_shapes_cuda, test/inductor/test_torchinductor_dynamic_shapes.py::DynamicShapesGPUTests::test_dropout_trivial_1_dynamic_shapes_cuda, test/inductor/test_torchinductor_dynamic_shapes.py::DynamicShapesGPUTests::test_dtypeview_bfloat16_float32_dynamic_shapes_cuda, test/inductor/test_torchinductor_dynamic_shapes.py::DynamicShapesGPUTests::test_dtypeview_bfloat16_int32_dynamic_shapes_cuda, test/inductor/test_torchinductor_dynamic_shapes.py::DynamicShapesGPUTests::test_dtypeview_float16_int64_dynamic_shapes_cuda, test/inductor/test_torchinductor_dynamic_shapes.py::DynamicShapesGPUTests::test_dtypeview_float32_bfloat16_dynamic_shapes_cuda, test/inductor/test_torchinductor_dynamic_shapes.py::DynamicShapesGPUTests::test_dtypeview_float64_bfloat16_dynamic_shapes_cuda, test/inductor/test_torchinductor_dynamic_shapes.py::DynamicShapesGPUTests::test_dtypeview_float64_float16_dynamic_shapes_cuda, test/inductor/test_torchinductor_dynamic_shapes.py::DynamicShapesGPUTests::test_dtypeview_float64_uint8_dynamic_shapes_cuda, test/inductor/test_torchinductor_dynamic_shapes.py::DynamicShapesGPUTests::test_dtypeview_int16_float16_dynamic_shapes_cuda, test/inductor/test_torchinductor_dynamic_shapes.py::DynamicShapesGPUTests::test_dtypeview_int16_int16_dynamic_shapes_cuda, test/inductor/test_torchinductor_dynamic_shapes.py::DynamicShapesGPUTests::test_dtypeview_int32_int64_dynamic_shapes_cuda, test/inductor/test_torchinductor_dynamic_shapes.py::DynamicShapesGPUTests::test_dtypeview_int32_uint8_dynamic_shapes_cuda, test/inductor/test_torchinductor_dynamic_shapes.py::DynamicShapesGPUTests::test_dtypeview_int64_float64_dynamic_shapes_cuda, test/inductor/test_torchinductor_dynamic_shapes.py::DynamicShapesGPUTests::test_dtypeview_int64_int32_dynamic_shapes_cuda, test/inductor/test_torchinductor_dynamic_shapes.py::DynamicShapesGPUTests::test_dtypeview_int8_float16_dynamic_shapes_cuda, test/inductor/test_torchinductor_dynamic_shapes.py::DynamicShapesGPUTests::test_dtypeview_int8_uint8_dynamic_shapes_cuda, test/inductor/test_torchinductor_dynamic_shapes.py::DynamicShapesGPUTests::test_dtypeview_uint8_int32_dynamic_shapes_cuda, test/inductor/test_torchinductor_dynamic_shapes.py::DynamicShapesGPUTests::test_dtypeview_uint8_uint8_dynamic_shapes_cuda, test/inductor/test_torchinductor_dynamic_shapes.py::DynamicShapesGPUTests::test_erfc_dynamic_shapes_cuda, test/inductor/test_torchinductor_dynamic_shapes.py::DynamicShapesGPUTests::test_fallback_mutable_op_list_tensor_dynamic_shapes_cuda, test/inductor/test_torchinductor_dynamic_shapes.py::DynamicShapesGPUTests::test_fallback_mutable_op_no_mutated_tensors_dynamic_shapes_cuda, test/inductor/test_torchinductor_dynamic_shapes.py::DynamicShapesGPUTests::test_fractional_max_pool2d1_dynamic_shapes_cuda, test/inductor/test_torchinductor_dynamic_shapes.py::DynamicShapesGPUTests::test_fractional_max_pool2d4_dynamic_shapes_cuda, test/inductor/test_torchinductor_dynamic_shapes.py::DynamicShapesGPUTests::test_full_like_sliced_dynamic_shapes_cuda, test/inductor/test_torchinductor_dynamic_shapes.py::DynamicShapesGPUTests::test_fuse_large_params_dynamic_shapes_cuda, test/inductor/test_torchinductor_dynamic_shapes.py::DynamicShapesGPUTests::test_fusing_write_into_disjoint_read_dynamic_shapes_cuda, test/inductor/test_torchinductor_dynamic_shapes.py::DynamicShapesGPUTests::test_graph_partition_both_scalars_dynamic_shapes_cuda, test/inductor/test_torchinductor_dynamic_shapes.py::DynamicShapesGPUTests::test_graph_partition_misaligned_input_dynamic_shapes_cuda, test/inductor/test_torchinductor_dynamic_shapes.py::DynamicShapesGPUTests::test_graph_partition_pad_dynamic_dynamic_shapes_cuda, test/inductor/test_torchinductor_dynamic_shapes.py::DynamicShapesGPUTests::test_hardsigmoid_dynamic_shapes_cuda, test/inductor/test_torchinductor_dynamic_shapes.py::DynamicShapesGPUTests::test_hardtanh_dynamic_shapes_cuda, test/inductor/test_torchinductor_dynamic_shapes.py::DynamicShapesGPUTests::test_horizonal_fusion1_dynamic_shapes_cuda, test/inductor/test_torchinductor_dynamic_shapes.py::DynamicShapesGPUTests::test_index3_dynamic_shapes_cuda, test/inductor/test_torchinductor_dynamic_shapes.py::DynamicShapesGPUTests::test_index_dynamic_shapes_dynamic_shapes_cuda, test/inductor/test_torchinductor_dynamic_shapes.py::DynamicShapesGPUTests::test_index_propagation_dynamic_shapes_cuda, test/inductor/test_torchinductor_dynamic_shapes.py::DynamicShapesGPUTests::test_index_put_as_masked_fill_dynamic_shapes_cuda, test/inductor/test_torchinductor_dynamic_shapes.py::DynamicShapesGPUTests::test_index_put_deterministic_fallback_dynamic_shapes_cuda, test/inductor/test_torchinductor_dynamic_shapes.py::DynamicShapesGPUTests::test_index_put_reinplace_dynamic_shapes_cuda, test/inductor/test_torchinductor_dynamic_shapes.py::DynamicShapesGPUTests::test_inductor_multiple_specializations_dynamic_shapes_cuda, test/inductor/test_torchinductor_dynamic_shapes.py::DynamicShapesGPUTests::test_inductor_triton_bucketize_respects_masking_dynamic_shapes_cuda, test/inductor/test_torchinductor_dynamic_shapes.py::DynamicShapesGPUTests::test_inner_reduction_detection_dynamic_shapes_cuda, test/inductor/test_torchinductor_dynamic_shapes.py::DynamicShapesGPUTests::test_inplace_flip_dynamic_shapes_cuda, test/inductor/test_torchinductor_dynamic_shapes.py::DynamicShapesGPUTests::test_inplace_mixed_dtype_ops_dynamic_shapes_cuda, test/inductor/test_torchinductor_dynamic_shapes.py::DynamicShapesGPUTests::test_inplace_resize_as_dynamic_shapes_cuda, test/inductor/test_torchinductor_dynamic_shapes.py::DynamicShapesGPUTests::test_inplace_where_pointwise_dynamic_shapes_cuda, test/inductor/test_torchinductor_dynamic_shapes.py::DynamicShapesGPUTests::test_isinf2_dynamic_shapes_cuda, test/inductor/test_torchinductor_dynamic_shapes.py::DynamicShapesGPUTests::test_kernel_names_dynamic_shapes_cuda, test/inductor/test_torchinductor_dynamic_shapes.py::DynamicShapesGPUTests::test_lgamma_dynamic_shapes_cuda, test/inductor/test_torchinductor_dynamic_shapes.py::DynamicShapesGPUTests::test_like_rands2_dynamic_shapes_cuda, test/inductor/test_torchinductor_dynamic_shapes.py::DynamicShapesGPUTests::test_like_rands3_dynamic_shapes_cuda, test/inductor/test_torchinductor_dynamic_shapes.py::DynamicShapesGPUTests::test_linalg_eig_stride_consistency_dynamic_shapes_cuda, test/inductor/test_torchinductor_dynamic_shapes.py::DynamicShapesGPUTests::test_linear_dynamic_maxautotune_dynamic_shapes_cuda, test/inductor/test_torchinductor_dynamic_shapes.py::DynamicShapesGPUTests::test_linspace1_dynamic_shapes_cuda, test/inductor/test_torchinductor_dynamic_shapes.py::DynamicShapesGPUTests::test_linspace3_dynamic_shapes_cuda, test/inductor/test_torchinductor_dynamic_shapes.py::DynamicShapesGPUTests::test_lite_mode_not_decompose_dynamic_shapes_cuda, test/inductor/test_torchinductor_dynamic_shapes.py::DynamicShapesGPUTests::test_lite_regional_compile_repeated_blocks_dynamic_shapes_cuda, test/inductor/test_torchinductor_dynamic_shapes.py::DynamicShapesGPUTests::test_log_softmax_dynamic_shapes_cuda, test/inductor/test_torchinductor_dynamic_shapes.py::DynamicShapesGPUTests::test_masked_scatter_dynamic_shapes_cuda, test/inductor/test_torchinductor_dynamic_shapes.py::DynamicShapesGPUTests::test_max_pool2d3_dynamic_shapes_cuda, test/inductor/test_torchinductor_dynamic_shapes.py::DynamicShapesGPUTests::test_max_pool2d7_dynamic_shapes_cuda, test/inductor/test_torchinductor_dynamic_shapes.py::DynamicShapesGPUTests::test_max_pool2d_with_indices_backward2_dynamic_shapes_cuda, test/inductor/test_torchinductor_dynamic_shapes.py::DynamicShapesGPUTests::test_min_max_reduction_nan_dynamic_shapes_cuda, test/inductor/test_torchinductor_dynamic_shapes.py::DynamicShapesGPUTests::test_mixed_mm_dynamic_shapes_cuda, test/inductor/test_torchinductor_dynamic_shapes.py::DynamicShapesGPUTests::test_mm_views_dynamic_shapes_cuda, test/inductor/test_torchinductor_dynamic_shapes.py::DynamicShapesGPUTests::test_mul_softmax_symfloat_dynamic_shapes_cuda, test/inductor/test_torchinductor_dynamic_shapes.py::DynamicShapesGPUTests::test_multi_device_dynamic_shapes_cuda, test/inductor/test_torchinductor_dynamic_shapes.py::DynamicShapesGPUTests::test_multilayer_var_lowp_dynamic_shapes_cuda, test/inductor/test_torchinductor_dynamic_shapes.py::DynamicShapesGPUTests::test_nan_sort_stable_False_descending_False_dynamic_shapes_cuda, test/inductor/test_torchinductor_dynamic_shapes.py::DynamicShapesGPUTests::test_neg_max_uint8_dynamic_shapes_cuda, test/inductor/test_torchinductor_dynamic_shapes.py::DynamicShapesGPUTests::test_new_empty_dynamic_shapes_cuda, test/inductor/test_torchinductor_dynamic_shapes.py::DynamicShapesGPUTests::test_new_empty_strided_dynamic_shapes_cuda, test/inductor/test_torchinductor_dynamic_shapes.py::DynamicShapesGPUTests::test_pad_cast_dynamic_shapes_cuda, test/inductor/test_torchinductor_dynamic_shapes.py::DynamicShapesGPUTests::test_philox_rand_dynamic_shapes_cuda, test/inductor/test_torchinductor_dynamic_shapes.py::DynamicShapesGPUTests::test_pointwise_airy_ai_dynamic_shapes_cuda, test/inductor/test_torchinductor_dynamic_shapes.py::DynamicShapesGPUTests::test_pointwise_bessel_y0_dynamic_shapes_cuda, test/inductor/test_torchinductor_dynamic_shapes.py::DynamicShapesGPUTests::test_pointwise_expit_dynamic_shapes_cuda, test/inductor/test_torchinductor_dynamic_shapes.py::DynamicShapesGPUTests::test_pointwise_expm1_dynamic_shapes_cuda, test/inductor/test_torchinductor_dynamic_shapes.py::DynamicShapesGPUTests::test_pointwise_hermite_polynomial_he_dynamic_shapes_cuda, test/inductor/test_torchinductor_dynamic_shapes.py::DynamicShapesGPUTests::test_pointwise_i0_dynamic_shapes_cuda, test/inductor/test_torchinductor_dynamic_shapes.py::DynamicShapesGPUTests::test_pointwise_i1e_dynamic_shapes_cuda, test/inductor/test_torchinductor_dynamic_shapes.py::DynamicShapesGPUTests::test_pointwise_modified_bessel_k1_dynamic_shapes_cuda, test/inductor/test_torchinductor_dynamic_shapes.py::DynamicShapesGPUTests::test_pointwise_spherical_bessel_j0_dynamic_shapes_cuda, test/inductor/test_torchinductor_dynamic_shapes.py::DynamicShapesGPUTests::test_pointwise_xlogy_dynamic_shapes_cuda, test/inductor/test_torchinductor_dynamic_shapes.py::DynamicShapesGPUTests::test_pow1_dynamic_shapes_cuda, test/inductor/test_torchinductor_dynamic_shapes.py::DynamicShapesGPUTests::test_pow3_dynamic_shapes_cuda, test/inductor/test_torchinductor_dynamic_shapes.py::DynamicShapesGPUTests::test_prepare_softmax_with_fast_math_dynamic_shapes_cuda, test/inductor/test_torchinductor_dynamic_shapes.py::DynamicShapesGPUTests::test_randint_kernel_count_dynamic_shapes_cuda, test/inductor/test_torchinductor_dynamic_shapes.py::DynamicShapesGPUTests::test_randn_with_dtype_and_device_dynamic_shapes_cuda, test/inductor/test_torchinductor_dynamic_shapes.py::DynamicShapesGPUTests::test_remainder_dynamic_shapes_cuda, test/inductor/test_torchinductor_dynamic_shapes.py::DynamicShapesGPUTests::test_remove_noop_copy_dynamic_shapes_cuda, test/inductor/test_torchinductor_dynamic_shapes.py::DynamicShapesGPUTests::test_remove_noop_view_dtype_dynamic_shapes_cuda, test/inductor/test_torchinductor_dynamic_shapes.py::DynamicShapesGPUTests::test_repeat_interleave_dynamic_shapes_cuda, test/inductor/test_torchinductor_dynamic_shapes.py::DynamicShapesGPUTests::test_replication_pad_errors_with_bool_dynamic_shapes_cuda, test/inductor/test_torchinductor_dynamic_shapes.py::DynamicShapesGPUTests::test_resize_as_dynamic_shapes_cuda, test/inductor/test_torchinductor_dynamic_shapes.py::DynamicShapesGPUTests::test_reuse_buffers_with_aliasing_dynamic_shapes_cuda, test/inductor/test_torchinductor_dynamic_shapes.py::DynamicShapesGPUTests::test_rsqrt_dynamic_shapes_cuda, test/inductor/test_torchinductor_dynamic_shapes.py::DynamicShapesGPUTests::test_scatter_reduce3_dynamic_shapes_cuda, test/inductor/test_torchinductor_dynamic_shapes.py::DynamicShapesGPUTests::test_scheduler_vertical_fusion1_dynamic_shapes_cuda, test/inductor/test_torchinductor_dynamic_shapes.py::DynamicShapesGPUTests::test_sdpa_prefer_nd_tiling_True_use_block_ptr_False_dynamic_shapes_cuda, test/inductor/test_torchinductor_dynamic_shapes.py::DynamicShapesGPUTests::test_sdpa_unaligned_mask_dynamic_shapes_cuda, test/inductor/test_torchinductor_dynamic_shapes.py::DynamicShapesGPUTests::test_searchsorted_broadcast_dynamic_shapes_cuda, test/inductor/test_torchinductor_dynamic_shapes.py::DynamicShapesGPUTests::test_sgn_dynamic_shapes_cuda, test/inductor/test_torchinductor_dynamic_shapes.py::DynamicShapesGPUTests::test_sgn_extremal_dynamic_shapes_cuda, test/inductor/test_torchinductor_dynamic_shapes.py::DynamicShapesGPUTests::test_should_pad_bench_for_bmm_dynamic_shapes_cuda, test/inductor/test_torchinductor_dynamic_shapes.py::DynamicShapesGPUTests::test_sigmoid_dynamic_shapes_cuda, test/inductor/test_torchinductor_dynamic_shapes.py::DynamicShapesGPUTests::test_simplify_loops_dynamic_shapes_cuda, test/inductor/test_torchinductor_dynamic_shapes.py::DynamicShapesGPUTests::test_sin_dynamic_shapes_cuda, test/inductor/test_torchinductor_dynamic_shapes.py::DynamicShapesGPUTests::test_single_elem_dynamic_shapes_cuda, test/inductor/test_torchinductor_dynamic_shapes.py::DynamicShapesGPUTests::test_sizehint_issue1_dynamic_shapes_cuda, test/inductor/test_torchinductor_dynamic_shapes.py::DynamicShapesGPUTests::test_slice2_dynamic_shapes_cuda, test/inductor/test_torchinductor_dynamic_shapes.py::DynamicShapesGPUTests::test_slice3_dynamic_shapes_cuda, test/inductor/test_torchinductor_dynamic_shapes.py::DynamicShapesGPUTests::test_slice_mutation1_dynamic_shapes_cuda, test/inductor/test_torchinductor_dynamic_shapes.py::DynamicShapesGPUTests::test_softmax_backward_data_dynamic_shapes_cuda, test/inductor/test_torchinductor_dynamic_shapes.py::DynamicShapesGPUTests::test_softmax_dynamic_shapes_cuda, test/inductor/test_torchinductor_dynamic_shapes.py::DynamicShapesGPUTests::test_special_polygamma_dynamic_shapes_cuda, test/inductor/test_torchinductor_dynamic_shapes.py::DynamicShapesGPUTests::test_split_cumprod_dynamic_shapes_cuda, test/inductor/test_torchinductor_dynamic_shapes.py::DynamicShapesGPUTests::test_split_cumsum_dynamic_shapes_cuda, test/inductor/test_torchinductor_dynamic_shapes.py::DynamicShapesGPUTests::test_squeeze1_dynamic_shapes_cuda, test/inductor/test_torchinductor_dynamic_shapes.py::DynamicShapesGPUTests::test_squeeze2_dynamic_shapes_cuda, test/inductor/test_torchinductor_dynamic_shapes.py::DynamicShapesGPUTests::test_squeeze_varargs_dynamic_shapes_cuda, test/inductor/test_torchinductor_dynamic_shapes.py::DynamicShapesGPUTests::test_strided_inputs_dynamic_shapes_cuda, test/inductor/test_torchinductor_dynamic_shapes.py::DynamicShapesGPUTests::test_sum4_dynamic_shapes_cuda, test/inductor/test_torchinductor_dynamic_shapes.py::DynamicShapesGPUTests::test_sum5_dynamic_shapes_cuda, test/inductor/test_torchinductor_dynamic_shapes.py::DynamicShapesGPUTests::test_sum_keepdims_dynamic_shapes_cuda, test/inductor/test_torchinductor_dynamic_shapes.py::DynamicShapesGPUTests::test_tensor2_dynamic_shapes_cuda, test/inductor/test_torchinductor_dynamic_shapes.py::DynamicShapesGPUTests::test_triu_dynamic_shapes_cuda, test/inductor/test_torchinductor_dynamic_shapes.py::DynamicShapesGPUTests::test_unbacked_float_item_dynamic_shapes_cuda, test/inductor/test_torchinductor_dynamic_shapes.py::DynamicShapesGPUTests::test_unspec_inputs_float16_dynamic_shapes_cuda, test/inductor/test_torchinductor_dynamic_shapes.py::DynamicShapesGPUTests::test_unspec_inputs_float64_dynamic_shapes_cuda, test/inductor/test_torchinductor_dynamic_shapes.py::DynamicShapesGPUTests::test_unspec_inputs_int8_dynamic_shapes_cuda, test/inductor/test_torchinductor_dynamic_shapes.py::DynamicShapesGPUTests::test_var_mean_div_by_dynamic_shapes_cuda, test/inductor/test_torchinductor_dynamic_shapes.py::DynamicShapesGPUTests::test_vdd_clamp_dynamic_shapes_cuda, test/inductor/test_torchinductor_dynamic_shapes.py::DynamicShapesGPUTests::test_vectorized_ops_masked_dynamic_shapes_cuda, test/inductor/test_torchinductor_dynamic_shapes.py::DynamicShapesGPUTests::test_view_as_real_dynamic_shapes_cuda, test/inductor/test_torchinductor_dynamic_shapes.py::DynamicShapesGPUTests::test_views2_dynamic_shapes_cuda, test/inductor/test_torchinductor_dynamic_shapes.py::TestInductorDynamicCUDA::test_arithmetic_constant_folding_cuda, test/inductor/test_torchinductor_dynamic_shapes.py::TestInductorDynamicCUDA::test_dynamic_stride_nobreak_cuda, test/inductor/test_torchinductor_dynamic_shapes.py::TestInductorDynamicCUDA::test_float_is_integer_cuda, test/inductor/test_torchinductor_dynamic_shapes.py::TestInductorDynamicCUDA::test_float_item_neginf_cuda, test/inductor/test_torchinductor_dynamic_shapes.py::TestInductorDynamicCUDA::test_floor_cuda, test/inductor/test_torchinductor_dynamic_shapes.py::TestInductorDynamicCUDA::test_item_bool_nobreak_cuda, test/inductor/test_torchinductor_dynamic_shapes.py::TestInductorDynamicCUDA::test_item_to_inputs_kernel_nobreak_cuda, test/inductor/test_torchinductor_dynamic_shapes.py::TestInductorDynamicCUDA::test_math_ops_op3_cuda, test/inductor/test_torchinductor_dynamic_shapes.py::TestInductorDynamicCUDA::test_nonzero_no_realloc_cuda, test/inductor/test_torchinductor_dynamic_shapes.py::TestInductorDynamicCUDA::test_pad_dynamic_cuda, test/inductor/test_torchinductor_dynamic_shapes.py::TestInductorDynamicCUDA::test_sort_dynamic_shape_with_check_cuda, test/inductor/test_torchinductor_dynamic_shapes.py::TestInductorDynamicCUDA::test_sym_sum_unbacked_cuda, test/inductor/test_torchinductor_dynamic_shapes.py::TestInductorDynamicCUDA::test_unbacked_cat_backwards_save_data_dependent_cuda, test/inductor/test_torchinductor_dynamic_shapes.py::TestInductorDynamicCUDA::test_unbacked_matmul_cuda, test/inductor/test_torchinductor_dynamic_shapes.py::TestInductorDynamicCUDA::test_unbacked_reduction_cuda, test/inductor/test_torchinductor_dynamic_shapes.py::TestInductorDynamicCUDA::test_unspecialized_float_dynamic_cuda, test/inductor/test_torchinductor_dynamic_shapes.py::TestInductorDynamicCUDA::test_unspecialized_float_fallback_specialization_cuda, test/inductor/test_torchinductor_dynamic_shapes.py::TestInductorDynamicCUDA::test_unspecialized_float_softshrink_cuda 2025-12-04T10:06:52.9715510Z 2025-12-04T10:06:52.9715811Z Finished inductor/test_torchinductor_dynamic_shapes 1/4 ... [2025-12-04 10:06:52.940832][1660.883047645], took 7.71min 2025-12-04T10:06:52.9716757Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/inductor.test_torchinductor_dynamic_shapes/inductor.test_torchinductor_dynamic_shapes-1c753e0c38ee0fc9.xml 2025-12-04T10:06:53.0582629Z Running inductor/test_kernel_benchmark 1/1 ... [2025-12-04 10:06:53.058006][1661.000223589] 2025-12-04T10:06:53.0583120Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2025-12-04T10:06:53.0585980Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'inductor/test_kernel_benchmark.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-12-04 10:06:53.058308] 2025-12-04T10:08:57.5539967Z 2025-12-04T10:08:57.5541704Z inductor/test_kernel_benchmark 1/1 was successful, full logs can be found in artifacts with path test/test-reports/inductor.test_kernel_benchmark_1.1_7a99c0fc76c1cb85_.log 2025-12-04T10:08:57.5551560Z Running 18 items in this shard: test/inductor/test_kernel_benchmark.py::TestKernelBenchmark::test_fused_layernorm_bandwidth_computation, test/inductor/test_kernel_benchmark.py::TestKernelBenchmark::test_matmul_bandwidth_computation, test/inductor/test_kernel_benchmark.py::TestKernelBenchmark::test_matmul_triton_kernel_benchmark, test/inductor/test_kernel_benchmark.py::TestKernelBenchmark::test_mm_slice_add_bandwidth_computation, test/inductor/test_kernel_benchmark.py::TestKernelBenchmark::test_mm_slice_add_bandwidth_computation_2, test/inductor/test_kernel_benchmark.py::TestKernelBenchmark::test_mm_triton_kernel_benchmark, test/inductor/test_kernel_benchmark.py::TestKernelBenchmark::test_pw_kernel_benchmark, test/inductor/test_kernel_benchmark.py::TestKernelBenchmark::test_reduction_bandwidth_computation, test/inductor/test_kernel_benchmark.py::TestKernelBenchmark::test_remove_inductor_deps, test/inductor/test_kernel_benchmark.py::TestKernelBenchmark::test_remove_inductor_deps_multiple_kernels, test/inductor/test_kernel_benchmark.py::TestKernelBenchmark::test_remove_inductor_deps_scalar, test/inductor/test_kernel_benchmark.py::TestKernelBenchmark::test_remove_inductor_deps_templates, test/inductor/test_kernel_benchmark.py::TestKernelBenchmark::test_slice_add_bandwidth_computation, test/inductor/test_kernel_benchmark.py::TestKernelBenchmark::test_slice_add_cat_bandwidth_computation, test/inductor/test_kernel_benchmark.py::TestKernelBenchmark::test_slice_mm_bandwidth_computation, test/inductor/test_kernel_benchmark.py::TestKernelBenchmark::test_split_scan, test/inductor/test_kernel_benchmark.py::TestKernelBenchmark::test_star_dep, test/inductor/test_kernel_benchmark.py::TestKernelBenchmark::test_unused_input_bandwidth_computation 2025-12-04T10:08:57.5557164Z 2025-12-04T10:08:57.5557422Z Finished inductor/test_kernel_benchmark 1/1 ... [2025-12-04 10:08:57.553515][1785.495726701], took 2.07min 2025-12-04T10:08:57.5558281Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/inductor.test_kernel_benchmark/inductor.test_kernel_benchmark-a01600ecae601eef.xml 2025-12-04T10:08:57.6303671Z Running inductor/test_torchinductor_opinfo 4/14 ... [2025-12-04 10:08:57.630106][1785.572324582] 2025-12-04T10:08:57.6304312Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2025-12-04T10:08:57.6307129Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'inductor/test_torchinductor_opinfo.py', '--shard-id=4', '--num-shards=14', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-12-04 10:08:57.630430] 2025-12-04T10:18:09.7043885Z 2025-12-04T10:18:09.7045122Z inductor/test_torchinductor_opinfo 4/14 was successful, full logs can be found in artifacts with path test/test-reports/inductor.test_torchinductor_opinfo_4.14_d6c80c60df76d1ae_.log 2025-12-04T10:18:09.7183501Z Running 246 items in this shard: test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive___rmul___cuda_float16, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive___rxor___cuda_int32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive__batch_norm_with_update_cuda_float16, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive__segment_reduce_lengths_cuda_float32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive__softmax_backward_data_cuda_float16, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive__softmax_backward_data_cuda_float32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_abs_cuda_float16, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_acos_cuda_float64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_acosh_cuda_bool, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_addbmm_cuda_float32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_addcdiv_cuda_float32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_addmm_cuda_float16, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_amax_cuda_bool, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_amax_cuda_int64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_amax_cuda_uint8, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_aminmax_cuda_float16, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_angle_cuda_uint8, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_arange_cuda_uint8, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_argmin_cuda_uint8, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_as_strided_cuda_float16, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_as_strided_scatter_cuda_bool, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_atan_cuda_int32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_atleast_1d_cuda_bool, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_atleast_2d_cuda_float16, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_atleast_3d_cuda_uint8, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_block_diag_cuda_float64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_broadcast_tensors_cuda_float32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_broadcast_to_cuda_float16, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_cat_cuda_int64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_ceil_cuda_float16, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_ceil_cuda_float64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_ceil_cuda_int64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_char_cuda_uint8, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_chunk_cuda_uint8, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_clamp_cuda_float32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_combinations_cuda_float16, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_complex_cuda_float64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_conj_cuda_float16, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_contiguous_cuda_int64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_count_nonzero_cuda_int64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_cov_cuda_int32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_cummin_cuda_bool, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_cumsum_cuda_uint8, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_deg2rad_cuda_float64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_diagflat_cuda_float32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_diff_cuda_float32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_double_cuda_float32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_dsplit_cuda_float64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_empty_like_cuda_uint8, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_equal_cuda_int64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_equal_cuda_uint8, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_expand_copy_cuda_int64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_expand_cuda_bool, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_expm1_cuda_float32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_fft_fft2_cuda_float64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_fft_fftshift_cuda_int32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_fft_hfft_cuda_bool, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_fft_ifft_cuda_float64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_fft_ifftn_cuda_int32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_fft_ihfft2_cuda_bool, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_fft_ihfft_cuda_float32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_fft_ihfftn_cuda_float16, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_fft_irfft2_cuda_float64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_fft_rfftn_cuda_bool, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_fft_rfftn_cuda_int32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_flip_cuda_int64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_fmax_cuda_float16, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_full_cuda_int64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_full_like_cuda_float64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_gt_cuda_int32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_histc_cuda_int32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_hstack_cuda_uint8, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_i0_cuda_bool, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_igamma_cuda_float64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_index_add_cuda_int64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_index_reduce_amax_cuda_int32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_index_reduce_amin_cuda_uint8, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_index_reduce_prod_cuda_int32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_isinf_cuda_int64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_isnan_cuda_float32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_jiterator_4inputs_with_extra_args_cuda_int64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_jiterator_binary_return_by_ref_cuda_int64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_kron_cuda_int32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_kthvalue_cuda_float16, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_ldexp_cuda_uint8, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_le_cuda_float64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_lerp_cuda_float16, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_lgamma_cuda_float64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_linalg_cholesky_cuda_float32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_linalg_householder_product_cuda_float64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_linalg_matrix_norm_cuda_float32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_linalg_solve_ex_cuda_float64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_linalg_tensorinv_cuda_float64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_log2_cuda_float16, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_logaddexp_cuda_float16, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_logical_not_cuda_float16, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_logical_or_cuda_bool, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_logical_xor_cuda_float32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_logsumexp_cuda_int32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_logsumexp_cuda_uint8, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_long_cuda_int64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_lt_cuda_float16, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_lt_cuda_int32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_lt_cuda_int64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_mH_cuda_float16, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_masked_cumprod_cuda_float32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_masked_fill_cuda_int64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_masked_norm_cuda_float32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_masked_normalize_cuda_float64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_masked_scatter_cuda_bool, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_masked_var_cuda_float32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_masked_var_cuda_int64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_masked_var_cuda_uint8, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_matrix_exp_cuda_float32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_matrix_exp_cuda_float64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_max_reduction_with_dim_cuda_float64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_maximum_cuda_bool, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_maximum_cuda_float64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_mean_cuda_float16, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_meshgrid_variadic_tensors_cuda_int32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_min_reduction_with_dim_cuda_int32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_minimum_cuda_int32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_mode_cuda_bool, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_mul_cuda_uint8, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_multinomial_cuda_float64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_mvlgamma_mvlgamma_p_5_cuda_int64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_nan_to_num_cuda_float16, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_nanmedian_cuda_uint8, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_nansum_cuda_int32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_narrow_copy_cuda_uint8, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_narrow_cuda_bool, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_native_layer_norm_cuda_float32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_nn_functional_celu_cuda_float64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_nn_functional_conv_transpose2d_cuda_float16, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_nn_functional_cosine_similarity_cuda_float32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_nn_functional_embedding_bag_cuda_float16, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_nn_functional_embedding_bag_cuda_float64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_nn_functional_feature_alpha_dropout_without_train_cuda_float32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_nn_functional_feature_alpha_dropout_without_train_cuda_float64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_nn_functional_gaussian_nll_loss_cuda_float64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_nn_functional_group_norm_cuda_float32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_nn_functional_hardswish_cuda_float32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_nn_functional_huber_loss_cuda_float16, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_nn_functional_huber_loss_cuda_float32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_nn_functional_interpolate_nearest_cuda_float16, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_nn_functional_interpolate_trilinear_cuda_float32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_nn_functional_kl_div_cuda_float32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_nn_functional_logsigmoid_cuda_float32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_nn_functional_max_unpool2d_grad_cuda_float64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_nn_functional_max_unpool3d_cuda_float32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_nn_functional_max_unpool3d_grad_cuda_float64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_nn_functional_normalize_cuda_float32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_nn_functional_normalize_cuda_float64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_nn_functional_pad_circular_cuda_int32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_nn_functional_pad_constant_cuda_bool, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_nn_functional_pad_constant_cuda_int64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_nn_functional_pixel_unshuffle_cuda_bool, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_nn_functional_relu6_cuda_int64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_nn_functional_softshrink_cuda_float16, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_nn_functional_softsign_cuda_int32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_nn_functional_softsign_cuda_uint8, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_nn_functional_triplet_margin_loss_cuda_float64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_nn_functional_upsample_bilinear_cuda_float16, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_nn_functional_upsample_nearest_cuda_float64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_nonzero_cuda_bool, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_norm_fro_cuda_float16, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_norm_fro_cuda_float32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_normal_cuda_float32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_normal_in_place_cuda_float32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_normal_in_place_cuda_float64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_ones_like_cuda_float16, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_outer_cuda_float64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_pca_lowrank_cuda_float64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_permute_cuda_bool, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_polygamma_polygamma_n_2_cuda_bool, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_polygamma_polygamma_n_3_cuda_int32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_put_cuda_float16, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_put_cuda_float64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_randint_cuda_float32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_randint_like_cuda_uint8, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_randn_cuda_float32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_real_cuda_bool, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_resize__cuda_float64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_resolve_conj_cuda_float64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_resolve_conj_cuda_int64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_resolve_neg_cuda_bool, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_rot90_cuda_float64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_rsqrt_cuda_uint8, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_scatter_reduce_prod_cuda_float16, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_sgn_cuda_float16, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_sgn_cuda_float64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_short_cuda_float16, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_short_cuda_int64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_sigmoid_cuda_float64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_sign_cuda_bool, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_sign_cuda_float32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_signal_windows_exponential_cuda_float64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_signal_windows_general_hamming_cuda_float64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_sin_cuda_int32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_sparse_mm_reduce_cuda_float32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_sparse_sampled_addmm_cuda_float32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_special_airy_ai_cuda_float64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_special_entr_cuda_bool, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_special_erfcx_cuda_int32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_special_hermite_polynomial_h_cuda_int32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_special_hermite_polynomial_he_cuda_bool, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_special_i0e_cuda_int32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_special_laguerre_polynomial_l_cuda_int32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_special_legendre_polynomial_p_cuda_float32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_special_modified_bessel_i1_cuda_float32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_special_modified_bessel_i1_cuda_int32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_special_modified_bessel_k0_cuda_int64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_special_ndtr_cuda_float64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_special_scaled_modified_bessel_k1_cuda_int32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_special_shifted_chebyshev_polynomial_u_cuda_int64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_split_cuda_bool, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_split_list_args_cuda_int64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_split_with_sizes_copy_cuda_float16, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_split_with_sizes_copy_cuda_float32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_sqrt_cuda_int32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_sqrt_cuda_uint8, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_square_cuda_int32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_squeeze_copy_cuda_float64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_squeeze_multiple_cuda_int32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_stack_cuda_float64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_stack_cuda_int64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_std_unbiased_cuda_float32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_sum_to_size_cuda_float64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_sum_to_size_cuda_int32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_t_cuda_bool, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_tan_cuda_uint8, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_tile_cuda_int64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_to_sparse_cuda_float32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_trapezoid_cuda_int64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_trapz_cuda_uint8, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_true_divide_cuda_float64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_unbind_copy_cuda_float16, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_unflatten_cuda_bool, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_unflatten_cuda_uint8, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_unsafe_split_cuda_float64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_unsqueeze_copy_cuda_float16, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_unsqueeze_cuda_float32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_vdot_cuda_float32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_view_as_cuda_int32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_view_copy_cuda_int64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_zeros_like_cuda_bool 2025-12-04T10:18:09.7308100Z 2025-12-04T10:18:09.7308394Z Finished inductor/test_torchinductor_opinfo 4/14 ... [2025-12-04 10:18:09.704838][2337.647054417], took 9.20min 2025-12-04T10:18:09.7309296Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/inductor.test_torchinductor_opinfo/inductor.test_torchinductor_opinfo-d9c51466d8748c04.xml 2025-12-04T10:18:09.7785226Z Running inductor/test_torchinductor_opinfo 11/14 ... [2025-12-04 10:18:09.778275][2337.720493611] 2025-12-04T10:18:09.7785759Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2025-12-04T10:18:09.7788454Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'inductor/test_torchinductor_opinfo.py', '--shard-id=11', '--num-shards=14', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-12-04 10:18:09.778595] 2025-12-04T10:25:27.0020359Z 2025-12-04T10:25:27.0023778Z inductor/test_torchinductor_opinfo 11/14 was successful, full logs can be found in artifacts with path test/test-reports/inductor.test_torchinductor_opinfo_11.14_4593ceecb4e38359_.log 2025-12-04T10:25:27.0105489Z Running 239 items in this shard: test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_H_cuda_int64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive___getitem___cuda_float16, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive___getitem___cuda_float64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive___radd___cuda_float16, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive___rmod___cuda_int32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive___rmod___cuda_int64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive___rpow___cuda_float64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive___rsub___cuda_float32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive__chunk_cat_cuda_int32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive__native_batch_norm_legit_cuda_float32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive__unsafe_masked_index_put_accumulate_cuda_float16, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive__upsample_bilinear2d_aa_cuda_float16, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive__upsample_bilinear2d_aa_cuda_float64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_acosh_cuda_float64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_addbmm_cuda_float16, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_addmv_cuda_float16, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_alias_copy_cuda_int32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_amin_cuda_int64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_any_cuda_int32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_any_cuda_int64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_argsort_cuda_bool, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_argwhere_cuda_float64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_as_strided_scatter_cuda_uint8, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_atan2_cuda_uint8, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_atanh_cuda_float64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_atanh_cuda_int64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_baddbmm_cuda_float64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_bernoulli_cuda_float16, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_bitwise_and_cuda_bool, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_bitwise_and_cuda_uint8, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_block_diag_cuda_bool, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_byte_cuda_float64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_cholesky_cuda_float64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_clamp_cuda_int32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_clamp_max_cuda_float16, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_clamp_max_cuda_float32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_clamp_max_cuda_int64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_column_stack_cuda_float32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_combinations_cuda_float32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_corrcoef_cuda_int32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_cos_cuda_int32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_count_nonzero_cuda_float64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_cross_cuda_float16, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_cummax_cuda_int64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_cummin_cuda_float32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_cumprod_cuda_float32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_deg2rad_cuda_int32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_diagonal_copy_cuda_float16, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_diagonal_copy_cuda_float32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_diagonal_cuda_bool, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_diagonal_cuda_float16, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_digamma_cuda_int64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_digamma_cuda_uint8, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_div_no_rounding_mode_cuda_int64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_dot_cuda_float16, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_dsplit_cuda_int32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_empty_cuda_uint8, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_empty_permuted_cuda_float64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_erf_cuda_bool, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_erfc_cuda_bool, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_erfc_cuda_int32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_erfinv_cuda_int32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_expm1_cuda_int32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_fft_fft2_cuda_uint8, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_fft_ifft2_cuda_int64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_fft_ifft_cuda_bool, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_fft_ifft_cuda_float32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_fft_ihfftn_cuda_int32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_fft_ihfftn_cuda_int64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_fft_irfft_cuda_uint8, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_fft_irfftn_cuda_int64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_fft_rfft_cuda_uint8, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_flatten_cuda_float16, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_float_cuda_bool, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_float_power_cuda_int32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_fmax_cuda_int64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_frac_cuda_float16, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_gather_cuda_int64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_gather_cuda_uint8, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_gcd_cuda_uint8, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_ge_cuda_float64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_hash_tensor_cuda_float64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_histc_cuda_uint8, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_hsplit_cuda_float32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_hsplit_cuda_uint8, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_hstack_cuda_float32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_hypot_cuda_float16, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_index_reduce_mean_cuda_float32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_isinf_cuda_float64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_isreal_cuda_uint8, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_kron_cuda_float64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_ldexp_cuda_int32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_linalg_cross_cuda_float32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_linalg_eigh_cuda_float32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_linalg_ldl_factor_cuda_float32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_linalg_lu_factor_cuda_float32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_linalg_solve_ex_cuda_float32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_log10_cuda_int32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_log2_cuda_bool, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_log_cuda_float64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_log_normal_cuda_float64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_log_softmax_with_dtype_cuda_bool, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_logcumsumexp_cuda_float16, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_logdet_cuda_float64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_logical_xor_cuda_float64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_logspace_tensor_overload_cuda_int64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_long_cuda_float32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_masked_fill_cuda_bool, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_masked_fill_cuda_float64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_masked_norm_cuda_float16, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_masked_prod_cuda_uint8, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_masked_std_cuda_float16, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_masked_sum_cuda_int64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_matmul_cuda_float32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_max_binary_cuda_bool, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_max_binary_cuda_float32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_max_binary_cuda_int64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_max_reduction_no_dim_cuda_float32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_meshgrid_variadic_tensors_cuda_bool, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_min_reduction_no_dim_cuda_float64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_minimum_cuda_uint8, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_mm_cuda_float32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_movedim_cuda_float32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_msort_cuda_int32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_mvlgamma_mvlgamma_p_1_cuda_int64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_nan_to_num_cuda_float64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_nanmedian_cuda_int32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_narrow_cuda_uint8, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_native_batch_norm_cuda_float32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_neg_cuda_float32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_new_full_cuda_float16, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_nn_functional_batch_norm_cuda_float32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_nn_functional_binary_cross_entropy_with_logits_cuda_float64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_nn_functional_channel_shuffle_cuda_uint8, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_nn_functional_dropout_cuda_float32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_nn_functional_elu_cuda_float64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_nn_functional_embedding_cuda_float64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_nn_functional_fractional_max_pool3d_cuda_float64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_nn_functional_glu_cuda_float64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_nn_functional_interpolate_bicubic_cuda_float16, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_nn_functional_interpolate_bilinear_cuda_float32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_nn_functional_leaky_relu_cuda_float64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_nn_functional_logsigmoid_cuda_float16, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_nn_functional_max_pool1d_cuda_float16, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_nn_functional_max_pool2d_cuda_float32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_nn_functional_max_unpool3d_cuda_float64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_nn_functional_mse_loss_cuda_float16, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_nn_functional_mse_loss_cuda_float64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_nn_functional_multilabel_soft_margin_loss_cuda_float64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_nn_functional_pairwise_distance_cuda_float64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_nn_functional_pixel_unshuffle_cuda_float64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_nn_functional_pixel_unshuffle_cuda_uint8, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_nn_functional_relu_cuda_float64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_nn_functional_rms_norm_cuda_float64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_nn_functional_softmin_with_dtype_cuda_float32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_nn_functional_softmin_with_dtype_cuda_float64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_nn_functional_tanhshrink_cuda_float32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_nn_functional_tanhshrink_cuda_float64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_nonzero_static_cuda_bool, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_ones_cuda_float64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_outer_cuda_int32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_pca_lowrank_cuda_float32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_polygamma_polygamma_n_2_cuda_float64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_real_cuda_float16, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_remainder_cuda_float32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_repeat_interleave_cuda_uint8, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_reshape_as_cuda_uint8, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_resize_as__cuda_float32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_round_decimals_0_cuda_float32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_rsqrt_cuda_float16, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_scatter_add_cuda_bool, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_scatter_reduce_mean_cuda_int64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_scatter_reduce_mean_cuda_uint8, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_scatter_reduce_prod_cuda_float32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_scatter_reduce_sum_cuda_float16, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_searchsorted_cuda_int64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_select_scatter_cuda_int64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_sgn_cuda_float32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_signal_windows_bartlett_cuda_float32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_signal_windows_hamming_cuda_float64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_sinh_cuda_float16, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_slice_cuda_int32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_sparse_mm_reduce_cuda_float64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_special_bessel_j0_cuda_float64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_special_bessel_j1_cuda_float64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_special_chebyshev_polynomial_u_cuda_int64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_special_chebyshev_polynomial_v_cuda_float32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_special_erfcx_cuda_float64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_special_hermite_polynomial_h_cuda_bool, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_special_i1e_cuda_bool, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_special_log_ndtr_cuda_int64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_special_modified_bessel_i0_cuda_bool, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_special_modified_bessel_i0_cuda_int32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_special_ndtr_cuda_int32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_special_polygamma_special_polygamma_n_0_cuda_uint8, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_special_scaled_modified_bessel_k1_cuda_float64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_special_shifted_chebyshev_polynomial_t_cuda_uint8, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_special_shifted_chebyshev_polynomial_w_cuda_float64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_special_xlog1py_cuda_bool, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_special_zeta_cuda_float32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_special_zeta_cuda_int64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_split_cuda_int64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_split_list_args_cuda_float32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_split_with_sizes_copy_cuda_uint8, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_sqrt_cuda_int64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_square_cuda_int64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_squeeze_copy_cuda_float16, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_squeeze_multiple_cuda_uint8, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_stack_cuda_float16, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_stft_cuda_float32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_sub_cuda_float16, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_sub_cuda_float32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_sum_cuda_float32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_sum_to_size_cuda_uint8, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_svd_lowrank_cuda_float64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_t_copy_cuda_bool, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_take_along_dim_cuda_bool, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_take_cuda_bool, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_tan_cuda_int32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_tensor_split_cuda_float64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_topk_cuda_float16, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_torch_ops_aten__efficient_attention_forward_cuda_float32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_transpose_copy_cuda_bool, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_tril_cuda_float16, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_tril_cuda_uint8, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_unbind_copy_cuda_int32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_unbind_cuda_float32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_unbind_cuda_int64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_unfold_cuda_float64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_unfold_cuda_int32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_unique_cuda_float32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_unsqueeze_copy_cuda_bool, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_var_unbiased_cuda_float64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_view_as_cuda_bool, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_view_copy_cuda_float16, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_view_copy_cuda_uint8, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_view_cuda_float64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_xlogy_cuda_bool, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_zero__cuda_float16 2025-12-04T10:25:27.0184275Z 2025-12-04T10:25:27.0184549Z Finished inductor/test_torchinductor_opinfo 11/14 ... [2025-12-04 10:25:27.002269][2774.944483962], took 7.29min 2025-12-04T10:25:27.0185425Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/inductor.test_torchinductor_opinfo/inductor.test_torchinductor_opinfo-eb72b3d6271f1e7e.xml 2025-12-04T10:25:27.2392212Z Uploading artifacts took 0.16 seconds 2025-12-04T10:25:27.2395257Z Running dynamo/test_repros 1/1 ... [2025-12-04 10:25:27.239310][2775.181527632] 2025-12-04T10:25:27.2395707Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2025-12-04T10:25:27.2398867Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'dynamo/test_repros.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-12-04 10:25:27.239597] 2025-12-04T10:27:23.8389897Z 2025-12-04T10:27:23.8391405Z dynamo/test_repros 1/1 was successful, full logs can be found in artifacts with path test/test-reports/dynamo.test_repros_1.1_e85852541ac7eea0_.log 2025-12-04T10:27:23.8461708Z Running 351 items in this shard: test/dynamo/test_repros.py::LRUCacheWarningTests::test_lru_cache_warning_issued_during_tracing, test/dynamo/test_repros.py::ReproTests::test_312_local_cell_overlap, test/dynamo/test_repros.py::ReproTests::test_Size, test/dynamo/test_repros.py::ReproTests::test_abc_setattr, test/dynamo/test_repros.py::ReproTests::test_add_complex_conj, test/dynamo/test_repros.py::ReproTests::test_add_sub_alpha_out, test/dynamo/test_repros.py::ReproTests::test_addr_alpha_beta_out, test/dynamo/test_repros.py::ReproTests::test_amp_foreach_fake_impl, test/dynamo/test_repros.py::ReproTests::test_aot_autograd_runtime_wrapper_prologue_profiled, test/dynamo/test_repros.py::ReproTests::test_as_strided_on_base_with_mutation_works, test/dynamo/test_repros.py::ReproTests::test_as_strided_on_existing_view_banned, test/dynamo/test_repros.py::ReproTests::test_attached_attribute_in_dir, test/dynamo/test_repros.py::ReproTests::test_autograd_function_graph_break, test/dynamo/test_repros.py::ReproTests::test_avoid_dupe_specialization, test/dynamo/test_repros.py::ReproTests::test_batch_encoding_clone_inputs, test/dynamo/test_repros.py::ReproTests::test_batch_norm_act, test/dynamo/test_repros.py::ReproTests::test_batchnorm_e2e, test/dynamo/test_repros.py::ReproTests::test_bigbird_unsqueeze_inplace, test/dynamo/test_repros.py::ReproTests::test_bitwise_op_guard, test/dynamo/test_repros.py::ReproTests::test_bitwise_print_precedence, test/dynamo/test_repros.py::ReproTests::test_boxes_len, test/dynamo/test_repros.py::ReproTests::test_build_map_unpack_with_call, test/dynamo/test_repros.py::ReproTests::test_c_defined_metaclass, test/dynamo/test_repros.py::ReproTests::test_cells_unsupported_step_exception, test/dynamo/test_repros.py::ReproTests::test_changing_stride, test/dynamo/test_repros.py::ReproTests::test_chunk_reformer_ff, test/dynamo/test_repros.py::ReproTests::test_class_member, test/dynamo/test_repros.py::ReproTests::test_classmethod_with_slots, test/dynamo/test_repros.py::ReproTests::test_clone_not_memory_dense, test/dynamo/test_repros.py::ReproTests::test_compilation_metrics_on_error, test/dynamo/test_repros.py::ReproTests::test_compile_complex_conj, test/dynamo/test_repros.py::ReproTests::test_compile_copy__int_overload, test/dynamo/test_repros.py::ReproTests::test_compiled_module_truthiness, test/dynamo/test_repros.py::ReproTests::test_const_dict_keyerror, test/dynamo/test_repros.py::ReproTests::test_contains_range_constprop, test/dynamo/test_repros.py::ReproTests::test_convert_boxes_to_pooler_format, test/dynamo/test_repros.py::ReproTests::test_copy_weird_strides, test/dynamo/test_repros.py::ReproTests::test_create_rand_mask_from_inputs, test/dynamo/test_repros.py::ReproTests::test_dalle2_maybe, test/dynamo/test_repros.py::ReproTests::test_data_attr_mutation_after_saved_for_bw, test/dynamo/test_repros.py::ReproTests::test_dataclass_in_module, test/dynamo/test_repros.py::ReproTests::test_dataclass_init_with_default_factory_with_inputs, test/dynamo/test_repros.py::ReproTests::test_ddp_checkpoint, test/dynamo/test_repros.py::ReproTests::test_dedup_global, test/dynamo/test_repros.py::ReproTests::test_deferred_runtime_asserts, test/dynamo/test_repros.py::ReproTests::test_delattr, test/dynamo/test_repros.py::ReproTests::test_delattr_raises, test/dynamo/test_repros.py::ReproTests::test_delattr_return, test/dynamo/test_repros.py::ReproTests::test_delete_local_error, test/dynamo/test_repros.py::ReproTests::test_deleted_compile_wrapper_segfault, test/dynamo/test_repros.py::ReproTests::test_delsubscr, test/dynamo/test_repros.py::ReproTests::test_delsubscr_raises, test/dynamo/test_repros.py::ReproTests::test_detectron2_instances_cat, test/dynamo/test_repros.py::ReproTests::test_disabling_unpack_hooks_within_compiled_region, test/dynamo/test_repros.py::ReproTests::test_distributions_subclass, test/dynamo/test_repros.py::ReproTests::test_do_paste_mask, test/dynamo/test_repros.py::ReproTests::test_dont_aggressively_write_assert, test/dynamo/test_repros.py::ReproTests::test_dont_dce_rand, test/dynamo/test_repros.py::ReproTests::test_dropout_inline, test/dynamo/test_repros.py::ReproTests::test_dynamic_shape_disable_duck_size, test/dynamo/test_repros.py::ReproTests::test_dynamic_shapes_double_not_equal, test/dynamo/test_repros.py::ReproTests::test_dynamic_shapes_float_guard, test/dynamo/test_repros.py::ReproTests::test_dynamic_shapes_implicit_guard, test/dynamo/test_repros.py::ReproTests::test_dynamic_shapes_right_side, test/dynamo/test_repros.py::ReproTests::test_dynamo_default_lru_cache_behavior, test/dynamo/test_repros.py::ReproTests::test_dynamo_disable_lru_cache_behavior, test/dynamo/test_repros.py::ReproTests::test_dynamo_set_recursion_limit, test/dynamo/test_repros.py::ReproTests::test_dynamo_set_recursion_limit_usage, test/dynamo/test_repros.py::ReproTests::test_ellipsis, test/dynamo/test_repros.py::ReproTests::test_embedding_backward_broadcasting_decomp, test/dynamo/test_repros.py::ReproTests::test_empty_graph_nested_calls_fullgraph_False, test/dynamo/test_repros.py::ReproTests::test_empty_graph_nested_calls_fullgraph_True, test/dynamo/test_repros.py::ReproTests::test_empty_list_contains_with_jump, test/dynamo/test_repros.py::ReproTests::test_empty_out_dynamic, test/dynamo/test_repros.py::ReproTests::test_enum, test/dynamo/test_repros.py::ReproTests::test_ephemeral_module, test/dynamo/test_repros.py::ReproTests::test_error_return_without_exception_set, test/dynamo/test_repros.py::ReproTests::test_exception_in_dynamo_handling, test/dynamo/test_repros.py::ReproTests::test_exec_import, test/dynamo/test_repros.py::ReproTests::test_exec_wildcard_import, test/dynamo/test_repros.py::ReproTests::test_export_vs_dynamo_for_multiheadattention, test/dynamo/test_repros.py::ReproTests::test_flip_bad_accuracy, test/dynamo/test_repros.py::ReproTests::test_for_loop_graph_break, test/dynamo/test_repros.py::ReproTests::test_for_loop_graph_break_before, test/dynamo/test_repros.py::ReproTests::test_foreach_decomp_arg_names, test/dynamo/test_repros.py::ReproTests::test_fsdp_set_input_mutation_applied_when_input_gets_no_gradients, test/dynamo/test_repros.py::ReproTests::test_function_in_skipfiles, test/dynamo/test_repros.py::ReproTests::test_functools_wraps, test/dynamo/test_repros.py::ReproTests::test_gan_repro_trying_to_backward_through_the_graph_a_second_time, test/dynamo/test_repros.py::ReproTests::test_generator_dealloc, test/dynamo/test_repros.py::ReproTests::test_get_parameter_dtype, test/dynamo/test_repros.py::ReproTests::test_get_type_hints, test/dynamo/test_repros.py::ReproTests::test_global_fn_mutation, test/dynamo/test_repros.py::ReproTests::test_grad, test/dynamo/test_repros.py::ReproTests::test_grad_mode_carrying_correct_state_after_graph_break, test/dynamo/test_repros.py::ReproTests::test_grad_references_cleared, test/dynamo/test_repros.py::ReproTests::test_graph_break_on_jit_isinstance, test/dynamo/test_repros.py::ReproTests::test_graph_break_on_jit_isinstance_pep585, test/dynamo/test_repros.py::ReproTests::test_graph_break_unsupported_fake, test/dynamo/test_repros.py::ReproTests::test_guard_default_device, test/dynamo/test_repros.py::ReproTests::test_guard_fail_nested_tuple, test/dynamo/test_repros.py::ReproTests::test_guard_fail_tensor_bool, test/dynamo/test_repros.py::ReproTests::test_guard_ordering_shape_fail, test/dynamo/test_repros.py::ReproTests::test_guard_same_frame_fail_message, test/dynamo/test_repros.py::ReproTests::test_guard_with_tuple_mutation, test/dynamo/test_repros.py::ReproTests::test_hasattr_builtin, test/dynamo/test_repros.py::ReproTests::test_hf_bigbird_unsqueeze, test/dynamo/test_repros.py::ReproTests::test_hf_classinstantier, test/dynamo/test_repros.py::ReproTests::test_hf_gelu_inline, test/dynamo/test_repros.py::ReproTests::test_hf_model_output, test/dynamo/test_repros.py::ReproTests::test_hf_t5_forward, test/dynamo/test_repros.py::ReproTests::test_hf_xsoftmax_inference, test/dynamo/test_repros.py::ReproTests::test_hf_xsoftmax_training, test/dynamo/test_repros.py::ReproTests::test_iadd_graph_break, test/dynamo/test_repros.py::ReproTests::test_incompatible_configs, test/dynamo/test_repros.py::ReproTests::test_indexing_with_list, test/dynamo/test_repros.py::ReproTests::test_inductor_dynamic_shapes_broadcasting, test/dynamo/test_repros.py::ReproTests::test_inductor_no_recursionerror_on_for_loops, test/dynamo/test_repros.py::ReproTests::test_inductor_rng_default_dtype, test/dynamo/test_repros.py::ReproTests::test_inference_mode_dynamic_shapes, test/dynamo/test_repros.py::ReproTests::test_inlining_cornercase, test/dynamo/test_repros.py::ReproTests::test_inplace_unsqueeze_input, test/dynamo/test_repros.py::ReproTests::test_int_format, test/dynamo/test_repros.py::ReproTests::test_intermediate_leaf_requires_grad, test/dynamo/test_repros.py::ReproTests::test_invalid_seq_unpack, test/dynamo/test_repros.py::ReproTests::test_is_make_fx_tracing, test/dynamo/test_repros.py::ReproTests::test_is_symbolic_tracing, test/dynamo/test_repros.py::ReproTests::test_isinstance_dtype, test/dynamo/test_repros.py::ReproTests::test_isinstance_storage, test/dynamo/test_repros.py::ReproTests::test_issue111522, test/dynamo/test_repros.py::ReproTests::test_issue111918, test/dynamo/test_repros.py::ReproTests::test_issue114171, test/dynamo/test_repros.py::ReproTests::test_issue126128, test/dynamo/test_repros.py::ReproTests::test_issue134451, test/dynamo/test_repros.py::ReproTests::test_issue1466_size_aot_autograd, test/dynamo/test_repros.py::ReproTests::test_issue164247_backend_eager, test/dynamo/test_repros.py::ReproTests::test_issue164247_backend_inductor, test/dynamo/test_repros.py::ReproTests::test_issue175, test/dynamo/test_repros.py::ReproTests::test_jit_script_defaults, test/dynamo/test_repros.py::ReproTests::test_jit_trace_errors, test/dynamo/test_repros.py::ReproTests::test_kwargs_out_list_variable, test/dynamo/test_repros.py::ReproTests::test_list_aliasing, test/dynamo/test_repros.py::ReproTests::test_list_index, test/dynamo/test_repros.py::ReproTests::test_list_index_not_found, test/dynamo/test_repros.py::ReproTests::test_list_index_tensor_unsupported, test/dynamo/test_repros.py::ReproTests::test_list_reverse, test/dynamo/test_repros.py::ReproTests::test_list_self_reference, test/dynamo/test_repros.py::ReproTests::test_listcomp, test/dynamo/test_repros.py::ReproTests::test_longformer_chunk, test/dynamo/test_repros.py::ReproTests::test_longtensor_list, test/dynamo/test_repros.py::ReproTests::test_lru_cache_tracing, test/dynamo/test_repros.py::ReproTests::test_maml_item_capture, test/dynamo/test_repros.py::ReproTests::test_maml_no_item_capture, test/dynamo/test_repros.py::ReproTests::test_many_overlapping_inputs_does_not_explode_guards, test/dynamo/test_repros.py::ReproTests::test_many_views_with_mutation, test/dynamo/test_repros.py::ReproTests::test_map_with_multiple_args, test/dynamo/test_repros.py::ReproTests::test_maybe_multiply_symint, test/dynamo/test_repros.py::ReproTests::test_mem_leak_guards, test/dynamo/test_repros.py::ReproTests::test_merge_criteria_processor_list1, test/dynamo/test_repros.py::ReproTests::test_merge_criteria_processor_list2, test/dynamo/test_repros.py::ReproTests::test_method_overriding, test/dynamo/test_repros.py::ReproTests::test_module_in_skipfiles, test/dynamo/test_repros.py::ReproTests::test_modules, test/dynamo/test_repros.py::ReproTests::test_multi_dot_import, test/dynamo/test_repros.py::ReproTests::test_multi_import, test/dynamo/test_repros.py::ReproTests::test_named_buffers, test/dynamo/test_repros.py::ReproTests::test_nanmean_out, test/dynamo/test_repros.py::ReproTests::test_negative_floor_div_solve, test/dynamo/test_repros.py::ReproTests::test_negative_shape_guard, test/dynamo/test_repros.py::ReproTests::test_nested_while_loop_graph_break, test/dynamo/test_repros.py::ReproTests::test_nn_module_callable, test/dynamo/test_repros.py::ReproTests::test_nn_module_property_closure, test/dynamo/test_repros.py::ReproTests::test_nn_module_stack_bc, test/dynamo/test_repros.py::ReproTests::test_nn_param_freevar_codegen, test/dynamo/test_repros.py::ReproTests::test_nn_parameter, test/dynamo/test_repros.py::ReproTests::test_nn_parameter_ctor_graph_breaks, test/dynamo/test_repros.py::ReproTests::test_nn_parametrize, test/dynamo/test_repros.py::ReproTests::test_no_grad_inline, test/dynamo/test_repros.py::ReproTests::test_no_tracing_into_eval_frame, test/dynamo/test_repros.py::ReproTests::test_no_tracing_into_eval_frame_ctx_manager, test/dynamo/test_repros.py::ReproTests::test_nonconst_issubclass, test/dynamo/test_repros.py::ReproTests::test_not_rewrite_assert_for_other_errors, test/dynamo/test_repros.py::ReproTests::test_nullcontext1, test/dynamo/test_repros.py::ReproTests::test_nullcontext2, test/dynamo/test_repros.py::ReproTests::test_numpy_not_ndarray_recompiles, test/dynamo/test_repros.py::ReproTests::test_numpy_tobytes_no_error, test/dynamo/test_repros.py::ReproTests::test_odict_get_item_index_name, test/dynamo/test_repros.py::ReproTests::test_omegaconf_dictconfig, test/dynamo/test_repros.py::ReproTests::test_omegaconf_listconfig_contains, test/dynamo/test_repros.py::ReproTests::test_omegaconf_listconfig_iter, test/dynamo/test_repros.py::ReproTests::test_ones_out_dynamic, test/dynamo/test_repros.py::ReproTests::test_optim_state_references_cleared, test/dynamo/test_repros.py::ReproTests::test_optimized_deepcopy, test/dynamo/test_repros.py::ReproTests::test_optimized_module_patched_init, test/dynamo/test_repros.py::ReproTests::test_optimized_module_training, test/dynamo/test_repros.py::ReproTests::test_os_fspath, test/dynamo/test_repros.py::ReproTests::test_out_nested_cell_shape_change, test/dynamo/test_repros.py::ReproTests::test_out_nested_cell_tuple_shape_change, test/dynamo/test_repros.py::ReproTests::test_out_none, test/dynamo/test_repros.py::ReproTests::test_out_overload_non_contiguous, test/dynamo/test_repros.py::ReproTests::test_out_root_cell_shape_change, test/dynamo/test_repros.py::ReproTests::test_out_root_cell_tuple_shape_change, test/dynamo/test_repros.py::ReproTests::test_output_aliases_intermediate, test/dynamo/test_repros.py::ReproTests::test_overlapping_inputs_with_dynamic_shapes_error, test/dynamo/test_repros.py::ReproTests::test_overwriting_params, test/dynamo/test_repros.py::ReproTests::test_partially_initialized_module_property, test/dynamo/test_repros.py::ReproTests::test_partitioner_activation_memory_budget_with_unbacked_symints, test/dynamo/test_repros.py::ReproTests::test_partitioner_cse_respects_mutation_boundaries, test/dynamo/test_repros.py::ReproTests::test_pointless_graph_removal, test/dynamo/test_repros.py::ReproTests::test_preserve_stride_with_clone, test/dynamo/test_repros.py::ReproTests::test_primtorch, test/dynamo/test_repros.py::ReproTests::test_primtorch_no_graph_break, test/dynamo/test_repros.py::ReproTests::test_randint_out_dynamic, test/dynamo/test_repros.py::ReproTests::test_recursive_map, test/dynamo/test_repros.py::ReproTests::test_reformer_eval, test/dynamo/test_repros.py::ReproTests::test_reformer_min_chunk_len, test/dynamo/test_repros.py::ReproTests::test_reformer_sorting, test/dynamo/test_repros.py::ReproTests::test_reformer_train, test/dynamo/test_repros.py::ReproTests::test_reinplacing, test/dynamo/test_repros.py::ReproTests::test_relative_import, test/dynamo/test_repros.py::ReproTests::test_relative_import_no_modulename, test/dynamo/test_repros.py::ReproTests::test_requires_grad_guards_with_grad_mode1, test/dynamo/test_repros.py::ReproTests::test_requires_grad_guards_with_grad_mode2, test/dynamo/test_repros.py::ReproTests::test_restricted_list_subclass1, test/dynamo/test_repros.py::ReproTests::test_restricted_list_subclass2, test/dynamo/test_repros.py::ReproTests::test_restricted_list_subclass3, test/dynamo/test_repros.py::ReproTests::test_return_value_duplication_mixed_grad, test/dynamo/test_repros.py::ReproTests::test_return_value_duplication_scalar, test/dynamo/test_repros.py::ReproTests::test_return_value_duplication_tensor, test/dynamo/test_repros.py::ReproTests::test_return_weakref, test/dynamo/test_repros.py::ReproTests::test_rewrite_assert_dont_change_bytecode, test/dynamo/test_repros.py::ReproTests::test_rewrite_assert_noop, test/dynamo/test_repros.py::ReproTests::test_rewrite_assert_with_msg, test/dynamo/test_repros.py::ReproTests::test_rewrite_assert_with_non_string_msg, test/dynamo/test_repros.py::ReproTests::test_rewrite_assert_without_msg, test/dynamo/test_repros.py::ReproTests::test_rng_state, test/dynamo/test_repros.py::ReproTests::test_seq_append_list, test/dynamo/test_repros.py::ReproTests::test_setattr_requires_grad_graph_breaks, test/dynamo/test_repros.py::ReproTests::test_setitem_boolean_mask_diff, test/dynamo/test_repros.py::ReproTests::test_setitem_tensor_prop, test/dynamo/test_repros.py::ReproTests::test_setitem_tuple_boolean_mask_diff, test/dynamo/test_repros.py::ReproTests::test_sigmoid_out, test/dynamo/test_repros.py::ReproTests::test_sigmoid_out2, test/dynamo/test_repros.py::ReproTests::test_size_typematch, test/dynamo/test_repros.py::ReproTests::test_slice_into_list_mutable, test/dynamo/test_repros.py::ReproTests::test_slicing_dynamic_shape, test/dynamo/test_repros.py::ReproTests::test_slicing_dynamic_shape_setitem, test/dynamo/test_repros.py::ReproTests::test_sort_out, test/dynamo/test_repros.py::ReproTests::test_sort_out2, test/dynamo/test_repros.py::ReproTests::test_specialized_stride, test/dynamo/test_repros.py::ReproTests::test_split_with_sizes_aot_autograd, test/dynamo/test_repros.py::ReproTests::test_staticmethod_allow_in_graph, test/dynamo/test_repros.py::ReproTests::test_stk_sdd_is_transposed, test/dynamo/test_repros.py::ReproTests::test_stop_iteration_reconstruct, test/dynamo/test_repros.py::ReproTests::test_str_isalnum, test/dynamo/test_repros.py::ReproTests::test_string_format, test/dynamo/test_repros.py::ReproTests::test_subclass_graph_output_repro, test/dynamo/test_repros.py::ReproTests::test_super_classmethod, test/dynamo/test_repros.py::ReproTests::test_super_classmethod_inheritance, test/dynamo/test_repros.py::ReproTests::test_super_diamond, test/dynamo/test_repros.py::ReproTests::test_super_in_staticmethod, test/dynamo/test_repros.py::ReproTests::test_super_staticmethod, test/dynamo/test_repros.py::ReproTests::test_swin_base_tensor_attr, test/dynamo/test_repros.py::ReproTests::test_symint_bitwise, test/dynamo/test_repros.py::ReproTests::test_symnode_is_not_op, test/dynamo/test_repros.py::ReproTests::test_symnode_is_op, test/dynamo/test_repros.py::ReproTests::test_sys_monitoring, test/dynamo/test_repros.py::ReproTests::test_tensor_data_kwarg, test/dynamo/test_repros.py::ReproTests::test_tensor_isinstance_tuple, test/dynamo/test_repros.py::ReproTests::test_tensor_item, test/dynamo/test_repros.py::ReproTests::test_tensor_random, test/dynamo/test_repros.py::ReproTests::test_tensor_set_data_backend_aot_eager_func_name_func1, test/dynamo/test_repros.py::ReproTests::test_tensor_set_data_backend_aot_eager_func_name_func2, test/dynamo/test_repros.py::ReproTests::test_tensor_set_data_backend_aot_eager_func_name_func3, test/dynamo/test_repros.py::ReproTests::test_tensor_set_data_backend_eager_func_name_func1, test/dynamo/test_repros.py::ReproTests::test_tensor_set_data_backend_eager_func_name_func2, test/dynamo/test_repros.py::ReproTests::test_tensor_set_data_backend_eager_func_name_func3, test/dynamo/test_repros.py::ReproTests::test_tensor_set_data_backend_inductor_func_name_func1, test/dynamo/test_repros.py::ReproTests::test_tensor_set_data_backend_inductor_func_name_func2, test/dynamo/test_repros.py::ReproTests::test_tensor_set_data_backend_inductor_func_name_func3, test/dynamo/test_repros.py::ReproTests::test_tensor_set_data_mismatched_dtype, test/dynamo/test_repros.py::ReproTests::test_tensor_split, test/dynamo/test_repros.py::ReproTests::test_tensor_split_within_device_cm, test/dynamo/test_repros.py::ReproTests::test_tensor_uniform, test/dynamo/test_repros.py::ReproTests::test_threading_local, test/dynamo/test_repros.py::ReproTests::test_tokenization, test/dynamo/test_repros.py::ReproTests::test_torch_compile_in_compile_frame, test/dynamo/test_repros.py::ReproTests::test_torch_ops_aten, test/dynamo/test_repros.py::ReproTests::test_torch_tensor_ops, test/dynamo/test_repros.py::ReproTests::test_torch_tensor_ops_no_graph_break, test/dynamo/test_repros.py::ReproTests::test_torch_variable_type, test/dynamo/test_repros.py::ReproTests::test_torchname, test/dynamo/test_repros.py::ReproTests::test_trace_functional_tensor_with, test/dynamo/test_repros.py::ReproTests::test_tuple_enum_as_key_dict, test/dynamo/test_repros.py::ReproTests::test_typed_dict, test/dynamo/test_repros.py::ReproTests::test_typed_dict_total, test/dynamo/test_repros.py::ReproTests::test_udf_classes_reconstruction, test/dynamo/test_repros.py::ReproTests::test_unbacked_arange_in_bounds, test/dynamo/test_repros.py::ReproTests::test_unbind_copy_out, test/dynamo/test_repros.py::ReproTests::test_unpack_hooks_can_be_disabled, test/dynamo/test_repros.py::ReproTests::test_unpack_hooks_dont_run_during_tracing, test/dynamo/test_repros.py::ReproTests::test_unspecialized_nn_module_with_torch_variable_attribute, test/dynamo/test_repros.py::ReproTests::test_unsqueeze_mul_strides, test/dynamo/test_repros.py::ReproTests::test_user_ctor_ctx_manager, test/dynamo/test_repros.py::ReproTests::test_user_ctor_ctx_manager_custom_init, test/dynamo/test_repros.py::ReproTests::test_user_ctor_ctx_manager_custom_init_graph_break, test/dynamo/test_repros.py::ReproTests::test_user_defined_iter, test/dynamo/test_repros.py::ReproTests::test_user_defined_object_callable, test/dynamo/test_repros.py::ReproTests::test_validate_model_kwargs, test/dynamo/test_repros.py::ReproTests::test_vc_bumped_in_inference_graph, test/dynamo/test_repros.py::ReproTests::test_vdd_duplicate_error, test/dynamo/test_repros.py::ReproTests::test_view_dtype_overload, test/dynamo/test_repros.py::ReproTests::test_weakref, test/dynamo/test_repros.py::ReproTests::test_weakref_callback, test/dynamo/test_repros.py::ReproTests::test_weakref_construction, test/dynamo/test_repros.py::ReproTests::test_weakref_del, test/dynamo/test_repros.py::ReproTests::test_weakref_proxy, test/dynamo/test_repros.py::ReproTests::test_weakref_reconstruct, test/dynamo/test_repros.py::ReproTests::test_while_loop_graph_break, test/dynamo/test_repros.py::ReproTests::test_while_loop_graph_break_inside_call_function, test/dynamo/test_repros.py::ReproTests::test_with_on_graph_break_inst, test/dynamo/test_repros.py::ReproTests::test_with_on_graph_break_nested, test/dynamo/test_repros.py::ReproTests::test_zeros_out_dynamic, test/dynamo/test_repros.py::ReproTestsDeviceCUDA::test_cuda_sync_cuda, test/dynamo/test_repros.py::ReproTestsDeviceCUDA::test_current_accelerator_cuda, test/dynamo/test_repros.py::ReproTestsDeviceCUDA::test_data_dependent_error_log_no_print_cuda, test/dynamo/test_repros.py::ReproTestsDeviceCUDA::test_deepcopy_constant_tensor_in_aot_bwd_cuda, test/dynamo/test_repros.py::ReproTestsDeviceCUDA::test_filter_safe_grad_warning_cuda, test/dynamo/test_repros.py::ReproTestsDeviceCUDA::test_filter_user_warnings_cuda, test/dynamo/test_repros.py::ReproTestsDeviceCUDA::test_filter_warnings_cuda, test/dynamo/test_repros.py::ReproTestsDeviceCUDA::test_flash_attn_backward_mixed_strides_cuda, test/dynamo/test_repros.py::ReproTestsDeviceCUDA::test_getattr_return_cuda, test/dynamo/test_repros.py::ReproTestsDeviceCUDA::test_guard_default_device_cuda, test/dynamo/test_repros.py::ReproTestsDeviceCUDA::test_megablocks_moe_cuda, test/dynamo/test_repros.py::ReproTestsDeviceCUDA::test_memleak_when_graph_input_has_tensor_attr_cuda, test/dynamo/test_repros.py::ReproTestsDeviceCUDA::test_module_attribute_error_cuda, test/dynamo/test_repros.py::ReproTestsDeviceCUDA::test_named_tuple_vt_clone_cuda, test/dynamo/test_repros.py::ReproTestsDeviceCUDA::test_norm_dtype_cuda, test/dynamo/test_repros.py::ReproTestsDeviceCUDA::test_partial_export_cuda, test/dynamo/test_repros.py::ReproTestsDeviceCUDA::test_partitioner_saves_weights_for_bw_cuda, test/dynamo/test_repros.py::ReproTestsDeviceCUDA::test_pytree_get_node_type_not_traced_cuda, test/dynamo/test_repros.py::ReproTestsDeviceCUDA::test_pytree_get_node_type_with_namedtuple_cuda, test/dynamo/test_repros.py::ReproTestsDeviceCUDA::test_pytree_tree_is_leaf_not_traced_cuda, test/dynamo/test_repros.py::ReproTestsDeviceCUDA::test_pytree_tree_is_leaf_with_namedtuple_cuda, test/dynamo/test_repros.py::ReproTestsDeviceCUDA::test_sdpa_dynamic_shapes_cuda, test/dynamo/test_repros.py::ReproTestsDeviceCUDA::test_sub_alpha_scalar_repro_cuda, test/dynamo/test_repros.py::ReproTestsDeviceCUDA::test_tensor_size_hasattr_cuda, test/dynamo/test_repros.py::ReproTestsDeviceCUDA::test_torch_cuda_is_initialized_cuda, test/dynamo/test_repros.py::ReproTestsDeviceCUDA::test_truthiness_of_symints_no_recompiles_cuda, test/dynamo/test_repros.py::ReproTestsDeviceCUDA::test_udf_class_source_cuda, test/dynamo/test_repros.py::ReproTestsDeviceCUDA::test_zero_dim_param_mixed_device_grad_cuda 2025-12-04T10:27:23.8529289Z 2025-12-04T10:27:23.8529503Z Finished dynamo/test_repros 1/1 ... [2025-12-04 10:27:23.839317][2891.781532759], took 1.94min 2025-12-04T10:27:23.8530227Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/dynamo.test_repros/dynamo.test_repros-df80b1b40a6198cc.xml 2025-12-04T10:27:23.9345561Z Running inductor/test_cuda_select_algorithm 1/1 ... [2025-12-04 10:27:23.934271][2891.876489256] 2025-12-04T10:27:23.9346076Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2025-12-04T10:27:23.9348905Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'inductor/test_cuda_select_algorithm.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-12-04 10:27:23.934610] 2025-12-04T11:13:49.9001470Z 2025-12-04T11:13:49.9002488Z PRINTING LOG FILE of inductor/test_cuda_select_algorithm 1/1 (test/test-reports/inductor.test_cuda_select_algorithm_1.1_4871a8a3689f9e63_.log) 2025-12-04T11:13:49.9004541Z W1204 10:27:29.010000 56470 site-packages/torch/_inductor/utils.py:1703] Not enough SMs to use max_autotune_gemm mode 2025-12-04T11:13:49.9005854Z Test results will be stored in test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-9c66153466707d74.xml 2025-12-04T11:13:49.9006748Z ============================= test session starts ============================== 2025-12-04T11:13:49.9007442Z platform linux -- Python 3.10.14, pytest-7.3.2, pluggy-1.6.0 -- /opt/conda/envs/py_3.10/bin/python 2025-12-04T11:13:49.9008003Z cachedir: .pytest_cache 2025-12-04T11:13:49.9008664Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow] 2025-12-04T11:13:49.9009510Z rootdir: /var/lib/jenkins/workspace 2025-12-04T11:13:49.9009892Z configfile: pytest.ini 2025-12-04T11:13:49.9010446Z plugins: hypothesis-6.56.4, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-14.0, subtests-0.13.1, xdist-3.3.1, xdoctest-1.3.0, typeguard-4.3.0 2025-12-04T11:13:49.9011179Z collecting ... collected 58 items 2025-12-04T11:13:49.9011617Z stepcurrent: Cannot find last run test, not skipping 2025-12-04T11:13:49.9055367Z Running 58 items in this shard: test/inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_concat_cuda_batch_size_1_mid_dim_1_in_features_128_out_features_64_cuda_bfloat16, test/inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_concat_cuda_batch_size_1_mid_dim_8_in_features_128_out_features_64_cuda_bfloat16, test/inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_concat_cuda_batch_size_32_mid_dim_1_in_features_128_out_features_64_cuda_bfloat16, test/inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_concat_cuda_batch_size_32_mid_dim_8_in_features_128_out_features_64_cuda_bfloat16, test/inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_17_mid_dim_1_in_features_1024_out_features_1024_cuda_bfloat16, test/inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_17_mid_dim_1_in_features_1024_out_features_64_cuda_bfloat16, test/inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_17_mid_dim_1_in_features_1024_out_features_65_cuda_bfloat16, test/inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_17_mid_dim_1_in_features_128_out_features_1024_cuda_bfloat16, test/inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_17_mid_dim_1_in_features_128_out_features_64_cuda_bfloat16, test/inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_17_mid_dim_1_in_features_128_out_features_65_cuda_bfloat16, test/inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_17_mid_dim_1_in_features_144_out_features_1024_cuda_bfloat16, test/inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_17_mid_dim_1_in_features_144_out_features_64_cuda_bfloat16, test/inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_17_mid_dim_1_in_features_144_out_features_65_cuda_bfloat16, test/inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_17_mid_dim_8_in_features_1024_out_features_1024_cuda_bfloat16, test/inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_17_mid_dim_8_in_features_1024_out_features_64_cuda_bfloat16, test/inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_17_mid_dim_8_in_features_1024_out_features_65_cuda_bfloat16, test/inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_17_mid_dim_8_in_features_128_out_features_1024_cuda_bfloat16, test/inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_17_mid_dim_8_in_features_128_out_features_64_cuda_bfloat16, test/inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_17_mid_dim_8_in_features_128_out_features_65_cuda_bfloat16, test/inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_17_mid_dim_8_in_features_144_out_features_1024_cuda_bfloat16, test/inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_17_mid_dim_8_in_features_144_out_features_64_cuda_bfloat16, test/inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_17_mid_dim_8_in_features_144_out_features_65_cuda_bfloat16, test/inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_1_mid_dim_1_in_features_1024_out_features_1024_cuda_bfloat16, test/inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_1_mid_dim_1_in_features_1024_out_features_64_cuda_bfloat16, test/inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_1_mid_dim_1_in_features_1024_out_features_65_cuda_bfloat16, test/inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_1_mid_dim_1_in_features_128_out_features_1024_cuda_bfloat16, test/inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_1_mid_dim_1_in_features_128_out_features_64_cuda_bfloat16, test/inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_1_mid_dim_1_in_features_128_out_features_65_cuda_bfloat16, test/inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_1_mid_dim_1_in_features_144_out_features_1024_cuda_bfloat16, test/inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_1_mid_dim_1_in_features_144_out_features_64_cuda_bfloat16, test/inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_1_mid_dim_1_in_features_144_out_features_65_cuda_bfloat16, test/inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_1_mid_dim_8_in_features_1024_out_features_1024_cuda_bfloat16, test/inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_1_mid_dim_8_in_features_1024_out_features_64_cuda_bfloat16, test/inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_1_mid_dim_8_in_features_1024_out_features_65_cuda_bfloat16, test/inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_1_mid_dim_8_in_features_128_out_features_1024_cuda_bfloat16, test/inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_1_mid_dim_8_in_features_128_out_features_64_cuda_bfloat16, test/inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_1_mid_dim_8_in_features_128_out_features_65_cuda_bfloat16, test/inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_1_mid_dim_8_in_features_144_out_features_1024_cuda_bfloat16, test/inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_1_mid_dim_8_in_features_144_out_features_64_cuda_bfloat16, test/inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_1_mid_dim_8_in_features_144_out_features_65_cuda_bfloat16, test/inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_32_mid_dim_1_in_features_1024_out_features_1024_cuda_bfloat16, test/inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_32_mid_dim_1_in_features_1024_out_features_64_cuda_bfloat16, test/inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_32_mid_dim_1_in_features_1024_out_features_65_cuda_bfloat16, test/inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_32_mid_dim_1_in_features_128_out_features_1024_cuda_bfloat16, test/inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_32_mid_dim_1_in_features_128_out_features_64_cuda_bfloat16, test/inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_32_mid_dim_1_in_features_128_out_features_65_cuda_bfloat16, test/inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_32_mid_dim_1_in_features_144_out_features_1024_cuda_bfloat16, test/inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_32_mid_dim_1_in_features_144_out_features_64_cuda_bfloat16, test/inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_32_mid_dim_1_in_features_144_out_features_65_cuda_bfloat16, test/inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_32_mid_dim_8_in_features_1024_out_features_1024_cuda_bfloat16, test/inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_32_mid_dim_8_in_features_1024_out_features_64_cuda_bfloat16, test/inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_32_mid_dim_8_in_features_1024_out_features_65_cuda_bfloat16, test/inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_32_mid_dim_8_in_features_128_out_features_1024_cuda_bfloat16, test/inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_32_mid_dim_8_in_features_128_out_features_64_cuda_bfloat16, test/inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_32_mid_dim_8_in_features_128_out_features_65_cuda_bfloat16, test/inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_32_mid_dim_8_in_features_144_out_features_1024_cuda_bfloat16, test/inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_32_mid_dim_8_in_features_144_out_features_64_cuda_bfloat16, test/inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_32_mid_dim_8_in_features_144_out_features_65_cuda_bfloat16 2025-12-04T11:13:49.9118405Z 2025-12-04T11:13:49.9119379Z inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_concat_cuda_batch_size_1_mid_dim_1_in_features_128_out_features_64_cuda_bfloat16 ('RERUN', {'yellow': True}) [2.0441s] [ 1%] 2025-12-04T11:13:49.9121598Z inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_concat_cuda_batch_size_1_mid_dim_1_in_features_128_out_features_64_cuda_bfloat16 ('RERUN', {'yellow': True}) [0.6146s] [ 1%] 2025-12-04T11:13:49.9123480Z inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_concat_cuda_batch_size_1_mid_dim_1_in_features_128_out_features_64_cuda_bfloat16 FAILED [0.6243s] [ 1%] 2025-12-04T11:13:49.9124446Z 2025-12-04T11:13:49.9124613Z ==================================== RERUNS ==================================== 2025-12-04T11:13:49.9125489Z _ TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_concat_cuda_batch_size_1_mid_dim_1_in_features_128_out_features_64_cuda_bfloat16 _ 2025-12-04T11:13:49.9126303Z Traceback (most recent call last): 2025-12-04T11:13:49.9127109Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:49.9128149Z method(*args, **kwargs) 2025-12-04T11:13:49.9129134Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:49.9129912Z method(*args, **kwargs) 2025-12-04T11:13:49.9130670Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T11:13:49.9131414Z with policy(): 2025-12-04T11:13:49.9132154Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T11:13:49.9132967Z raise RuntimeError(msg) 2025-12-04T11:13:49.9134594Z RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_concat_cuda_batch_size_1_mid_dim_1_in_features_128_out_features_64_cuda_bfloat16! Caching allocator allocated memory was 0 and is now reported as 24576 on device 0. CUDA driver allocated memory was 230686720 and is now 274726912. 2025-12-04T11:13:49.9136143Z 2025-12-04T11:13:49.9136383Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:49.9137614Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_concat_cuda_batch_size_1_mid_dim_1_in_features_128_out_features_64_cuda_bfloat16 2025-12-04T11:13:49.9138693Z 2025-12-04T11:13:49.9138982Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:49.9139567Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:49.9140091Z stats [('calls_captured', 18), ('unique_graphs', 1)] 2025-12-04T11:13:49.9141007Z inductor [('pattern_matcher_nodes', 18), ('woq_matcher_nodes', 12), ('pattern_matcher_count', 9), ('woq_matcher_count', 3), ('extern_calls', 3), ('fxgraph_cache_miss', 1)] 2025-12-04T11:13:49.9142015Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:49.9142471Z graph_break [] 2025-12-04T11:13:49.9143071Z _ TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_concat_cuda_batch_size_1_mid_dim_1_in_features_128_out_features_64_cuda_bfloat16 _ 2025-12-04T11:13:49.9144022Z Traceback (most recent call last): 2025-12-04T11:13:49.9144850Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:49.9145607Z method(*args, **kwargs) 2025-12-04T11:13:49.9146264Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:49.9146934Z method(*args, **kwargs) 2025-12-04T11:13:49.9147610Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T11:13:49.9148369Z with policy(): 2025-12-04T11:13:49.9149037Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T11:13:49.9149801Z raise RuntimeError(msg) 2025-12-04T11:13:49.9151416Z RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_concat_cuda_batch_size_1_mid_dim_1_in_features_128_out_features_64_cuda_bfloat16! Caching allocator allocated memory was 24576 and is now reported as 49152 on device 0. CUDA driver allocated memory was 274726912 and is now 276824064. 2025-12-04T11:13:49.9152933Z 2025-12-04T11:13:49.9153161Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:49.9154426Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_concat_cuda_batch_size_1_mid_dim_1_in_features_128_out_features_64_cuda_bfloat16 2025-12-04T11:13:49.9155480Z 2025-12-04T11:13:49.9155752Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:49.9156530Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:49.9157154Z stats [('calls_captured', 18), ('unique_graphs', 1)] 2025-12-04T11:13:49.9158066Z inductor [('pattern_matcher_nodes', 18), ('woq_matcher_nodes', 12), ('pattern_matcher_count', 9), ('woq_matcher_count', 3), ('extern_calls', 3), ('fxgraph_cache_miss', 1)] 2025-12-04T11:13:49.9159023Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:49.9159478Z graph_break [] 2025-12-04T11:13:49.9159849Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:49.9160370Z stats [('calls_captured', 18), ('unique_graphs', 1)] 2025-12-04T11:13:49.9160858Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:49.9161792Z inductor [('pattern_matcher_nodes', 18), ('woq_matcher_nodes', 12), ('pattern_matcher_count', 9), ('woq_matcher_count', 3), ('extern_calls', 3), ('fxgraph_cache_miss', 1)] 2025-12-04T11:13:49.9162643Z graph_break [] 2025-12-04T11:13:49.9162940Z =================================== FAILURES =================================== 2025-12-04T11:13:49.9163744Z _ TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_concat_cuda_batch_size_1_mid_dim_1_in_features_128_out_features_64_cuda_bfloat16 _ 2025-12-04T11:13:49.9164524Z Traceback (most recent call last): 2025-12-04T11:13:49.9165268Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:49.9166029Z method(*args, **kwargs) 2025-12-04T11:13:49.9166764Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:49.9167504Z method(*args, **kwargs) 2025-12-04T11:13:49.9168198Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T11:13:49.9168939Z with policy(): 2025-12-04T11:13:49.9169621Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T11:13:49.9170368Z raise RuntimeError(msg) 2025-12-04T11:13:49.9171982Z RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_concat_cuda_batch_size_1_mid_dim_1_in_features_128_out_features_64_cuda_bfloat16! Caching allocator allocated memory was 49152 and is now reported as 73728 on device 0. CUDA driver allocated memory was 276824064 and is now 278921216. 2025-12-04T11:13:49.9173512Z 2025-12-04T11:13:49.9173732Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:49.9175013Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_concat_cuda_batch_size_1_mid_dim_1_in_features_128_out_features_64_cuda_bfloat16 2025-12-04T11:13:49.9176057Z 2025-12-04T11:13:49.9176330Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:49.9176958Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:49.9177488Z stats [('calls_captured', 18), ('unique_graphs', 1)] 2025-12-04T11:13:49.9178626Z inductor [('pattern_matcher_nodes', 18), ('woq_matcher_nodes', 12), ('pattern_matcher_count', 9), ('woq_matcher_count', 3), ('extern_calls', 3), ('fxgraph_cache_miss', 1)] 2025-12-04T11:13:49.9179562Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:49.9180008Z graph_break [] 2025-12-04T11:13:49.9180377Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:49.9180892Z stats [('calls_captured', 18), ('unique_graphs', 1)] 2025-12-04T11:13:49.9181393Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:49.9182336Z inductor [('pattern_matcher_nodes', 18), ('woq_matcher_nodes', 12), ('pattern_matcher_count', 9), ('woq_matcher_count', 3), ('extern_calls', 3), ('fxgraph_cache_miss', 1)] 2025-12-04T11:13:49.9183373Z graph_break [] 2025-12-04T11:13:49.9183953Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:49.9184530Z stats [('calls_captured', 18), ('unique_graphs', 1)] 2025-12-04T11:13:49.9185026Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:49.9185975Z inductor [('pattern_matcher_nodes', 18), ('woq_matcher_nodes', 12), ('pattern_matcher_count', 9), ('woq_matcher_count', 3), ('extern_calls', 3), ('fxgraph_cache_miss', 1)] 2025-12-04T11:13:49.9186667Z graph_break [] 2025-12-04T11:13:49.9187635Z - generated xml file: /var/lib/jenkins/workspace/test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-9c66153466707d74.xml - 2025-12-04T11:13:49.9188813Z =========================== short test summary info ============================ 2025-12-04T11:13:49.9191479Z FAILED [0.6243s] inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_concat_cuda_batch_size_1_mid_dim_1_in_features_128_out_features_64_cuda_bfloat16 - RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_concat_cuda_batch_size_1_mid_dim_1_in_features_128_out_features_64_cuda_bfloat16! Caching allocator allocated memory was 49152 and is now reported as 73728 on device 0. CUDA driver allocated memory was 276824064 and is now 278921216. 2025-12-04T11:13:49.9193893Z 2025-12-04T11:13:49.9194143Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:49.9195307Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_concat_cuda_batch_size_1_mid_dim_1_in_features_128_out_features_64_cuda_bfloat16 2025-12-04T11:13:49.9196330Z 2025-12-04T11:13:49.9196519Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:49.9197085Z !!!!!!!!!!!!!!!!!!!!!!!!!! stopping after 1 failures !!!!!!!!!!!!!!!!!!!!!!!!!!! 2025-12-04T11:13:49.9197611Z ========================== 1 failed, 2 rerun in 3.31s ========================== 2025-12-04T11:13:49.9198031Z Got exit code 1 2025-12-04T11:13:49.9198315Z Retrying single test... 2025-12-04T11:13:49.9198899Z W1204 10:27:39.040000 56652 site-packages/torch/_inductor/utils.py:1703] Not enough SMs to use max_autotune_gemm mode 2025-12-04T11:13:49.9200151Z Test results will be stored in test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-11449d70dc195ff4.xml 2025-12-04T11:13:49.9201166Z ============================= test session starts ============================== 2025-12-04T11:13:49.9201818Z platform linux -- Python 3.10.14, pytest-7.3.2, pluggy-1.6.0 -- /opt/conda/envs/py_3.10/bin/python 2025-12-04T11:13:49.9202416Z cachedir: .pytest_cache 2025-12-04T11:13:49.9203103Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow] 2025-12-04T11:13:49.9203915Z rootdir: /var/lib/jenkins/workspace 2025-12-04T11:13:49.9204296Z configfile: pytest.ini 2025-12-04T11:13:49.9205064Z plugins: hypothesis-6.56.4, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-14.0, subtests-0.13.1, xdist-3.3.1, xdoctest-1.3.0, typeguard-4.3.0 2025-12-04T11:13:49.9205873Z collecting ... collected 58 items / 57 deselected / 1 selected 2025-12-04T11:13:49.9207032Z stepcurrent: skipping 0 already run items. Running only test/inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_concat_cuda_batch_size_1_mid_dim_1_in_features_128_out_features_64_cuda_bfloat16 2025-12-04T11:13:49.9207974Z Running 1 items in this shard 2025-12-04T11:13:49.9208212Z 2025-12-04T11:13:49.9209357Z inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_concat_cuda_batch_size_1_mid_dim_1_in_features_128_out_features_64_cuda_bfloat16 [W1204 10:27:40.182867441 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:49.9210999Z 2025-12-04T11:13:49.9211660Z [W1204 10:27:49.357522254 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:49.9212351Z 2025-12-04T11:13:49.9212885Z [W1204 10:27:49.357766340 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:49.9213557Z 2025-12-04T11:13:49.9214078Z [W1204 10:27:49.358327422 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:49.9214707Z 2025-12-04T11:13:49.9215201Z [W1204 10:27:49.358537426 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:49.9215848Z 2025-12-04T11:13:49.9216332Z [W1204 10:27:49.359634031 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:49.9216966Z 2025-12-04T11:13:49.9217474Z [W1204 10:27:49.359794044 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:49.9218134Z 2025-12-04T11:13:49.9218644Z [W1204 10:27:49.360134612 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:49.9219283Z 2025-12-04T11:13:49.9219773Z [W1204 10:27:49.360313346 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:49.9220425Z 2025-12-04T11:13:49.9220943Z [W1204 10:27:49.368463086 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:49.9221588Z 2025-12-04T11:13:49.9222101Z [W1204 10:27:49.368671770 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:49.9222768Z 2025-12-04T11:13:49.9223213Z [W1204 10:27:49.368843844 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:49.9223920Z 2025-12-04T11:13:49.9224405Z [W1204 10:27:49.369078150 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:49.9225059Z 2025-12-04T11:13:49.9225595Z [W1204 10:27:49.369220133 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:49.9226130Z 2025-12-04T11:13:49.9226522Z [W1204 10:27:49.369454908 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:49.9226919Z 2025-12-04T11:13:49.9227457Z [W1204 10:27:49.369596561 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:49.9228173Z 2025-12-04T11:13:49.9228726Z [W1204 10:27:49.369826346 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:49.9229327Z 2025-12-04T11:13:49.9229852Z [W1204 10:27:49.369975090 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:49.9230409Z 2025-12-04T11:13:49.9230854Z [W1204 10:27:49.456252538 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:49.9231518Z 2025-12-04T11:13:49.9232023Z [W1204 10:27:49.456465253 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:49.9232671Z 2025-12-04T11:13:49.9233189Z [W1204 10:27:49.456627416 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:49.9233822Z 2025-12-04T11:13:49.9234541Z [W1204 10:27:49.456839351 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:49.9235410Z 2025-12-04T11:13:49.9235982Z [W1204 10:27:49.456968044 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:49.9236671Z 2025-12-04T11:13:49.9237209Z [W1204 10:27:49.457184569 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:49.9237893Z 2025-12-04T11:13:49.9238435Z [W1204 10:27:49.457315881 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:49.9239110Z 2025-12-04T11:13:49.9239622Z [W1204 10:27:49.457521846 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:49.9240275Z 2025-12-04T11:13:49.9240743Z [W1204 10:27:49.457644599 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:49.9241432Z 2025-12-04T11:13:49.9241590Z ('RERUN', {'yellow': True}) [11.2049s] [100%] 2025-12-04T11:13:49.9242934Z inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_concat_cuda_batch_size_1_mid_dim_1_in_features_128_out_features_64_cuda_bfloat16 [W1204 10:27:50.697326034 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:49.9244250Z 2025-12-04T11:13:49.9244797Z [W1204 10:27:50.697588180 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:49.9245445Z 2025-12-04T11:13:49.9245943Z [W1204 10:27:50.697746693 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:49.9246583Z 2025-12-04T11:13:49.9247078Z [W1204 10:27:50.697962338 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:49.9247722Z 2025-12-04T11:13:49.9248236Z [W1204 10:27:50.698090011 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:49.9248873Z 2025-12-04T11:13:49.9249382Z [W1204 10:27:50.698313466 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:49.9250026Z 2025-12-04T11:13:49.9250543Z [W1204 10:27:50.698438089 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:49.9251179Z 2025-12-04T11:13:49.9251683Z [W1204 10:27:50.698642723 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:49.9252302Z 2025-12-04T11:13:49.9252808Z [W1204 10:27:50.698764696 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:49.9253460Z 2025-12-04T11:13:49.9253954Z [W1204 10:27:50.704971893 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:49.9254580Z 2025-12-04T11:13:49.9255082Z [W1204 10:27:50.705142217 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:49.9255715Z 2025-12-04T11:13:49.9256208Z [W1204 10:27:50.705292630 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:49.9256836Z 2025-12-04T11:13:49.9257339Z [W1204 10:27:50.705496304 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:49.9257961Z 2025-12-04T11:13:49.9258470Z [W1204 10:27:50.705620837 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:49.9259195Z 2025-12-04T11:13:49.9259569Z [W1204 10:27:50.705835462 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:49.9259951Z 2025-12-04T11:13:49.9260241Z [W1204 10:27:50.705959875 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:49.9260615Z 2025-12-04T11:13:49.9261027Z [W1204 10:27:50.706166390 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:49.9261593Z 2025-12-04T11:13:49.9262097Z [W1204 10:27:50.706288012 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:49.9262722Z 2025-12-04T11:13:49.9263218Z [W1204 10:27:50.788825679 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:49.9263936Z 2025-12-04T11:13:49.9264444Z [W1204 10:27:50.789049744 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:49.9265069Z 2025-12-04T11:13:49.9265561Z [W1204 10:27:50.789200007 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:49.9266198Z 2025-12-04T11:13:49.9266688Z [W1204 10:27:50.789405312 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:49.9267310Z 2025-12-04T11:13:49.9267806Z [W1204 10:27:50.789532625 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:49.9268419Z 2025-12-04T11:13:49.9268911Z [W1204 10:27:50.789744999 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:49.9269551Z 2025-12-04T11:13:49.9270063Z [W1204 10:27:50.789869852 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:49.9270716Z 2025-12-04T11:13:49.9271211Z [W1204 10:27:50.790095947 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:49.9271841Z 2025-12-04T11:13:49.9272337Z [W1204 10:27:50.790226820 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:49.9272966Z 2025-12-04T11:13:49.9273102Z ('RERUN', {'yellow': True}) [0.5675s] [100%] 2025-12-04T11:13:49.9274646Z inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_concat_cuda_batch_size_1_mid_dim_1_in_features_128_out_features_64_cuda_bfloat16 [W1204 10:27:51.259081754 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:49.9276028Z 2025-12-04T11:13:49.9276538Z [W1204 10:27:51.259293759 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:49.9277161Z 2025-12-04T11:13:49.9277654Z [W1204 10:27:51.259447432 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:49.9278487Z 2025-12-04T11:13:49.9278990Z [W1204 10:27:51.259659257 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:49.9279610Z 2025-12-04T11:13:49.9280112Z [W1204 10:27:51.259785430 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:49.9280587Z 2025-12-04T11:13:49.9280884Z [W1204 10:27:51.260022235 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:49.9281728Z 2025-12-04T11:13:49.9282389Z [W1204 10:27:51.260154738 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:49.9283020Z 2025-12-04T11:13:49.9283523Z [W1204 10:27:51.260369183 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:49.9284153Z 2025-12-04T11:13:49.9284651Z [W1204 10:27:51.260493395 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:49.9285278Z 2025-12-04T11:13:49.9285771Z [W1204 10:27:51.266475117 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:49.9286396Z 2025-12-04T11:13:49.9286899Z [W1204 10:27:51.266641730 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:49.9287535Z 2025-12-04T11:13:49.9288058Z [W1204 10:27:51.266787374 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:49.9288694Z 2025-12-04T11:13:49.9289190Z [W1204 10:27:51.266990238 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:49.9289834Z 2025-12-04T11:13:49.9290202Z [W1204 10:27:51.267114581 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:49.9290810Z 2025-12-04T11:13:49.9291268Z [W1204 10:27:51.267332855 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:49.9291714Z 2025-12-04T11:13:49.9292005Z [W1204 10:27:51.267461449 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:49.9292429Z 2025-12-04T11:13:49.9292946Z [W1204 10:27:51.267665203 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:49.9293584Z 2025-12-04T11:13:49.9294075Z [W1204 10:27:51.267788466 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:49.9294690Z 2025-12-04T11:13:49.9295178Z [W1204 10:27:51.349950844 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:49.9295801Z 2025-12-04T11:13:49.9296302Z [W1204 10:27:51.350146118 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:49.9296933Z 2025-12-04T11:13:49.9297428Z [W1204 10:27:51.350299472 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:49.9298070Z 2025-12-04T11:13:49.9298564Z [W1204 10:27:51.350505306 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:49.9299201Z 2025-12-04T11:13:49.9299691Z [W1204 10:27:51.350626989 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:49.9300315Z 2025-12-04T11:13:49.9300815Z [W1204 10:27:51.350840933 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:49.9301429Z 2025-12-04T11:13:49.9301919Z [W1204 10:27:51.350967046 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:49.9302550Z 2025-12-04T11:13:49.9303039Z [W1204 10:27:51.351168991 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:49.9303769Z 2025-12-04T11:13:49.9304262Z [W1204 10:27:51.351289874 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:49.9305010Z 2025-12-04T11:13:49.9305214Z FAILED [0.5568s] [100%] 2025-12-04T11:13:49.9305394Z 2025-12-04T11:13:49.9305544Z ==================================== RERUNS ==================================== 2025-12-04T11:13:49.9306399Z _ TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_concat_cuda_batch_size_1_mid_dim_1_in_features_128_out_features_64_cuda_bfloat16 _ 2025-12-04T11:13:49.9307235Z Traceback (most recent call last): 2025-12-04T11:13:49.9308000Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:49.9308782Z method(*args, **kwargs) 2025-12-04T11:13:49.9309474Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:49.9310200Z method(*args, **kwargs) 2025-12-04T11:13:49.9310891Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T11:13:49.9311592Z with policy(): 2025-12-04T11:13:49.9312307Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T11:13:49.9313089Z raise RuntimeError(msg) 2025-12-04T11:13:49.9314488Z RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_concat_cuda_batch_size_1_mid_dim_1_in_features_128_out_features_64_cuda_bfloat16! Caching allocator allocated memory was 0 and is now reported as 24576 on device 0. CUDA driver allocated memory was 230686720 and is now 274726912. 2025-12-04T11:13:49.9315650Z 2025-12-04T11:13:49.9315799Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:49.9316662Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_concat_cuda_batch_size_1_mid_dim_1_in_features_128_out_features_64_cuda_bfloat16 2025-12-04T11:13:49.9317636Z 2025-12-04T11:13:49.9317938Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:49.9318341Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:49.9318666Z stats [('calls_captured', 18), ('unique_graphs', 1)] 2025-12-04T11:13:49.9319549Z inductor [('pattern_matcher_nodes', 18), ('woq_matcher_nodes', 12), ('pattern_matcher_count', 9), ('woq_matcher_count', 3), ('extern_calls', 3), ('fxgraph_cache_miss', 1)] 2025-12-04T11:13:49.9320207Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:49.9320528Z graph_break [] 2025-12-04T11:13:49.9320894Z ----------------------------- Captured stderr call ----------------------------- 2025-12-04T11:13:49.9322416Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/constant_folding.py:256: UserWarning: Unsupported unwinding pattern: Address not in range (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/profiler/unwind/unwind.cpp:219.) 2025-12-04T11:13:49.9323871Z if out == self.unknown_value: 2025-12-04T11:13:49.9324652Z _ TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_concat_cuda_batch_size_1_mid_dim_1_in_features_128_out_features_64_cuda_bfloat16 _ 2025-12-04T11:13:49.9325447Z Traceback (most recent call last): 2025-12-04T11:13:49.9326194Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:49.9326943Z method(*args, **kwargs) 2025-12-04T11:13:49.9327636Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:49.9328319Z method(*args, **kwargs) 2025-12-04T11:13:49.9329025Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T11:13:49.9329748Z with policy(): 2025-12-04T11:13:49.9330575Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T11:13:49.9331414Z raise RuntimeError(msg) 2025-12-04T11:13:49.9332996Z RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_concat_cuda_batch_size_1_mid_dim_1_in_features_128_out_features_64_cuda_bfloat16! Caching allocator allocated memory was 24576 and is now reported as 49152 on device 0. CUDA driver allocated memory was 274726912 and is now 276824064. 2025-12-04T11:13:49.9334547Z 2025-12-04T11:13:49.9334788Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:49.9336054Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_concat_cuda_batch_size_1_mid_dim_1_in_features_128_out_features_64_cuda_bfloat16 2025-12-04T11:13:49.9337106Z 2025-12-04T11:13:49.9337369Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:49.9338000Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:49.9338508Z stats [('calls_captured', 18), ('unique_graphs', 1)] 2025-12-04T11:13:49.9339413Z inductor [('pattern_matcher_nodes', 18), ('woq_matcher_nodes', 12), ('pattern_matcher_count', 9), ('woq_matcher_count', 3), ('extern_calls', 3), ('fxgraph_cache_miss', 1)] 2025-12-04T11:13:49.9340378Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:49.9340838Z graph_break [] 2025-12-04T11:13:49.9341212Z ----------------------------- Captured stderr call ----------------------------- 2025-12-04T11:13:49.9342213Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/constant_folding.py:256: UserWarning: Unsupported unwinding pattern: Address not in range (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/profiler/unwind/unwind.cpp:219.) 2025-12-04T11:13:49.9343054Z if out == self.unknown_value: 2025-12-04T11:13:49.9343323Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:49.9343739Z stats [('calls_captured', 18), ('unique_graphs', 1)] 2025-12-04T11:13:49.9344042Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:49.9344603Z inductor [('pattern_matcher_nodes', 18), ('woq_matcher_nodes', 12), ('pattern_matcher_count', 9), ('woq_matcher_count', 3), ('extern_calls', 3), ('fxgraph_cache_miss', 1)] 2025-12-04T11:13:49.9345097Z graph_break [] 2025-12-04T11:13:49.9345285Z =================================== FAILURES =================================== 2025-12-04T11:13:49.9345769Z _ TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_concat_cuda_batch_size_1_mid_dim_1_in_features_128_out_features_64_cuda_bfloat16 _ 2025-12-04T11:13:49.9346235Z Traceback (most recent call last): 2025-12-04T11:13:49.9346685Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:49.9347135Z method(*args, **kwargs) 2025-12-04T11:13:49.9347558Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:49.9347998Z method(*args, **kwargs) 2025-12-04T11:13:49.9348433Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T11:13:49.9348879Z with policy(): 2025-12-04T11:13:49.9349283Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T11:13:49.9349729Z raise RuntimeError(msg) 2025-12-04T11:13:49.9350680Z RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_concat_cuda_batch_size_1_mid_dim_1_in_features_128_out_features_64_cuda_bfloat16! Caching allocator allocated memory was 49152 and is now reported as 73728 on device 0. CUDA driver allocated memory was 276824064 and is now 278921216. 2025-12-04T11:13:49.9351696Z 2025-12-04T11:13:49.9351831Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:49.9352656Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_concat_cuda_batch_size_1_mid_dim_1_in_features_128_out_features_64_cuda_bfloat16 2025-12-04T11:13:49.9353278Z 2025-12-04T11:13:49.9353447Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:49.9353822Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:49.9354133Z stats [('calls_captured', 18), ('unique_graphs', 1)] 2025-12-04T11:13:49.9354670Z inductor [('pattern_matcher_nodes', 18), ('woq_matcher_nodes', 12), ('pattern_matcher_count', 9), ('woq_matcher_count', 3), ('extern_calls', 3), ('fxgraph_cache_miss', 1)] 2025-12-04T11:13:49.9355239Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:49.9355519Z graph_break [] 2025-12-04T11:13:49.9355737Z ----------------------------- Captured stderr call ----------------------------- 2025-12-04T11:13:49.9356669Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/constant_folding.py:256: UserWarning: Unsupported unwinding pattern: Address not in range (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/profiler/unwind/unwind.cpp:219.) 2025-12-04T11:13:49.9357519Z if out == self.unknown_value: 2025-12-04T11:13:49.9357783Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:49.9358091Z stats [('calls_captured', 18), ('unique_graphs', 1)] 2025-12-04T11:13:49.9358396Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:49.9358958Z inductor [('pattern_matcher_nodes', 18), ('woq_matcher_nodes', 12), ('pattern_matcher_count', 9), ('woq_matcher_count', 3), ('extern_calls', 3), ('fxgraph_cache_miss', 1)] 2025-12-04T11:13:49.9359455Z graph_break [] 2025-12-04T11:13:49.9359687Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:49.9360001Z stats [('calls_captured', 18), ('unique_graphs', 1)] 2025-12-04T11:13:49.9360304Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:49.9360857Z inductor [('pattern_matcher_nodes', 18), ('woq_matcher_nodes', 12), ('pattern_matcher_count', 9), ('woq_matcher_count', 3), ('extern_calls', 3), ('fxgraph_cache_miss', 1)] 2025-12-04T11:13:49.9361346Z graph_break [] 2025-12-04T11:13:49.9361945Z - generated xml file: /var/lib/jenkins/workspace/test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-11449d70dc195ff4.xml - 2025-12-04T11:13:49.9362618Z =========================== short test summary info ============================ 2025-12-04T11:13:49.9364165Z FAILED [0.5568s] inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_concat_cuda_batch_size_1_mid_dim_1_in_features_128_out_features_64_cuda_bfloat16 - RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_concat_cuda_batch_size_1_mid_dim_1_in_features_128_out_features_64_cuda_bfloat16! Caching allocator allocated memory was 49152 and is now reported as 73728 on device 0. CUDA driver allocated memory was 276824064 and is now 278921216. 2025-12-04T11:13:49.9365555Z 2025-12-04T11:13:49.9365689Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:49.9366445Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_concat_cuda_batch_size_1_mid_dim_1_in_features_128_out_features_64_cuda_bfloat16 2025-12-04T11:13:49.9367063Z 2025-12-04T11:13:49.9367226Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:49.9367582Z !!!!!!!!!!!!!!!!!!!!!!!!!! stopping after 1 failures !!!!!!!!!!!!!!!!!!!!!!!!!!! 2025-12-04T11:13:49.9367984Z ================== 1 failed, 57 deselected, 2 rerun in 12.36s ================== 2025-12-04T11:13:49.9368253Z Got exit code 1 2025-12-04T11:13:49.9368492Z Retrying single test... 2025-12-04T11:13:49.9368882Z W1204 10:27:58.197000 56839 site-packages/torch/_inductor/utils.py:1703] Not enough SMs to use max_autotune_gemm mode 2025-12-04T11:13:49.9369631Z Test results will be stored in test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-6c093515a384d9fc.xml 2025-12-04T11:13:49.9370203Z ============================= test session starts ============================== 2025-12-04T11:13:49.9370611Z platform linux -- Python 3.10.14, pytest-7.3.2, pluggy-1.6.0 -- /opt/conda/envs/py_3.10/bin/python 2025-12-04T11:13:49.9370972Z cachedir: .pytest_cache 2025-12-04T11:13:49.9371402Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow] 2025-12-04T11:13:49.9371872Z rootdir: /var/lib/jenkins/workspace 2025-12-04T11:13:49.9372085Z configfile: pytest.ini 2025-12-04T11:13:49.9372528Z plugins: hypothesis-6.56.4, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-14.0, subtests-0.13.1, xdist-3.3.1, xdoctest-1.3.0, typeguard-4.3.0 2025-12-04T11:13:49.9373069Z collecting ... collected 58 items / 57 deselected / 1 selected 2025-12-04T11:13:49.9373868Z stepcurrent: skipping 0 already run items. Running only test/inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_concat_cuda_batch_size_1_mid_dim_1_in_features_128_out_features_64_cuda_bfloat16 2025-12-04T11:13:49.9374595Z Running 1 items in this shard 2025-12-04T11:13:49.9374733Z 2025-12-04T11:13:49.9375487Z inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_concat_cuda_batch_size_1_mid_dim_1_in_features_128_out_features_64_cuda_bfloat16 [W1204 10:27:59.331992495 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:49.9376316Z 2025-12-04T11:13:49.9376623Z [W1204 10:28:08.309895424 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:49.9377015Z 2025-12-04T11:13:49.9377308Z [W1204 10:28:08.310222151 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:49.9377683Z 2025-12-04T11:13:49.9378311Z [W1204 10:28:08.310847234 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:49.9378706Z 2025-12-04T11:13:49.9379009Z [W1204 10:28:08.311037439 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:49.9379379Z 2025-12-04T11:13:49.9379674Z [W1204 10:28:08.312203344 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:49.9380048Z 2025-12-04T11:13:49.9380346Z [W1204 10:28:08.312369168 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:49.9380721Z 2025-12-04T11:13:49.9381016Z [W1204 10:28:08.312674115 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:49.9381385Z 2025-12-04T11:13:49.9381680Z [W1204 10:28:08.312846678 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:49.9382050Z 2025-12-04T11:13:49.9382344Z [W1204 10:28:08.321077970 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:49.9382711Z 2025-12-04T11:13:49.9383002Z [W1204 10:28:08.321273564 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:49.9383373Z 2025-12-04T11:13:49.9383742Z [W1204 10:28:08.321445078 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:49.9384260Z 2025-12-04T11:13:49.9384665Z [W1204 10:28:08.321684813 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:49.9385038Z 2025-12-04T11:13:49.9385334Z [W1204 10:28:08.321844956 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:49.9385704Z 2025-12-04T11:13:49.9385998Z [W1204 10:28:08.322117393 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:49.9386365Z 2025-12-04T11:13:49.9386658Z [W1204 10:28:08.322259146 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:49.9387031Z 2025-12-04T11:13:49.9387323Z [W1204 10:28:08.322486571 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:49.9387706Z 2025-12-04T11:13:49.9388000Z [W1204 10:28:08.322629894 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:49.9388383Z 2025-12-04T11:13:49.9388681Z [W1204 10:28:08.408701914 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:49.9389052Z 2025-12-04T11:13:49.9389357Z [W1204 10:28:08.408915538 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:49.9389728Z 2025-12-04T11:13:49.9390017Z [W1204 10:28:08.409065122 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:49.9390391Z 2025-12-04T11:13:49.9390680Z [W1204 10:28:08.409273366 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:49.9391058Z 2025-12-04T11:13:49.9391352Z [W1204 10:28:08.409395299 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:49.9391733Z 2025-12-04T11:13:49.9392032Z [W1204 10:28:08.409608384 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:49.9392404Z 2025-12-04T11:13:49.9392700Z [W1204 10:28:08.409733207 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:49.9393066Z 2025-12-04T11:13:49.9393357Z [W1204 10:28:08.409940461 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:49.9393733Z 2025-12-04T11:13:49.9394030Z [W1204 10:28:08.410084604 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:49.9394412Z 2025-12-04T11:13:49.9394500Z ('RERUN', {'yellow': True}) [11.0067s] [100%] 2025-12-04T11:13:49.9395430Z inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_concat_cuda_batch_size_1_mid_dim_1_in_features_128_out_features_64_cuda_bfloat16 [W1204 10:28:09.654518397 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:49.9396248Z 2025-12-04T11:13:49.9396731Z [W1204 10:28:09.654779582 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:49.9397122Z 2025-12-04T11:13:49.9397415Z [W1204 10:28:09.654931296 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:49.9397793Z 2025-12-04T11:13:49.9398095Z [W1204 10:28:09.655148971 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:49.9398574Z 2025-12-04T11:13:49.9398941Z [W1204 10:28:09.655276113 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:49.9399323Z 2025-12-04T11:13:49.9399621Z [W1204 10:28:09.655491828 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:49.9399996Z 2025-12-04T11:13:49.9400296Z [W1204 10:28:09.655617711 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:49.9400666Z 2025-12-04T11:13:49.9400966Z [W1204 10:28:09.655822946 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:49.9401339Z 2025-12-04T11:13:49.9401630Z [W1204 10:28:09.655945768 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:49.9402015Z 2025-12-04T11:13:49.9402310Z [W1204 10:28:09.662009692 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:49.9402690Z 2025-12-04T11:13:49.9402981Z [W1204 10:28:09.662180146 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:49.9403354Z 2025-12-04T11:13:49.9403655Z [W1204 10:28:09.662329939 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:49.9404027Z 2025-12-04T11:13:49.9404325Z [W1204 10:28:09.662535494 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:49.9404698Z 2025-12-04T11:13:49.9404990Z [W1204 10:28:09.662667317 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:49.9405372Z 2025-12-04T11:13:49.9405670Z [W1204 10:28:09.662886272 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:49.9406047Z 2025-12-04T11:13:49.9406339Z [W1204 10:28:09.663010094 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:49.9406711Z 2025-12-04T11:13:49.9407008Z [W1204 10:28:09.663217389 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:49.9407378Z 2025-12-04T11:13:49.9407676Z [W1204 10:28:09.663338682 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:49.9408050Z 2025-12-04T11:13:49.9408339Z [W1204 10:28:09.744944503 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:49.9408719Z 2025-12-04T11:13:49.9409012Z [W1204 10:28:09.745173638 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:49.9409403Z 2025-12-04T11:13:49.9409696Z [W1204 10:28:09.745323941 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:49.9410069Z 2025-12-04T11:13:49.9410389Z [W1204 10:28:09.745538066 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:49.9410761Z 2025-12-04T11:13:49.9411058Z [W1204 10:28:09.745662439 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:49.9411434Z 2025-12-04T11:13:49.9411728Z [W1204 10:28:09.745874603 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:49.9412109Z 2025-12-04T11:13:49.9412401Z [W1204 10:28:09.745997086 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:49.9412863Z 2025-12-04T11:13:49.9413222Z [W1204 10:28:09.746198710 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:49.9413595Z 2025-12-04T11:13:49.9413894Z [W1204 10:28:09.746319323 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:49.9414266Z 2025-12-04T11:13:49.9414358Z ('RERUN', {'yellow': True}) [0.5685s] [100%] 2025-12-04T11:13:49.9415262Z inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_concat_cuda_batch_size_1_mid_dim_1_in_features_128_out_features_64_cuda_bfloat16 [W1204 10:28:10.215903966 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:49.9416086Z 2025-12-04T11:13:49.9416382Z [W1204 10:28:10.216115730 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:49.9416768Z 2025-12-04T11:13:49.9417065Z [W1204 10:28:10.216266864 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:49.9417447Z 2025-12-04T11:13:49.9417739Z [W1204 10:28:10.216478848 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:49.9418111Z 2025-12-04T11:13:49.9418409Z [W1204 10:28:10.216614001 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:49.9418781Z 2025-12-04T11:13:49.9419077Z [W1204 10:28:10.216831346 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:49.9419454Z 2025-12-04T11:13:49.9419748Z [W1204 10:28:10.216956019 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:49.9420129Z 2025-12-04T11:13:49.9420427Z [W1204 10:28:10.217157673 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:49.9420812Z 2025-12-04T11:13:49.9421105Z [W1204 10:28:10.217277966 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:49.9421480Z 2025-12-04T11:13:49.9421779Z [W1204 10:28:10.223289769 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:49.9422155Z 2025-12-04T11:13:49.9422462Z [W1204 10:28:10.223462443 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:49.9422835Z 2025-12-04T11:13:49.9423126Z [W1204 10:28:10.223611296 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:49.9423606Z 2025-12-04T11:13:49.9423906Z [W1204 10:28:10.223811961 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:49.9424290Z 2025-12-04T11:13:49.9424584Z [W1204 10:28:10.223939483 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:49.9424957Z 2025-12-04T11:13:49.9425256Z [W1204 10:28:10.224168649 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:49.9425631Z 2025-12-04T11:13:49.9425929Z [W1204 10:28:10.224294001 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:49.9426302Z 2025-12-04T11:13:49.9426594Z [W1204 10:28:10.224495956 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:49.9427056Z 2025-12-04T11:13:49.9427415Z [W1204 10:28:10.224629669 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:49.9427799Z 2025-12-04T11:13:49.9428091Z [W1204 10:28:10.305874702 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:49.9428467Z 2025-12-04T11:13:49.9428770Z [W1204 10:28:10.306058906 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:49.9429142Z 2025-12-04T11:13:49.9429440Z [W1204 10:28:10.306209719 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:49.9429813Z 2025-12-04T11:13:49.9430107Z [W1204 10:28:10.306413084 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:49.9430493Z 2025-12-04T11:13:49.9430789Z [W1204 10:28:10.306542647 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:49.9431168Z 2025-12-04T11:13:49.9431465Z [W1204 10:28:10.306756641 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:49.9431837Z 2025-12-04T11:13:49.9432133Z [W1204 10:28:10.306882854 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:49.9432507Z 2025-12-04T11:13:49.9432805Z [W1204 10:28:10.307087268 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:49.9433182Z 2025-12-04T11:13:49.9433476Z [W1204 10:28:10.307210171 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:49.9433859Z 2025-12-04T11:13:49.9433925Z FAILED [0.5602s] [100%] 2025-12-04T11:13:49.9434040Z 2025-12-04T11:13:49.9434138Z ==================================== RERUNS ==================================== 2025-12-04T11:13:49.9434632Z _ TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_concat_cuda_batch_size_1_mid_dim_1_in_features_128_out_features_64_cuda_bfloat16 _ 2025-12-04T11:13:49.9435098Z Traceback (most recent call last): 2025-12-04T11:13:49.9435563Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:49.9436028Z method(*args, **kwargs) 2025-12-04T11:13:49.9436459Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:49.9436896Z method(*args, **kwargs) 2025-12-04T11:13:49.9437305Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T11:13:49.9437745Z with policy(): 2025-12-04T11:13:49.9438145Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T11:13:49.9438590Z raise RuntimeError(msg) 2025-12-04T11:13:49.9439532Z RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_concat_cuda_batch_size_1_mid_dim_1_in_features_128_out_features_64_cuda_bfloat16! Caching allocator allocated memory was 0 and is now reported as 24576 on device 0. CUDA driver allocated memory was 230686720 and is now 274726912. 2025-12-04T11:13:49.9440411Z 2025-12-04T11:13:49.9440551Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:49.9441300Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_concat_cuda_batch_size_1_mid_dim_1_in_features_128_out_features_64_cuda_bfloat16 2025-12-04T11:13:49.9441910Z 2025-12-04T11:13:49.9442158Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:49.9442630Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:49.9442954Z stats [('calls_captured', 18), ('unique_graphs', 1)] 2025-12-04T11:13:49.9443489Z inductor [('pattern_matcher_nodes', 18), ('woq_matcher_nodes', 12), ('pattern_matcher_count', 9), ('woq_matcher_count', 3), ('extern_calls', 3), ('fxgraph_cache_miss', 1)] 2025-12-04T11:13:49.9444050Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:49.9444327Z graph_break [] 2025-12-04T11:13:49.9444557Z ----------------------------- Captured stderr call ----------------------------- 2025-12-04T11:13:49.9445480Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/constant_folding.py:256: UserWarning: Unsupported unwinding pattern: Address not in range (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/profiler/unwind/unwind.cpp:219.) 2025-12-04T11:13:49.9446340Z if out == self.unknown_value: 2025-12-04T11:13:49.9446792Z _ TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_concat_cuda_batch_size_1_mid_dim_1_in_features_128_out_features_64_cuda_bfloat16 _ 2025-12-04T11:13:49.9447262Z Traceback (most recent call last): 2025-12-04T11:13:49.9447707Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:49.9448153Z method(*args, **kwargs) 2025-12-04T11:13:49.9448571Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:49.9449019Z method(*args, **kwargs) 2025-12-04T11:13:49.9449433Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T11:13:49.9449871Z with policy(): 2025-12-04T11:13:49.9450283Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T11:13:49.9450729Z raise RuntimeError(msg) 2025-12-04T11:13:49.9451692Z RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_concat_cuda_batch_size_1_mid_dim_1_in_features_128_out_features_64_cuda_bfloat16! Caching allocator allocated memory was 24576 and is now reported as 49152 on device 0. CUDA driver allocated memory was 274726912 and is now 276824064. 2025-12-04T11:13:49.9452596Z 2025-12-04T11:13:49.9452729Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:49.9453481Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_concat_cuda_batch_size_1_mid_dim_1_in_features_128_out_features_64_cuda_bfloat16 2025-12-04T11:13:49.9454093Z 2025-12-04T11:13:49.9454273Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:49.9454651Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:49.9454969Z stats [('calls_captured', 18), ('unique_graphs', 1)] 2025-12-04T11:13:49.9455512Z inductor [('pattern_matcher_nodes', 18), ('woq_matcher_nodes', 12), ('pattern_matcher_count', 9), ('woq_matcher_count', 3), ('extern_calls', 3), ('fxgraph_cache_miss', 1)] 2025-12-04T11:13:49.9456077Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:49.9456343Z graph_break [] 2025-12-04T11:13:49.9456571Z ----------------------------- Captured stderr call ----------------------------- 2025-12-04T11:13:49.9457479Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/constant_folding.py:256: UserWarning: Unsupported unwinding pattern: Address not in range (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/profiler/unwind/unwind.cpp:219.) 2025-12-04T11:13:49.9458310Z if out == self.unknown_value: 2025-12-04T11:13:49.9458559Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:49.9458943Z stats [('calls_captured', 18), ('unique_graphs', 1)] 2025-12-04T11:13:49.9459384Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:49.9459944Z inductor [('pattern_matcher_nodes', 18), ('woq_matcher_nodes', 12), ('pattern_matcher_count', 9), ('woq_matcher_count', 3), ('extern_calls', 3), ('fxgraph_cache_miss', 1)] 2025-12-04T11:13:49.9460429Z graph_break [] 2025-12-04T11:13:49.9460613Z =================================== FAILURES =================================== 2025-12-04T11:13:49.9461092Z _ TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_concat_cuda_batch_size_1_mid_dim_1_in_features_128_out_features_64_cuda_bfloat16 _ 2025-12-04T11:13:49.9461558Z Traceback (most recent call last): 2025-12-04T11:13:49.9462002Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:49.9462443Z method(*args, **kwargs) 2025-12-04T11:13:49.9462851Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:49.9463300Z method(*args, **kwargs) 2025-12-04T11:13:49.9463771Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T11:13:49.9464203Z with policy(): 2025-12-04T11:13:49.9464593Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T11:13:49.9465030Z raise RuntimeError(msg) 2025-12-04T11:13:49.9465981Z RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_concat_cuda_batch_size_1_mid_dim_1_in_features_128_out_features_64_cuda_bfloat16! Caching allocator allocated memory was 49152 and is now reported as 73728 on device 0. CUDA driver allocated memory was 276824064 and is now 278921216. 2025-12-04T11:13:49.9466881Z 2025-12-04T11:13:49.9467016Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:49.9467759Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_concat_cuda_batch_size_1_mid_dim_1_in_features_128_out_features_64_cuda_bfloat16 2025-12-04T11:13:49.9468371Z 2025-12-04T11:13:49.9468537Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:49.9468903Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:49.9469213Z stats [('calls_captured', 18), ('unique_graphs', 1)] 2025-12-04T11:13:49.9469733Z inductor [('pattern_matcher_nodes', 18), ('woq_matcher_nodes', 12), ('pattern_matcher_count', 9), ('woq_matcher_count', 3), ('extern_calls', 3), ('fxgraph_cache_miss', 1)] 2025-12-04T11:13:49.9470288Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:49.9470556Z graph_break [] 2025-12-04T11:13:49.9470773Z ----------------------------- Captured stderr call ----------------------------- 2025-12-04T11:13:49.9471680Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/constant_folding.py:256: UserWarning: Unsupported unwinding pattern: Address not in range (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/profiler/unwind/unwind.cpp:219.) 2025-12-04T11:13:49.9472507Z if out == self.unknown_value: 2025-12-04T11:13:49.9472760Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:49.9473071Z stats [('calls_captured', 18), ('unique_graphs', 1)] 2025-12-04T11:13:49.9473360Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:49.9473912Z inductor [('pattern_matcher_nodes', 18), ('woq_matcher_nodes', 12), ('pattern_matcher_count', 9), ('woq_matcher_count', 3), ('extern_calls', 3), ('fxgraph_cache_miss', 1)] 2025-12-04T11:13:49.9474398Z graph_break [] 2025-12-04T11:13:49.9474609Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:49.9475000Z stats [('calls_captured', 18), ('unique_graphs', 1)] 2025-12-04T11:13:49.9475358Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:49.9475911Z inductor [('pattern_matcher_nodes', 18), ('woq_matcher_nodes', 12), ('pattern_matcher_count', 9), ('woq_matcher_count', 3), ('extern_calls', 3), ('fxgraph_cache_miss', 1)] 2025-12-04T11:13:49.9476390Z graph_break [] 2025-12-04T11:13:49.9476966Z - generated xml file: /var/lib/jenkins/workspace/test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-6c093515a384d9fc.xml - 2025-12-04T11:13:49.9477620Z =========================== short test summary info ============================ 2025-12-04T11:13:49.9479387Z FAILED [0.5602s] inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_concat_cuda_batch_size_1_mid_dim_1_in_features_128_out_features_64_cuda_bfloat16 - RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_concat_cuda_batch_size_1_mid_dim_1_in_features_128_out_features_64_cuda_bfloat16! Caching allocator allocated memory was 49152 and is now reported as 73728 on device 0. CUDA driver allocated memory was 276824064 and is now 278921216. 2025-12-04T11:13:49.9480782Z 2025-12-04T11:13:49.9480916Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:49.9481653Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_concat_cuda_batch_size_1_mid_dim_1_in_features_128_out_features_64_cuda_bfloat16 2025-12-04T11:13:49.9482269Z 2025-12-04T11:13:49.9482427Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:49.9482774Z !!!!!!!!!!!!!!!!!!!!!!!!!! stopping after 1 failures !!!!!!!!!!!!!!!!!!!!!!!!!!! 2025-12-04T11:13:49.9483075Z ================== 1 failed, 57 deselected, 2 rerun in 12.16s ================== 2025-12-04T11:13:49.9483332Z Got exit code 1 2025-12-04T11:13:49.9483925Z FAILED CONSISTENTLY: test/inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_concat_cuda_batch_size_1_mid_dim_1_in_features_128_out_features_64_cuda_bfloat16 2025-12-04T11:13:49.9484723Z Test failed consistently, continuing with the rest of the tests due to continue-through-error being set 2025-12-04T11:13:49.9485305Z W1204 10:28:17.168000 57026 site-packages/torch/_inductor/utils.py:1703] Not enough SMs to use max_autotune_gemm mode 2025-12-04T11:13:49.9486035Z Test results will be stored in test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-7ea688002f0554bd.xml 2025-12-04T11:13:49.9486590Z ============================= test session starts ============================== 2025-12-04T11:13:49.9486983Z platform linux -- Python 3.10.14, pytest-7.3.2, pluggy-1.6.0 -- /opt/conda/envs/py_3.10/bin/python 2025-12-04T11:13:49.9487338Z cachedir: .pytest_cache 2025-12-04T11:13:49.9487754Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow] 2025-12-04T11:13:49.9488217Z rootdir: /var/lib/jenkins/workspace 2025-12-04T11:13:49.9488425Z configfile: pytest.ini 2025-12-04T11:13:49.9488844Z plugins: hypothesis-6.56.4, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-14.0, subtests-0.13.1, xdist-3.3.1, xdoctest-1.3.0, typeguard-4.3.0 2025-12-04T11:13:49.9489363Z collecting ... collected 58 items / 1 deselected / 57 selected 2025-12-04T11:13:49.9489657Z stepcurrent: skipping 1 already run items. 2025-12-04T11:13:49.9489884Z Running 57 items in this shard 2025-12-04T11:13:49.9490010Z 2025-12-04T11:13:49.9490535Z inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_concat_cuda_batch_size_1_mid_dim_8_in_features_128_out_features_64_cuda_bfloat16 ('RERUN', {'yellow': True}) [2.0388s] [ 1%] 2025-12-04T11:13:49.9491620Z inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_concat_cuda_batch_size_1_mid_dim_8_in_features_128_out_features_64_cuda_bfloat16 ('RERUN', {'yellow': True}) [0.6171s] [ 1%] 2025-12-04T11:13:49.9492999Z inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_concat_cuda_batch_size_1_mid_dim_8_in_features_128_out_features_64_cuda_bfloat16 FAILED [0.6224s] [ 1%] 2025-12-04T11:13:49.9493547Z 2025-12-04T11:13:49.9493639Z ==================================== RERUNS ==================================== 2025-12-04T11:13:49.9494113Z _ TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_concat_cuda_batch_size_1_mid_dim_8_in_features_128_out_features_64_cuda_bfloat16 _ 2025-12-04T11:13:49.9494571Z Traceback (most recent call last): 2025-12-04T11:13:49.9495023Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:49.9495465Z method(*args, **kwargs) 2025-12-04T11:13:49.9495878Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:49.9496317Z method(*args, **kwargs) 2025-12-04T11:13:49.9496731Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T11:13:49.9497161Z with policy(): 2025-12-04T11:13:49.9497552Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T11:13:49.9498035Z raise RuntimeError(msg) 2025-12-04T11:13:49.9498978Z RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_concat_cuda_batch_size_1_mid_dim_8_in_features_128_out_features_64_cuda_bfloat16! Caching allocator allocated memory was 0 and is now reported as 24576 on device 0. CUDA driver allocated memory was 230686720 and is now 274726912. 2025-12-04T11:13:49.9499866Z 2025-12-04T11:13:49.9500005Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:49.9500760Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_concat_cuda_batch_size_1_mid_dim_8_in_features_128_out_features_64_cuda_bfloat16 2025-12-04T11:13:49.9501373Z 2025-12-04T11:13:49.9537081Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:49.9537565Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:49.9537902Z stats [('calls_captured', 18), ('unique_graphs', 1)] 2025-12-04T11:13:49.9538432Z inductor [('pattern_matcher_nodes', 18), ('woq_matcher_nodes', 12), ('pattern_matcher_count', 9), ('woq_matcher_count', 3), ('extern_calls', 3), ('fxgraph_cache_miss', 1)] 2025-12-04T11:13:49.9538997Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:49.9539267Z graph_break [] 2025-12-04T11:13:49.9539684Z _ TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_concat_cuda_batch_size_1_mid_dim_8_in_features_128_out_features_64_cuda_bfloat16 _ 2025-12-04T11:13:49.9540155Z Traceback (most recent call last): 2025-12-04T11:13:49.9540629Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:49.9541077Z method(*args, **kwargs) 2025-12-04T11:13:49.9541491Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:49.9541935Z method(*args, **kwargs) 2025-12-04T11:13:49.9542335Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T11:13:49.9542772Z with policy(): 2025-12-04T11:13:49.9543173Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T11:13:49.9543672Z raise RuntimeError(msg) 2025-12-04T11:13:49.9544741Z RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_concat_cuda_batch_size_1_mid_dim_8_in_features_128_out_features_64_cuda_bfloat16! Caching allocator allocated memory was 24576 and is now reported as 49152 on device 0. CUDA driver allocated memory was 274726912 and is now 276824064. 2025-12-04T11:13:49.9545712Z 2025-12-04T11:13:49.9545846Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:49.9546588Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_concat_cuda_batch_size_1_mid_dim_8_in_features_128_out_features_64_cuda_bfloat16 2025-12-04T11:13:49.9547203Z 2025-12-04T11:13:49.9547369Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:49.9547738Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:49.9548048Z stats [('calls_captured', 18), ('unique_graphs', 1)] 2025-12-04T11:13:49.9548573Z inductor [('pattern_matcher_nodes', 18), ('woq_matcher_nodes', 12), ('pattern_matcher_count', 9), ('woq_matcher_count', 3), ('extern_calls', 3), ('fxgraph_cache_miss', 1)] 2025-12-04T11:13:49.9549146Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:49.9549412Z graph_break [] 2025-12-04T11:13:49.9549643Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:49.9549952Z stats [('calls_captured', 18), ('unique_graphs', 1)] 2025-12-04T11:13:49.9550256Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:49.9550797Z inductor [('pattern_matcher_nodes', 18), ('woq_matcher_nodes', 12), ('pattern_matcher_count', 9), ('woq_matcher_count', 3), ('extern_calls', 3), ('fxgraph_cache_miss', 1)] 2025-12-04T11:13:49.9551276Z graph_break [] 2025-12-04T11:13:49.9551454Z =================================== FAILURES =================================== 2025-12-04T11:13:49.9551935Z _ TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_concat_cuda_batch_size_1_mid_dim_8_in_features_128_out_features_64_cuda_bfloat16 _ 2025-12-04T11:13:49.9552398Z Traceback (most recent call last): 2025-12-04T11:13:49.9552865Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:49.9553309Z method(*args, **kwargs) 2025-12-04T11:13:49.9553723Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:49.9554148Z method(*args, **kwargs) 2025-12-04T11:13:49.9554553Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T11:13:49.9554975Z with policy(): 2025-12-04T11:13:49.9555370Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T11:13:49.9555801Z raise RuntimeError(msg) 2025-12-04T11:13:49.9556753Z RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_concat_cuda_batch_size_1_mid_dim_8_in_features_128_out_features_64_cuda_bfloat16! Caching allocator allocated memory was 49152 and is now reported as 73728 on device 0. CUDA driver allocated memory was 276824064 and is now 278921216. 2025-12-04T11:13:49.9557645Z 2025-12-04T11:13:49.9557781Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:49.9558519Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_concat_cuda_batch_size_1_mid_dim_8_in_features_128_out_features_64_cuda_bfloat16 2025-12-04T11:13:49.9559130Z 2025-12-04T11:13:49.9559293Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:49.9559659Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:49.9559962Z stats [('calls_captured', 18), ('unique_graphs', 1)] 2025-12-04T11:13:49.9560655Z inductor [('pattern_matcher_nodes', 18), ('woq_matcher_nodes', 12), ('pattern_matcher_count', 9), ('woq_matcher_count', 3), ('extern_calls', 3), ('fxgraph_cache_miss', 1)] 2025-12-04T11:13:49.9561199Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:49.9561464Z graph_break [] 2025-12-04T11:13:49.9561682Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:49.9561976Z stats [('calls_captured', 18), ('unique_graphs', 1)] 2025-12-04T11:13:49.9562262Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:49.9562811Z inductor [('pattern_matcher_nodes', 18), ('woq_matcher_nodes', 12), ('pattern_matcher_count', 9), ('woq_matcher_count', 3), ('extern_calls', 3), ('fxgraph_cache_miss', 1)] 2025-12-04T11:13:49.9563286Z graph_break [] 2025-12-04T11:13:49.9563493Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:49.9563787Z stats [('calls_captured', 18), ('unique_graphs', 1)] 2025-12-04T11:13:49.9564079Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:49.9564619Z inductor [('pattern_matcher_nodes', 18), ('woq_matcher_nodes', 12), ('pattern_matcher_count', 9), ('woq_matcher_count', 3), ('extern_calls', 3), ('fxgraph_cache_miss', 1)] 2025-12-04T11:13:49.9565096Z graph_break [] 2025-12-04T11:13:49.9565676Z - generated xml file: /var/lib/jenkins/workspace/test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-7ea688002f0554bd.xml - 2025-12-04T11:13:49.9566354Z =========================== short test summary info ============================ 2025-12-04T11:13:49.9567922Z FAILED [0.6224s] inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_concat_cuda_batch_size_1_mid_dim_8_in_features_128_out_features_64_cuda_bfloat16 - RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_concat_cuda_batch_size_1_mid_dim_8_in_features_128_out_features_64_cuda_bfloat16! Caching allocator allocated memory was 49152 and is now reported as 73728 on device 0. CUDA driver allocated memory was 276824064 and is now 278921216. 2025-12-04T11:13:49.9569301Z 2025-12-04T11:13:49.9569437Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:49.9570176Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_concat_cuda_batch_size_1_mid_dim_8_in_features_128_out_features_64_cuda_bfloat16 2025-12-04T11:13:49.9570782Z 2025-12-04T11:13:49.9570942Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:49.9571285Z !!!!!!!!!!!!!!!!!!!!!!!!!! stopping after 1 failures !!!!!!!!!!!!!!!!!!!!!!!!!!! 2025-12-04T11:13:49.9571588Z =================== 1 failed, 1 deselected, 2 rerun in 3.30s =================== 2025-12-04T11:13:49.9571837Z Got exit code 1 2025-12-04T11:13:49.9572001Z Retrying single test... 2025-12-04T11:13:49.9572376Z W1204 10:28:27.260000 57208 site-packages/torch/_inductor/utils.py:1703] Not enough SMs to use max_autotune_gemm mode 2025-12-04T11:13:49.9573094Z Test results will be stored in test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-759714b1d86e3c4c.xml 2025-12-04T11:13:49.9573652Z ============================= test session starts ============================== 2025-12-04T11:13:49.9574056Z platform linux -- Python 3.10.14, pytest-7.3.2, pluggy-1.6.0 -- /opt/conda/envs/py_3.10/bin/python 2025-12-04T11:13:49.9574417Z cachedir: .pytest_cache 2025-12-04T11:13:49.9574834Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow] 2025-12-04T11:13:49.9575291Z rootdir: /var/lib/jenkins/workspace 2025-12-04T11:13:49.9575510Z configfile: pytest.ini 2025-12-04T11:13:49.9575938Z plugins: hypothesis-6.56.4, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-14.0, subtests-0.13.1, xdist-3.3.1, xdoctest-1.3.0, typeguard-4.3.0 2025-12-04T11:13:49.9576536Z collecting ... collected 58 items / 57 deselected / 1 selected 2025-12-04T11:13:49.9577472Z stepcurrent: skipping 1 already run items. Running only test/inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_concat_cuda_batch_size_1_mid_dim_8_in_features_128_out_features_64_cuda_bfloat16 2025-12-04T11:13:49.9578515Z Running 1 items in this shard 2025-12-04T11:13:49.9578646Z 2025-12-04T11:13:49.9579396Z inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_concat_cuda_batch_size_1_mid_dim_8_in_features_128_out_features_64_cuda_bfloat16 [W1204 10:28:28.399301900 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:49.9580214Z 2025-12-04T11:13:49.9580513Z [W1204 10:28:37.596655653 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:49.9580898Z 2025-12-04T11:13:49.9581197Z [W1204 10:28:37.596897128 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:49.9581565Z 2025-12-04T11:13:49.9581854Z [W1204 10:28:37.597479151 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:49.9582222Z 2025-12-04T11:13:49.9582518Z [W1204 10:28:37.597668155 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:49.9582883Z 2025-12-04T11:13:49.9583175Z [W1204 10:28:37.598842021 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:49.9583609Z 2025-12-04T11:13:49.9583899Z [W1204 10:28:37.599036675 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:49.9584274Z 2025-12-04T11:13:49.9584560Z [W1204 10:28:37.599326582 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:49.9584929Z 2025-12-04T11:13:49.9585213Z [W1204 10:28:37.599477975 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:49.9585578Z 2025-12-04T11:13:49.9585871Z [W1204 10:28:37.607852219 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:49.9586237Z 2025-12-04T11:13:49.9586526Z [W1204 10:28:37.608075064 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:49.9586895Z 2025-12-04T11:13:49.9587184Z [W1204 10:28:37.608246698 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:49.9587551Z 2025-12-04T11:13:49.9587841Z [W1204 10:28:37.608480983 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:49.9588210Z 2025-12-04T11:13:49.9588506Z [W1204 10:28:37.608637496 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:49.9588870Z 2025-12-04T11:13:49.9589165Z [W1204 10:28:37.608888872 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:49.9589529Z 2025-12-04T11:13:49.9589819Z [W1204 10:28:37.609033145 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:49.9590181Z 2025-12-04T11:13:49.9590465Z [W1204 10:28:37.609264680 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:49.9590834Z 2025-12-04T11:13:49.9591123Z [W1204 10:28:37.609405953 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:49.9591639Z 2025-12-04T11:13:49.9592047Z [W1204 10:28:37.696057270 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:49.9592419Z 2025-12-04T11:13:49.9592705Z [W1204 10:28:37.696271915 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:49.9593071Z 2025-12-04T11:13:49.9593362Z [W1204 10:28:37.696422558 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:49.9593725Z 2025-12-04T11:13:49.9594015Z [W1204 10:28:37.696639693 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:49.9594380Z 2025-12-04T11:13:49.9594667Z [W1204 10:28:37.696768306 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:49.9595057Z 2025-12-04T11:13:49.9595352Z [W1204 10:28:37.696983240 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:49.9595724Z 2025-12-04T11:13:49.9596016Z [W1204 10:28:37.697110593 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:49.9596381Z 2025-12-04T11:13:49.9596668Z [W1204 10:28:37.697316188 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:49.9597047Z 2025-12-04T11:13:49.9597336Z [W1204 10:28:37.697437530 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:49.9597703Z 2025-12-04T11:13:49.9597793Z ('RERUN', {'yellow': True}) [11.2342s] [100%] 2025-12-04T11:13:49.9598698Z inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_concat_cuda_batch_size_1_mid_dim_8_in_features_128_out_features_64_cuda_bfloat16 [W1204 10:28:38.946949743 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:49.9599526Z 2025-12-04T11:13:49.9599818Z [W1204 10:28:38.947207989 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:49.9600181Z 2025-12-04T11:13:49.9600480Z [W1204 10:28:38.947363962 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:49.9600847Z 2025-12-04T11:13:49.9601137Z [W1204 10:28:38.947580607 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:49.9601499Z 2025-12-04T11:13:49.9601786Z [W1204 10:28:38.947707750 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:49.9602161Z 2025-12-04T11:13:49.9602456Z [W1204 10:28:38.947927954 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:49.9602828Z 2025-12-04T11:13:49.9603118Z [W1204 10:28:38.948053727 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:49.9603483Z 2025-12-04T11:13:49.9603772Z [W1204 10:28:38.948261642 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:49.9604137Z 2025-12-04T11:13:49.9604434Z [W1204 10:28:38.948382855 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:49.9604793Z 2025-12-04T11:13:49.9605078Z [W1204 10:28:39.954541220 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:49.9605558Z 2025-12-04T11:13:49.9605923Z [W1204 10:28:39.954716264 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:49.9606297Z 2025-12-04T11:13:49.9606585Z [W1204 10:28:39.954862707 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:49.9606955Z 2025-12-04T11:13:49.9607244Z [W1204 10:28:39.955062661 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:49.9607619Z 2025-12-04T11:13:49.9607908Z [W1204 10:28:39.955186584 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:49.9608272Z 2025-12-04T11:13:49.9608561Z [W1204 10:28:39.955395709 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:49.9608932Z 2025-12-04T11:13:49.9609226Z [W1204 10:28:39.955526841 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:49.9609602Z 2025-12-04T11:13:49.9609889Z [W1204 10:28:39.955729056 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:49.9610257Z 2025-12-04T11:13:49.9610548Z [W1204 10:28:39.955848628 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:49.9610918Z 2025-12-04T11:13:49.9611203Z [W1204 10:28:39.039643403 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:49.9611563Z 2025-12-04T11:13:49.9611860Z [W1204 10:28:39.039865348 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:49.9612232Z 2025-12-04T11:13:49.9612530Z [W1204 10:28:39.040040001 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:49.9612894Z 2025-12-04T11:13:49.9613180Z [W1204 10:28:39.040268686 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:49.9613547Z 2025-12-04T11:13:49.9613836Z [W1204 10:28:39.040396009 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:49.9614202Z 2025-12-04T11:13:49.9614488Z [W1204 10:28:39.040625124 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:49.9614854Z 2025-12-04T11:13:49.9615148Z [W1204 10:28:39.040752107 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:49.9615513Z 2025-12-04T11:13:49.9615807Z [W1204 10:28:39.040961112 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:49.9616192Z 2025-12-04T11:13:49.9616483Z [W1204 10:28:39.041084634 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:49.9616852Z 2025-12-04T11:13:49.9616935Z ('RERUN', {'yellow': True}) [0.5721s] [100%] 2025-12-04T11:13:49.9617832Z inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_concat_cuda_batch_size_1_mid_dim_8_in_features_128_out_features_64_cuda_bfloat16 [W1204 10:28:39.514887799 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:49.9618651Z 2025-12-04T11:13:49.9618943Z [W1204 10:28:39.515099973 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:49.9619307Z 2025-12-04T11:13:49.9619690Z [W1204 10:28:39.515251807 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:49.9620128Z 2025-12-04T11:13:49.9620419Z [W1204 10:28:39.515462421 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:49.9620791Z 2025-12-04T11:13:49.9621078Z [W1204 10:28:39.515592644 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:49.9621442Z 2025-12-04T11:13:49.9621729Z [W1204 10:28:39.515808869 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:49.9622091Z 2025-12-04T11:13:49.9622381Z [W1204 10:28:39.515933332 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:49.9622746Z 2025-12-04T11:13:49.9623037Z [W1204 10:28:39.516139706 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:49.9623406Z 2025-12-04T11:13:49.9623756Z [W1204 10:28:39.516262489 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:49.9624121Z 2025-12-04T11:13:49.9624408Z [W1204 10:28:39.522313183 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:49.9624780Z 2025-12-04T11:13:49.9625065Z [W1204 10:28:39.522484086 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:49.9625434Z 2025-12-04T11:13:49.9625726Z [W1204 10:28:39.522633050 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:49.9626091Z 2025-12-04T11:13:49.9626378Z [W1204 10:28:39.522835414 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:49.9626748Z 2025-12-04T11:13:49.9627046Z [W1204 10:28:39.522957717 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:49.9627409Z 2025-12-04T11:13:49.9627696Z [W1204 10:28:39.523170411 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:49.9628063Z 2025-12-04T11:13:49.9628353Z [W1204 10:28:39.523294384 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:49.9628718Z 2025-12-04T11:13:49.9629003Z [W1204 10:28:39.523509919 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:49.9629372Z 2025-12-04T11:13:49.9629662Z [W1204 10:28:39.523630601 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:49.9630029Z 2025-12-04T11:13:49.9630325Z [W1204 10:28:39.607135199 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:49.9630689Z 2025-12-04T11:13:49.9630975Z [W1204 10:28:39.607323333 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:49.9631343Z 2025-12-04T11:13:49.9631630Z [W1204 10:28:39.607474916 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:49.9631995Z 2025-12-04T11:13:49.9632279Z [W1204 10:28:39.607685071 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:49.9632644Z 2025-12-04T11:13:49.9632935Z [W1204 10:28:39.607809154 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:49.9633385Z 2025-12-04T11:13:49.9633740Z [W1204 10:28:39.608026169 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:49.9634108Z 2025-12-04T11:13:49.9634393Z [W1204 10:28:39.608148801 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:49.9634768Z 2025-12-04T11:13:49.9635057Z [W1204 10:28:39.608352206 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:49.9635426Z 2025-12-04T11:13:49.9635733Z [W1204 10:28:39.608475639 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:49.9636103Z 2025-12-04T11:13:49.9636177Z FAILED [0.5676s] [100%] 2025-12-04T11:13:49.9636288Z 2025-12-04T11:13:49.9636381Z ==================================== RERUNS ==================================== 2025-12-04T11:13:49.9636880Z _ TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_concat_cuda_batch_size_1_mid_dim_8_in_features_128_out_features_64_cuda_bfloat16 _ 2025-12-04T11:13:49.9637348Z Traceback (most recent call last): 2025-12-04T11:13:49.9637807Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:49.9638250Z method(*args, **kwargs) 2025-12-04T11:13:49.9638667Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:49.9639106Z method(*args, **kwargs) 2025-12-04T11:13:49.9639511Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T11:13:49.9639944Z with policy(): 2025-12-04T11:13:49.9640338Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T11:13:49.9640783Z raise RuntimeError(msg) 2025-12-04T11:13:49.9641718Z RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_concat_cuda_batch_size_1_mid_dim_8_in_features_128_out_features_64_cuda_bfloat16! Caching allocator allocated memory was 0 and is now reported as 24576 on device 0. CUDA driver allocated memory was 230686720 and is now 274726912. 2025-12-04T11:13:49.9642603Z 2025-12-04T11:13:49.9642736Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:49.9643484Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_concat_cuda_batch_size_1_mid_dim_8_in_features_128_out_features_64_cuda_bfloat16 2025-12-04T11:13:49.9644091Z 2025-12-04T11:13:49.9644262Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:49.9644635Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:49.9644947Z stats [('calls_captured', 18), ('unique_graphs', 1)] 2025-12-04T11:13:49.9645487Z inductor [('pattern_matcher_nodes', 18), ('woq_matcher_nodes', 12), ('pattern_matcher_count', 9), ('woq_matcher_count', 3), ('extern_calls', 3), ('fxgraph_cache_miss', 1)] 2025-12-04T11:13:49.9646044Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:49.9646311Z graph_break [] 2025-12-04T11:13:49.9646541Z ----------------------------- Captured stderr call ----------------------------- 2025-12-04T11:13:49.9647520Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/constant_folding.py:256: UserWarning: Unsupported unwinding pattern: Address not in range (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/profiler/unwind/unwind.cpp:219.) 2025-12-04T11:13:49.9648542Z if out == self.unknown_value: 2025-12-04T11:13:49.9648983Z _ TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_concat_cuda_batch_size_1_mid_dim_8_in_features_128_out_features_64_cuda_bfloat16 _ 2025-12-04T11:13:49.9649564Z Traceback (most recent call last): 2025-12-04T11:13:49.9650074Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:49.9650530Z method(*args, **kwargs) 2025-12-04T11:13:49.9650938Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:49.9651369Z method(*args, **kwargs) 2025-12-04T11:13:49.9651775Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T11:13:49.9652200Z with policy(): 2025-12-04T11:13:49.9652596Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T11:13:49.9653040Z raise RuntimeError(msg) 2025-12-04T11:13:49.9654004Z RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_concat_cuda_batch_size_1_mid_dim_8_in_features_128_out_features_64_cuda_bfloat16! Caching allocator allocated memory was 24576 and is now reported as 49152 on device 0. CUDA driver allocated memory was 274726912 and is now 276824064. 2025-12-04T11:13:49.9654919Z 2025-12-04T11:13:49.9655057Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:49.9655805Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_concat_cuda_batch_size_1_mid_dim_8_in_features_128_out_features_64_cuda_bfloat16 2025-12-04T11:13:49.9656419Z 2025-12-04T11:13:49.9656583Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:49.9656958Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:49.9657274Z stats [('calls_captured', 18), ('unique_graphs', 1)] 2025-12-04T11:13:49.9657803Z inductor [('pattern_matcher_nodes', 18), ('woq_matcher_nodes', 12), ('pattern_matcher_count', 9), ('woq_matcher_count', 3), ('extern_calls', 3), ('fxgraph_cache_miss', 1)] 2025-12-04T11:13:49.9658376Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:49.9658648Z graph_break [] 2025-12-04T11:13:49.9658867Z ----------------------------- Captured stderr call ----------------------------- 2025-12-04T11:13:49.9659798Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/constant_folding.py:256: UserWarning: Unsupported unwinding pattern: Address not in range (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/profiler/unwind/unwind.cpp:219.) 2025-12-04T11:13:49.9660633Z if out == self.unknown_value: 2025-12-04T11:13:49.9660889Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:49.9661192Z stats [('calls_captured', 18), ('unique_graphs', 1)] 2025-12-04T11:13:49.9661498Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:49.9662064Z inductor [('pattern_matcher_nodes', 18), ('woq_matcher_nodes', 12), ('pattern_matcher_count', 9), ('woq_matcher_count', 3), ('extern_calls', 3), ('fxgraph_cache_miss', 1)] 2025-12-04T11:13:49.9662550Z graph_break [] 2025-12-04T11:13:49.9662730Z =================================== FAILURES =================================== 2025-12-04T11:13:49.9663214Z _ TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_concat_cuda_batch_size_1_mid_dim_8_in_features_128_out_features_64_cuda_bfloat16 _ 2025-12-04T11:13:49.9663765Z Traceback (most recent call last): 2025-12-04T11:13:49.9664214Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:49.9664660Z method(*args, **kwargs) 2025-12-04T11:13:49.9665078Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:49.9665508Z method(*args, **kwargs) 2025-12-04T11:13:49.9665994Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T11:13:49.9666492Z with policy(): 2025-12-04T11:13:49.9666897Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T11:13:49.9667333Z raise RuntimeError(msg) 2025-12-04T11:13:49.9668283Z RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_concat_cuda_batch_size_1_mid_dim_8_in_features_128_out_features_64_cuda_bfloat16! Caching allocator allocated memory was 49152 and is now reported as 73728 on device 0. CUDA driver allocated memory was 276824064 and is now 278921216. 2025-12-04T11:13:49.9669192Z 2025-12-04T11:13:49.9669320Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:49.9670062Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_concat_cuda_batch_size_1_mid_dim_8_in_features_128_out_features_64_cuda_bfloat16 2025-12-04T11:13:49.9670678Z 2025-12-04T11:13:49.9670849Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:49.9671212Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:49.9671518Z stats [('calls_captured', 18), ('unique_graphs', 1)] 2025-12-04T11:13:49.9672043Z inductor [('pattern_matcher_nodes', 18), ('woq_matcher_nodes', 12), ('pattern_matcher_count', 9), ('woq_matcher_count', 3), ('extern_calls', 3), ('fxgraph_cache_miss', 1)] 2025-12-04T11:13:49.9672596Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:49.9672858Z graph_break [] 2025-12-04T11:13:49.9673077Z ----------------------------- Captured stderr call ----------------------------- 2025-12-04T11:13:49.9673980Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/constant_folding.py:256: UserWarning: Unsupported unwinding pattern: Address not in range (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/profiler/unwind/unwind.cpp:219.) 2025-12-04T11:13:49.9674819Z if out == self.unknown_value: 2025-12-04T11:13:49.9675071Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:49.9675393Z stats [('calls_captured', 18), ('unique_graphs', 1)] 2025-12-04T11:13:49.9675697Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:49.9676248Z inductor [('pattern_matcher_nodes', 18), ('woq_matcher_nodes', 12), ('pattern_matcher_count', 9), ('woq_matcher_count', 3), ('extern_calls', 3), ('fxgraph_cache_miss', 1)] 2025-12-04T11:13:49.9676733Z graph_break [] 2025-12-04T11:13:49.9676953Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:49.9677255Z stats [('calls_captured', 18), ('unique_graphs', 1)] 2025-12-04T11:13:49.9677549Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:49.9678378Z inductor [('pattern_matcher_nodes', 18), ('woq_matcher_nodes', 12), ('pattern_matcher_count', 9), ('woq_matcher_count', 3), ('extern_calls', 3), ('fxgraph_cache_miss', 1)] 2025-12-04T11:13:49.9678867Z graph_break [] 2025-12-04T11:13:49.9679460Z - generated xml file: /var/lib/jenkins/workspace/test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-759714b1d86e3c4c.xml - 2025-12-04T11:13:49.9680126Z =========================== short test summary info ============================ 2025-12-04T11:13:49.9681635Z FAILED [0.5676s] inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_concat_cuda_batch_size_1_mid_dim_8_in_features_128_out_features_64_cuda_bfloat16 - RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_concat_cuda_batch_size_1_mid_dim_8_in_features_128_out_features_64_cuda_bfloat16! Caching allocator allocated memory was 49152 and is now reported as 73728 on device 0. CUDA driver allocated memory was 276824064 and is now 278921216. 2025-12-04T11:13:49.9683154Z 2025-12-04T11:13:49.9683376Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:49.9684123Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_concat_cuda_batch_size_1_mid_dim_8_in_features_128_out_features_64_cuda_bfloat16 2025-12-04T11:13:49.9684728Z 2025-12-04T11:13:49.9684896Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:49.9685239Z !!!!!!!!!!!!!!!!!!!!!!!!!! stopping after 1 failures !!!!!!!!!!!!!!!!!!!!!!!!!!! 2025-12-04T11:13:49.9685548Z ================== 1 failed, 57 deselected, 2 rerun in 12.40s ================== 2025-12-04T11:13:49.9685827Z Got exit code 1 2025-12-04T11:13:49.9685990Z Retrying single test... 2025-12-04T11:13:49.9686372Z W1204 10:28:46.437000 57395 site-packages/torch/_inductor/utils.py:1703] Not enough SMs to use max_autotune_gemm mode 2025-12-04T11:13:49.9687113Z Test results will be stored in test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-7a190e1b598a9023.xml 2025-12-04T11:13:49.9687682Z ============================= test session starts ============================== 2025-12-04T11:13:49.9688077Z platform linux -- Python 3.10.14, pytest-7.3.2, pluggy-1.6.0 -- /opt/conda/envs/py_3.10/bin/python 2025-12-04T11:13:49.9688441Z cachedir: .pytest_cache 2025-12-04T11:13:49.9688861Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow] 2025-12-04T11:13:49.9689326Z rootdir: /var/lib/jenkins/workspace 2025-12-04T11:13:49.9689536Z configfile: pytest.ini 2025-12-04T11:13:49.9689968Z plugins: hypothesis-6.56.4, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-14.0, subtests-0.13.1, xdist-3.3.1, xdoctest-1.3.0, typeguard-4.3.0 2025-12-04T11:13:49.9690495Z collecting ... collected 58 items / 57 deselected / 1 selected 2025-12-04T11:13:49.9691294Z stepcurrent: skipping 1 already run items. Running only test/inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_concat_cuda_batch_size_1_mid_dim_8_in_features_128_out_features_64_cuda_bfloat16 2025-12-04T11:13:49.9692016Z Running 1 items in this shard 2025-12-04T11:13:49.9692150Z 2025-12-04T11:13:49.9692897Z inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_concat_cuda_batch_size_1_mid_dim_8_in_features_128_out_features_64_cuda_bfloat16 [W1204 10:28:47.577159407 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:49.9693716Z 2025-12-04T11:13:49.9694017Z [W1204 10:28:56.640655500 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:49.9694392Z 2025-12-04T11:13:49.9694691Z [W1204 10:28:56.640924766 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:49.9695065Z 2025-12-04T11:13:49.9695365Z [W1204 10:28:56.641480588 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:49.9695733Z 2025-12-04T11:13:49.9696035Z [W1204 10:28:56.641667062 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:49.9698976Z 2025-12-04T11:13:49.9699285Z [W1204 10:28:56.642893539 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:49.9699671Z 2025-12-04T11:13:49.9699967Z [W1204 10:28:56.643060963 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:49.9700334Z 2025-12-04T11:13:49.9700630Z [W1204 10:28:56.643317059 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:49.9701081Z 2025-12-04T11:13:49.9701446Z [W1204 10:28:56.643468192 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:49.9701815Z 2025-12-04T11:13:49.9702103Z [W1204 10:28:56.651942909 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:49.9702477Z 2025-12-04T11:13:49.9702768Z [W1204 10:28:56.652140353 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:49.9703138Z 2025-12-04T11:13:49.9703425Z [W1204 10:28:56.652311666 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:49.9703867Z 2025-12-04T11:13:49.9704166Z [W1204 10:28:56.652546892 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:49.9704546Z 2025-12-04T11:13:49.9704856Z [W1204 10:28:56.652696245 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:49.9705228Z 2025-12-04T11:13:49.9705521Z [W1204 10:28:56.652941980 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:49.9705900Z 2025-12-04T11:13:49.9706189Z [W1204 10:28:56.653090114 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:49.9706560Z 2025-12-04T11:13:49.9706849Z [W1204 10:28:56.653319619 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:49.9707217Z 2025-12-04T11:13:49.9707514Z [W1204 10:28:56.653459012 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:49.9707888Z 2025-12-04T11:13:49.9708186Z [W1204 10:28:56.742858274 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:49.9708559Z 2025-12-04T11:13:49.9708847Z [W1204 10:28:56.743070618 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:49.9709221Z 2025-12-04T11:13:49.9709512Z [W1204 10:28:56.743219011 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:49.9709886Z 2025-12-04T11:13:49.9710176Z [W1204 10:28:56.743425736 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:49.9710561Z 2025-12-04T11:13:49.9710853Z [W1204 10:28:56.743550139 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:49.9711226Z 2025-12-04T11:13:49.9711529Z [W1204 10:28:56.743771763 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:49.9711896Z 2025-12-04T11:13:49.9712186Z [W1204 10:28:56.743895446 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:49.9712560Z 2025-12-04T11:13:49.9712849Z [W1204 10:28:56.744108321 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:49.9713218Z 2025-12-04T11:13:49.9713509Z [W1204 10:28:56.744227164 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:49.9713883Z 2025-12-04T11:13:49.9713970Z ('RERUN', {'yellow': True}) [11.1039s] [100%] 2025-12-04T11:13:49.9714975Z inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_concat_cuda_batch_size_1_mid_dim_8_in_features_128_out_features_64_cuda_bfloat16 [W1204 10:28:58.993477887 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:49.9715855Z 2025-12-04T11:13:49.9716157Z [W1204 10:28:58.993733813 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:49.9716525Z 2025-12-04T11:13:49.9716815Z [W1204 10:28:58.993884626 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:49.9717188Z 2025-12-04T11:13:49.9717479Z [W1204 10:28:58.994095851 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:49.9717849Z 2025-12-04T11:13:49.9718144Z [W1204 10:28:58.994219944 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:49.9718509Z 2025-12-04T11:13:49.9718812Z [W1204 10:28:58.994433578 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:49.9719184Z 2025-12-04T11:13:49.9719481Z [W1204 10:28:58.994555541 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:49.9719848Z 2025-12-04T11:13:49.9720138Z [W1204 10:28:58.994754135 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:49.9720689Z 2025-12-04T11:13:49.9720982Z [W1204 10:28:58.994876628 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:49.9721354Z 2025-12-04T11:13:49.9721646Z [W1204 10:28:58.001173666 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:49.9722013Z 2025-12-04T11:13:49.9722311Z [W1204 10:28:58.001349560 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:49.9722684Z 2025-12-04T11:13:49.9722983Z [W1204 10:28:58.001496624 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:49.9723350Z 2025-12-04T11:13:49.9723643Z [W1204 10:28:58.001698058 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:49.9724013Z 2025-12-04T11:13:49.9724302Z [W1204 10:28:58.001820491 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:49.9724676Z 2025-12-04T11:13:49.9724964Z [W1204 10:28:58.002034865 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:49.9725333Z 2025-12-04T11:13:49.9725629Z [W1204 10:28:58.002157268 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:49.9725997Z 2025-12-04T11:13:49.9726297Z [W1204 10:28:58.002357413 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:49.9726661Z 2025-12-04T11:13:49.9726950Z [W1204 10:28:58.002478625 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:49.9727325Z 2025-12-04T11:13:49.9727613Z [W1204 10:28:58.086823415 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:49.9727987Z 2025-12-04T11:13:49.9728277Z [W1204 10:28:58.087035330 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:49.9728642Z 2025-12-04T11:13:49.9728938Z [W1204 10:28:58.087185693 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:49.9729398Z 2025-12-04T11:13:49.9729765Z [W1204 10:28:58.087393828 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:49.9730136Z 2025-12-04T11:13:49.9730423Z [W1204 10:28:58.087517621 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:49.9730793Z 2025-12-04T11:13:49.9731083Z [W1204 10:28:58.087732446 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:49.9731452Z 2025-12-04T11:13:49.9731741Z [W1204 10:28:58.087856468 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:49.9732111Z 2025-12-04T11:13:49.9732401Z [W1204 10:28:58.088062923 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:49.9732775Z 2025-12-04T11:13:49.9733075Z [W1204 10:28:58.088183945 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:49.9733441Z 2025-12-04T11:13:49.9733526Z ('RERUN', {'yellow': True}) [0.5760s] [100%] 2025-12-04T11:13:49.9734435Z inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_concat_cuda_batch_size_1_mid_dim_8_in_features_128_out_features_64_cuda_bfloat16 [W1204 10:28:58.565588111 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:49.9735252Z 2025-12-04T11:13:49.9735557Z [W1204 10:28:58.565811746 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:49.9735935Z 2025-12-04T11:13:49.9736226Z [W1204 10:28:58.565960500 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:49.9736601Z 2025-12-04T11:13:49.9736901Z [W1204 10:28:58.566172814 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:49.9737268Z 2025-12-04T11:13:49.9737566Z [W1204 10:28:58.566297267 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:49.9737931Z 2025-12-04T11:13:49.9738218Z [W1204 10:28:58.566511022 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:49.9738593Z 2025-12-04T11:13:49.9738883Z [W1204 10:28:58.566631814 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:49.9739254Z 2025-12-04T11:13:49.9739542Z [W1204 10:28:58.566835719 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:49.9739911Z 2025-12-04T11:13:49.9740213Z [W1204 10:28:58.566959482 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:49.9740580Z 2025-12-04T11:13:49.9740874Z [W1204 10:28:58.573165709 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:49.9741241Z 2025-12-04T11:13:49.9741531Z [W1204 10:28:58.573335712 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:49.9741905Z 2025-12-04T11:13:49.9742198Z [W1204 10:28:58.573483725 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:49.9742570Z 2025-12-04T11:13:49.9742861Z [W1204 10:28:58.573687790 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:49.9743308Z 2025-12-04T11:13:49.9743742Z [W1204 10:28:58.573816143 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:49.9744119Z 2025-12-04T11:13:49.9744419Z [W1204 10:28:58.574028768 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:49.9744788Z 2025-12-04T11:13:49.9745078Z [W1204 10:28:58.574152860 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:49.9745448Z 2025-12-04T11:13:49.9745737Z [W1204 10:28:58.574355505 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:49.9746111Z 2025-12-04T11:13:49.9746397Z [W1204 10:28:58.574477858 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:49.9746771Z 2025-12-04T11:13:49.9747067Z [W1204 10:28:58.658958000 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:49.9747441Z 2025-12-04T11:13:49.9747733Z [W1204 10:28:58.659146904 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:49.9748103Z 2025-12-04T11:13:49.9748391Z [W1204 10:28:58.659292467 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:49.9748764Z 2025-12-04T11:13:49.9749054Z [W1204 10:28:58.659498822 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:49.9749429Z 2025-12-04T11:13:49.9749716Z [W1204 10:28:58.659621194 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:49.9750082Z 2025-12-04T11:13:49.9750380Z [W1204 10:28:58.659832449 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:49.9750750Z 2025-12-04T11:13:49.9751043Z [W1204 10:28:58.659953362 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:49.9751410Z 2025-12-04T11:13:49.9751702Z [W1204 10:28:58.660183307 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:49.9752070Z 2025-12-04T11:13:49.9752357Z [W1204 10:28:58.660308870 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:49.9752729Z 2025-12-04T11:13:49.9752796Z FAILED [0.5723s] [100%] 2025-12-04T11:13:49.9752906Z 2025-12-04T11:13:49.9753004Z ==================================== RERUNS ==================================== 2025-12-04T11:13:49.9753489Z _ TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_concat_cuda_batch_size_1_mid_dim_8_in_features_128_out_features_64_cuda_bfloat16 _ 2025-12-04T11:13:49.9753958Z Traceback (most recent call last): 2025-12-04T11:13:49.9754424Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:49.9754886Z method(*args, **kwargs) 2025-12-04T11:13:49.9755300Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:49.9755739Z method(*args, **kwargs) 2025-12-04T11:13:49.9756144Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T11:13:49.9756573Z with policy(): 2025-12-04T11:13:49.9756964Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T11:13:49.9757405Z raise RuntimeError(msg) 2025-12-04T11:13:49.9758509Z RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_concat_cuda_batch_size_1_mid_dim_8_in_features_128_out_features_64_cuda_bfloat16! Caching allocator allocated memory was 0 and is now reported as 24576 on device 0. CUDA driver allocated memory was 230686720 and is now 274726912. 2025-12-04T11:13:49.9759394Z 2025-12-04T11:13:49.9759532Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:49.9760277Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_concat_cuda_batch_size_1_mid_dim_8_in_features_128_out_features_64_cuda_bfloat16 2025-12-04T11:13:49.9760886Z 2025-12-04T11:13:49.9761050Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:49.9761427Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:49.9761742Z stats [('calls_captured', 18), ('unique_graphs', 1)] 2025-12-04T11:13:49.9762278Z inductor [('pattern_matcher_nodes', 18), ('woq_matcher_nodes', 12), ('pattern_matcher_count', 9), ('woq_matcher_count', 3), ('extern_calls', 3), ('fxgraph_cache_miss', 1)] 2025-12-04T11:13:49.9762837Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:49.9763109Z graph_break [] 2025-12-04T11:13:49.9763344Z ----------------------------- Captured stderr call ----------------------------- 2025-12-04T11:13:49.9764249Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/constant_folding.py:256: UserWarning: Unsupported unwinding pattern: Address not in range (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/profiler/unwind/unwind.cpp:219.) 2025-12-04T11:13:49.9765084Z if out == self.unknown_value: 2025-12-04T11:13:49.9765530Z _ TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_concat_cuda_batch_size_1_mid_dim_8_in_features_128_out_features_64_cuda_bfloat16 _ 2025-12-04T11:13:49.9765994Z Traceback (most recent call last): 2025-12-04T11:13:49.9766440Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:49.9766890Z method(*args, **kwargs) 2025-12-04T11:13:49.9767303Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:49.9767734Z method(*args, **kwargs) 2025-12-04T11:13:49.9768144Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T11:13:49.9768573Z with policy(): 2025-12-04T11:13:49.9768969Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T11:13:49.9769404Z raise RuntimeError(msg) 2025-12-04T11:13:49.9770352Z RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_concat_cuda_batch_size_1_mid_dim_8_in_features_128_out_features_64_cuda_bfloat16! Caching allocator allocated memory was 24576 and is now reported as 49152 on device 0. CUDA driver allocated memory was 274726912 and is now 276824064. 2025-12-04T11:13:49.9771253Z 2025-12-04T11:13:49.9771382Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:49.9772126Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_concat_cuda_batch_size_1_mid_dim_8_in_features_128_out_features_64_cuda_bfloat16 2025-12-04T11:13:49.9772734Z 2025-12-04T11:13:49.9772899Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:49.9773263Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:49.9773576Z stats [('calls_captured', 18), ('unique_graphs', 1)] 2025-12-04T11:13:49.9774103Z inductor [('pattern_matcher_nodes', 18), ('woq_matcher_nodes', 12), ('pattern_matcher_count', 9), ('woq_matcher_count', 3), ('extern_calls', 3), ('fxgraph_cache_miss', 1)] 2025-12-04T11:13:49.9774740Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:49.9775076Z graph_break [] 2025-12-04T11:13:49.9775300Z ----------------------------- Captured stderr call ----------------------------- 2025-12-04T11:13:49.9776202Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/constant_folding.py:256: UserWarning: Unsupported unwinding pattern: Address not in range (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/profiler/unwind/unwind.cpp:219.) 2025-12-04T11:13:49.9777041Z if out == self.unknown_value: 2025-12-04T11:13:49.9777298Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:49.9777600Z stats [('calls_captured', 18), ('unique_graphs', 1)] 2025-12-04T11:13:49.9778130Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:49.9778686Z inductor [('pattern_matcher_nodes', 18), ('woq_matcher_nodes', 12), ('pattern_matcher_count', 9), ('woq_matcher_count', 3), ('extern_calls', 3), ('fxgraph_cache_miss', 1)] 2025-12-04T11:13:49.9779179Z graph_break [] 2025-12-04T11:13:49.9779365Z =================================== FAILURES =================================== 2025-12-04T11:13:49.9779843Z _ TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_concat_cuda_batch_size_1_mid_dim_8_in_features_128_out_features_64_cuda_bfloat16 _ 2025-12-04T11:13:49.9780305Z Traceback (most recent call last): 2025-12-04T11:13:49.9780751Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:49.9781194Z method(*args, **kwargs) 2025-12-04T11:13:49.9781601Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:49.9782038Z method(*args, **kwargs) 2025-12-04T11:13:49.9782442Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T11:13:49.9782875Z with policy(): 2025-12-04T11:13:49.9783271Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T11:13:49.9783775Z raise RuntimeError(msg) 2025-12-04T11:13:49.9784732Z RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_concat_cuda_batch_size_1_mid_dim_8_in_features_128_out_features_64_cuda_bfloat16! Caching allocator allocated memory was 49152 and is now reported as 73728 on device 0. CUDA driver allocated memory was 276824064 and is now 278921216. 2025-12-04T11:13:49.9785626Z 2025-12-04T11:13:49.9785762Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:49.9786496Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_concat_cuda_batch_size_1_mid_dim_8_in_features_128_out_features_64_cuda_bfloat16 2025-12-04T11:13:49.9787117Z 2025-12-04T11:13:49.9787280Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:49.9787654Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:49.9787967Z stats [('calls_captured', 18), ('unique_graphs', 1)] 2025-12-04T11:13:49.9788491Z inductor [('pattern_matcher_nodes', 18), ('woq_matcher_nodes', 12), ('pattern_matcher_count', 9), ('woq_matcher_count', 3), ('extern_calls', 3), ('fxgraph_cache_miss', 1)] 2025-12-04T11:13:49.9789050Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:49.9789321Z graph_break [] 2025-12-04T11:13:49.9789541Z ----------------------------- Captured stderr call ----------------------------- 2025-12-04T11:13:49.9790458Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/constant_folding.py:256: UserWarning: Unsupported unwinding pattern: Address not in range (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/profiler/unwind/unwind.cpp:219.) 2025-12-04T11:13:49.9791414Z if out == self.unknown_value: 2025-12-04T11:13:49.9791760Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:49.9792069Z stats [('calls_captured', 18), ('unique_graphs', 1)] 2025-12-04T11:13:49.9792370Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:49.9792928Z inductor [('pattern_matcher_nodes', 18), ('woq_matcher_nodes', 12), ('pattern_matcher_count', 9), ('woq_matcher_count', 3), ('extern_calls', 3), ('fxgraph_cache_miss', 1)] 2025-12-04T11:13:49.9793415Z graph_break [] 2025-12-04T11:13:49.9793631Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:49.9793929Z stats [('calls_captured', 18), ('unique_graphs', 1)] 2025-12-04T11:13:49.9794226Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:49.9794779Z inductor [('pattern_matcher_nodes', 18), ('woq_matcher_nodes', 12), ('pattern_matcher_count', 9), ('woq_matcher_count', 3), ('extern_calls', 3), ('fxgraph_cache_miss', 1)] 2025-12-04T11:13:49.9795269Z graph_break [] 2025-12-04T11:13:49.9795860Z - generated xml file: /var/lib/jenkins/workspace/test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-7a190e1b598a9023.xml - 2025-12-04T11:13:49.9796524Z =========================== short test summary info ============================ 2025-12-04T11:13:49.9798031Z FAILED [0.5723s] inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_concat_cuda_batch_size_1_mid_dim_8_in_features_128_out_features_64_cuda_bfloat16 - RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_concat_cuda_batch_size_1_mid_dim_8_in_features_128_out_features_64_cuda_bfloat16! Caching allocator allocated memory was 49152 and is now reported as 73728 on device 0. CUDA driver allocated memory was 276824064 and is now 278921216. 2025-12-04T11:13:49.9799408Z 2025-12-04T11:13:49.9799537Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:49.9800276Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_concat_cuda_batch_size_1_mid_dim_8_in_features_128_out_features_64_cuda_bfloat16 2025-12-04T11:13:49.9800888Z 2025-12-04T11:13:49.9801049Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:49.9801391Z !!!!!!!!!!!!!!!!!!!!!!!!!! stopping after 1 failures !!!!!!!!!!!!!!!!!!!!!!!!!!! 2025-12-04T11:13:49.9801694Z ================== 1 failed, 57 deselected, 2 rerun in 12.28s ================== 2025-12-04T11:13:49.9801950Z Got exit code 1 2025-12-04T11:13:49.9802535Z FAILED CONSISTENTLY: test/inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_concat_cuda_batch_size_1_mid_dim_8_in_features_128_out_features_64_cuda_bfloat16 2025-12-04T11:13:49.9803345Z Test failed consistently, continuing with the rest of the tests due to continue-through-error being set 2025-12-04T11:13:49.9803931Z W1204 10:29:05.510000 57582 site-packages/torch/_inductor/utils.py:1703] Not enough SMs to use max_autotune_gemm mode 2025-12-04T11:13:49.9804654Z Test results will be stored in test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-c45e67249599e6c2.xml 2025-12-04T11:13:49.9805224Z ============================= test session starts ============================== 2025-12-04T11:13:49.9805628Z platform linux -- Python 3.10.14, pytest-7.3.2, pluggy-1.6.0 -- /opt/conda/envs/py_3.10/bin/python 2025-12-04T11:13:49.9805980Z cachedir: .pytest_cache 2025-12-04T11:13:49.9806404Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow] 2025-12-04T11:13:49.9806865Z rootdir: /var/lib/jenkins/workspace 2025-12-04T11:13:49.9807075Z configfile: pytest.ini 2025-12-04T11:13:49.9807580Z plugins: hypothesis-6.56.4, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-14.0, subtests-0.13.1, xdist-3.3.1, xdoctest-1.3.0, typeguard-4.3.0 2025-12-04T11:13:49.9808181Z collecting ... collected 58 items / 2 deselected / 56 selected 2025-12-04T11:13:49.9808476Z stepcurrent: skipping 2 already run items. 2025-12-04T11:13:49.9808702Z Running 56 items in this shard 2025-12-04T11:13:49.9808831Z 2025-12-04T11:13:49.9809361Z inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_concat_cuda_batch_size_32_mid_dim_1_in_features_128_out_features_64_cuda_bfloat16 ('RERUN', {'yellow': True}) [2.0580s] [ 1%] 2025-12-04T11:13:49.9810445Z inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_concat_cuda_batch_size_32_mid_dim_1_in_features_128_out_features_64_cuda_bfloat16 ('RERUN', {'yellow': True}) [0.6098s] [ 1%] 2025-12-04T11:13:49.9811484Z inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_concat_cuda_batch_size_32_mid_dim_1_in_features_128_out_features_64_cuda_bfloat16 FAILED [0.6191s] [ 1%] 2025-12-04T11:13:49.9812026Z 2025-12-04T11:13:49.9812127Z ==================================== RERUNS ==================================== 2025-12-04T11:13:49.9812610Z _ TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_concat_cuda_batch_size_32_mid_dim_1_in_features_128_out_features_64_cuda_bfloat16 _ 2025-12-04T11:13:49.9813076Z Traceback (most recent call last): 2025-12-04T11:13:49.9813528Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:49.9813968Z method(*args, **kwargs) 2025-12-04T11:13:49.9814387Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:49.9814819Z method(*args, **kwargs) 2025-12-04T11:13:49.9815225Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T11:13:49.9815665Z with policy(): 2025-12-04T11:13:49.9816075Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T11:13:49.9816513Z raise RuntimeError(msg) 2025-12-04T11:13:49.9817461Z RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_concat_cuda_batch_size_32_mid_dim_1_in_features_128_out_features_64_cuda_bfloat16! Caching allocator allocated memory was 0 and is now reported as 24576 on device 0. CUDA driver allocated memory was 230686720 and is now 274726912. 2025-12-04T11:13:49.9818345Z 2025-12-04T11:13:49.9818477Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:49.9819223Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_concat_cuda_batch_size_32_mid_dim_1_in_features_128_out_features_64_cuda_bfloat16 2025-12-04T11:13:49.9819835Z 2025-12-04T11:13:49.9820005Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:49.9820378Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:49.9820683Z stats [('calls_captured', 18), ('unique_graphs', 1)] 2025-12-04T11:13:49.9821214Z inductor [('pattern_matcher_nodes', 18), ('woq_matcher_nodes', 12), ('pattern_matcher_count', 9), ('woq_matcher_count', 3), ('extern_calls', 3), ('fxgraph_cache_miss', 1)] 2025-12-04T11:13:49.9821769Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:49.9822036Z graph_break [] 2025-12-04T11:13:49.9822436Z _ TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_concat_cuda_batch_size_32_mid_dim_1_in_features_128_out_features_64_cuda_bfloat16 _ 2025-12-04T11:13:49.9822896Z Traceback (most recent call last): 2025-12-04T11:13:49.9823347Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:49.9824019Z method(*args, **kwargs) 2025-12-04T11:13:49.9824517Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:49.9824953Z method(*args, **kwargs) 2025-12-04T11:13:49.9825358Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T11:13:49.9825785Z with policy(): 2025-12-04T11:13:49.9826176Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T11:13:49.9826612Z raise RuntimeError(msg) 2025-12-04T11:13:49.9827561Z RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_concat_cuda_batch_size_32_mid_dim_1_in_features_128_out_features_64_cuda_bfloat16! Caching allocator allocated memory was 24576 and is now reported as 49152 on device 0. CUDA driver allocated memory was 274726912 and is now 276824064. 2025-12-04T11:13:49.9828465Z 2025-12-04T11:13:49.9828596Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:49.9829350Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_concat_cuda_batch_size_32_mid_dim_1_in_features_128_out_features_64_cuda_bfloat16 2025-12-04T11:13:49.9829962Z 2025-12-04T11:13:49.9830121Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:49.9830489Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:49.9830796Z stats [('calls_captured', 18), ('unique_graphs', 1)] 2025-12-04T11:13:49.9831331Z inductor [('pattern_matcher_nodes', 18), ('woq_matcher_nodes', 12), ('pattern_matcher_count', 9), ('woq_matcher_count', 3), ('extern_calls', 3), ('fxgraph_cache_miss', 1)] 2025-12-04T11:13:49.9831888Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:49.9832158Z graph_break [] 2025-12-04T11:13:49.9832376Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:49.9832675Z stats [('calls_captured', 18), ('unique_graphs', 1)] 2025-12-04T11:13:49.9832970Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:49.9833518Z inductor [('pattern_matcher_nodes', 18), ('woq_matcher_nodes', 12), ('pattern_matcher_count', 9), ('woq_matcher_count', 3), ('extern_calls', 3), ('fxgraph_cache_miss', 1)] 2025-12-04T11:13:49.9833993Z graph_break [] 2025-12-04T11:13:49.9834169Z =================================== FAILURES =================================== 2025-12-04T11:13:49.9834647Z _ TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_concat_cuda_batch_size_32_mid_dim_1_in_features_128_out_features_64_cuda_bfloat16 _ 2025-12-04T11:13:49.9835101Z Traceback (most recent call last): 2025-12-04T11:13:49.9835554Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:49.9836005Z method(*args, **kwargs) 2025-12-04T11:13:49.9836420Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:49.9836853Z method(*args, **kwargs) 2025-12-04T11:13:49.9837254Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T11:13:49.9837682Z with policy(): 2025-12-04T11:13:49.9838072Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T11:13:49.9838508Z raise RuntimeError(msg) 2025-12-04T11:13:49.9839454Z RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_concat_cuda_batch_size_32_mid_dim_1_in_features_128_out_features_64_cuda_bfloat16! Caching allocator allocated memory was 49152 and is now reported as 73728 on device 0. CUDA driver allocated memory was 276824064 and is now 278921216. 2025-12-04T11:13:49.9840432Z 2025-12-04T11:13:49.9840631Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:49.9841374Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_concat_cuda_batch_size_32_mid_dim_1_in_features_128_out_features_64_cuda_bfloat16 2025-12-04T11:13:49.9841982Z 2025-12-04T11:13:49.9842142Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:49.9842507Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:49.9842814Z stats [('calls_captured', 18), ('unique_graphs', 1)] 2025-12-04T11:13:49.9843332Z inductor [('pattern_matcher_nodes', 18), ('woq_matcher_nodes', 12), ('pattern_matcher_count', 9), ('woq_matcher_count', 3), ('extern_calls', 3), ('fxgraph_cache_miss', 1)] 2025-12-04T11:13:49.9843877Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:49.9844150Z graph_break [] 2025-12-04T11:13:49.9844373Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:49.9844675Z stats [('calls_captured', 18), ('unique_graphs', 1)] 2025-12-04T11:13:49.9844971Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:49.9845515Z inductor [('pattern_matcher_nodes', 18), ('woq_matcher_nodes', 12), ('pattern_matcher_count', 9), ('woq_matcher_count', 3), ('extern_calls', 3), ('fxgraph_cache_miss', 1)] 2025-12-04T11:13:49.9845994Z graph_break [] 2025-12-04T11:13:49.9846215Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:49.9846515Z stats [('calls_captured', 18), ('unique_graphs', 1)] 2025-12-04T11:13:49.9846807Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:49.9847342Z inductor [('pattern_matcher_nodes', 18), ('woq_matcher_nodes', 12), ('pattern_matcher_count', 9), ('woq_matcher_count', 3), ('extern_calls', 3), ('fxgraph_cache_miss', 1)] 2025-12-04T11:13:49.9847824Z graph_break [] 2025-12-04T11:13:49.9848407Z - generated xml file: /var/lib/jenkins/workspace/test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-c45e67249599e6c2.xml - 2025-12-04T11:13:49.9849059Z =========================== short test summary info ============================ 2025-12-04T11:13:49.9850560Z FAILED [0.6191s] inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_concat_cuda_batch_size_32_mid_dim_1_in_features_128_out_features_64_cuda_bfloat16 - RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_concat_cuda_batch_size_32_mid_dim_1_in_features_128_out_features_64_cuda_bfloat16! Caching allocator allocated memory was 49152 and is now reported as 73728 on device 0. CUDA driver allocated memory was 276824064 and is now 278921216. 2025-12-04T11:13:49.9851934Z 2025-12-04T11:13:49.9852066Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:49.9852808Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_concat_cuda_batch_size_32_mid_dim_1_in_features_128_out_features_64_cuda_bfloat16 2025-12-04T11:13:49.9853421Z 2025-12-04T11:13:49.9853579Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:49.9853919Z !!!!!!!!!!!!!!!!!!!!!!!!!! stopping after 1 failures !!!!!!!!!!!!!!!!!!!!!!!!!!! 2025-12-04T11:13:49.9854221Z =================== 1 failed, 2 deselected, 2 rerun in 3.31s =================== 2025-12-04T11:13:49.9854472Z Got exit code 1 2025-12-04T11:13:49.9854636Z Retrying single test... 2025-12-04T11:13:49.9855010Z W1204 10:29:15.630000 57764 site-packages/torch/_inductor/utils.py:1703] Not enough SMs to use max_autotune_gemm mode 2025-12-04T11:13:49.9855723Z Test results will be stored in test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-8112676a6b085940.xml 2025-12-04T11:13:49.9856426Z ============================= test session starts ============================== 2025-12-04T11:13:49.9856817Z platform linux -- Python 3.10.14, pytest-7.3.2, pluggy-1.6.0 -- /opt/conda/envs/py_3.10/bin/python 2025-12-04T11:13:49.9857166Z cachedir: .pytest_cache 2025-12-04T11:13:49.9857574Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow] 2025-12-04T11:13:49.9858031Z rootdir: /var/lib/jenkins/workspace 2025-12-04T11:13:49.9858256Z configfile: pytest.ini 2025-12-04T11:13:49.9858682Z plugins: hypothesis-6.56.4, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-14.0, subtests-0.13.1, xdist-3.3.1, xdoctest-1.3.0, typeguard-4.3.0 2025-12-04T11:13:49.9859202Z collecting ... collected 58 items / 57 deselected / 1 selected 2025-12-04T11:13:49.9860003Z stepcurrent: skipping 2 already run items. Running only test/inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_concat_cuda_batch_size_32_mid_dim_1_in_features_128_out_features_64_cuda_bfloat16 2025-12-04T11:13:49.9860737Z Running 1 items in this shard 2025-12-04T11:13:49.9860864Z 2025-12-04T11:13:49.9861613Z inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_concat_cuda_batch_size_32_mid_dim_1_in_features_128_out_features_64_cuda_bfloat16 [W1204 10:29:16.772520042 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:49.9862433Z 2025-12-04T11:13:49.9862734Z [W1204 10:29:25.829821682 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:49.9863115Z 2025-12-04T11:13:49.9863408Z [W1204 10:29:25.830109088 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:49.9863855Z 2025-12-04T11:13:49.9864153Z [W1204 10:29:25.830680181 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:49.9864527Z 2025-12-04T11:13:49.9864828Z [W1204 10:29:25.830883326 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:49.9865196Z 2025-12-04T11:13:49.9865491Z [W1204 10:29:25.832068221 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:49.9865856Z 2025-12-04T11:13:49.9866145Z [W1204 10:29:25.832243985 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:49.9866514Z 2025-12-04T11:13:49.9866805Z [W1204 10:29:25.832511371 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:49.9867174Z 2025-12-04T11:13:49.9867465Z [W1204 10:29:25.832685385 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:49.9867836Z 2025-12-04T11:13:49.9868133Z [W1204 10:29:25.841194322 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:49.9868502Z 2025-12-04T11:13:49.9868793Z [W1204 10:29:25.841404566 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:49.9869160Z 2025-12-04T11:13:49.9869463Z [W1204 10:29:25.841578180 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:49.9869831Z 2025-12-04T11:13:49.9870120Z [W1204 10:29:25.841813466 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:49.9870489Z 2025-12-04T11:13:49.9870777Z [W1204 10:29:25.841963569 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:49.9871222Z 2025-12-04T11:13:49.9871588Z [W1204 10:29:25.842213104 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:49.9871961Z 2025-12-04T11:13:49.9872255Z [W1204 10:29:25.842355638 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:49.9872626Z 2025-12-04T11:13:49.9872915Z [W1204 10:29:25.842581283 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:49.9873290Z 2025-12-04T11:13:49.9873578Z [W1204 10:29:25.842706665 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:49.9873956Z 2025-12-04T11:13:49.9874246Z [W1204 10:29:25.933714938 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:49.9874615Z 2025-12-04T11:13:49.9874913Z [W1204 10:29:25.933929573 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:49.9875289Z 2025-12-04T11:13:49.9875578Z [W1204 10:29:25.934080236 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:49.9875944Z 2025-12-04T11:13:49.9876234Z [W1204 10:29:25.934294001 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:49.9876607Z 2025-12-04T11:13:49.9876909Z [W1204 10:29:25.934422773 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:49.9877286Z 2025-12-04T11:13:49.9877577Z [W1204 10:29:25.934642208 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:49.9878173Z 2025-12-04T11:13:49.9878473Z [W1204 10:29:25.934768711 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:49.9878841Z 2025-12-04T11:13:49.9879135Z [W1204 10:29:25.934979005 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:49.9879505Z 2025-12-04T11:13:49.9879797Z [W1204 10:29:25.935103858 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:49.9880160Z 2025-12-04T11:13:49.9880244Z ('RERUN', {'yellow': True}) [11.1075s] [100%] 2025-12-04T11:13:49.9881149Z inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_concat_cuda_batch_size_32_mid_dim_1_in_features_128_out_features_64_cuda_bfloat16 [W1204 10:29:27.193476753 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:49.9881974Z 2025-12-04T11:13:49.9882271Z [W1204 10:29:27.193736999 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:49.9882637Z 2025-12-04T11:13:49.9882931Z [W1204 10:29:27.193895682 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:49.9883295Z 2025-12-04T11:13:49.9883591Z [W1204 10:29:27.194108857 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:49.9883956Z 2025-12-04T11:13:49.9884243Z [W1204 10:29:27.194237490 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:49.9884619Z 2025-12-04T11:13:49.9884909Z [W1204 10:29:27.194456975 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:49.9885423Z 2025-12-04T11:13:49.9885814Z [W1204 10:29:27.194582797 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:49.9886196Z 2025-12-04T11:13:49.9886494Z [W1204 10:29:27.194793132 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:49.9886863Z 2025-12-04T11:13:49.9887157Z [W1204 10:29:27.194913645 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:49.9887524Z 2025-12-04T11:13:49.9887813Z [W1204 10:29:27.201122932 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:49.9888179Z 2025-12-04T11:13:49.9888469Z [W1204 10:29:27.201294866 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:49.9888846Z 2025-12-04T11:13:49.9889151Z [W1204 10:29:27.201445059 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:49.9889516Z 2025-12-04T11:13:49.9889810Z [W1204 10:29:27.201650204 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:49.9890176Z 2025-12-04T11:13:49.9890473Z [W1204 10:29:27.201780336 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:49.9890838Z 2025-12-04T11:13:49.9891128Z [W1204 10:29:27.201995691 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:49.9891496Z 2025-12-04T11:13:49.9891786Z [W1204 10:29:27.202121134 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:49.9892155Z 2025-12-04T11:13:49.9892453Z [W1204 10:29:27.202325318 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:49.9892827Z 2025-12-04T11:13:49.9893120Z [W1204 10:29:27.202447441 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:49.9893489Z 2025-12-04T11:13:49.9893787Z [W1204 10:29:27.285747313 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:49.9894151Z 2025-12-04T11:13:49.9894436Z [W1204 10:29:27.285968027 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:49.9894816Z 2025-12-04T11:13:49.9895106Z [W1204 10:29:27.286117691 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:49.9895476Z 2025-12-04T11:13:49.9895766Z [W1204 10:29:27.286323805 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:49.9896171Z 2025-12-04T11:13:49.9896467Z [W1204 10:29:27.286447088 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:49.9896835Z 2025-12-04T11:13:49.9897125Z [W1204 10:29:27.286662443 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:49.9897490Z 2025-12-04T11:13:49.9897783Z [W1204 10:29:27.286786396 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:49.9898150Z 2025-12-04T11:13:49.9898441Z [W1204 10:29:27.286998090 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:49.9898808Z 2025-12-04T11:13:49.9899100Z [W1204 10:29:27.287117973 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:49.9899750Z 2025-12-04T11:13:49.9899899Z ('RERUN', {'yellow': True}) [0.5752s] [100%] 2025-12-04T11:13:49.9900809Z inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_concat_cuda_batch_size_32_mid_dim_1_in_features_128_out_features_64_cuda_bfloat16 [W1204 10:29:27.757978883 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:49.9901630Z 2025-12-04T11:13:49.9901922Z [W1204 10:29:27.758187438 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:49.9902290Z 2025-12-04T11:13:49.9902579Z [W1204 10:29:27.758341371 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:49.9902950Z 2025-12-04T11:13:49.9903239Z [W1204 10:29:27.758568736 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:49.9903667Z 2025-12-04T11:13:49.9903967Z [W1204 10:29:27.758699559 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:49.9904334Z 2025-12-04T11:13:49.9904626Z [W1204 10:29:27.758919084 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:49.9904989Z 2025-12-04T11:13:49.9905289Z [W1204 10:29:27.759044456 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:49.9905655Z 2025-12-04T11:13:49.9905941Z [W1204 10:29:27.759249581 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:49.9906315Z 2025-12-04T11:13:49.9906604Z [W1204 10:29:27.759373744 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:49.9906983Z 2025-12-04T11:13:49.9907277Z [W1204 10:29:27.765415627 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:49.9907643Z 2025-12-04T11:13:49.9907939Z [W1204 10:29:27.765587691 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:49.9908307Z 2025-12-04T11:13:49.9908603Z [W1204 10:29:27.765739744 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:49.9908971Z 2025-12-04T11:13:49.9909260Z [W1204 10:29:27.765948238 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:49.9909633Z 2025-12-04T11:13:49.9909927Z [W1204 10:29:27.766079261 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:49.9910308Z 2025-12-04T11:13:49.9910603Z [W1204 10:29:27.766295386 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:49.9910973Z 2025-12-04T11:13:49.9911267Z [W1204 10:29:27.766422739 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:49.9911636Z 2025-12-04T11:13:49.9911931Z [W1204 10:29:27.766628443 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:49.9912297Z 2025-12-04T11:13:49.9912589Z [W1204 10:29:27.766750326 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:49.9912960Z 2025-12-04T11:13:49.9913250Z [W1204 10:29:27.849427834 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:49.9913706Z 2025-12-04T11:13:49.9914063Z [W1204 10:29:27.849616898 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:49.9914437Z 2025-12-04T11:13:49.9914726Z [W1204 10:29:27.849764791 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:49.9915090Z 2025-12-04T11:13:49.9915385Z [W1204 10:29:27.849970716 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:49.9915750Z 2025-12-04T11:13:49.9916041Z [W1204 10:29:27.850113809 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:49.9916414Z 2025-12-04T11:13:49.9916701Z [W1204 10:29:27.850339044 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:49.9917077Z 2025-12-04T11:13:49.9917369Z [W1204 10:29:27.850463667 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:49.9917740Z 2025-12-04T11:13:49.9918027Z [W1204 10:29:27.850670601 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:49.9918392Z 2025-12-04T11:13:49.9918687Z [W1204 10:29:27.850794174 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:49.9919054Z 2025-12-04T11:13:49.9919125Z FAILED [0.5628s] [100%] 2025-12-04T11:13:49.9919237Z 2025-12-04T11:13:49.9919328Z ==================================== RERUNS ==================================== 2025-12-04T11:13:49.9919818Z _ TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_concat_cuda_batch_size_32_mid_dim_1_in_features_128_out_features_64_cuda_bfloat16 _ 2025-12-04T11:13:49.9920299Z Traceback (most recent call last): 2025-12-04T11:13:49.9920759Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:49.9921217Z method(*args, **kwargs) 2025-12-04T11:13:49.9921635Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:49.9922073Z method(*args, **kwargs) 2025-12-04T11:13:49.9922477Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T11:13:49.9922916Z with policy(): 2025-12-04T11:13:49.9923315Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T11:13:49.9923780Z raise RuntimeError(msg) 2025-12-04T11:13:49.9924734Z RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_concat_cuda_batch_size_32_mid_dim_1_in_features_128_out_features_64_cuda_bfloat16! Caching allocator allocated memory was 0 and is now reported as 24576 on device 0. CUDA driver allocated memory was 230686720 and is now 274726912. 2025-12-04T11:13:49.9925634Z 2025-12-04T11:13:49.9925768Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:49.9926531Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_concat_cuda_batch_size_32_mid_dim_1_in_features_128_out_features_64_cuda_bfloat16 2025-12-04T11:13:49.9927147Z 2025-12-04T11:13:49.9927319Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:49.9927691Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:49.9928010Z stats [('calls_captured', 18), ('unique_graphs', 1)] 2025-12-04T11:13:49.9928542Z inductor [('pattern_matcher_nodes', 18), ('woq_matcher_nodes', 12), ('pattern_matcher_count', 9), ('woq_matcher_count', 3), ('extern_calls', 3), ('fxgraph_cache_miss', 1)] 2025-12-04T11:13:49.9929191Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:49.9929527Z graph_break [] 2025-12-04T11:13:49.9929755Z ----------------------------- Captured stderr call ----------------------------- 2025-12-04T11:13:49.9930667Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/constant_folding.py:256: UserWarning: Unsupported unwinding pattern: Address not in range (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/profiler/unwind/unwind.cpp:219.) 2025-12-04T11:13:49.9931507Z if out == self.unknown_value: 2025-12-04T11:13:49.9931948Z _ TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_concat_cuda_batch_size_32_mid_dim_1_in_features_128_out_features_64_cuda_bfloat16 _ 2025-12-04T11:13:49.9932421Z Traceback (most recent call last): 2025-12-04T11:13:49.9932868Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:49.9933313Z method(*args, **kwargs) 2025-12-04T11:13:49.9933737Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:49.9934175Z method(*args, **kwargs) 2025-12-04T11:13:49.9934586Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T11:13:49.9935020Z with policy(): 2025-12-04T11:13:49.9935416Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T11:13:49.9935870Z raise RuntimeError(msg) 2025-12-04T11:13:49.9936812Z RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_concat_cuda_batch_size_32_mid_dim_1_in_features_128_out_features_64_cuda_bfloat16! Caching allocator allocated memory was 24576 and is now reported as 49152 on device 0. CUDA driver allocated memory was 274726912 and is now 276824064. 2025-12-04T11:13:49.9937717Z 2025-12-04T11:13:49.9937846Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:49.9938594Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_concat_cuda_batch_size_32_mid_dim_1_in_features_128_out_features_64_cuda_bfloat16 2025-12-04T11:13:49.9939207Z 2025-12-04T11:13:49.9939368Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:49.9939735Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:49.9940039Z stats [('calls_captured', 18), ('unique_graphs', 1)] 2025-12-04T11:13:49.9940566Z inductor [('pattern_matcher_nodes', 18), ('woq_matcher_nodes', 12), ('pattern_matcher_count', 9), ('woq_matcher_count', 3), ('extern_calls', 3), ('fxgraph_cache_miss', 1)] 2025-12-04T11:13:49.9941123Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:49.9941398Z graph_break [] 2025-12-04T11:13:49.9941615Z ----------------------------- Captured stderr call ----------------------------- 2025-12-04T11:13:49.9942517Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/constant_folding.py:256: UserWarning: Unsupported unwinding pattern: Address not in range (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/profiler/unwind/unwind.cpp:219.) 2025-12-04T11:13:49.9943348Z if out == self.unknown_value: 2025-12-04T11:13:49.9943669Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:49.9943974Z stats [('calls_captured', 18), ('unique_graphs', 1)] 2025-12-04T11:13:49.9944275Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:49.9944842Z inductor [('pattern_matcher_nodes', 18), ('woq_matcher_nodes', 12), ('pattern_matcher_count', 9), ('woq_matcher_count', 3), ('extern_calls', 3), ('fxgraph_cache_miss', 1)] 2025-12-04T11:13:49.9945432Z graph_break [] 2025-12-04T11:13:49.9945613Z =================================== FAILURES =================================== 2025-12-04T11:13:49.9946167Z _ TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_concat_cuda_batch_size_32_mid_dim_1_in_features_128_out_features_64_cuda_bfloat16 _ 2025-12-04T11:13:49.9946637Z Traceback (most recent call last): 2025-12-04T11:13:49.9947076Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:49.9947520Z method(*args, **kwargs) 2025-12-04T11:13:49.9947935Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:49.9948367Z method(*args, **kwargs) 2025-12-04T11:13:49.9948773Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T11:13:49.9949207Z with policy(): 2025-12-04T11:13:49.9949604Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T11:13:49.9950052Z raise RuntimeError(msg) 2025-12-04T11:13:49.9951013Z RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_concat_cuda_batch_size_32_mid_dim_1_in_features_128_out_features_64_cuda_bfloat16! Caching allocator allocated memory was 49152 and is now reported as 73728 on device 0. CUDA driver allocated memory was 276824064 and is now 278921216. 2025-12-04T11:13:49.9951914Z 2025-12-04T11:13:49.9952056Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:49.9952805Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_concat_cuda_batch_size_32_mid_dim_1_in_features_128_out_features_64_cuda_bfloat16 2025-12-04T11:13:49.9953416Z 2025-12-04T11:13:49.9953582Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:49.9953951Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:49.9954269Z stats [('calls_captured', 18), ('unique_graphs', 1)] 2025-12-04T11:13:49.9954874Z inductor [('pattern_matcher_nodes', 18), ('woq_matcher_nodes', 12), ('pattern_matcher_count', 9), ('woq_matcher_count', 3), ('extern_calls', 3), ('fxgraph_cache_miss', 1)] 2025-12-04T11:13:49.9993272Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:49.9993581Z graph_break [] 2025-12-04T11:13:49.9993821Z ----------------------------- Captured stderr call ----------------------------- 2025-12-04T11:13:49.9994765Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/constant_folding.py:256: UserWarning: Unsupported unwinding pattern: Address not in range (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/profiler/unwind/unwind.cpp:219.) 2025-12-04T11:13:49.9995628Z if out == self.unknown_value: 2025-12-04T11:13:49.9995910Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:49.9996236Z stats [('calls_captured', 18), ('unique_graphs', 1)] 2025-12-04T11:13:49.9996552Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:49.9997118Z inductor [('pattern_matcher_nodes', 18), ('woq_matcher_nodes', 12), ('pattern_matcher_count', 9), ('woq_matcher_count', 3), ('extern_calls', 3), ('fxgraph_cache_miss', 1)] 2025-12-04T11:13:49.9997606Z graph_break [] 2025-12-04T11:13:49.9997829Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:49.9998135Z stats [('calls_captured', 18), ('unique_graphs', 1)] 2025-12-04T11:13:49.9998430Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:49.9998978Z inductor [('pattern_matcher_nodes', 18), ('woq_matcher_nodes', 12), ('pattern_matcher_count', 9), ('woq_matcher_count', 3), ('extern_calls', 3), ('fxgraph_cache_miss', 1)] 2025-12-04T11:13:49.9999634Z graph_break [] 2025-12-04T11:13:50.0000337Z - generated xml file: /var/lib/jenkins/workspace/test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-8112676a6b085940.xml - 2025-12-04T11:13:50.0001012Z =========================== short test summary info ============================ 2025-12-04T11:13:50.0002538Z FAILED [0.5628s] inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_concat_cuda_batch_size_32_mid_dim_1_in_features_128_out_features_64_cuda_bfloat16 - RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_concat_cuda_batch_size_32_mid_dim_1_in_features_128_out_features_64_cuda_bfloat16! Caching allocator allocated memory was 49152 and is now reported as 73728 on device 0. CUDA driver allocated memory was 276824064 and is now 278921216. 2025-12-04T11:13:50.0003932Z 2025-12-04T11:13:50.0004064Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.0004821Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_concat_cuda_batch_size_32_mid_dim_1_in_features_128_out_features_64_cuda_bfloat16 2025-12-04T11:13:50.0005432Z 2025-12-04T11:13:50.0005599Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.0005942Z !!!!!!!!!!!!!!!!!!!!!!!!!! stopping after 1 failures !!!!!!!!!!!!!!!!!!!!!!!!!!! 2025-12-04T11:13:50.0006245Z ================== 1 failed, 57 deselected, 2 rerun in 12.27s ================== 2025-12-04T11:13:50.0006497Z Got exit code 1 2025-12-04T11:13:50.0006656Z Retrying single test... 2025-12-04T11:13:50.0007045Z W1204 10:29:34.681000 57951 site-packages/torch/_inductor/utils.py:1703] Not enough SMs to use max_autotune_gemm mode 2025-12-04T11:13:50.0007774Z Test results will be stored in test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-2280a7982827e03d.xml 2025-12-04T11:13:50.0008342Z ============================= test session starts ============================== 2025-12-04T11:13:50.0008747Z platform linux -- Python 3.10.14, pytest-7.3.2, pluggy-1.6.0 -- /opt/conda/envs/py_3.10/bin/python 2025-12-04T11:13:50.0009099Z cachedir: .pytest_cache 2025-12-04T11:13:50.0009527Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow] 2025-12-04T11:13:50.0009985Z rootdir: /var/lib/jenkins/workspace 2025-12-04T11:13:50.0010190Z configfile: pytest.ini 2025-12-04T11:13:50.0010616Z plugins: hypothesis-6.56.4, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-14.0, subtests-0.13.1, xdist-3.3.1, xdoctest-1.3.0, typeguard-4.3.0 2025-12-04T11:13:50.0011135Z collecting ... collected 58 items / 57 deselected / 1 selected 2025-12-04T11:13:50.0011937Z stepcurrent: skipping 2 already run items. Running only test/inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_concat_cuda_batch_size_32_mid_dim_1_in_features_128_out_features_64_cuda_bfloat16 2025-12-04T11:13:50.0012680Z Running 1 items in this shard 2025-12-04T11:13:50.0012810Z 2025-12-04T11:13:50.0013573Z inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_concat_cuda_batch_size_32_mid_dim_1_in_features_128_out_features_64_cuda_bfloat16 [W1204 10:29:35.834344587 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0014391Z 2025-12-04T11:13:50.0014692Z [W1204 10:29:45.119182739 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0015070Z 2025-12-04T11:13:50.0015362Z [W1204 10:29:45.119454185 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0015728Z 2025-12-04T11:13:50.0016017Z [W1204 10:29:45.120066139 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0016463Z 2025-12-04T11:13:50.0016819Z [W1204 10:29:45.120274173 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0017186Z 2025-12-04T11:13:50.0017478Z [W1204 10:29:45.121401078 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0017847Z 2025-12-04T11:13:50.0018134Z [W1204 10:29:45.121577232 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0018506Z 2025-12-04T11:13:50.0018793Z [W1204 10:29:45.121847308 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0019161Z 2025-12-04T11:13:50.0019449Z [W1204 10:29:45.122003211 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0019821Z 2025-12-04T11:13:50.0020121Z [W1204 10:29:45.130264753 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0020487Z 2025-12-04T11:13:50.0020773Z [W1204 10:29:45.130459717 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0021143Z 2025-12-04T11:13:50.0021431Z [W1204 10:29:45.130636681 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0021799Z 2025-12-04T11:13:50.0022085Z [W1204 10:29:45.130866006 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0022447Z 2025-12-04T11:13:50.0022739Z [W1204 10:29:45.131005829 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0023106Z 2025-12-04T11:13:50.0023398Z [W1204 10:29:45.131246754 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0023871Z 2025-12-04T11:13:50.0024158Z [W1204 10:29:45.131382887 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0024527Z 2025-12-04T11:13:50.0024813Z [W1204 10:29:45.131609592 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0025182Z 2025-12-04T11:13:50.0025467Z [W1204 10:29:45.131733665 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0025828Z 2025-12-04T11:13:50.0026118Z [W1204 10:29:45.220743623 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0026487Z 2025-12-04T11:13:50.0026780Z [W1204 10:29:45.220957608 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0027147Z 2025-12-04T11:13:50.0027435Z [W1204 10:29:45.221106211 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0027806Z 2025-12-04T11:13:50.0028099Z [W1204 10:29:45.221316946 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0028468Z 2025-12-04T11:13:50.0028754Z [W1204 10:29:45.221437069 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0029124Z 2025-12-04T11:13:50.0029409Z [W1204 10:29:45.221655494 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0029772Z 2025-12-04T11:13:50.0030144Z [W1204 10:29:45.221780466 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0030506Z 2025-12-04T11:13:50.0030949Z [W1204 10:29:45.221989201 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0031321Z 2025-12-04T11:13:50.0031607Z [W1204 10:29:45.222114104 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0031979Z 2025-12-04T11:13:50.0032062Z ('RERUN', {'yellow': True}) [11.3339s] [100%] 2025-12-04T11:13:50.0032966Z inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_concat_cuda_batch_size_32_mid_dim_1_in_features_128_out_features_64_cuda_bfloat16 [W1204 10:29:46.463849282 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0033784Z 2025-12-04T11:13:50.0034084Z [W1204 10:29:46.464080077 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0034455Z 2025-12-04T11:13:50.0034742Z [W1204 10:29:46.464229391 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0035110Z 2025-12-04T11:13:50.0035395Z [W1204 10:29:46.464439665 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0035761Z 2025-12-04T11:13:50.0036048Z [W1204 10:29:46.464562438 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0036415Z 2025-12-04T11:13:50.0036702Z [W1204 10:29:46.464790023 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0037068Z 2025-12-04T11:13:50.0037359Z [W1204 10:29:46.464912796 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0037727Z 2025-12-04T11:13:50.0038023Z [W1204 10:29:46.465116480 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0038388Z 2025-12-04T11:13:50.0038674Z [W1204 10:29:46.465238383 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0039038Z 2025-12-04T11:13:50.0039322Z [W1204 10:29:46.471500570 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0039689Z 2025-12-04T11:13:50.0039973Z [W1204 10:29:46.471700325 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0040337Z 2025-12-04T11:13:50.0040626Z [W1204 10:29:46.471849938 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0040994Z 2025-12-04T11:13:50.0041295Z [W1204 10:29:46.472053432 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0041660Z 2025-12-04T11:13:50.0041948Z [W1204 10:29:46.472177255 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0042313Z 2025-12-04T11:13:50.0042598Z [W1204 10:29:46.472390870 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0042972Z 2025-12-04T11:13:50.0043269Z [W1204 10:29:46.472517373 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0043644Z 2025-12-04T11:13:50.0043937Z [W1204 10:29:46.472732247 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0044389Z 2025-12-04T11:13:50.0044748Z [W1204 10:29:46.472860360 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0045117Z 2025-12-04T11:13:50.0045404Z [W1204 10:29:46.556677364 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0045772Z 2025-12-04T11:13:50.0046058Z [W1204 10:29:46.556905359 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0046423Z 2025-12-04T11:13:50.0046711Z [W1204 10:29:46.557056993 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0047078Z 2025-12-04T11:13:50.0047366Z [W1204 10:29:46.557264247 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0047738Z 2025-12-04T11:13:50.0048036Z [W1204 10:29:46.557383870 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0048398Z 2025-12-04T11:13:50.0048686Z [W1204 10:29:46.557596285 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0049070Z 2025-12-04T11:13:50.0049367Z [W1204 10:29:46.557717727 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0049737Z 2025-12-04T11:13:50.0050022Z [W1204 10:29:46.557920272 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0050392Z 2025-12-04T11:13:50.0050679Z [W1204 10:29:46.558038604 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0051050Z 2025-12-04T11:13:50.0051135Z ('RERUN', {'yellow': True}) [0.5649s] [100%] 2025-12-04T11:13:50.0052039Z inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_concat_cuda_batch_size_32_mid_dim_1_in_features_128_out_features_64_cuda_bfloat16 [W1204 10:29:47.024446946 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0052858Z 2025-12-04T11:13:50.0053148Z [W1204 10:29:47.024667531 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0053523Z 2025-12-04T11:13:50.0053811Z [W1204 10:29:47.024818054 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0054184Z 2025-12-04T11:13:50.0054469Z [W1204 10:29:47.025027899 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0054841Z 2025-12-04T11:13:50.0055136Z [W1204 10:29:47.025150501 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0055500Z 2025-12-04T11:13:50.0055791Z [W1204 10:29:47.025365226 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0056157Z 2025-12-04T11:13:50.0056448Z [W1204 10:29:47.025490429 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0056820Z 2025-12-04T11:13:50.0057108Z [W1204 10:29:47.025694973 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0057478Z 2025-12-04T11:13:50.0057765Z [W1204 10:29:47.025821896 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0058211Z 2025-12-04T11:13:50.0058588Z [W1204 10:29:47.031934220 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0058955Z 2025-12-04T11:13:50.0059246Z [W1204 10:29:47.032105304 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0059611Z 2025-12-04T11:13:50.0059898Z [W1204 10:29:47.032254247 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0060263Z 2025-12-04T11:13:50.0060551Z [W1204 10:29:47.032457492 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0060921Z 2025-12-04T11:13:50.0061208Z [W1204 10:29:47.032591345 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0061572Z 2025-12-04T11:13:50.0061873Z [W1204 10:29:47.032807489 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0062243Z 2025-12-04T11:13:50.0062532Z [W1204 10:29:47.032931242 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0062896Z 2025-12-04T11:13:50.0063183Z [W1204 10:29:47.033135077 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0063634Z 2025-12-04T11:13:50.0063926Z [W1204 10:29:47.033270550 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0064294Z 2025-12-04T11:13:50.0064581Z [W1204 10:29:47.116788217 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0064947Z 2025-12-04T11:13:50.0065238Z [W1204 10:29:47.116963901 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0065607Z 2025-12-04T11:13:50.0065904Z [W1204 10:29:47.117112814 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0066269Z 2025-12-04T11:13:50.0066557Z [W1204 10:29:47.117319489 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0066926Z 2025-12-04T11:13:50.0067213Z [W1204 10:29:47.117441852 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0067583Z 2025-12-04T11:13:50.0067872Z [W1204 10:29:47.117657667 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0068243Z 2025-12-04T11:13:50.0068530Z [W1204 10:29:47.117780519 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0068900Z 2025-12-04T11:13:50.0069194Z [W1204 10:29:47.117983404 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0069561Z 2025-12-04T11:13:50.0069853Z [W1204 10:29:47.118105186 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0070219Z 2025-12-04T11:13:50.0070282Z FAILED [0.5581s] [100%] 2025-12-04T11:13:50.0070393Z 2025-12-04T11:13:50.0070483Z ==================================== RERUNS ==================================== 2025-12-04T11:13:50.0070976Z _ TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_concat_cuda_batch_size_32_mid_dim_1_in_features_128_out_features_64_cuda_bfloat16 _ 2025-12-04T11:13:50.0071444Z Traceback (most recent call last): 2025-12-04T11:13:50.0071899Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.0072423Z method(*args, **kwargs) 2025-12-04T11:13:50.0072906Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.0073338Z method(*args, **kwargs) 2025-12-04T11:13:50.0073756Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T11:13:50.0074189Z with policy(): 2025-12-04T11:13:50.0074586Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T11:13:50.0075024Z raise RuntimeError(msg) 2025-12-04T11:13:50.0075961Z RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_concat_cuda_batch_size_32_mid_dim_1_in_features_128_out_features_64_cuda_bfloat16! Caching allocator allocated memory was 0 and is now reported as 24576 on device 0. CUDA driver allocated memory was 230686720 and is now 274726912. 2025-12-04T11:13:50.0076861Z 2025-12-04T11:13:50.0076997Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.0077743Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_concat_cuda_batch_size_32_mid_dim_1_in_features_128_out_features_64_cuda_bfloat16 2025-12-04T11:13:50.0078594Z 2025-12-04T11:13:50.0078763Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.0079139Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.0079463Z stats [('calls_captured', 18), ('unique_graphs', 1)] 2025-12-04T11:13:50.0079994Z inductor [('pattern_matcher_nodes', 18), ('woq_matcher_nodes', 12), ('pattern_matcher_count', 9), ('woq_matcher_count', 3), ('extern_calls', 3), ('fxgraph_cache_miss', 1)] 2025-12-04T11:13:50.0080555Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.0080819Z graph_break [] 2025-12-04T11:13:50.0081046Z ----------------------------- Captured stderr call ----------------------------- 2025-12-04T11:13:50.0081953Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/constant_folding.py:256: UserWarning: Unsupported unwinding pattern: Address not in range (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/profiler/unwind/unwind.cpp:219.) 2025-12-04T11:13:50.0082784Z if out == self.unknown_value: 2025-12-04T11:13:50.0083237Z _ TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_concat_cuda_batch_size_32_mid_dim_1_in_features_128_out_features_64_cuda_bfloat16 _ 2025-12-04T11:13:50.0083702Z Traceback (most recent call last): 2025-12-04T11:13:50.0084149Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.0084590Z method(*args, **kwargs) 2025-12-04T11:13:50.0085003Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.0085443Z method(*args, **kwargs) 2025-12-04T11:13:50.0085843Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T11:13:50.0086270Z with policy(): 2025-12-04T11:13:50.0086670Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T11:13:50.0087113Z raise RuntimeError(msg) 2025-12-04T11:13:50.0088060Z RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_concat_cuda_batch_size_32_mid_dim_1_in_features_128_out_features_64_cuda_bfloat16! Caching allocator allocated memory was 24576 and is now reported as 49152 on device 0. CUDA driver allocated memory was 274726912 and is now 276824064. 2025-12-04T11:13:50.0088962Z 2025-12-04T11:13:50.0089231Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.0090101Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_concat_cuda_batch_size_32_mid_dim_1_in_features_128_out_features_64_cuda_bfloat16 2025-12-04T11:13:50.0090714Z 2025-12-04T11:13:50.0090881Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.0091246Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.0091555Z stats [('calls_captured', 18), ('unique_graphs', 1)] 2025-12-04T11:13:50.0092085Z inductor [('pattern_matcher_nodes', 18), ('woq_matcher_nodes', 12), ('pattern_matcher_count', 9), ('woq_matcher_count', 3), ('extern_calls', 3), ('fxgraph_cache_miss', 1)] 2025-12-04T11:13:50.0092638Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.0092900Z graph_break [] 2025-12-04T11:13:50.0093119Z ----------------------------- Captured stderr call ----------------------------- 2025-12-04T11:13:50.0094034Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/constant_folding.py:256: UserWarning: Unsupported unwinding pattern: Address not in range (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/profiler/unwind/unwind.cpp:219.) 2025-12-04T11:13:50.0094869Z if out == self.unknown_value: 2025-12-04T11:13:50.0095115Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.0095417Z stats [('calls_captured', 18), ('unique_graphs', 1)] 2025-12-04T11:13:50.0095731Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.0096282Z inductor [('pattern_matcher_nodes', 18), ('woq_matcher_nodes', 12), ('pattern_matcher_count', 9), ('woq_matcher_count', 3), ('extern_calls', 3), ('fxgraph_cache_miss', 1)] 2025-12-04T11:13:50.0096765Z graph_break [] 2025-12-04T11:13:50.0096943Z =================================== FAILURES =================================== 2025-12-04T11:13:50.0097432Z _ TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_concat_cuda_batch_size_32_mid_dim_1_in_features_128_out_features_64_cuda_bfloat16 _ 2025-12-04T11:13:50.0097890Z Traceback (most recent call last): 2025-12-04T11:13:50.0098335Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.0098775Z method(*args, **kwargs) 2025-12-04T11:13:50.0099184Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.0099619Z method(*args, **kwargs) 2025-12-04T11:13:50.0100023Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T11:13:50.0100449Z with policy(): 2025-12-04T11:13:50.0100840Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T11:13:50.0101283Z raise RuntimeError(msg) 2025-12-04T11:13:50.0102253Z RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_concat_cuda_batch_size_32_mid_dim_1_in_features_128_out_features_64_cuda_bfloat16! Caching allocator allocated memory was 49152 and is now reported as 73728 on device 0. CUDA driver allocated memory was 276824064 and is now 278921216. 2025-12-04T11:13:50.0103158Z 2025-12-04T11:13:50.0103290Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.0104102Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_concat_cuda_batch_size_32_mid_dim_1_in_features_128_out_features_64_cuda_bfloat16 2025-12-04T11:13:50.0104714Z 2025-12-04T11:13:50.0104879Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.0105250Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.0105640Z stats [('calls_captured', 18), ('unique_graphs', 1)] 2025-12-04T11:13:50.0106234Z inductor [('pattern_matcher_nodes', 18), ('woq_matcher_nodes', 12), ('pattern_matcher_count', 9), ('woq_matcher_count', 3), ('extern_calls', 3), ('fxgraph_cache_miss', 1)] 2025-12-04T11:13:50.0106788Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.0107054Z graph_break [] 2025-12-04T11:13:50.0107275Z ----------------------------- Captured stderr call ----------------------------- 2025-12-04T11:13:50.0108170Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/constant_folding.py:256: UserWarning: Unsupported unwinding pattern: Address not in range (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/profiler/unwind/unwind.cpp:219.) 2025-12-04T11:13:50.0109003Z if out == self.unknown_value: 2025-12-04T11:13:50.0109269Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.0109575Z stats [('calls_captured', 18), ('unique_graphs', 1)] 2025-12-04T11:13:50.0109876Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.0110427Z inductor [('pattern_matcher_nodes', 18), ('woq_matcher_nodes', 12), ('pattern_matcher_count', 9), ('woq_matcher_count', 3), ('extern_calls', 3), ('fxgraph_cache_miss', 1)] 2025-12-04T11:13:50.0110906Z graph_break [] 2025-12-04T11:13:50.0111134Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.0111431Z stats [('calls_captured', 18), ('unique_graphs', 1)] 2025-12-04T11:13:50.0111722Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.0112275Z inductor [('pattern_matcher_nodes', 18), ('woq_matcher_nodes', 12), ('pattern_matcher_count', 9), ('woq_matcher_count', 3), ('extern_calls', 3), ('fxgraph_cache_miss', 1)] 2025-12-04T11:13:50.0112753Z graph_break [] 2025-12-04T11:13:50.0113330Z - generated xml file: /var/lib/jenkins/workspace/test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-2280a7982827e03d.xml - 2025-12-04T11:13:50.0114000Z =========================== short test summary info ============================ 2025-12-04T11:13:50.0115515Z FAILED [0.5581s] inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_concat_cuda_batch_size_32_mid_dim_1_in_features_128_out_features_64_cuda_bfloat16 - RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_concat_cuda_batch_size_32_mid_dim_1_in_features_128_out_features_64_cuda_bfloat16! Caching allocator allocated memory was 49152 and is now reported as 73728 on device 0. CUDA driver allocated memory was 276824064 and is now 278921216. 2025-12-04T11:13:50.0116896Z 2025-12-04T11:13:50.0117027Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.0117772Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_concat_cuda_batch_size_32_mid_dim_1_in_features_128_out_features_64_cuda_bfloat16 2025-12-04T11:13:50.0118395Z 2025-12-04T11:13:50.0118554Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.0118894Z !!!!!!!!!!!!!!!!!!!!!!!!!! stopping after 1 failures !!!!!!!!!!!!!!!!!!!!!!!!!!! 2025-12-04T11:13:50.0119198Z ================== 1 failed, 57 deselected, 2 rerun in 12.48s ================== 2025-12-04T11:13:50.0119449Z Got exit code 1 2025-12-04T11:13:50.0120055Z FAILED CONSISTENTLY: test/inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_concat_cuda_batch_size_32_mid_dim_1_in_features_128_out_features_64_cuda_bfloat16 2025-12-04T11:13:50.0120859Z Test failed consistently, continuing with the rest of the tests due to continue-through-error being set 2025-12-04T11:13:50.0121442Z W1204 10:29:53.992000 58138 site-packages/torch/_inductor/utils.py:1703] Not enough SMs to use max_autotune_gemm mode 2025-12-04T11:13:50.0122325Z Test results will be stored in test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-ecd8523a161788bb.xml 2025-12-04T11:13:50.0122890Z ============================= test session starts ============================== 2025-12-04T11:13:50.0123276Z platform linux -- Python 3.10.14, pytest-7.3.2, pluggy-1.6.0 -- /opt/conda/envs/py_3.10/bin/python 2025-12-04T11:13:50.0123645Z cachedir: .pytest_cache 2025-12-04T11:13:50.0124070Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow] 2025-12-04T11:13:50.0124549Z rootdir: /var/lib/jenkins/workspace 2025-12-04T11:13:50.0124763Z configfile: pytest.ini 2025-12-04T11:13:50.0125191Z plugins: hypothesis-6.56.4, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-14.0, subtests-0.13.1, xdist-3.3.1, xdoctest-1.3.0, typeguard-4.3.0 2025-12-04T11:13:50.0125717Z collecting ... collected 58 items / 3 deselected / 55 selected 2025-12-04T11:13:50.0126019Z stepcurrent: skipping 3 already run items. 2025-12-04T11:13:50.0126246Z Running 55 items in this shard 2025-12-04T11:13:50.0126376Z 2025-12-04T11:13:50.0126899Z inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_concat_cuda_batch_size_32_mid_dim_8_in_features_128_out_features_64_cuda_bfloat16 ('RERUN', {'yellow': True}) [2.0514s] [ 1%] 2025-12-04T11:13:50.0128006Z inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_concat_cuda_batch_size_32_mid_dim_8_in_features_128_out_features_64_cuda_bfloat16 ('RERUN', {'yellow': True}) [0.6134s] [ 1%] 2025-12-04T11:13:50.0129171Z inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_concat_cuda_batch_size_32_mid_dim_8_in_features_128_out_features_64_cuda_bfloat16 FAILED [0.6260s] [ 1%] 2025-12-04T11:13:50.0129722Z 2025-12-04T11:13:50.0129823Z ==================================== RERUNS ==================================== 2025-12-04T11:13:50.0130314Z _ TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_concat_cuda_batch_size_32_mid_dim_8_in_features_128_out_features_64_cuda_bfloat16 _ 2025-12-04T11:13:50.0130784Z Traceback (most recent call last): 2025-12-04T11:13:50.0131237Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.0131686Z method(*args, **kwargs) 2025-12-04T11:13:50.0132097Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.0132534Z method(*args, **kwargs) 2025-12-04T11:13:50.0132939Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T11:13:50.0133371Z with policy(): 2025-12-04T11:13:50.0133763Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T11:13:50.0134204Z raise RuntimeError(msg) 2025-12-04T11:13:50.0135152Z RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_concat_cuda_batch_size_32_mid_dim_8_in_features_128_out_features_64_cuda_bfloat16! Caching allocator allocated memory was 0 and is now reported as 24576 on device 0. CUDA driver allocated memory was 230686720 and is now 274726912. 2025-12-04T11:13:50.0136039Z 2025-12-04T11:13:50.0136174Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.0136917Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_concat_cuda_batch_size_32_mid_dim_8_in_features_128_out_features_64_cuda_bfloat16 2025-12-04T11:13:50.0137534Z 2025-12-04T11:13:50.0137692Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.0138060Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.0138494Z stats [('calls_captured', 18), ('unique_graphs', 1)] 2025-12-04T11:13:50.0139082Z inductor [('pattern_matcher_nodes', 18), ('woq_matcher_nodes', 12), ('pattern_matcher_count', 9), ('woq_matcher_count', 3), ('extern_calls', 3), ('fxgraph_cache_miss', 1)] 2025-12-04T11:13:50.0139650Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.0139919Z graph_break [] 2025-12-04T11:13:50.0140333Z _ TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_concat_cuda_batch_size_32_mid_dim_8_in_features_128_out_features_64_cuda_bfloat16 _ 2025-12-04T11:13:50.0140803Z Traceback (most recent call last): 2025-12-04T11:13:50.0141247Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.0141690Z method(*args, **kwargs) 2025-12-04T11:13:50.0142096Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.0142539Z method(*args, **kwargs) 2025-12-04T11:13:50.0142950Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T11:13:50.0143377Z with policy(): 2025-12-04T11:13:50.0143833Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T11:13:50.0144271Z raise RuntimeError(msg) 2025-12-04T11:13:50.0145222Z RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_concat_cuda_batch_size_32_mid_dim_8_in_features_128_out_features_64_cuda_bfloat16! Caching allocator allocated memory was 24576 and is now reported as 49152 on device 0. CUDA driver allocated memory was 274726912 and is now 276824064. 2025-12-04T11:13:50.0146121Z 2025-12-04T11:13:50.0146258Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.0146999Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_concat_cuda_batch_size_32_mid_dim_8_in_features_128_out_features_64_cuda_bfloat16 2025-12-04T11:13:50.0147624Z 2025-12-04T11:13:50.0147782Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.0148151Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.0148460Z stats [('calls_captured', 18), ('unique_graphs', 1)] 2025-12-04T11:13:50.0148982Z inductor [('pattern_matcher_nodes', 18), ('woq_matcher_nodes', 12), ('pattern_matcher_count', 9), ('woq_matcher_count', 3), ('extern_calls', 3), ('fxgraph_cache_miss', 1)] 2025-12-04T11:13:50.0149542Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.0149809Z graph_break [] 2025-12-04T11:13:50.0150031Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.0150330Z stats [('calls_captured', 18), ('unique_graphs', 1)] 2025-12-04T11:13:50.0150628Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.0151183Z inductor [('pattern_matcher_nodes', 18), ('woq_matcher_nodes', 12), ('pattern_matcher_count', 9), ('woq_matcher_count', 3), ('extern_calls', 3), ('fxgraph_cache_miss', 1)] 2025-12-04T11:13:50.0151658Z graph_break [] 2025-12-04T11:13:50.0151837Z =================================== FAILURES =================================== 2025-12-04T11:13:50.0152316Z _ TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_concat_cuda_batch_size_32_mid_dim_8_in_features_128_out_features_64_cuda_bfloat16 _ 2025-12-04T11:13:50.0152790Z Traceback (most recent call last): 2025-12-04T11:13:50.0153230Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.0153672Z method(*args, **kwargs) 2025-12-04T11:13:50.0154083Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.0154607Z method(*args, **kwargs) 2025-12-04T11:13:50.0155073Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T11:13:50.0155504Z with policy(): 2025-12-04T11:13:50.0155896Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T11:13:50.0156330Z raise RuntimeError(msg) 2025-12-04T11:13:50.0157285Z RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_concat_cuda_batch_size_32_mid_dim_8_in_features_128_out_features_64_cuda_bfloat16! Caching allocator allocated memory was 49152 and is now reported as 73728 on device 0. CUDA driver allocated memory was 276824064 and is now 278921216. 2025-12-04T11:13:50.0158184Z 2025-12-04T11:13:50.0158313Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.0159060Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_concat_cuda_batch_size_32_mid_dim_8_in_features_128_out_features_64_cuda_bfloat16 2025-12-04T11:13:50.0159676Z 2025-12-04T11:13:50.0159841Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.0160202Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.0160511Z stats [('calls_captured', 18), ('unique_graphs', 1)] 2025-12-04T11:13:50.0161035Z inductor [('pattern_matcher_nodes', 18), ('woq_matcher_nodes', 12), ('pattern_matcher_count', 9), ('woq_matcher_count', 3), ('extern_calls', 3), ('fxgraph_cache_miss', 1)] 2025-12-04T11:13:50.0161590Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.0161845Z graph_break [] 2025-12-04T11:13:50.0162062Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.0162364Z stats [('calls_captured', 18), ('unique_graphs', 1)] 2025-12-04T11:13:50.0162679Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.0163232Z inductor [('pattern_matcher_nodes', 18), ('woq_matcher_nodes', 12), ('pattern_matcher_count', 9), ('woq_matcher_count', 3), ('extern_calls', 3), ('fxgraph_cache_miss', 1)] 2025-12-04T11:13:50.0163711Z graph_break [] 2025-12-04T11:13:50.0163925Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.0164220Z stats [('calls_captured', 18), ('unique_graphs', 1)] 2025-12-04T11:13:50.0164509Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.0165054Z inductor [('pattern_matcher_nodes', 18), ('woq_matcher_nodes', 12), ('pattern_matcher_count', 9), ('woq_matcher_count', 3), ('extern_calls', 3), ('fxgraph_cache_miss', 1)] 2025-12-04T11:13:50.0165531Z graph_break [] 2025-12-04T11:13:50.0166109Z - generated xml file: /var/lib/jenkins/workspace/test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-ecd8523a161788bb.xml - 2025-12-04T11:13:50.0166779Z =========================== short test summary info ============================ 2025-12-04T11:13:50.0168114Z FAILED [0.6260s] inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_concat_cuda_batch_size_32_mid_dim_8_in_features_128_out_features_64_cuda_bfloat16 - RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_concat_cuda_batch_size_32_mid_dim_8_in_features_128_out_features_64_cuda_bfloat16! Caching allocator allocated memory was 49152 and is now reported as 73728 on device 0. CUDA driver allocated memory was 276824064 and is now 278921216. 2025-12-04T11:13:50.0168122Z 2025-12-04T11:13:50.0168249Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.0168791Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_concat_cuda_batch_size_32_mid_dim_8_in_features_128_out_features_64_cuda_bfloat16 2025-12-04T11:13:50.0168871Z 2025-12-04T11:13:50.0169096Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.0169212Z !!!!!!!!!!!!!!!!!!!!!!!!!! stopping after 1 failures !!!!!!!!!!!!!!!!!!!!!!!!!!! 2025-12-04T11:13:50.0169333Z =================== 1 failed, 3 deselected, 2 rerun in 3.32s =================== 2025-12-04T11:13:50.0169396Z Got exit code 1 2025-12-04T11:13:50.0169461Z Retrying single test... 2025-12-04T11:13:50.0169731Z W1204 10:30:04.100000 58320 site-packages/torch/_inductor/utils.py:1703] Not enough SMs to use max_autotune_gemm mode 2025-12-04T11:13:50.0170119Z Test results will be stored in test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-d90aab22855519e4.xml 2025-12-04T11:13:50.0170213Z ============================= test session starts ============================== 2025-12-04T11:13:50.0170431Z platform linux -- Python 3.10.14, pytest-7.3.2, pluggy-1.6.0 -- /opt/conda/envs/py_3.10/bin/python 2025-12-04T11:13:50.0170498Z cachedir: .pytest_cache 2025-12-04T11:13:50.0170814Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow] 2025-12-04T11:13:50.0170890Z rootdir: /var/lib/jenkins/workspace 2025-12-04T11:13:50.0170956Z configfile: pytest.ini 2025-12-04T11:13:50.0171277Z plugins: hypothesis-6.56.4, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-14.0, subtests-0.13.1, xdist-3.3.1, xdoctest-1.3.0, typeguard-4.3.0 2025-12-04T11:13:50.0171405Z collecting ... collected 58 items / 57 deselected / 1 selected 2025-12-04T11:13:50.0172009Z stepcurrent: skipping 3 already run items. Running only test/inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_concat_cuda_batch_size_32_mid_dim_8_in_features_128_out_features_64_cuda_bfloat16 2025-12-04T11:13:50.0172081Z Running 1 items in this shard 2025-12-04T11:13:50.0172089Z 2025-12-04T11:13:50.0172846Z inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_concat_cuda_batch_size_32_mid_dim_8_in_features_128_out_features_64_cuda_bfloat16 [W1204 10:30:05.233061951 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0172853Z 2025-12-04T11:13:50.0173154Z [W1204 10:30:14.360084689 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0173157Z 2025-12-04T11:13:50.0173449Z [W1204 10:30:14.360329754 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0173453Z 2025-12-04T11:13:50.0173746Z [W1204 10:30:14.360901967 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0173749Z 2025-12-04T11:13:50.0174035Z [W1204 10:30:14.361094251 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0174042Z 2025-12-04T11:13:50.0174335Z [W1204 10:30:14.362276596 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0174338Z 2025-12-04T11:13:50.0174625Z [W1204 10:30:14.362430849 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0174628Z 2025-12-04T11:13:50.0174917Z [W1204 10:30:14.362693854 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0174921Z 2025-12-04T11:13:50.0175210Z [W1204 10:30:14.362843217 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0175213Z 2025-12-04T11:13:50.0175499Z [W1204 10:30:14.371032791 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0175599Z 2025-12-04T11:13:50.0175955Z [W1204 10:30:14.371261406 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0175959Z 2025-12-04T11:13:50.0176247Z [W1204 10:30:14.371428119 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0176250Z 2025-12-04T11:13:50.0176540Z [W1204 10:30:14.371674574 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0176543Z 2025-12-04T11:13:50.0176830Z [W1204 10:30:14.371815477 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0176833Z 2025-12-04T11:13:50.0177124Z [W1204 10:30:14.372052832 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0177132Z 2025-12-04T11:13:50.0177425Z [W1204 10:30:14.372195015 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0177428Z 2025-12-04T11:13:50.0177719Z [W1204 10:30:14.372431850 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0177722Z 2025-12-04T11:13:50.0178302Z [W1204 10:30:14.372572913 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0178307Z 2025-12-04T11:13:50.0178616Z [W1204 10:30:14.459897420 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0178619Z 2025-12-04T11:13:50.0178912Z [W1204 10:30:14.460135105 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0178922Z 2025-12-04T11:13:50.0179221Z [W1204 10:30:14.460290688 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0179229Z 2025-12-04T11:13:50.0179514Z [W1204 10:30:14.460504333 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0179517Z 2025-12-04T11:13:50.0179807Z [W1204 10:30:14.460641496 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0179811Z 2025-12-04T11:13:50.0180108Z [W1204 10:30:14.460866430 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0180112Z 2025-12-04T11:13:50.0180403Z [W1204 10:30:14.460988023 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0180409Z 2025-12-04T11:13:50.0180703Z [W1204 10:30:14.461199348 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0180706Z 2025-12-04T11:13:50.0180993Z [W1204 10:30:14.461318090 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0180996Z 2025-12-04T11:13:50.0181086Z ('RERUN', {'yellow': True}) [11.1520s] [100%] 2025-12-04T11:13:50.0181839Z inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_concat_cuda_batch_size_32_mid_dim_8_in_features_128_out_features_64_cuda_bfloat16 [W1204 10:30:15.701577094 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0181843Z 2025-12-04T11:13:50.0182152Z [W1204 10:30:15.701830270 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0182286Z 2025-12-04T11:13:50.0182679Z [W1204 10:30:15.701984993 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0182683Z 2025-12-04T11:13:50.0182976Z [W1204 10:30:15.702199457 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0182983Z 2025-12-04T11:13:50.0183269Z [W1204 10:30:15.702328230 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0183273Z 2025-12-04T11:13:50.0183621Z [W1204 10:30:15.702548175 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0183624Z 2025-12-04T11:13:50.0183917Z [W1204 10:30:15.702671387 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0183926Z 2025-12-04T11:13:50.0184216Z [W1204 10:30:15.702876812 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0184220Z 2025-12-04T11:13:50.0184510Z [W1204 10:30:15.702999084 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0184513Z 2025-12-04T11:13:50.0184801Z [W1204 10:30:15.709208846 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0184804Z 2025-12-04T11:13:50.0185094Z [W1204 10:30:15.709382950 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0185098Z 2025-12-04T11:13:50.0185385Z [W1204 10:30:15.709528562 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0185389Z 2025-12-04T11:13:50.0185679Z [W1204 10:30:15.709737267 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0185690Z 2025-12-04T11:13:50.0185978Z [W1204 10:30:15.709862670 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0185982Z 2025-12-04T11:13:50.0186268Z [W1204 10:30:15.710102685 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0186271Z 2025-12-04T11:13:50.0186564Z [W1204 10:30:15.710234788 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0186567Z 2025-12-04T11:13:50.0186853Z [W1204 10:30:15.710449692 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0186856Z 2025-12-04T11:13:50.0187146Z [W1204 10:30:15.710573515 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0187155Z 2025-12-04T11:13:50.0187443Z [W1204 10:30:15.794872727 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0187446Z 2025-12-04T11:13:50.0187735Z [W1204 10:30:15.795099252 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0187738Z 2025-12-04T11:13:50.0188025Z [W1204 10:30:15.795248765 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0188028Z 2025-12-04T11:13:50.0188315Z [W1204 10:30:15.795456399 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0188318Z 2025-12-04T11:13:50.0188604Z [W1204 10:30:15.795582042 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0188682Z 2025-12-04T11:13:50.0189037Z [W1204 10:30:15.795794977 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0189045Z 2025-12-04T11:13:50.0189332Z [W1204 10:30:15.795914269 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0189335Z 2025-12-04T11:13:50.0189628Z [W1204 10:30:15.796114843 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0189631Z 2025-12-04T11:13:50.0189921Z [W1204 10:30:15.796233286 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0189924Z 2025-12-04T11:13:50.0190005Z ('RERUN', {'yellow': True}) [0.5672s] [100%] 2025-12-04T11:13:50.0190780Z inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_concat_cuda_batch_size_32_mid_dim_8_in_features_128_out_features_64_cuda_bfloat16 [W1204 10:30:16.263771586 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0190784Z 2025-12-04T11:13:50.0191074Z [W1204 10:30:16.263983210 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0191078Z 2025-12-04T11:13:50.0191366Z [W1204 10:30:16.264136003 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0191370Z 2025-12-04T11:13:50.0191653Z [W1204 10:30:16.264347898 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0191657Z 2025-12-04T11:13:50.0191950Z [W1204 10:30:16.264474640 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0191957Z 2025-12-04T11:13:50.0192246Z [W1204 10:30:16.264709325 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0192249Z 2025-12-04T11:13:50.0192535Z [W1204 10:30:16.264836248 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0192545Z 2025-12-04T11:13:50.0192830Z [W1204 10:30:16.265042963 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0192833Z 2025-12-04T11:13:50.0193120Z [W1204 10:30:16.265165875 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0193123Z 2025-12-04T11:13:50.0193414Z [W1204 10:30:16.271364117 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0193421Z 2025-12-04T11:13:50.0193709Z [W1204 10:30:16.271537240 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0193713Z 2025-12-04T11:13:50.0194003Z [W1204 10:30:16.271685583 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0194006Z 2025-12-04T11:13:50.0194294Z [W1204 10:30:16.271892908 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0194298Z 2025-12-04T11:13:50.0194587Z [W1204 10:30:16.272015780 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0194590Z 2025-12-04T11:13:50.0194876Z [W1204 10:30:16.272232695 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0194954Z 2025-12-04T11:13:50.0195336Z [W1204 10:30:16.272355598 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0195342Z 2025-12-04T11:13:50.0195632Z [W1204 10:30:16.272561742 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0195636Z 2025-12-04T11:13:50.0195924Z [W1204 10:30:16.272694235 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0195927Z 2025-12-04T11:13:50.0196218Z [W1204 10:30:16.356802293 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0196221Z 2025-12-04T11:13:50.0196507Z [W1204 10:30:16.356994697 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0196515Z 2025-12-04T11:13:50.0196807Z [W1204 10:30:16.357143170 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0196810Z 2025-12-04T11:13:50.0197098Z [W1204 10:30:16.357353595 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0197101Z 2025-12-04T11:13:50.0197391Z [W1204 10:30:16.357478007 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0197394Z 2025-12-04T11:13:50.0197684Z [W1204 10:30:16.357692202 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0197687Z 2025-12-04T11:13:50.0197984Z [W1204 10:30:16.357815784 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0197991Z 2025-12-04T11:13:50.0198277Z [W1204 10:30:16.358021459 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0198284Z 2025-12-04T11:13:50.0198571Z [W1204 10:30:16.358141691 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0198576Z 2025-12-04T11:13:50.0198641Z FAILED [0.5606s] [100%] 2025-12-04T11:13:50.0198645Z 2025-12-04T11:13:50.0198734Z ==================================== RERUNS ==================================== 2025-12-04T11:13:50.0199060Z _ TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_concat_cuda_batch_size_32_mid_dim_8_in_features_128_out_features_64_cuda_bfloat16 _ 2025-12-04T11:13:50.0199138Z Traceback (most recent call last): 2025-12-04T11:13:50.0199454Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.0199528Z method(*args, **kwargs) 2025-12-04T11:13:50.0199829Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.0199898Z method(*args, **kwargs) 2025-12-04T11:13:50.0200191Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T11:13:50.0200252Z with policy(): 2025-12-04T11:13:50.0200551Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T11:13:50.0200616Z raise RuntimeError(msg) 2025-12-04T11:13:50.0201447Z RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_concat_cuda_batch_size_32_mid_dim_8_in_features_128_out_features_64_cuda_bfloat16! Caching allocator allocated memory was 0 and is now reported as 24576 on device 0. CUDA driver allocated memory was 230686720 and is now 274726912. 2025-12-04T11:13:50.0201525Z 2025-12-04T11:13:50.0201663Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.0202269Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_concat_cuda_batch_size_32_mid_dim_8_in_features_128_out_features_64_cuda_bfloat16 2025-12-04T11:13:50.0202279Z 2025-12-04T11:13:50.0202445Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.0202580Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.0202683Z stats [('calls_captured', 18), ('unique_graphs', 1)] 2025-12-04T11:13:50.0203041Z inductor [('pattern_matcher_nodes', 18), ('woq_matcher_nodes', 12), ('pattern_matcher_count', 9), ('woq_matcher_count', 3), ('extern_calls', 3), ('fxgraph_cache_miss', 1)] 2025-12-04T11:13:50.0203169Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.0203239Z graph_break [] 2025-12-04T11:13:50.0203372Z ----------------------------- Captured stderr call ----------------------------- 2025-12-04T11:13:50.0204086Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/constant_folding.py:256: UserWarning: Unsupported unwinding pattern: Address not in range (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/profiler/unwind/unwind.cpp:219.) 2025-12-04T11:13:50.0204166Z if out == self.unknown_value: 2025-12-04T11:13:50.0204480Z _ TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_concat_cuda_batch_size_32_mid_dim_8_in_features_128_out_features_64_cuda_bfloat16 _ 2025-12-04T11:13:50.0204565Z Traceback (most recent call last): 2025-12-04T11:13:50.0204870Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.0204942Z method(*args, **kwargs) 2025-12-04T11:13:50.0205241Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.0205310Z method(*args, **kwargs) 2025-12-04T11:13:50.0205607Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T11:13:50.0205669Z with policy(): 2025-12-04T11:13:50.0205966Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T11:13:50.0206041Z raise RuntimeError(msg) 2025-12-04T11:13:50.0206872Z RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_concat_cuda_batch_size_32_mid_dim_8_in_features_128_out_features_64_cuda_bfloat16! Caching allocator allocated memory was 24576 and is now reported as 49152 on device 0. CUDA driver allocated memory was 274726912 and is now 276824064. 2025-12-04T11:13:50.0206876Z 2025-12-04T11:13:50.0207013Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.0207555Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_concat_cuda_batch_size_32_mid_dim_8_in_features_128_out_features_64_cuda_bfloat16 2025-12-04T11:13:50.0207561Z 2025-12-04T11:13:50.0207727Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.0207859Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.0207958Z stats [('calls_captured', 18), ('unique_graphs', 1)] 2025-12-04T11:13:50.0208317Z inductor [('pattern_matcher_nodes', 18), ('woq_matcher_nodes', 12), ('pattern_matcher_count', 9), ('woq_matcher_count', 3), ('extern_calls', 3), ('fxgraph_cache_miss', 1)] 2025-12-04T11:13:50.0208446Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.0208507Z graph_break [] 2025-12-04T11:13:50.0208641Z ----------------------------- Captured stderr call ----------------------------- 2025-12-04T11:13:50.0209415Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/constant_folding.py:256: UserWarning: Unsupported unwinding pattern: Address not in range (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/profiler/unwind/unwind.cpp:219.) 2025-12-04T11:13:50.0209559Z if out == self.unknown_value: 2025-12-04T11:13:50.0209685Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.0209780Z stats [('calls_captured', 18), ('unique_graphs', 1)] 2025-12-04T11:13:50.0209908Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.0210259Z inductor [('pattern_matcher_nodes', 18), ('woq_matcher_nodes', 12), ('pattern_matcher_count', 9), ('woq_matcher_count', 3), ('extern_calls', 3), ('fxgraph_cache_miss', 1)] 2025-12-04T11:13:50.0210324Z graph_break [] 2025-12-04T11:13:50.0210409Z =================================== FAILURES =================================== 2025-12-04T11:13:50.0210725Z _ TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_concat_cuda_batch_size_32_mid_dim_8_in_features_128_out_features_64_cuda_bfloat16 _ 2025-12-04T11:13:50.0210811Z Traceback (most recent call last): 2025-12-04T11:13:50.0211125Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.0211195Z method(*args, **kwargs) 2025-12-04T11:13:50.0211496Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.0211561Z method(*args, **kwargs) 2025-12-04T11:13:50.0211853Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T11:13:50.0211914Z with policy(): 2025-12-04T11:13:50.0212208Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T11:13:50.0212284Z raise RuntimeError(msg) 2025-12-04T11:13:50.0213114Z RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_concat_cuda_batch_size_32_mid_dim_8_in_features_128_out_features_64_cuda_bfloat16! Caching allocator allocated memory was 49152 and is now reported as 73728 on device 0. CUDA driver allocated memory was 276824064 and is now 278921216. 2025-12-04T11:13:50.0213123Z 2025-12-04T11:13:50.0213258Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.0213797Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_concat_cuda_batch_size_32_mid_dim_8_in_features_128_out_features_64_cuda_bfloat16 2025-12-04T11:13:50.0213801Z 2025-12-04T11:13:50.0213965Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.0214096Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.0214192Z stats [('calls_captured', 18), ('unique_graphs', 1)] 2025-12-04T11:13:50.0214551Z inductor [('pattern_matcher_nodes', 18), ('woq_matcher_nodes', 12), ('pattern_matcher_count', 9), ('woq_matcher_count', 3), ('extern_calls', 3), ('fxgraph_cache_miss', 1)] 2025-12-04T11:13:50.0214676Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.0214737Z graph_break [] 2025-12-04T11:13:50.0214868Z ----------------------------- Captured stderr call ----------------------------- 2025-12-04T11:13:50.0215555Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/constant_folding.py:256: UserWarning: Unsupported unwinding pattern: Address not in range (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/profiler/unwind/unwind.cpp:219.) 2025-12-04T11:13:50.0215634Z if out == self.unknown_value: 2025-12-04T11:13:50.0215759Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.0215855Z stats [('calls_captured', 18), ('unique_graphs', 1)] 2025-12-04T11:13:50.0215984Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.0216551Z inductor [('pattern_matcher_nodes', 18), ('woq_matcher_nodes', 12), ('pattern_matcher_count', 9), ('woq_matcher_count', 3), ('extern_calls', 3), ('fxgraph_cache_miss', 1)] 2025-12-04T11:13:50.0216618Z graph_break [] 2025-12-04T11:13:50.0216744Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.0216838Z stats [('calls_captured', 18), ('unique_graphs', 1)] 2025-12-04T11:13:50.0216966Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.0217313Z inductor [('pattern_matcher_nodes', 18), ('woq_matcher_nodes', 12), ('pattern_matcher_count', 9), ('woq_matcher_count', 3), ('extern_calls', 3), ('fxgraph_cache_miss', 1)] 2025-12-04T11:13:50.0217372Z graph_break [] 2025-12-04T11:13:50.0217864Z - generated xml file: /var/lib/jenkins/workspace/test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-d90aab22855519e4.xml - 2025-12-04T11:13:50.0217969Z =========================== short test summary info ============================ 2025-12-04T11:13:50.0219315Z FAILED [0.5606s] inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_concat_cuda_batch_size_32_mid_dim_8_in_features_128_out_features_64_cuda_bfloat16 - RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_concat_cuda_batch_size_32_mid_dim_8_in_features_128_out_features_64_cuda_bfloat16! Caching allocator allocated memory was 49152 and is now reported as 73728 on device 0. CUDA driver allocated memory was 276824064 and is now 278921216. 2025-12-04T11:13:50.0219320Z 2025-12-04T11:13:50.0219447Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.0219990Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_concat_cuda_batch_size_32_mid_dim_8_in_features_128_out_features_64_cuda_bfloat16 2025-12-04T11:13:50.0219998Z 2025-12-04T11:13:50.0220157Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.0220267Z !!!!!!!!!!!!!!!!!!!!!!!!!! stopping after 1 failures !!!!!!!!!!!!!!!!!!!!!!!!!!! 2025-12-04T11:13:50.0220392Z ================== 1 failed, 57 deselected, 2 rerun in 12.30s ================== 2025-12-04T11:13:50.0220453Z Got exit code 1 2025-12-04T11:13:50.0220528Z Retrying single test... 2025-12-04T11:13:50.0220795Z W1204 10:30:23.258000 58507 site-packages/torch/_inductor/utils.py:1703] Not enough SMs to use max_autotune_gemm mode 2025-12-04T11:13:50.0221192Z Test results will be stored in test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-5cb074eac86f05e5.xml 2025-12-04T11:13:50.0221294Z ============================= test session starts ============================== 2025-12-04T11:13:50.0221507Z platform linux -- Python 3.10.14, pytest-7.3.2, pluggy-1.6.0 -- /opt/conda/envs/py_3.10/bin/python 2025-12-04T11:13:50.0221584Z cachedir: .pytest_cache 2025-12-04T11:13:50.0221900Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow] 2025-12-04T11:13:50.0221981Z rootdir: /var/lib/jenkins/workspace 2025-12-04T11:13:50.0222056Z configfile: pytest.ini 2025-12-04T11:13:50.0222375Z plugins: hypothesis-6.56.4, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-14.0, subtests-0.13.1, xdist-3.3.1, xdoctest-1.3.0, typeguard-4.3.0 2025-12-04T11:13:50.0222513Z collecting ... collected 58 items / 57 deselected / 1 selected 2025-12-04T11:13:50.0223111Z stepcurrent: skipping 3 already run items. Running only test/inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_concat_cuda_batch_size_32_mid_dim_8_in_features_128_out_features_64_cuda_bfloat16 2025-12-04T11:13:50.0223185Z Running 1 items in this shard 2025-12-04T11:13:50.0223189Z 2025-12-04T11:13:50.0224087Z inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_concat_cuda_batch_size_32_mid_dim_8_in_features_128_out_features_64_cuda_bfloat16 [W1204 10:30:24.382403260 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0224172Z 2025-12-04T11:13:50.0224480Z [W1204 10:30:33.529984157 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0224484Z 2025-12-04T11:13:50.0224781Z [W1204 10:30:33.530273874 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0224784Z 2025-12-04T11:13:50.0225073Z [W1204 10:30:33.530853146 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0225076Z 2025-12-04T11:13:50.0225370Z [W1204 10:30:33.531035020 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0225380Z 2025-12-04T11:13:50.0225673Z [W1204 10:30:33.532280818 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0225677Z 2025-12-04T11:13:50.0225967Z [W1204 10:30:33.532480192 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0225976Z 2025-12-04T11:13:50.0226264Z [W1204 10:30:33.532810229 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0226267Z 2025-12-04T11:13:50.0226556Z [W1204 10:30:33.532966933 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0226559Z 2025-12-04T11:13:50.0226853Z [W1204 10:30:33.541268455 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0226860Z 2025-12-04T11:13:50.0227155Z [W1204 10:30:33.541482050 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0227158Z 2025-12-04T11:13:50.0227454Z [W1204 10:30:33.541652944 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0227458Z 2025-12-04T11:13:50.0227746Z [W1204 10:30:33.541884329 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0227750Z 2025-12-04T11:13:50.0228047Z [W1204 10:30:33.542022392 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0228050Z 2025-12-04T11:13:50.0228340Z [W1204 10:30:33.542262267 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0228346Z 2025-12-04T11:13:50.0228643Z [W1204 10:30:33.542403280 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0228647Z 2025-12-04T11:13:50.0228935Z [W1204 10:30:33.542637855 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0228938Z 2025-12-04T11:13:50.0229235Z [W1204 10:30:33.542776188 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0229239Z 2025-12-04T11:13:50.0229532Z [W1204 10:30:33.630288914 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0229535Z 2025-12-04T11:13:50.0229825Z [W1204 10:30:33.630496579 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0229828Z 2025-12-04T11:13:50.0230213Z [W1204 10:30:33.630644932 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0230282Z 2025-12-04T11:13:50.0230577Z [W1204 10:30:33.630862427 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0230580Z 2025-12-04T11:13:50.0230877Z [W1204 10:30:33.630989949 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0230881Z 2025-12-04T11:13:50.0231169Z [W1204 10:30:33.631206914 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0231172Z 2025-12-04T11:13:50.0231467Z [W1204 10:30:33.631330817 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0231470Z 2025-12-04T11:13:50.0231761Z [W1204 10:30:33.631536832 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0231769Z 2025-12-04T11:13:50.0232060Z [W1204 10:30:33.631659164 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0232069Z 2025-12-04T11:13:50.0232154Z ('RERUN', {'yellow': True}) [11.1724s] [100%] 2025-12-04T11:13:50.0232903Z inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_concat_cuda_batch_size_32_mid_dim_8_in_features_128_out_features_64_cuda_bfloat16 [W1204 10:30:34.875109340 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0232906Z 2025-12-04T11:13:50.0233205Z [W1204 10:30:34.875336865 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0233208Z 2025-12-04T11:13:50.0233496Z [W1204 10:30:34.875489619 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0233503Z 2025-12-04T11:13:50.0233802Z [W1204 10:30:34.875699953 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0233806Z 2025-12-04T11:13:50.0234095Z [W1204 10:30:34.875831076 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0234098Z 2025-12-04T11:13:50.0234391Z [W1204 10:30:34.876046881 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0234394Z 2025-12-04T11:13:50.0234685Z [W1204 10:30:34.876170144 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0234689Z 2025-12-04T11:13:50.0234981Z [W1204 10:30:34.876371028 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0234989Z 2025-12-04T11:13:50.0235278Z [W1204 10:30:34.876492211 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0235282Z 2025-12-04T11:13:50.0235568Z [W1204 10:30:34.882492703 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0235578Z 2025-12-04T11:13:50.0235865Z [W1204 10:30:34.882661246 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0235868Z 2025-12-04T11:13:50.0236154Z [W1204 10:30:34.882808470 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0236157Z 2025-12-04T11:13:50.0236448Z [W1204 10:30:34.883006624 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0236521Z 2025-12-04T11:13:50.0236877Z [W1204 10:30:34.883127207 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0236880Z 2025-12-04T11:13:50.0237174Z [W1204 10:30:34.883335151 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0237178Z 2025-12-04T11:13:50.0237465Z [W1204 10:30:34.883456254 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0237468Z 2025-12-04T11:13:50.0237760Z [W1204 10:30:34.883657418 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0237763Z 2025-12-04T11:13:50.0238054Z [W1204 10:30:34.883777161 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0238061Z 2025-12-04T11:13:50.0238360Z [W1204 10:30:35.964925136 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0238363Z 2025-12-04T11:13:50.0238655Z [W1204 10:30:35.965148331 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0238658Z 2025-12-04T11:13:50.0238945Z [W1204 10:30:35.965297174 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0238949Z 2025-12-04T11:13:50.0239240Z [W1204 10:30:35.965504069 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0239243Z 2025-12-04T11:13:50.0239530Z [W1204 10:30:35.965627471 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0239537Z 2025-12-04T11:13:50.0239830Z [W1204 10:30:35.965839906 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0239833Z 2025-12-04T11:13:50.0240119Z [W1204 10:30:35.965962289 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0240123Z 2025-12-04T11:13:50.0240416Z [W1204 10:30:35.966163274 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0240420Z 2025-12-04T11:13:50.0240708Z [W1204 10:30:35.966282506 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0240711Z 2025-12-04T11:13:50.0240810Z ('RERUN', {'yellow': True}) [0.5592s] [100%] 2025-12-04T11:13:50.0241565Z inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_concat_cuda_batch_size_32_mid_dim_8_in_features_128_out_features_64_cuda_bfloat16 [W1204 10:30:35.430180103 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0241573Z 2025-12-04T11:13:50.0241866Z [W1204 10:30:35.430392587 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0241874Z 2025-12-04T11:13:50.0242165Z [W1204 10:30:35.430541880 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0242168Z 2025-12-04T11:13:50.0242457Z [W1204 10:30:35.430751525 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0242460Z 2025-12-04T11:13:50.0242752Z [W1204 10:30:35.430889378 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0242828Z 2025-12-04T11:13:50.0243210Z [W1204 10:30:35.431104463 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0243213Z 2025-12-04T11:13:50.0243509Z [W1204 10:30:35.431227346 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0243512Z 2025-12-04T11:13:50.0243802Z [W1204 10:30:35.431426130 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0243805Z 2025-12-04T11:13:50.0244096Z [W1204 10:30:35.431546143 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0244099Z 2025-12-04T11:13:50.0244387Z [W1204 10:30:35.437409031 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0244395Z 2025-12-04T11:13:50.0244691Z [W1204 10:30:35.437574535 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0244700Z 2025-12-04T11:13:50.0244986Z [W1204 10:30:35.437722538 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0244990Z 2025-12-04T11:13:50.0245279Z [W1204 10:30:35.437923493 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0245287Z 2025-12-04T11:13:50.0245574Z [W1204 10:30:35.438044455 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0245577Z 2025-12-04T11:13:50.0245863Z [W1204 10:30:35.438253110 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0245866Z 2025-12-04T11:13:50.0246167Z [W1204 10:30:35.438375613 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0246171Z 2025-12-04T11:13:50.0246462Z [W1204 10:30:35.438575187 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0246465Z 2025-12-04T11:13:50.0246758Z [W1204 10:30:35.438694710 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0246762Z 2025-12-04T11:13:50.0247049Z [W1204 10:30:35.519520577 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0247052Z 2025-12-04T11:13:50.0247345Z [W1204 10:30:35.519694761 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0247348Z 2025-12-04T11:13:50.0247638Z [W1204 10:30:35.519841384 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0247644Z 2025-12-04T11:13:50.0247934Z [W1204 10:30:35.520063189 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0247943Z 2025-12-04T11:13:50.0248233Z [W1204 10:30:35.520189072 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0248235Z 2025-12-04T11:13:50.0248524Z [W1204 10:30:35.520403586 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0248527Z 2025-12-04T11:13:50.0248825Z [W1204 10:30:35.520523499 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0248828Z 2025-12-04T11:13:50.0249120Z [W1204 10:30:35.520733934 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0249213Z 2025-12-04T11:13:50.0249574Z [W1204 10:30:35.520852216 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0249578Z 2025-12-04T11:13:50.0249643Z FAILED [0.5547s] [100%] 2025-12-04T11:13:50.0249646Z 2025-12-04T11:13:50.0249753Z ==================================== RERUNS ==================================== 2025-12-04T11:13:50.0250072Z _ TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_concat_cuda_batch_size_32_mid_dim_8_in_features_128_out_features_64_cuda_bfloat16 _ 2025-12-04T11:13:50.0250148Z Traceback (most recent call last): 2025-12-04T11:13:50.0250467Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.0250536Z method(*args, **kwargs) 2025-12-04T11:13:50.0250840Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.0250911Z method(*args, **kwargs) 2025-12-04T11:13:50.0251207Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T11:13:50.0251276Z with policy(): 2025-12-04T11:13:50.0251574Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T11:13:50.0251643Z raise RuntimeError(msg) 2025-12-04T11:13:50.0252473Z RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_concat_cuda_batch_size_32_mid_dim_8_in_features_128_out_features_64_cuda_bfloat16! Caching allocator allocated memory was 0 and is now reported as 24576 on device 0. CUDA driver allocated memory was 230686720 and is now 274726912. 2025-12-04T11:13:50.0252477Z 2025-12-04T11:13:50.0252610Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.0253164Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_concat_cuda_batch_size_32_mid_dim_8_in_features_128_out_features_64_cuda_bfloat16 2025-12-04T11:13:50.0253168Z 2025-12-04T11:13:50.0253330Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.0253469Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.0253571Z stats [('calls_captured', 18), ('unique_graphs', 1)] 2025-12-04T11:13:50.0253926Z inductor [('pattern_matcher_nodes', 18), ('woq_matcher_nodes', 12), ('pattern_matcher_count', 9), ('woq_matcher_count', 3), ('extern_calls', 3), ('fxgraph_cache_miss', 1)] 2025-12-04T11:13:50.0254064Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.0254126Z graph_break [] 2025-12-04T11:13:50.0254255Z ----------------------------- Captured stderr call ----------------------------- 2025-12-04T11:13:50.0254962Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/constant_folding.py:256: UserWarning: Unsupported unwinding pattern: Address not in range (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/profiler/unwind/unwind.cpp:219.) 2025-12-04T11:13:50.0255040Z if out == self.unknown_value: 2025-12-04T11:13:50.0255364Z _ TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_concat_cuda_batch_size_32_mid_dim_8_in_features_128_out_features_64_cuda_bfloat16 _ 2025-12-04T11:13:50.0255443Z Traceback (most recent call last): 2025-12-04T11:13:50.0255745Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.0255818Z method(*args, **kwargs) 2025-12-04T11:13:50.0256114Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.0256187Z method(*args, **kwargs) 2025-12-04T11:13:50.0256557Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T11:13:50.0256685Z with policy(): 2025-12-04T11:13:50.0256993Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T11:13:50.0257059Z raise RuntimeError(msg) 2025-12-04T11:13:50.0257898Z RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_concat_cuda_batch_size_32_mid_dim_8_in_features_128_out_features_64_cuda_bfloat16! Caching allocator allocated memory was 24576 and is now reported as 49152 on device 0. CUDA driver allocated memory was 274726912 and is now 276824064. 2025-12-04T11:13:50.0257903Z 2025-12-04T11:13:50.0258029Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.0258564Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_concat_cuda_batch_size_32_mid_dim_8_in_features_128_out_features_64_cuda_bfloat16 2025-12-04T11:13:50.0258573Z 2025-12-04T11:13:50.0258741Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.0258867Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.0258975Z stats [('calls_captured', 18), ('unique_graphs', 1)] 2025-12-04T11:13:50.0259325Z inductor [('pattern_matcher_nodes', 18), ('woq_matcher_nodes', 12), ('pattern_matcher_count', 9), ('woq_matcher_count', 3), ('extern_calls', 3), ('fxgraph_cache_miss', 1)] 2025-12-04T11:13:50.0259452Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.0259514Z graph_break [] 2025-12-04T11:13:50.0259639Z ----------------------------- Captured stderr call ----------------------------- 2025-12-04T11:13:50.0260337Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/constant_folding.py:256: UserWarning: Unsupported unwinding pattern: Address not in range (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/profiler/unwind/unwind.cpp:219.) 2025-12-04T11:13:50.0260416Z if out == self.unknown_value: 2025-12-04T11:13:50.0260544Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.0260645Z stats [('calls_captured', 18), ('unique_graphs', 1)] 2025-12-04T11:13:50.0260771Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.0261125Z inductor [('pattern_matcher_nodes', 18), ('woq_matcher_nodes', 12), ('pattern_matcher_count', 9), ('woq_matcher_count', 3), ('extern_calls', 3), ('fxgraph_cache_miss', 1)] 2025-12-04T11:13:50.0261190Z graph_break [] 2025-12-04T11:13:50.0261277Z =================================== FAILURES =================================== 2025-12-04T11:13:50.0261596Z _ TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_concat_cuda_batch_size_32_mid_dim_8_in_features_128_out_features_64_cuda_bfloat16 _ 2025-12-04T11:13:50.0261674Z Traceback (most recent call last): 2025-12-04T11:13:50.0261981Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.0262053Z method(*args, **kwargs) 2025-12-04T11:13:50.0262351Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.0262419Z method(*args, **kwargs) 2025-12-04T11:13:50.0262709Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T11:13:50.0262771Z with policy(): 2025-12-04T11:13:50.0263069Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T11:13:50.0263136Z raise RuntimeError(msg) 2025-12-04T11:13:50.0264120Z RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_concat_cuda_batch_size_32_mid_dim_8_in_features_128_out_features_64_cuda_bfloat16! Caching allocator allocated memory was 49152 and is now reported as 73728 on device 0. CUDA driver allocated memory was 276824064 and is now 278921216. 2025-12-04T11:13:50.0264196Z 2025-12-04T11:13:50.0264327Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.0264867Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_concat_cuda_batch_size_32_mid_dim_8_in_features_128_out_features_64_cuda_bfloat16 2025-12-04T11:13:50.0264871Z 2025-12-04T11:13:50.0265035Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.0265165Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.0265267Z stats [('calls_captured', 18), ('unique_graphs', 1)] 2025-12-04T11:13:50.0265620Z inductor [('pattern_matcher_nodes', 18), ('woq_matcher_nodes', 12), ('pattern_matcher_count', 9), ('woq_matcher_count', 3), ('extern_calls', 3), ('fxgraph_cache_miss', 1)] 2025-12-04T11:13:50.0265758Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.0265825Z graph_break [] 2025-12-04T11:13:50.0265952Z ----------------------------- Captured stderr call ----------------------------- 2025-12-04T11:13:50.0266652Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/constant_folding.py:256: UserWarning: Unsupported unwinding pattern: Address not in range (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/profiler/unwind/unwind.cpp:219.) 2025-12-04T11:13:50.0266722Z if out == self.unknown_value: 2025-12-04T11:13:50.0266849Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.0266948Z stats [('calls_captured', 18), ('unique_graphs', 1)] 2025-12-04T11:13:50.0267072Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.0267420Z inductor [('pattern_matcher_nodes', 18), ('woq_matcher_nodes', 12), ('pattern_matcher_count', 9), ('woq_matcher_count', 3), ('extern_calls', 3), ('fxgraph_cache_miss', 1)] 2025-12-04T11:13:50.0267495Z graph_break [] 2025-12-04T11:13:50.0267620Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.0267718Z stats [('calls_captured', 18), ('unique_graphs', 1)] 2025-12-04T11:13:50.0267842Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.0268182Z inductor [('pattern_matcher_nodes', 18), ('woq_matcher_nodes', 12), ('pattern_matcher_count', 9), ('woq_matcher_count', 3), ('extern_calls', 3), ('fxgraph_cache_miss', 1)] 2025-12-04T11:13:50.0268247Z graph_break [] 2025-12-04T11:13:50.0268739Z - generated xml file: /var/lib/jenkins/workspace/test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-5cb074eac86f05e5.xml - 2025-12-04T11:13:50.0268846Z =========================== short test summary info ============================ 2025-12-04T11:13:50.0270198Z FAILED [0.5547s] inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_concat_cuda_batch_size_32_mid_dim_8_in_features_128_out_features_64_cuda_bfloat16 - RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_concat_cuda_batch_size_32_mid_dim_8_in_features_128_out_features_64_cuda_bfloat16! Caching allocator allocated memory was 49152 and is now reported as 73728 on device 0. CUDA driver allocated memory was 276824064 and is now 278921216. 2025-12-04T11:13:50.0270203Z 2025-12-04T11:13:50.0270336Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.0270877Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_concat_cuda_batch_size_32_mid_dim_8_in_features_128_out_features_64_cuda_bfloat16 2025-12-04T11:13:50.0270881Z 2025-12-04T11:13:50.0271040Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.0271292Z !!!!!!!!!!!!!!!!!!!!!!!!!! stopping after 1 failures !!!!!!!!!!!!!!!!!!!!!!!!!!! 2025-12-04T11:13:50.0271416Z ================== 1 failed, 57 deselected, 2 rerun in 12.31s ================== 2025-12-04T11:13:50.0271483Z Got exit code 1 2025-12-04T11:13:50.0271979Z FAILED CONSISTENTLY: test/inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_concat_cuda_batch_size_32_mid_dim_8_in_features_128_out_features_64_cuda_bfloat16 2025-12-04T11:13:50.0272223Z Test failed consistently, continuing with the rest of the tests due to continue-through-error being set 2025-12-04T11:13:50.0272495Z W1204 10:30:42.349000 58694 site-packages/torch/_inductor/utils.py:1703] Not enough SMs to use max_autotune_gemm mode 2025-12-04T11:13:50.0272885Z Test results will be stored in test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-7681ec3fadfa4a43.xml 2025-12-04T11:13:50.0272992Z ============================= test session starts ============================== 2025-12-04T11:13:50.0273208Z platform linux -- Python 3.10.14, pytest-7.3.2, pluggy-1.6.0 -- /opt/conda/envs/py_3.10/bin/python 2025-12-04T11:13:50.0273278Z cachedir: .pytest_cache 2025-12-04T11:13:50.0273593Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow] 2025-12-04T11:13:50.0273675Z rootdir: /var/lib/jenkins/workspace 2025-12-04T11:13:50.0273744Z configfile: pytest.ini 2025-12-04T11:13:50.0274067Z plugins: hypothesis-6.56.4, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-14.0, subtests-0.13.1, xdist-3.3.1, xdoctest-1.3.0, typeguard-4.3.0 2025-12-04T11:13:50.0274197Z collecting ... collected 58 items / 4 deselected / 54 selected 2025-12-04T11:13:50.0274287Z stepcurrent: skipping 4 already run items. 2025-12-04T11:13:50.0274358Z Running 54 items in this shard 2025-12-04T11:13:50.0274362Z 2025-12-04T11:13:50.0274880Z inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_17_mid_dim_1_in_features_1024_out_features_1024_cuda_bfloat16 ('RERUN', {'yellow': True}) [2.1160s] [ 1%] 2025-12-04T11:13:50.0275386Z inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_17_mid_dim_1_in_features_1024_out_features_1024_cuda_bfloat16 ('RERUN', {'yellow': True}) [0.6602s] [ 1%] 2025-12-04T11:13:50.0275843Z inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_17_mid_dim_1_in_features_1024_out_features_1024_cuda_bfloat16 FAILED [0.6548s] [ 1%] 2025-12-04T11:13:50.0275847Z 2025-12-04T11:13:50.0275936Z ==================================== RERUNS ==================================== 2025-12-04T11:13:50.0276235Z _ TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_1_in_features_1024_out_features_1024_cuda_bfloat16 _ 2025-12-04T11:13:50.0276311Z Traceback (most recent call last): 2025-12-04T11:13:50.0276632Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.0276703Z method(*args, **kwargs) 2025-12-04T11:13:50.0277009Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.0277076Z method(*args, **kwargs) 2025-12-04T11:13:50.0277367Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T11:13:50.0277434Z with policy(): 2025-12-04T11:13:50.0277730Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T11:13:50.0277975Z raise RuntimeError(msg) 2025-12-04T11:13:50.0278814Z RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_1_in_features_1024_out_features_1024_cuda_bfloat16! Caching allocator allocated memory was 0 and is now reported as 1048576 on device 0. CUDA driver allocated memory was 230686720 and is now 274726912. 2025-12-04T11:13:50.0279033Z 2025-12-04T11:13:50.0279173Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.0279738Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_1_in_features_1024_out_features_1024_cuda_bfloat16 2025-12-04T11:13:50.0279742Z 2025-12-04T11:13:50.0279906Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.0280041Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.0280137Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.0280491Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.0280629Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.0280689Z graph_break [] 2025-12-04T11:13:50.0280998Z _ TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_1_in_features_1024_out_features_1024_cuda_bfloat16 _ 2025-12-04T11:13:50.0281079Z Traceback (most recent call last): 2025-12-04T11:13:50.0281382Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.0281454Z method(*args, **kwargs) 2025-12-04T11:13:50.0281747Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.0281810Z method(*args, **kwargs) 2025-12-04T11:13:50.0282106Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T11:13:50.0282166Z with policy(): 2025-12-04T11:13:50.0282470Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T11:13:50.0282539Z raise RuntimeError(msg) 2025-12-04T11:13:50.0283373Z RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_1_in_features_1024_out_features_1024_cuda_bfloat16! Caching allocator allocated memory was 1048576 and is now reported as 2097152 on device 0. CUDA driver allocated memory was 274726912 and is now 276824064. 2025-12-04T11:13:50.0283377Z 2025-12-04T11:13:50.0283511Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.0284036Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_1_in_features_1024_out_features_1024_cuda_bfloat16 2025-12-04T11:13:50.0284039Z 2025-12-04T11:13:50.0284203Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.0284335Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.0284433Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.0284784Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.0284914Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.0284983Z graph_break [] 2025-12-04T11:13:50.0285120Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.0285214Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.0285345Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.0285691Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.0285873Z graph_break [] 2025-12-04T11:13:50.0286121Z =================================== FAILURES =================================== 2025-12-04T11:13:50.0286472Z _ TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_1_in_features_1024_out_features_1024_cuda_bfloat16 _ 2025-12-04T11:13:50.0286555Z Traceback (most recent call last): 2025-12-04T11:13:50.0286855Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.0286923Z method(*args, **kwargs) 2025-12-04T11:13:50.0287221Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.0287286Z method(*args, **kwargs) 2025-12-04T11:13:50.0287586Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T11:13:50.0287645Z with policy(): 2025-12-04T11:13:50.0287943Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T11:13:50.0288021Z raise RuntimeError(msg) 2025-12-04T11:13:50.0288858Z RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_1_in_features_1024_out_features_1024_cuda_bfloat16! Caching allocator allocated memory was 2097152 and is now reported as 3145728 on device 0. CUDA driver allocated memory was 276824064 and is now 278921216. 2025-12-04T11:13:50.0288862Z 2025-12-04T11:13:50.0289002Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.0289528Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_1_in_features_1024_out_features_1024_cuda_bfloat16 2025-12-04T11:13:50.0289532Z 2025-12-04T11:13:50.0289696Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.0289841Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.0289938Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.0290293Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.0290425Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.0290485Z graph_break [] 2025-12-04T11:13:50.0290619Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.0290711Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.0290835Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.0291184Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.0291247Z graph_break [] 2025-12-04T11:13:50.0291379Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.0291468Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.0291588Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.0291938Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.0292002Z graph_break [] 2025-12-04T11:13:50.0292506Z - generated xml file: /var/lib/jenkins/workspace/test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-7681ec3fadfa4a43.xml - 2025-12-04T11:13:50.0292616Z =========================== short test summary info ============================ 2025-12-04T11:13:50.0294021Z FAILED [0.6548s] inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_17_mid_dim_1_in_features_1024_out_features_1024_cuda_bfloat16 - RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_1_in_features_1024_out_features_1024_cuda_bfloat16! Caching allocator allocated memory was 2097152 and is now reported as 3145728 on device 0. CUDA driver allocated memory was 276824064 and is now 278921216. 2025-12-04T11:13:50.0294098Z 2025-12-04T11:13:50.0294231Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.0294760Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_1_in_features_1024_out_features_1024_cuda_bfloat16 2025-12-04T11:13:50.0294764Z 2025-12-04T11:13:50.0294927Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.0295034Z !!!!!!!!!!!!!!!!!!!!!!!!!! stopping after 1 failures !!!!!!!!!!!!!!!!!!!!!!!!!!! 2025-12-04T11:13:50.0295157Z =================== 1 failed, 4 deselected, 2 rerun in 3.46s =================== 2025-12-04T11:13:50.0295262Z Got exit code 1 2025-12-04T11:13:50.0295365Z Retrying single test... 2025-12-04T11:13:50.0295721Z W1204 10:30:52.658000 58883 site-packages/torch/_inductor/utils.py:1703] Not enough SMs to use max_autotune_gemm mode 2025-12-04T11:13:50.0296143Z Test results will be stored in test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-49c0a334019b663e.xml 2025-12-04T11:13:50.0296311Z ============================= test session starts ============================== 2025-12-04T11:13:50.0296559Z platform linux -- Python 3.10.14, pytest-7.3.2, pluggy-1.6.0 -- /opt/conda/envs/py_3.10/bin/python 2025-12-04T11:13:50.0296788Z cachedir: .pytest_cache 2025-12-04T11:13:50.0297249Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow] 2025-12-04T11:13:50.0297399Z rootdir: /var/lib/jenkins/workspace 2025-12-04T11:13:50.0297497Z configfile: pytest.ini 2025-12-04T11:13:50.0297848Z plugins: hypothesis-6.56.4, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-14.0, subtests-0.13.1, xdist-3.3.1, xdoctest-1.3.0, typeguard-4.3.0 2025-12-04T11:13:50.0298038Z collecting ... collected 58 items / 57 deselected / 1 selected 2025-12-04T11:13:50.0298722Z stepcurrent: skipping 4 already run items. Running only test/inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_17_mid_dim_1_in_features_1024_out_features_1024_cuda_bfloat16 2025-12-04T11:13:50.0298890Z Running 1 items in this shard 2025-12-04T11:13:50.0298894Z 2025-12-04T11:13:50.0299679Z inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_17_mid_dim_1_in_features_1024_out_features_1024_cuda_bfloat16 [W1204 10:30:54.170564497 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0299687Z 2025-12-04T11:13:50.0300060Z [W1204 10:31:03.406085657 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0300064Z 2025-12-04T11:13:50.0300389Z [W1204 10:31:03.406338052 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0300393Z 2025-12-04T11:13:50.0300698Z [W1204 10:31:03.412212691 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0300847Z 2025-12-04T11:13:50.0301192Z [W1204 10:31:03.412820114 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0301196Z 2025-12-04T11:13:50.0301517Z [W1204 10:31:03.413005309 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0301601Z 2025-12-04T11:13:50.0302030Z [W1204 10:31:03.418700684 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0302034Z 2025-12-04T11:13:50.0302375Z [W1204 10:31:03.419260116 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0302378Z 2025-12-04T11:13:50.0302723Z [W1204 10:31:03.419437950 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0302727Z 2025-12-04T11:13:50.0302912Z ('RERUN', {'yellow': True}) [11.3613s] [100%] 2025-12-04T11:13:50.0303824Z inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_17_mid_dim_1_in_features_1024_out_features_1024_cuda_bfloat16 [W1204 10:31:04.597966679 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0303833Z 2025-12-04T11:13:50.0304168Z [W1204 10:31:04.598513921 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0304172Z 2025-12-04T11:13:50.0304547Z [W1204 10:31:04.598651194 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0304551Z 2025-12-04T11:13:50.0304956Z [W1204 10:31:04.601716572 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0304959Z 2025-12-04T11:13:50.0305262Z [W1204 10:31:04.602292354 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0305362Z 2025-12-04T11:13:50.0305698Z [W1204 10:31:04.602428137 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0305705Z 2025-12-04T11:13:50.0306028Z [W1204 10:31:04.607203532 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0306031Z 2025-12-04T11:13:50.0306402Z [W1204 10:31:04.607676103 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0306405Z 2025-12-04T11:13:50.0306722Z [W1204 10:31:04.607814306 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0306725Z 2025-12-04T11:13:50.0306856Z ('RERUN', {'yellow': True}) [0.6036s] [100%] 2025-12-04T11:13:50.0307683Z inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_17_mid_dim_1_in_features_1024_out_features_1024_cuda_bfloat16 [W1204 10:31:05.198290147 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0307690Z 2025-12-04T11:13:50.0308068Z [W1204 10:31:05.198829218 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0308071Z 2025-12-04T11:13:50.0308391Z [W1204 10:31:05.198968391 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0308395Z 2025-12-04T11:13:50.0308763Z [W1204 10:31:05.202062219 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0308766Z 2025-12-04T11:13:50.0309088Z [W1204 10:31:05.202630451 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0309092Z 2025-12-04T11:13:50.0309402Z [W1204 10:31:05.202770065 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0309566Z 2025-12-04T11:13:50.0309987Z [W1204 10:31:05.207517879 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0309991Z 2025-12-04T11:13:50.0310312Z [W1204 10:31:05.207993659 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0310315Z 2025-12-04T11:13:50.0310671Z [W1204 10:31:05.208130482 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0310674Z 2025-12-04T11:13:50.0310769Z FAILED [0.5986s] [100%] 2025-12-04T11:13:50.0310773Z 2025-12-04T11:13:50.0310914Z ==================================== RERUNS ==================================== 2025-12-04T11:13:50.0311303Z _ TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_1_in_features_1024_out_features_1024_cuda_bfloat16 _ 2025-12-04T11:13:50.0311457Z Traceback (most recent call last): 2025-12-04T11:13:50.0311918Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.0312021Z method(*args, **kwargs) 2025-12-04T11:13:50.0312382Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.0312463Z method(*args, **kwargs) 2025-12-04T11:13:50.0312839Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T11:13:50.0313012Z with policy(): 2025-12-04T11:13:50.0313342Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T11:13:50.0313477Z raise RuntimeError(msg) 2025-12-04T11:13:50.0314331Z RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_1_in_features_1024_out_features_1024_cuda_bfloat16! Caching allocator allocated memory was 0 and is now reported as 1048576 on device 0. CUDA driver allocated memory was 230686720 and is now 274726912. 2025-12-04T11:13:50.0314338Z 2025-12-04T11:13:50.0314489Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.0315178Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_1_in_features_1024_out_features_1024_cuda_bfloat16 2025-12-04T11:13:50.0315182Z 2025-12-04T11:13:50.0315375Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.0315574Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.0315701Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.0316084Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.0316340Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.0316450Z graph_break [] 2025-12-04T11:13:50.0316644Z ----------------------------- Captured stderr call ----------------------------- 2025-12-04T11:13:50.0317377Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/constant_folding.py:256: UserWarning: Unsupported unwinding pattern: Address not in range (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/profiler/unwind/unwind.cpp:219.) 2025-12-04T11:13:50.0317481Z if out == self.unknown_value: 2025-12-04T11:13:50.0317858Z _ TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_1_in_features_1024_out_features_1024_cuda_bfloat16 _ 2025-12-04T11:13:50.0318030Z Traceback (most recent call last): 2025-12-04T11:13:50.0318424Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.0318686Z method(*args, **kwargs) 2025-12-04T11:13:50.0319096Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.0319233Z method(*args, **kwargs) 2025-12-04T11:13:50.0319550Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T11:13:50.0319758Z with policy(): 2025-12-04T11:13:50.0320087Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T11:13:50.0320187Z raise RuntimeError(msg) 2025-12-04T11:13:50.0321099Z RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_1_in_features_1024_out_features_1024_cuda_bfloat16! Caching allocator allocated memory was 1048576 and is now reported as 2097152 on device 0. CUDA driver allocated memory was 274726912 and is now 276824064. 2025-12-04T11:13:50.0321107Z 2025-12-04T11:13:50.0321267Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.0321903Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_1_in_features_1024_out_features_1024_cuda_bfloat16 2025-12-04T11:13:50.0321908Z 2025-12-04T11:13:50.0322115Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.0322326Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.0322462Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.0322842Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.0323025Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.0323169Z graph_break [] 2025-12-04T11:13:50.0323342Z ----------------------------- Captured stderr call ----------------------------- 2025-12-04T11:13:50.0324127Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/constant_folding.py:256: UserWarning: Unsupported unwinding pattern: Address not in range (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/profiler/unwind/unwind.cpp:219.) 2025-12-04T11:13:50.0324229Z if out == self.unknown_value: 2025-12-04T11:13:50.0324423Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.0324533Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.0324737Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.0325252Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.0325352Z graph_break [] 2025-12-04T11:13:50.0325506Z =================================== FAILURES =================================== 2025-12-04T11:13:50.0325840Z _ TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_1_in_features_1024_out_features_1024_cuda_bfloat16 _ 2025-12-04T11:13:50.0325932Z Traceback (most recent call last): 2025-12-04T11:13:50.0326361Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.0326477Z method(*args, **kwargs) 2025-12-04T11:13:50.0326836Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.0326932Z method(*args, **kwargs) 2025-12-04T11:13:50.0327265Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T11:13:50.0327425Z with policy(): 2025-12-04T11:13:50.0327785Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T11:13:50.0327966Z raise RuntimeError(msg) 2025-12-04T11:13:50.0328931Z RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_1_in_features_1024_out_features_1024_cuda_bfloat16! Caching allocator allocated memory was 2097152 and is now reported as 3145728 on device 0. CUDA driver allocated memory was 276824064 and is now 278921216. 2025-12-04T11:13:50.0328936Z 2025-12-04T11:13:50.0329098Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.0329694Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_1_in_features_1024_out_features_1024_cuda_bfloat16 2025-12-04T11:13:50.0329698Z 2025-12-04T11:13:50.0329938Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.0330151Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.0330277Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.0330655Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.0330953Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.0331029Z graph_break [] 2025-12-04T11:13:50.0331293Z ----------------------------- Captured stderr call ----------------------------- 2025-12-04T11:13:50.0332016Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/constant_folding.py:256: UserWarning: Unsupported unwinding pattern: Address not in range (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/profiler/unwind/unwind.cpp:219.) 2025-12-04T11:13:50.0332121Z if out == self.unknown_value: 2025-12-04T11:13:50.0332315Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.0332460Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.0332700Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.0333093Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.0333186Z graph_break [] 2025-12-04T11:13:50.0333375Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.0333512Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.0333668Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.0334118Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.0334224Z graph_break [] 2025-12-04T11:13:50.0334779Z - generated xml file: /var/lib/jenkins/workspace/test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-49c0a334019b663e.xml - 2025-12-04T11:13:50.0334910Z =========================== short test summary info ============================ 2025-12-04T11:13:50.0336315Z FAILED [0.5986s] inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_17_mid_dim_1_in_features_1024_out_features_1024_cuda_bfloat16 - RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_1_in_features_1024_out_features_1024_cuda_bfloat16! Caching allocator allocated memory was 2097152 and is now reported as 3145728 on device 0. CUDA driver allocated memory was 276824064 and is now 278921216. 2025-12-04T11:13:50.0336321Z 2025-12-04T11:13:50.0336467Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.0337207Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_1_in_features_1024_out_features_1024_cuda_bfloat16 2025-12-04T11:13:50.0337277Z 2025-12-04T11:13:50.0337491Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.0337701Z !!!!!!!!!!!!!!!!!!!!!!!!!! stopping after 1 failures !!!!!!!!!!!!!!!!!!!!!!!!!!! 2025-12-04T11:13:50.0337884Z ================== 1 failed, 57 deselected, 2 rerun in 12.59s ================== 2025-12-04T11:13:50.0337975Z Got exit code 1 2025-12-04T11:13:50.0338059Z Retrying single test... 2025-12-04T11:13:50.0338472Z W1204 10:31:12.082000 59077 site-packages/torch/_inductor/utils.py:1703] Not enough SMs to use max_autotune_gemm mode 2025-12-04T11:13:50.0338909Z Test results will be stored in test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-a4c3d2e69c8841e3.xml 2025-12-04T11:13:50.0339069Z ============================= test session starts ============================== 2025-12-04T11:13:50.0339321Z platform linux -- Python 3.10.14, pytest-7.3.2, pluggy-1.6.0 -- /opt/conda/envs/py_3.10/bin/python 2025-12-04T11:13:50.0339421Z cachedir: .pytest_cache 2025-12-04T11:13:50.0339849Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow] 2025-12-04T11:13:50.0339990Z rootdir: /var/lib/jenkins/workspace 2025-12-04T11:13:50.0340124Z configfile: pytest.ini 2025-12-04T11:13:50.0340470Z plugins: hypothesis-6.56.4, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-14.0, subtests-0.13.1, xdist-3.3.1, xdoctest-1.3.0, typeguard-4.3.0 2025-12-04T11:13:50.0340632Z collecting ... collected 58 items / 57 deselected / 1 selected 2025-12-04T11:13:50.0341259Z stepcurrent: skipping 4 already run items. Running only test/inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_17_mid_dim_1_in_features_1024_out_features_1024_cuda_bfloat16 2025-12-04T11:13:50.0341421Z Running 1 items in this shard 2025-12-04T11:13:50.0341425Z 2025-12-04T11:13:50.0342268Z inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_17_mid_dim_1_in_features_1024_out_features_1024_cuda_bfloat16 [W1204 10:31:13.557458424 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0342273Z 2025-12-04T11:13:50.0342601Z [W1204 10:31:22.438532523 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0342605Z 2025-12-04T11:13:50.0342960Z [W1204 10:31:22.438767278 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0342964Z 2025-12-04T11:13:50.0343290Z [W1204 10:31:22.444419279 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0343300Z 2025-12-04T11:13:50.0343748Z [W1204 10:31:22.444989580 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0343752Z 2025-12-04T11:13:50.0344116Z [W1204 10:31:22.445147174 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0344120Z 2025-12-04T11:13:50.0344441Z [W1204 10:31:22.450439027 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0344444Z 2025-12-04T11:13:50.0344878Z [W1204 10:31:22.450984608 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0344882Z 2025-12-04T11:13:50.0345200Z [W1204 10:31:22.451157582 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0345204Z 2025-12-04T11:13:50.0345438Z ('RERUN', {'yellow': True}) [10.9673s] [100%] 2025-12-04T11:13:50.0346336Z inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_17_mid_dim_1_in_features_1024_out_features_1024_cuda_bfloat16 [W1204 10:31:23.614524924 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0346341Z 2025-12-04T11:13:50.0346714Z [W1204 10:31:23.615063995 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0346717Z 2025-12-04T11:13:50.0347035Z [W1204 10:31:23.615205428 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0347039Z 2025-12-04T11:13:50.0347407Z [W1204 10:31:23.618138090 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0347413Z 2025-12-04T11:13:50.0347734Z [W1204 10:31:23.618702532 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0347741Z 2025-12-04T11:13:50.0348132Z [W1204 10:31:23.618850065 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0348135Z 2025-12-04T11:13:50.0348466Z [W1204 10:31:23.623405292 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0348471Z 2025-12-04T11:13:50.0348788Z [W1204 10:31:23.623874382 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0348828Z 2025-12-04T11:13:50.0349146Z [W1204 10:31:23.624009465 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0349150Z 2025-12-04T11:13:50.0349279Z ('RERUN', {'yellow': True}) [0.5825s] [100%] 2025-12-04T11:13:50.0350076Z inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_17_mid_dim_1_in_features_1024_out_features_1024_cuda_bfloat16 [W1204 10:31:24.189113084 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0350080Z 2025-12-04T11:13:50.0350450Z [W1204 10:31:24.189645405 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0350453Z 2025-12-04T11:13:50.0350821Z [W1204 10:31:24.189783888 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0350825Z 2025-12-04T11:13:50.0351141Z [W1204 10:31:24.192743601 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0351144Z 2025-12-04T11:13:50.0351516Z [W1204 10:31:24.193313663 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0351523Z 2025-12-04T11:13:50.0351844Z [W1204 10:31:24.193451176 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0351848Z 2025-12-04T11:13:50.0352241Z [W1204 10:31:24.198008443 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0352245Z 2025-12-04T11:13:50.0352657Z [W1204 10:31:24.198473183 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0352661Z 2025-12-04T11:13:50.0352988Z [W1204 10:31:24.198606896 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0353028Z 2025-12-04T11:13:50.0353139Z FAILED [0.5767s] [100%] 2025-12-04T11:13:50.0353218Z 2025-12-04T11:13:50.0353338Z ==================================== RERUNS ==================================== 2025-12-04T11:13:50.0353762Z _ TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_1_in_features_1024_out_features_1024_cuda_bfloat16 _ 2025-12-04T11:13:50.0353931Z Traceback (most recent call last): 2025-12-04T11:13:50.0354304Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.0354439Z method(*args, **kwargs) 2025-12-04T11:13:50.0354764Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.0354893Z method(*args, **kwargs) 2025-12-04T11:13:50.0355203Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T11:13:50.0355370Z with policy(): 2025-12-04T11:13:50.0355781Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T11:13:50.0355886Z raise RuntimeError(msg) 2025-12-04T11:13:50.0356769Z RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_1_in_features_1024_out_features_1024_cuda_bfloat16! Caching allocator allocated memory was 0 and is now reported as 1048576 on device 0. CUDA driver allocated memory was 230686720 and is now 274726912. 2025-12-04T11:13:50.0356773Z 2025-12-04T11:13:50.0356934Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.0357564Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_1_in_features_1024_out_features_1024_cuda_bfloat16 2025-12-04T11:13:50.0357568Z 2025-12-04T11:13:50.0357790Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.0357952Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.0358126Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.0358514Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.0358771Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.0358929Z graph_break [] 2025-12-04T11:13:50.0359102Z ----------------------------- Captured stderr call ----------------------------- 2025-12-04T11:13:50.0359856Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/constant_folding.py:256: UserWarning: Unsupported unwinding pattern: Address not in range (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/profiler/unwind/unwind.cpp:219.) 2025-12-04T11:13:50.0359966Z if out == self.unknown_value: 2025-12-04T11:13:50.0360304Z _ TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_1_in_features_1024_out_features_1024_cuda_bfloat16 _ 2025-12-04T11:13:50.0360433Z Traceback (most recent call last): 2025-12-04T11:13:50.0360825Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.0360971Z method(*args, **kwargs) 2025-12-04T11:13:50.0361295Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.0361390Z method(*args, **kwargs) 2025-12-04T11:13:50.0361757Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T11:13:50.0361851Z with policy(): 2025-12-04T11:13:50.0362288Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T11:13:50.0362390Z raise RuntimeError(msg) 2025-12-04T11:13:50.0363402Z RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_1_in_features_1024_out_features_1024_cuda_bfloat16! Caching allocator allocated memory was 1048576 and is now reported as 2097152 on device 0. CUDA driver allocated memory was 274726912 and is now 276824064. 2025-12-04T11:13:50.0363460Z 2025-12-04T11:13:50.0363622Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.0364178Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_1_in_features_1024_out_features_1024_cuda_bfloat16 2025-12-04T11:13:50.0364182Z 2025-12-04T11:13:50.0364456Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.0364634Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.0364869Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.0365272Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.0365432Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.0365541Z graph_break [] 2025-12-04T11:13:50.0365749Z ----------------------------- Captured stderr call ----------------------------- 2025-12-04T11:13:50.0366485Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/constant_folding.py:256: UserWarning: Unsupported unwinding pattern: Address not in range (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/profiler/unwind/unwind.cpp:219.) 2025-12-04T11:13:50.0366649Z if out == self.unknown_value: 2025-12-04T11:13:50.0366809Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.0366966Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.0367112Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.0367548Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.0367695Z graph_break [] 2025-12-04T11:13:50.0367836Z =================================== FAILURES =================================== 2025-12-04T11:13:50.0368202Z _ TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_1_in_features_1024_out_features_1024_cuda_bfloat16 _ 2025-12-04T11:13:50.0368309Z Traceback (most recent call last): 2025-12-04T11:13:50.0368627Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.0368830Z method(*args, **kwargs) 2025-12-04T11:13:50.0369173Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.0369307Z method(*args, **kwargs) 2025-12-04T11:13:50.0369634Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T11:13:50.0369727Z with policy(): 2025-12-04T11:13:50.0370146Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T11:13:50.0370263Z raise RuntimeError(msg) 2025-12-04T11:13:50.0371263Z RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_1_in_features_1024_out_features_1024_cuda_bfloat16! Caching allocator allocated memory was 2097152 and is now reported as 3145728 on device 0. CUDA driver allocated memory was 276824064 and is now 278921216. 2025-12-04T11:13:50.0371268Z 2025-12-04T11:13:50.0371424Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.0372056Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_1_in_features_1024_out_features_1024_cuda_bfloat16 2025-12-04T11:13:50.0372121Z 2025-12-04T11:13:50.0372341Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.0372580Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.0372754Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.0373136Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.0373293Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.0373422Z graph_break [] 2025-12-04T11:13:50.0373579Z ----------------------------- Captured stderr call ----------------------------- 2025-12-04T11:13:50.0374395Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/constant_folding.py:256: UserWarning: Unsupported unwinding pattern: Address not in range (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/profiler/unwind/unwind.cpp:219.) 2025-12-04T11:13:50.0374502Z if out == self.unknown_value: 2025-12-04T11:13:50.0374660Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.0374820Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.0374991Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.0375442Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.0375549Z graph_break [] 2025-12-04T11:13:50.0375705Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.0375861Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.0376019Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.0376410Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.0376576Z graph_break [] 2025-12-04T11:13:50.0377192Z - generated xml file: /var/lib/jenkins/workspace/test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-a4c3d2e69c8841e3.xml - 2025-12-04T11:13:50.0377355Z =========================== short test summary info ============================ 2025-12-04T11:13:50.0378900Z FAILED [0.5767s] inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_17_mid_dim_1_in_features_1024_out_features_1024_cuda_bfloat16 - RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_1_in_features_1024_out_features_1024_cuda_bfloat16! Caching allocator allocated memory was 2097152 and is now reported as 3145728 on device 0. CUDA driver allocated memory was 276824064 and is now 278921216. 2025-12-04T11:13:50.0378909Z 2025-12-04T11:13:50.0379124Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.0379669Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_1_in_features_1024_out_features_1024_cuda_bfloat16 2025-12-04T11:13:50.0379673Z 2025-12-04T11:13:50.0379980Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.0380132Z !!!!!!!!!!!!!!!!!!!!!!!!!! stopping after 1 failures !!!!!!!!!!!!!!!!!!!!!!!!!!! 2025-12-04T11:13:50.0380285Z ================== 1 failed, 57 deselected, 2 rerun in 12.15s ================== 2025-12-04T11:13:50.0380427Z Got exit code 1 2025-12-04T11:13:50.0380940Z FAILED CONSISTENTLY: test/inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_17_mid_dim_1_in_features_1024_out_features_1024_cuda_bfloat16 2025-12-04T11:13:50.0381521Z Test failed consistently, continuing with the rest of the tests due to continue-through-error being set 2025-12-04T11:13:50.0381846Z W1204 10:31:31.103000 59271 site-packages/torch/_inductor/utils.py:1703] Not enough SMs to use max_autotune_gemm mode 2025-12-04T11:13:50.0382287Z Test results will be stored in test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-0f60f7321324a86a.xml 2025-12-04T11:13:50.0382454Z ============================= test session starts ============================== 2025-12-04T11:13:50.0382700Z platform linux -- Python 3.10.14, pytest-7.3.2, pluggy-1.6.0 -- /opt/conda/envs/py_3.10/bin/python 2025-12-04T11:13:50.0382818Z cachedir: .pytest_cache 2025-12-04T11:13:50.0383213Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow] 2025-12-04T11:13:50.0383342Z rootdir: /var/lib/jenkins/workspace 2025-12-04T11:13:50.0383494Z configfile: pytest.ini 2025-12-04T11:13:50.0383929Z plugins: hypothesis-6.56.4, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-14.0, subtests-0.13.1, xdist-3.3.1, xdoctest-1.3.0, typeguard-4.3.0 2025-12-04T11:13:50.0384093Z collecting ... collected 58 items / 5 deselected / 53 selected 2025-12-04T11:13:50.0384312Z stepcurrent: skipping 5 already run items. 2025-12-04T11:13:50.0384469Z Running 53 items in this shard 2025-12-04T11:13:50.0384473Z 2025-12-04T11:13:50.0385074Z inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_17_mid_dim_1_in_features_1024_out_features_64_cuda_bfloat16 ('RERUN', {'yellow': True}) [1.9956s] [ 1%] 2025-12-04T11:13:50.0385594Z inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_17_mid_dim_1_in_features_1024_out_features_64_cuda_bfloat16 ('RERUN', {'yellow': True}) [0.5581s] [ 1%] 2025-12-04T11:13:50.0386112Z inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_17_mid_dim_1_in_features_1024_out_features_64_cuda_bfloat16 FAILED [0.5425s] [ 1%] 2025-12-04T11:13:50.0386116Z 2025-12-04T11:13:50.0386236Z ==================================== RERUNS ==================================== 2025-12-04T11:13:50.0386549Z _ TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_1_in_features_1024_out_features_64_cuda_bfloat16 _ 2025-12-04T11:13:50.0386783Z Traceback (most recent call last): 2025-12-04T11:13:50.0387123Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.0387254Z method(*args, **kwargs) 2025-12-04T11:13:50.0387582Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.0387676Z method(*args, **kwargs) 2025-12-04T11:13:50.0388078Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T11:13:50.0388204Z with policy(): 2025-12-04T11:13:50.0388542Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T11:13:50.0388676Z raise RuntimeError(msg) 2025-12-04T11:13:50.0389517Z RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_1_in_features_1024_out_features_64_cuda_bfloat16! Caching allocator allocated memory was 0 and is now reported as 65536 on device 0. CUDA driver allocated memory was 230686720 and is now 274726912. 2025-12-04T11:13:50.0389521Z 2025-12-04T11:13:50.0389703Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.0390333Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_1_in_features_1024_out_features_64_cuda_bfloat16 2025-12-04T11:13:50.0390417Z 2025-12-04T11:13:50.0390726Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.0390964Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.0391093Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.0391730Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('benchmarking.InductorBenchmarker.benchmark_gpu', 2), ('pad_mm_bench', 1), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.0391876Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.0392092Z graph_break [] 2025-12-04T11:13:50.0392425Z _ TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_1_in_features_1024_out_features_64_cuda_bfloat16 _ 2025-12-04T11:13:50.0392531Z Traceback (most recent call last): 2025-12-04T11:13:50.0392923Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.0393020Z method(*args, **kwargs) 2025-12-04T11:13:50.0393434Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.0393545Z method(*args, **kwargs) 2025-12-04T11:13:50.0393868Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T11:13:50.0394009Z with policy(): 2025-12-04T11:13:50.0394335Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T11:13:50.0394433Z raise RuntimeError(msg) 2025-12-04T11:13:50.0395368Z RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_1_in_features_1024_out_features_64_cuda_bfloat16! Caching allocator allocated memory was 65536 and is now reported as 131072 on device 0. CUDA driver allocated memory was 274726912 and is now 276824064. 2025-12-04T11:13:50.0395375Z 2025-12-04T11:13:50.0395554Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.0396264Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_1_in_features_1024_out_features_64_cuda_bfloat16 2025-12-04T11:13:50.0396270Z 2025-12-04T11:13:50.0396562Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.0396759Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.0396872Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.0397530Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('benchmarking.InductorBenchmarker.benchmark_gpu', 2), ('pad_mm_bench', 1), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.0397752Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.0397934Z graph_break [] 2025-12-04T11:13:50.0398128Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.0398251Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.0398391Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.0399072Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('benchmarking.InductorBenchmarker.benchmark_gpu', 2), ('pad_mm_bench', 1), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.0399166Z graph_break [] 2025-12-04T11:13:50.0399331Z =================================== FAILURES =================================== 2025-12-04T11:13:50.0399750Z _ TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_1_in_features_1024_out_features_64_cuda_bfloat16 _ 2025-12-04T11:13:50.0399921Z Traceback (most recent call last): 2025-12-04T11:13:50.0400346Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.0400479Z method(*args, **kwargs) 2025-12-04T11:13:50.0400806Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.0400938Z method(*args, **kwargs) 2025-12-04T11:13:50.0401260Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T11:13:50.0401373Z with policy(): 2025-12-04T11:13:50.0401766Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T11:13:50.0401883Z raise RuntimeError(msg) 2025-12-04T11:13:50.0402797Z RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_1_in_features_1024_out_features_64_cuda_bfloat16! Caching allocator allocated memory was 131072 and is now reported as 196608 on device 0. CUDA driver allocated memory was 276824064 and is now 278921216. 2025-12-04T11:13:50.0402803Z 2025-12-04T11:13:50.0402966Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.0403560Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_1_in_features_1024_out_features_64_cuda_bfloat16 2025-12-04T11:13:50.0403564Z 2025-12-04T11:13:50.0403744Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.0404117Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.0404248Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.0404835Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('benchmarking.InductorBenchmarker.benchmark_gpu', 2), ('pad_mm_bench', 1), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.0405027Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.0405118Z graph_break [] 2025-12-04T11:13:50.0405376Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.0405513Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.0405668Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.0406280Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('benchmarking.InductorBenchmarker.benchmark_gpu', 2), ('pad_mm_bench', 1), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.0406377Z graph_break [] 2025-12-04T11:13:50.0406549Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.0406744Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.0406911Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.0407518Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('benchmarking.InductorBenchmarker.benchmark_gpu', 2), ('pad_mm_bench', 1), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.0407607Z graph_break [] 2025-12-04T11:13:50.0408134Z - generated xml file: /var/lib/jenkins/workspace/test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-0f60f7321324a86a.xml - 2025-12-04T11:13:50.0408285Z =========================== short test summary info ============================ 2025-12-04T11:13:50.0424188Z FAILED [0.5425s] inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_17_mid_dim_1_in_features_1024_out_features_64_cuda_bfloat16 - RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_1_in_features_1024_out_features_64_cuda_bfloat16! Caching allocator allocated memory was 131072 and is now reported as 196608 on device 0. CUDA driver allocated memory was 276824064 and is now 278921216. 2025-12-04T11:13:50.0424269Z 2025-12-04T11:13:50.0424444Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.0425003Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_1_in_features_1024_out_features_64_cuda_bfloat16 2025-12-04T11:13:50.0425007Z 2025-12-04T11:13:50.0425182Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.0425310Z !!!!!!!!!!!!!!!!!!!!!!!!!! stopping after 1 failures !!!!!!!!!!!!!!!!!!!!!!!!!!! 2025-12-04T11:13:50.0425445Z =================== 1 failed, 5 deselected, 2 rerun in 3.12s =================== 2025-12-04T11:13:50.0425511Z Got exit code 1 2025-12-04T11:13:50.0425587Z Retrying single test... 2025-12-04T11:13:50.0425863Z W1204 10:31:41.401000 59460 site-packages/torch/_inductor/utils.py:1703] Not enough SMs to use max_autotune_gemm mode 2025-12-04T11:13:50.0426266Z Test results will be stored in test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-294e2abb077cceb2.xml 2025-12-04T11:13:50.0426387Z ============================= test session starts ============================== 2025-12-04T11:13:50.0426607Z platform linux -- Python 3.10.14, pytest-7.3.2, pluggy-1.6.0 -- /opt/conda/envs/py_3.10/bin/python 2025-12-04T11:13:50.0426683Z cachedir: .pytest_cache 2025-12-04T11:13:50.0427005Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow] 2025-12-04T11:13:50.0427094Z rootdir: /var/lib/jenkins/workspace 2025-12-04T11:13:50.0427166Z configfile: pytest.ini 2025-12-04T11:13:50.0427494Z plugins: hypothesis-6.56.4, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-14.0, subtests-0.13.1, xdist-3.3.1, xdoctest-1.3.0, typeguard-4.3.0 2025-12-04T11:13:50.0427635Z collecting ... collected 58 items / 57 deselected / 1 selected 2025-12-04T11:13:50.0428273Z stepcurrent: skipping 5 already run items. Running only test/inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_17_mid_dim_1_in_features_1024_out_features_64_cuda_bfloat16 2025-12-04T11:13:50.0428349Z Running 1 items in this shard 2025-12-04T11:13:50.0428354Z 2025-12-04T11:13:50.0429106Z inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_17_mid_dim_1_in_features_1024_out_features_64_cuda_bfloat16 [W1204 10:31:43.053104391 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0429114Z 2025-12-04T11:13:50.0429423Z [W1204 10:31:52.123418445 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0429427Z 2025-12-04T11:13:50.0429738Z [W1204 10:31:52.123672381 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0429741Z 2025-12-04T11:13:50.0430034Z [W1204 10:31:52.129392997 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0430037Z 2025-12-04T11:13:50.0430334Z [W1204 10:31:52.129952869 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0430337Z 2025-12-04T11:13:50.0430631Z [W1204 10:31:52.130169934 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0430711Z 2025-12-04T11:13:50.0431072Z [W1204 10:31:52.135500201 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0431077Z 2025-12-04T11:13:50.0431369Z [W1204 10:31:52.136015072 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0431372Z 2025-12-04T11:13:50.0431663Z [W1204 10:31:52.136170446 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0431670Z 2025-12-04T11:13:50.0431755Z ('RERUN', {'yellow': True}) [11.0690s] [100%] 2025-12-04T11:13:50.0432478Z inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_17_mid_dim_1_in_features_1024_out_features_64_cuda_bfloat16 [W1204 10:31:52.951991615 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0432485Z 2025-12-04T11:13:50.0432784Z [W1204 10:31:52.952523296 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0432787Z 2025-12-04T11:13:50.0433079Z [W1204 10:31:52.952671000 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0433082Z 2025-12-04T11:13:50.0433377Z [W1204 10:31:52.955654795 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0433380Z 2025-12-04T11:13:50.0433668Z [W1204 10:31:52.956117195 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0433671Z 2025-12-04T11:13:50.0433967Z [W1204 10:31:52.956256889 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0433972Z 2025-12-04T11:13:50.0434265Z [W1204 10:31:53.960955682 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0434268Z 2025-12-04T11:13:50.0434563Z [W1204 10:31:53.961427922 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0434567Z 2025-12-04T11:13:50.0434859Z [W1204 10:31:53.961564095 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0434862Z 2025-12-04T11:13:50.0434946Z ('RERUN', {'yellow': True}) [0.5097s] [100%] 2025-12-04T11:13:50.0435676Z inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_17_mid_dim_1_in_features_1024_out_features_64_cuda_bfloat16 [W1204 10:31:53.458369740 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0435682Z 2025-12-04T11:13:50.0435976Z [W1204 10:31:53.458892752 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0435979Z 2025-12-04T11:13:50.0436274Z [W1204 10:31:53.459037185 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0436277Z 2025-12-04T11:13:50.0436566Z [W1204 10:31:53.461987780 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0436570Z 2025-12-04T11:13:50.0436863Z [W1204 10:31:53.462453690 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0436866Z 2025-12-04T11:13:50.0437156Z [W1204 10:31:53.462591473 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0437244Z 2025-12-04T11:13:50.0437614Z [W1204 10:31:53.467279006 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0437618Z 2025-12-04T11:13:50.0437911Z [W1204 10:31:53.467749736 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0437915Z 2025-12-04T11:13:50.0438210Z [W1204 10:31:53.467890509 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0438213Z 2025-12-04T11:13:50.0438283Z FAILED [0.5067s] [100%] 2025-12-04T11:13:50.0438287Z 2025-12-04T11:13:50.0438376Z ==================================== RERUNS ==================================== 2025-12-04T11:13:50.0438716Z _ TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_1_in_features_1024_out_features_64_cuda_bfloat16 _ 2025-12-04T11:13:50.0438798Z Traceback (most recent call last): 2025-12-04T11:13:50.0439124Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.0439200Z method(*args, **kwargs) 2025-12-04T11:13:50.0439497Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.0439567Z method(*args, **kwargs) 2025-12-04T11:13:50.0439891Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T11:13:50.0440007Z with policy(): 2025-12-04T11:13:50.0440403Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T11:13:50.0440559Z raise RuntimeError(msg) 2025-12-04T11:13:50.0444336Z RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_1_in_features_1024_out_features_64_cuda_bfloat16! Caching allocator allocated memory was 0 and is now reported as 65536 on device 0. CUDA driver allocated memory was 230686720 and is now 274726912. 2025-12-04T11:13:50.0444353Z 2025-12-04T11:13:50.0444521Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.0445068Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_1_in_features_1024_out_features_64_cuda_bfloat16 2025-12-04T11:13:50.0445076Z 2025-12-04T11:13:50.0445243Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.0445383Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.0445488Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.0446041Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('benchmarking.InductorBenchmarker.benchmark_gpu', 2), ('pad_mm_bench', 1), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.0446191Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.0446253Z graph_break [] 2025-12-04T11:13:50.0446385Z ----------------------------- Captured stderr call ----------------------------- 2025-12-04T11:13:50.0447088Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/constant_folding.py:256: UserWarning: Unsupported unwinding pattern: Address not in range (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/profiler/unwind/unwind.cpp:219.) 2025-12-04T11:13:50.0447163Z if out == self.unknown_value: 2025-12-04T11:13:50.0447468Z _ TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_1_in_features_1024_out_features_64_cuda_bfloat16 _ 2025-12-04T11:13:50.0447548Z Traceback (most recent call last): 2025-12-04T11:13:50.0447854Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.0448021Z method(*args, **kwargs) 2025-12-04T11:13:50.0448398Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.0448462Z method(*args, **kwargs) 2025-12-04T11:13:50.0448776Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T11:13:50.0448836Z with policy(): 2025-12-04T11:13:50.0449133Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T11:13:50.0449202Z raise RuntimeError(msg) 2025-12-04T11:13:50.0450039Z RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_1_in_features_1024_out_features_64_cuda_bfloat16! Caching allocator allocated memory was 65536 and is now reported as 131072 on device 0. CUDA driver allocated memory was 274726912 and is now 276824064. 2025-12-04T11:13:50.0450047Z 2025-12-04T11:13:50.0450183Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.0450715Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_1_in_features_1024_out_features_64_cuda_bfloat16 2025-12-04T11:13:50.0450719Z 2025-12-04T11:13:50.0450890Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.0451021Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.0451118Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.0451666Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('benchmarking.InductorBenchmarker.benchmark_gpu', 2), ('pad_mm_bench', 1), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.0451797Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.0451858Z graph_break [] 2025-12-04T11:13:50.0451986Z ----------------------------- Captured stderr call ----------------------------- 2025-12-04T11:13:50.0452741Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/constant_folding.py:256: UserWarning: Unsupported unwinding pattern: Address not in range (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/profiler/unwind/unwind.cpp:219.) 2025-12-04T11:13:50.0452816Z if out == self.unknown_value: 2025-12-04T11:13:50.0452941Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.0453034Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.0474049Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.0475396Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('benchmarking.InductorBenchmarker.benchmark_gpu', 2), ('pad_mm_bench', 1), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.0475562Z graph_break [] 2025-12-04T11:13:50.0475764Z =================================== FAILURES =================================== 2025-12-04T11:13:50.0476472Z _ TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_1_in_features_1024_out_features_64_cuda_bfloat16 _ 2025-12-04T11:13:50.0476656Z Traceback (most recent call last): 2025-12-04T11:13:50.0477388Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.0477548Z method(*args, **kwargs) 2025-12-04T11:13:50.0478551Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.0478729Z method(*args, **kwargs) 2025-12-04T11:13:50.0479526Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T11:13:50.0480144Z with policy(): 2025-12-04T11:13:50.0480878Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T11:13:50.0480961Z raise RuntimeError(msg) 2025-12-04T11:13:50.0481828Z RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_1_in_features_1024_out_features_64_cuda_bfloat16! Caching allocator allocated memory was 131072 and is now reported as 196608 on device 0. CUDA driver allocated memory was 276824064 and is now 278921216. 2025-12-04T11:13:50.0481833Z 2025-12-04T11:13:50.0481966Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.0482492Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_1_in_features_1024_out_features_64_cuda_bfloat16 2025-12-04T11:13:50.0482499Z 2025-12-04T11:13:50.0482661Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.0482788Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.0482880Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.0483425Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('benchmarking.InductorBenchmarker.benchmark_gpu', 2), ('pad_mm_bench', 1), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.0483548Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.0483608Z graph_break [] 2025-12-04T11:13:50.0483732Z ----------------------------- Captured stderr call ----------------------------- 2025-12-04T11:13:50.0484433Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/constant_folding.py:256: UserWarning: Unsupported unwinding pattern: Address not in range (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/profiler/unwind/unwind.cpp:219.) 2025-12-04T11:13:50.0484509Z if out == self.unknown_value: 2025-12-04T11:13:50.0484631Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.0484722Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.0484845Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.0485396Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('benchmarking.InductorBenchmarker.benchmark_gpu', 2), ('pad_mm_bench', 1), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.0485456Z graph_break [] 2025-12-04T11:13:50.0485581Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.0485675Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.0485803Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.0486347Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('benchmarking.InductorBenchmarker.benchmark_gpu', 2), ('pad_mm_bench', 1), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.0486406Z graph_break [] 2025-12-04T11:13:50.0486904Z - generated xml file: /var/lib/jenkins/workspace/test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-294e2abb077cceb2.xml - 2025-12-04T11:13:50.0487012Z =========================== short test summary info ============================ 2025-12-04T11:13:50.0488406Z FAILED [0.5067s] inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_17_mid_dim_1_in_features_1024_out_features_64_cuda_bfloat16 - RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_1_in_features_1024_out_features_64_cuda_bfloat16! Caching allocator allocated memory was 131072 and is now reported as 196608 on device 0. CUDA driver allocated memory was 276824064 and is now 278921216. 2025-12-04T11:13:50.0488477Z 2025-12-04T11:13:50.0488613Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.0489138Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_1_in_features_1024_out_features_64_cuda_bfloat16 2025-12-04T11:13:50.0489142Z 2025-12-04T11:13:50.0489311Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.0489420Z !!!!!!!!!!!!!!!!!!!!!!!!!! stopping after 1 failures !!!!!!!!!!!!!!!!!!!!!!!!!!! 2025-12-04T11:13:50.0489537Z ================== 1 failed, 57 deselected, 2 rerun in 12.11s ================== 2025-12-04T11:13:50.0489602Z Got exit code 1 2025-12-04T11:13:50.0489673Z Retrying single test... 2025-12-04T11:13:50.0489945Z W1204 10:32:00.286000 59654 site-packages/torch/_inductor/utils.py:1703] Not enough SMs to use max_autotune_gemm mode 2025-12-04T11:13:50.0490332Z Test results will be stored in test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-b24025703d1a5b1f.xml 2025-12-04T11:13:50.0490426Z ============================= test session starts ============================== 2025-12-04T11:13:50.0490638Z platform linux -- Python 3.10.14, pytest-7.3.2, pluggy-1.6.0 -- /opt/conda/envs/py_3.10/bin/python 2025-12-04T11:13:50.0490705Z cachedir: .pytest_cache 2025-12-04T11:13:50.0491017Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow] 2025-12-04T11:13:50.0491099Z rootdir: /var/lib/jenkins/workspace 2025-12-04T11:13:50.0491166Z configfile: pytest.ini 2025-12-04T11:13:50.0491489Z plugins: hypothesis-6.56.4, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-14.0, subtests-0.13.1, xdist-3.3.1, xdoctest-1.3.0, typeguard-4.3.0 2025-12-04T11:13:50.0491624Z collecting ... collected 58 items / 57 deselected / 1 selected 2025-12-04T11:13:50.0492197Z stepcurrent: skipping 5 already run items. Running only test/inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_17_mid_dim_1_in_features_1024_out_features_64_cuda_bfloat16 2025-12-04T11:13:50.0492272Z Running 1 items in this shard 2025-12-04T11:13:50.0492276Z 2025-12-04T11:13:50.0493010Z inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_17_mid_dim_1_in_features_1024_out_features_64_cuda_bfloat16 [W1204 10:32:01.958250138 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0493014Z 2025-12-04T11:13:50.0493316Z [W1204 10:32:10.838302532 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0493322Z 2025-12-04T11:13:50.0493616Z [W1204 10:32:10.838555897 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0493620Z 2025-12-04T11:13:50.0493913Z [W1204 10:32:10.844873956 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0493917Z 2025-12-04T11:13:50.0494204Z [W1204 10:32:10.845443878 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0494207Z 2025-12-04T11:13:50.0494499Z [W1204 10:32:10.845630393 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0494502Z 2025-12-04T11:13:50.0494789Z [W1204 10:32:10.851011541 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0495353Z 2025-12-04T11:13:50.0495720Z [W1204 10:32:10.851534012 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0495729Z 2025-12-04T11:13:50.0496018Z [W1204 10:32:10.851694636 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0496022Z 2025-12-04T11:13:50.0496106Z ('RERUN', {'yellow': True}) [10.8945s] [100%] 2025-12-04T11:13:50.0496841Z inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_17_mid_dim_1_in_features_1024_out_features_64_cuda_bfloat16 [W1204 10:32:11.667176670 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0496845Z 2025-12-04T11:13:50.0497134Z [W1204 10:32:11.667705611 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0497141Z 2025-12-04T11:13:50.0497436Z [W1204 10:32:11.667848314 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0497439Z 2025-12-04T11:13:50.0497727Z [W1204 10:32:11.670880121 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0497731Z 2025-12-04T11:13:50.0498022Z [W1204 10:32:11.671351371 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0498026Z 2025-12-04T11:13:50.0498311Z [W1204 10:32:11.671491304 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0498314Z 2025-12-04T11:13:50.0498607Z [W1204 10:32:11.676199688 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0498612Z 2025-12-04T11:13:50.0498908Z [W1204 10:32:11.676685609 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0498914Z 2025-12-04T11:13:50.0499201Z [W1204 10:32:11.676822992 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0499209Z 2025-12-04T11:13:50.0499298Z ('RERUN', {'yellow': True}) [0.5095s] [100%] 2025-12-04T11:13:50.0500020Z inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_17_mid_dim_1_in_features_1024_out_features_64_cuda_bfloat16 [W1204 10:32:12.174040971 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0500024Z 2025-12-04T11:13:50.0500320Z [W1204 10:32:12.174571513 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0500323Z 2025-12-04T11:13:50.0500614Z [W1204 10:32:12.174711526 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0500620Z 2025-12-04T11:13:50.0500913Z [W1204 10:32:12.177694261 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0500916Z 2025-12-04T11:13:50.0501202Z [W1204 10:32:12.178167142 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0501205Z 2025-12-04T11:13:50.0501492Z [W1204 10:32:12.178304455 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0501495Z 2025-12-04T11:13:50.0501780Z [W1204 10:32:12.183115981 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0501783Z 2025-12-04T11:13:50.0502069Z [W1204 10:32:12.183595881 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0502171Z 2025-12-04T11:13:50.0502525Z [W1204 10:32:12.183733614 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0502529Z 2025-12-04T11:13:50.0502592Z FAILED [0.5062s] [100%] 2025-12-04T11:13:50.0502595Z 2025-12-04T11:13:50.0502687Z ==================================== RERUNS ==================================== 2025-12-04T11:13:50.0502983Z _ TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_1_in_features_1024_out_features_64_cuda_bfloat16 _ 2025-12-04T11:13:50.0503062Z Traceback (most recent call last): 2025-12-04T11:13:50.0503368Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.0503435Z method(*args, **kwargs) 2025-12-04T11:13:50.0503812Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.0503884Z method(*args, **kwargs) 2025-12-04T11:13:50.0504178Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T11:13:50.0504243Z with policy(): 2025-12-04T11:13:50.0504538Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T11:13:50.0504610Z raise RuntimeError(msg) 2025-12-04T11:13:50.0505417Z RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_1_in_features_1024_out_features_64_cuda_bfloat16! Caching allocator allocated memory was 0 and is now reported as 65536 on device 0. CUDA driver allocated memory was 230686720 and is now 274726912. 2025-12-04T11:13:50.0505422Z 2025-12-04T11:13:50.0505553Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.0506086Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_1_in_features_1024_out_features_64_cuda_bfloat16 2025-12-04T11:13:50.0506091Z 2025-12-04T11:13:50.0506250Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.0506387Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.0506483Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.0507034Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('benchmarking.InductorBenchmarker.benchmark_gpu', 2), ('pad_mm_bench', 1), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.0507166Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.0507227Z graph_break [] 2025-12-04T11:13:50.0507357Z ----------------------------- Captured stderr call ----------------------------- 2025-12-04T11:13:50.0508051Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/constant_folding.py:256: UserWarning: Unsupported unwinding pattern: Address not in range (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/profiler/unwind/unwind.cpp:219.) 2025-12-04T11:13:50.0508122Z if out == self.unknown_value: 2025-12-04T11:13:50.0508417Z _ TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_1_in_features_1024_out_features_64_cuda_bfloat16 _ 2025-12-04T11:13:50.0508490Z Traceback (most recent call last): 2025-12-04T11:13:50.0508792Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.0508856Z method(*args, **kwargs) 2025-12-04T11:13:50.0509147Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.0509294Z method(*args, **kwargs) 2025-12-04T11:13:50.0509648Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T11:13:50.0509718Z with policy(): 2025-12-04T11:13:50.0510021Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T11:13:50.0510087Z raise RuntimeError(msg) 2025-12-04T11:13:50.0510910Z RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_1_in_features_1024_out_features_64_cuda_bfloat16! Caching allocator allocated memory was 65536 and is now reported as 131072 on device 0. CUDA driver allocated memory was 274726912 and is now 276824064. 2025-12-04T11:13:50.0510914Z 2025-12-04T11:13:50.0511043Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.0511571Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_1_in_features_1024_out_features_64_cuda_bfloat16 2025-12-04T11:13:50.0511578Z 2025-12-04T11:13:50.0511736Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.0511865Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.0511962Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.0512508Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('benchmarking.InductorBenchmarker.benchmark_gpu', 2), ('pad_mm_bench', 1), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.0512640Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.0512698Z graph_break [] 2025-12-04T11:13:50.0512824Z ----------------------------- Captured stderr call ----------------------------- 2025-12-04T11:13:50.0513520Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/constant_folding.py:256: UserWarning: Unsupported unwinding pattern: Address not in range (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/profiler/unwind/unwind.cpp:219.) 2025-12-04T11:13:50.0513593Z if out == self.unknown_value: 2025-12-04T11:13:50.0513720Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.0513813Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.0513937Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.0514482Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('benchmarking.InductorBenchmarker.benchmark_gpu', 2), ('pad_mm_bench', 1), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.0514542Z graph_break [] 2025-12-04T11:13:50.0514631Z =================================== FAILURES =================================== 2025-12-04T11:13:50.0514931Z _ TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_1_in_features_1024_out_features_64_cuda_bfloat16 _ 2025-12-04T11:13:50.0515007Z Traceback (most recent call last): 2025-12-04T11:13:50.0515308Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.0515372Z method(*args, **kwargs) 2025-12-04T11:13:50.0515661Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.0515730Z method(*args, **kwargs) 2025-12-04T11:13:50.0516017Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T11:13:50.0516083Z with policy(): 2025-12-04T11:13:50.0516375Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T11:13:50.0516520Z raise RuntimeError(msg) 2025-12-04T11:13:50.0517406Z RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_1_in_features_1024_out_features_64_cuda_bfloat16! Caching allocator allocated memory was 131072 and is now reported as 196608 on device 0. CUDA driver allocated memory was 276824064 and is now 278921216. 2025-12-04T11:13:50.0517411Z 2025-12-04T11:13:50.0517536Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.0518065Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_1_in_features_1024_out_features_64_cuda_bfloat16 2025-12-04T11:13:50.0518071Z 2025-12-04T11:13:50.0518230Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.0518356Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.0518455Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.0519003Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('benchmarking.InductorBenchmarker.benchmark_gpu', 2), ('pad_mm_bench', 1), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.0519132Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.0519191Z graph_break [] 2025-12-04T11:13:50.0519316Z ----------------------------- Captured stderr call ----------------------------- 2025-12-04T11:13:50.0520005Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/constant_folding.py:256: UserWarning: Unsupported unwinding pattern: Address not in range (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/profiler/unwind/unwind.cpp:219.) 2025-12-04T11:13:50.0520073Z if out == self.unknown_value: 2025-12-04T11:13:50.0520201Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.0520294Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.0520420Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.0520959Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('benchmarking.InductorBenchmarker.benchmark_gpu', 2), ('pad_mm_bench', 1), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.0521018Z graph_break [] 2025-12-04T11:13:50.0521157Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.0521247Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.0521371Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.0521910Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('benchmarking.InductorBenchmarker.benchmark_gpu', 2), ('pad_mm_bench', 1), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.0521972Z graph_break [] 2025-12-04T11:13:50.0522462Z - generated xml file: /var/lib/jenkins/workspace/test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-b24025703d1a5b1f.xml - 2025-12-04T11:13:50.0522566Z =========================== short test summary info ============================ 2025-12-04T11:13:50.0523868Z FAILED [0.5062s] inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_17_mid_dim_1_in_features_1024_out_features_64_cuda_bfloat16 - RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_1_in_features_1024_out_features_64_cuda_bfloat16! Caching allocator allocated memory was 131072 and is now reported as 196608 on device 0. CUDA driver allocated memory was 276824064 and is now 278921216. 2025-12-04T11:13:50.0523952Z 2025-12-04T11:13:50.0524079Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.0524665Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_1_in_features_1024_out_features_64_cuda_bfloat16 2025-12-04T11:13:50.0524669Z 2025-12-04T11:13:50.0524830Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.0524937Z !!!!!!!!!!!!!!!!!!!!!!!!!! stopping after 1 failures !!!!!!!!!!!!!!!!!!!!!!!!!!! 2025-12-04T11:13:50.0525058Z ================== 1 failed, 57 deselected, 2 rerun in 11.94s ================== 2025-12-04T11:13:50.0525118Z Got exit code 1 2025-12-04T11:13:50.0525594Z FAILED CONSISTENTLY: test/inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_17_mid_dim_1_in_features_1024_out_features_64_cuda_bfloat16 2025-12-04T11:13:50.0525848Z Test failed consistently, continuing with the rest of the tests due to continue-through-error being set 2025-12-04T11:13:50.0526116Z W1204 10:32:18.975000 59848 site-packages/torch/_inductor/utils.py:1703] Not enough SMs to use max_autotune_gemm mode 2025-12-04T11:13:50.0526503Z Test results will be stored in test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-531381cfa4745e03.xml 2025-12-04T11:13:50.0526603Z ============================= test session starts ============================== 2025-12-04T11:13:50.0526810Z platform linux -- Python 3.10.14, pytest-7.3.2, pluggy-1.6.0 -- /opt/conda/envs/py_3.10/bin/python 2025-12-04T11:13:50.0526885Z cachedir: .pytest_cache 2025-12-04T11:13:50.0527197Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow] 2025-12-04T11:13:50.0527275Z rootdir: /var/lib/jenkins/workspace 2025-12-04T11:13:50.0527346Z configfile: pytest.ini 2025-12-04T11:13:50.0527662Z plugins: hypothesis-6.56.4, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-14.0, subtests-0.13.1, xdist-3.3.1, xdoctest-1.3.0, typeguard-4.3.0 2025-12-04T11:13:50.0527802Z collecting ... collected 58 items / 6 deselected / 52 selected 2025-12-04T11:13:50.0527889Z stepcurrent: skipping 6 already run items. 2025-12-04T11:13:50.0527959Z Running 52 items in this shard 2025-12-04T11:13:50.0527963Z 2025-12-04T11:13:50.0528470Z inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_17_mid_dim_1_in_features_1024_out_features_65_cuda_bfloat16 ('RERUN', {'yellow': True}) [1.9672s] [ 1%] 2025-12-04T11:13:50.0528959Z inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_17_mid_dim_1_in_features_1024_out_features_65_cuda_bfloat16 ('RERUN', {'yellow': True}) [0.5552s] [ 1%] 2025-12-04T11:13:50.0529418Z inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_17_mid_dim_1_in_features_1024_out_features_65_cuda_bfloat16 FAILED [0.5507s] [ 1%] 2025-12-04T11:13:50.0529424Z 2025-12-04T11:13:50.0529508Z ==================================== RERUNS ==================================== 2025-12-04T11:13:50.0529801Z _ TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_1_in_features_1024_out_features_65_cuda_bfloat16 _ 2025-12-04T11:13:50.0529881Z Traceback (most recent call last): 2025-12-04T11:13:50.0530186Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.0530255Z method(*args, **kwargs) 2025-12-04T11:13:50.0530546Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.0530610Z method(*args, **kwargs) 2025-12-04T11:13:50.0530899Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T11:13:50.0530971Z with policy(): 2025-12-04T11:13:50.0531360Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T11:13:50.0531495Z raise RuntimeError(msg) 2025-12-04T11:13:50.0532299Z RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_1_in_features_1024_out_features_65_cuda_bfloat16! Caching allocator allocated memory was 0 and is now reported as 66560 on device 0. CUDA driver allocated memory was 230686720 and is now 274726912. 2025-12-04T11:13:50.0532304Z 2025-12-04T11:13:50.0532434Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.0532961Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_1_in_features_1024_out_features_65_cuda_bfloat16 2025-12-04T11:13:50.0532964Z 2025-12-04T11:13:50.0533126Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.0533256Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.0533352Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.0533901Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('benchmarking.InductorBenchmarker.benchmark_gpu', 2), ('pad_mm_bench', 1), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.0534029Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.0534090Z graph_break [] 2025-12-04T11:13:50.0534381Z _ TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_1_in_features_1024_out_features_65_cuda_bfloat16 _ 2025-12-04T11:13:50.0534458Z Traceback (most recent call last): 2025-12-04T11:13:50.0534757Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.0534823Z method(*args, **kwargs) 2025-12-04T11:13:50.0535116Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.0535185Z method(*args, **kwargs) 2025-12-04T11:13:50.0535484Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T11:13:50.0535549Z with policy(): 2025-12-04T11:13:50.0535843Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T11:13:50.0535909Z raise RuntimeError(msg) 2025-12-04T11:13:50.0536726Z RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_1_in_features_1024_out_features_65_cuda_bfloat16! Caching allocator allocated memory was 66560 and is now reported as 133120 on device 0. CUDA driver allocated memory was 274726912 and is now 276824064. 2025-12-04T11:13:50.0536733Z 2025-12-04T11:13:50.0536858Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.0537383Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_1_in_features_1024_out_features_65_cuda_bfloat16 2025-12-04T11:13:50.0537387Z 2025-12-04T11:13:50.0537543Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.0537671Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.0537766Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.0538310Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('benchmarking.InductorBenchmarker.benchmark_gpu', 2), ('pad_mm_bench', 1), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.0538442Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.0538578Z graph_break [] 2025-12-04T11:13:50.0538769Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.0538863Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.0538985Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.0539526Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('benchmarking.InductorBenchmarker.benchmark_gpu', 2), ('pad_mm_bench', 1), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.0539588Z graph_break [] 2025-12-04T11:13:50.0539671Z =================================== FAILURES =================================== 2025-12-04T11:13:50.0539964Z _ TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_1_in_features_1024_out_features_65_cuda_bfloat16 _ 2025-12-04T11:13:50.0540039Z Traceback (most recent call last): 2025-12-04T11:13:50.0540340Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.0540406Z method(*args, **kwargs) 2025-12-04T11:13:50.0540697Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.0540763Z method(*args, **kwargs) 2025-12-04T11:13:50.0541051Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T11:13:50.0541109Z with policy(): 2025-12-04T11:13:50.0541404Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T11:13:50.0541469Z raise RuntimeError(msg) 2025-12-04T11:13:50.0542294Z RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_1_in_features_1024_out_features_65_cuda_bfloat16! Caching allocator allocated memory was 133120 and is now reported as 199680 on device 0. CUDA driver allocated memory was 276824064 and is now 278921216. 2025-12-04T11:13:50.0542300Z 2025-12-04T11:13:50.0542424Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.0542945Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_1_in_features_1024_out_features_65_cuda_bfloat16 2025-12-04T11:13:50.0542952Z 2025-12-04T11:13:50.0543106Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.0543232Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.0543327Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.0543911Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('benchmarking.InductorBenchmarker.benchmark_gpu', 2), ('pad_mm_bench', 1), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.0544045Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.0544117Z graph_break [] 2025-12-04T11:13:50.0544245Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.0544336Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.0544455Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.0544993Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('benchmarking.InductorBenchmarker.benchmark_gpu', 2), ('pad_mm_bench', 1), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.0545055Z graph_break [] 2025-12-04T11:13:50.0545176Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.0545269Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.0545467Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.0546069Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('benchmarking.InductorBenchmarker.benchmark_gpu', 2), ('pad_mm_bench', 1), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.0546132Z graph_break [] 2025-12-04T11:13:50.0546623Z - generated xml file: /var/lib/jenkins/workspace/test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-531381cfa4745e03.xml - 2025-12-04T11:13:50.0546726Z =========================== short test summary info ============================ 2025-12-04T11:13:50.0548026Z FAILED [0.5507s] inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_17_mid_dim_1_in_features_1024_out_features_65_cuda_bfloat16 - RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_1_in_features_1024_out_features_65_cuda_bfloat16! Caching allocator allocated memory was 133120 and is now reported as 199680 on device 0. CUDA driver allocated memory was 276824064 and is now 278921216. 2025-12-04T11:13:50.0548033Z 2025-12-04T11:13:50.0548161Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.0548676Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_1_in_features_1024_out_features_65_cuda_bfloat16 2025-12-04T11:13:50.0548679Z 2025-12-04T11:13:50.0548833Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.0548942Z !!!!!!!!!!!!!!!!!!!!!!!!!! stopping after 1 failures !!!!!!!!!!!!!!!!!!!!!!!!!!! 2025-12-04T11:13:50.0549057Z =================== 1 failed, 6 deselected, 2 rerun in 3.10s =================== 2025-12-04T11:13:50.0549123Z Got exit code 1 2025-12-04T11:13:50.0549189Z Retrying single test... 2025-12-04T11:13:50.0549456Z W1204 10:32:29.018000 60037 site-packages/torch/_inductor/utils.py:1703] Not enough SMs to use max_autotune_gemm mode 2025-12-04T11:13:50.0549849Z Test results will be stored in test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-bb14ed3958363693.xml 2025-12-04T11:13:50.0549944Z ============================= test session starts ============================== 2025-12-04T11:13:50.0550153Z platform linux -- Python 3.10.14, pytest-7.3.2, pluggy-1.6.0 -- /opt/conda/envs/py_3.10/bin/python 2025-12-04T11:13:50.0550221Z cachedir: .pytest_cache 2025-12-04T11:13:50.0550526Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow] 2025-12-04T11:13:50.0550609Z rootdir: /var/lib/jenkins/workspace 2025-12-04T11:13:50.0550673Z configfile: pytest.ini 2025-12-04T11:13:50.0550986Z plugins: hypothesis-6.56.4, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-14.0, subtests-0.13.1, xdist-3.3.1, xdoctest-1.3.0, typeguard-4.3.0 2025-12-04T11:13:50.0551136Z collecting ... collected 58 items / 57 deselected / 1 selected 2025-12-04T11:13:50.0551711Z stepcurrent: skipping 6 already run items. Running only test/inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_17_mid_dim_1_in_features_1024_out_features_65_cuda_bfloat16 2025-12-04T11:13:50.0551788Z Running 1 items in this shard 2025-12-04T11:13:50.0551791Z 2025-12-04T11:13:50.0552528Z inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_17_mid_dim_1_in_features_1024_out_features_65_cuda_bfloat16 [W1204 10:32:30.649556821 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0552532Z 2025-12-04T11:13:50.0552831Z [W1204 10:32:39.713021361 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0552914Z 2025-12-04T11:13:50.0553290Z [W1204 10:32:39.713269247 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0553294Z 2025-12-04T11:13:50.0553583Z [W1204 10:32:39.719214867 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0553587Z 2025-12-04T11:13:50.0553877Z [W1204 10:32:39.719782050 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0553880Z 2025-12-04T11:13:50.0554167Z [W1204 10:32:39.719951584 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0554171Z 2025-12-04T11:13:50.0554464Z [W1204 10:32:39.725193449 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0554471Z 2025-12-04T11:13:50.0554763Z [W1204 10:32:39.725708530 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0554766Z 2025-12-04T11:13:50.0555057Z [W1204 10:32:39.725867863 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0555060Z 2025-12-04T11:13:50.0555143Z ('RERUN', {'yellow': True}) [11.0393s] [100%] 2025-12-04T11:13:50.0555891Z inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_17_mid_dim_1_in_features_1024_out_features_65_cuda_bfloat16 [W1204 10:32:40.542127455 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0555895Z 2025-12-04T11:13:50.0556189Z [W1204 10:32:40.542647816 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0556195Z 2025-12-04T11:13:50.0556488Z [W1204 10:32:40.542788839 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0556500Z 2025-12-04T11:13:50.0556796Z [W1204 10:32:40.545736313 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0556799Z 2025-12-04T11:13:50.0557088Z [W1204 10:32:40.546196983 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0557091Z 2025-12-04T11:13:50.0557385Z [W1204 10:32:40.546334316 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0557389Z 2025-12-04T11:13:50.0557676Z [W1204 10:32:40.550996088 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0557682Z 2025-12-04T11:13:50.0557978Z [W1204 10:32:40.551472689 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0557981Z 2025-12-04T11:13:50.0558268Z [W1204 10:32:40.551607172 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0558271Z 2025-12-04T11:13:50.0558356Z ('RERUN', {'yellow': True}) [0.5087s] [100%] 2025-12-04T11:13:50.0559082Z inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_17_mid_dim_1_in_features_1024_out_features_65_cuda_bfloat16 [W1204 10:32:41.047974128 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0559086Z 2025-12-04T11:13:50.0559380Z [W1204 10:32:41.048499949 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0559456Z 2025-12-04T11:13:50.0559812Z [W1204 10:32:41.048646843 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0559816Z 2025-12-04T11:13:50.0560107Z [W1204 10:32:41.051616808 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0560110Z 2025-12-04T11:13:50.0560401Z [W1204 10:32:41.052079228 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0560404Z 2025-12-04T11:13:50.0560693Z [W1204 10:32:41.052215051 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0560697Z 2025-12-04T11:13:50.0560986Z [W1204 10:32:41.056725230 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0560992Z 2025-12-04T11:13:50.0561281Z [W1204 10:32:41.057189800 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0561284Z 2025-12-04T11:13:50.0561577Z [W1204 10:32:41.057323863 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0561581Z 2025-12-04T11:13:50.0561644Z FAILED [0.5060s] [100%] 2025-12-04T11:13:50.0561647Z 2025-12-04T11:13:50.0561735Z ==================================== RERUNS ==================================== 2025-12-04T11:13:50.0562033Z _ TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_1_in_features_1024_out_features_65_cuda_bfloat16 _ 2025-12-04T11:13:50.0562110Z Traceback (most recent call last): 2025-12-04T11:13:50.0562431Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.0562497Z method(*args, **kwargs) 2025-12-04T11:13:50.0562794Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.0562868Z method(*args, **kwargs) 2025-12-04T11:13:50.0563157Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T11:13:50.0563223Z with policy(): 2025-12-04T11:13:50.0563517Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T11:13:50.0563585Z raise RuntimeError(msg) 2025-12-04T11:13:50.0564396Z RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_1_in_features_1024_out_features_65_cuda_bfloat16! Caching allocator allocated memory was 0 and is now reported as 66560 on device 0. CUDA driver allocated memory was 230686720 and is now 274726912. 2025-12-04T11:13:50.0564403Z 2025-12-04T11:13:50.0564533Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.0565062Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_1_in_features_1024_out_features_65_cuda_bfloat16 2025-12-04T11:13:50.0565066Z 2025-12-04T11:13:50.0565227Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.0565361Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.0565466Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.0566017Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('benchmarking.InductorBenchmarker.benchmark_gpu', 2), ('pad_mm_bench', 1), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.0566151Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.0566288Z graph_break [] 2025-12-04T11:13:50.0566415Z ----------------------------- Captured stderr call ----------------------------- 2025-12-04T11:13:50.0567184Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/constant_folding.py:256: UserWarning: Unsupported unwinding pattern: Address not in range (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/profiler/unwind/unwind.cpp:219.) 2025-12-04T11:13:50.0567259Z if out == self.unknown_value: 2025-12-04T11:13:50.0567560Z _ TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_1_in_features_1024_out_features_65_cuda_bfloat16 _ 2025-12-04T11:13:50.0567634Z Traceback (most recent call last): 2025-12-04T11:13:50.0567931Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.0568000Z method(*args, **kwargs) 2025-12-04T11:13:50.0568290Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.0568361Z method(*args, **kwargs) 2025-12-04T11:13:50.0568649Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T11:13:50.0568709Z with policy(): 2025-12-04T11:13:50.0569006Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T11:13:50.0569073Z raise RuntimeError(msg) 2025-12-04T11:13:50.0569890Z RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_1_in_features_1024_out_features_65_cuda_bfloat16! Caching allocator allocated memory was 66560 and is now reported as 133120 on device 0. CUDA driver allocated memory was 274726912 and is now 276824064. 2025-12-04T11:13:50.0569900Z 2025-12-04T11:13:50.0570029Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.0570557Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_1_in_features_1024_out_features_65_cuda_bfloat16 2025-12-04T11:13:50.0570560Z 2025-12-04T11:13:50.0570722Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.0570860Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.0570961Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.0571510Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('benchmarking.InductorBenchmarker.benchmark_gpu', 2), ('pad_mm_bench', 1), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.0571638Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.0571704Z graph_break [] 2025-12-04T11:13:50.0571828Z ----------------------------- Captured stderr call ----------------------------- 2025-12-04T11:13:50.0572526Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/constant_folding.py:256: UserWarning: Unsupported unwinding pattern: Address not in range (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/profiler/unwind/unwind.cpp:219.) 2025-12-04T11:13:50.0572597Z if out == self.unknown_value: 2025-12-04T11:13:50.0572721Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.0572823Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.0572947Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.0573488Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('benchmarking.InductorBenchmarker.benchmark_gpu', 2), ('pad_mm_bench', 1), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.0573554Z graph_break [] 2025-12-04T11:13:50.0573713Z =================================== FAILURES =================================== 2025-12-04T11:13:50.0574076Z _ TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_1_in_features_1024_out_features_65_cuda_bfloat16 _ 2025-12-04T11:13:50.0574153Z Traceback (most recent call last): 2025-12-04T11:13:50.0574450Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.0574521Z method(*args, **kwargs) 2025-12-04T11:13:50.0574810Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.0574878Z method(*args, **kwargs) 2025-12-04T11:13:50.0575167Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T11:13:50.0575230Z with policy(): 2025-12-04T11:13:50.0575531Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T11:13:50.0575601Z raise RuntimeError(msg) 2025-12-04T11:13:50.0576419Z RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_1_in_features_1024_out_features_65_cuda_bfloat16! Caching allocator allocated memory was 133120 and is now reported as 199680 on device 0. CUDA driver allocated memory was 276824064 and is now 278921216. 2025-12-04T11:13:50.0576428Z 2025-12-04T11:13:50.0576553Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.0577084Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_1_in_features_1024_out_features_65_cuda_bfloat16 2025-12-04T11:13:50.0577088Z 2025-12-04T11:13:50.0577251Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.0577381Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.0577479Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.0578178Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('benchmarking.InductorBenchmarker.benchmark_gpu', 2), ('pad_mm_bench', 1), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.0578310Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.0578384Z graph_break [] 2025-12-04T11:13:50.0578511Z ----------------------------- Captured stderr call ----------------------------- 2025-12-04T11:13:50.0579202Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/constant_folding.py:256: UserWarning: Unsupported unwinding pattern: Address not in range (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/profiler/unwind/unwind.cpp:219.) 2025-12-04T11:13:50.0579273Z if out == self.unknown_value: 2025-12-04T11:13:50.0579400Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.0579500Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.0579623Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.0580167Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('benchmarking.InductorBenchmarker.benchmark_gpu', 2), ('pad_mm_bench', 1), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.0580232Z graph_break [] 2025-12-04T11:13:50.0580355Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.0580451Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.0580574Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.0581115Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('benchmarking.InductorBenchmarker.benchmark_gpu', 2), ('pad_mm_bench', 1), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.0581398Z graph_break [] 2025-12-04T11:13:50.0581898Z - generated xml file: /var/lib/jenkins/workspace/test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-bb14ed3958363693.xml - 2025-12-04T11:13:50.0582004Z =========================== short test summary info ============================ 2025-12-04T11:13:50.0583321Z FAILED [0.5060s] inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_17_mid_dim_1_in_features_1024_out_features_65_cuda_bfloat16 - RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_1_in_features_1024_out_features_65_cuda_bfloat16! Caching allocator allocated memory was 133120 and is now reported as 199680 on device 0. CUDA driver allocated memory was 276824064 and is now 278921216. 2025-12-04T11:13:50.0583329Z 2025-12-04T11:13:50.0583463Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.0584025Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_1_in_features_1024_out_features_65_cuda_bfloat16 2025-12-04T11:13:50.0584028Z 2025-12-04T11:13:50.0584192Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.0584298Z !!!!!!!!!!!!!!!!!!!!!!!!!! stopping after 1 failures !!!!!!!!!!!!!!!!!!!!!!!!!!! 2025-12-04T11:13:50.0584415Z ================== 1 failed, 57 deselected, 2 rerun in 12.08s ================== 2025-12-04T11:13:50.0584481Z Got exit code 1 2025-12-04T11:13:50.0584546Z Retrying single test... 2025-12-04T11:13:50.0584813Z W1204 10:32:47.923000 60231 site-packages/torch/_inductor/utils.py:1703] Not enough SMs to use max_autotune_gemm mode 2025-12-04T11:13:50.0585209Z Test results will be stored in test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-ec0356b9a2627168.xml 2025-12-04T11:13:50.0585307Z ============================= test session starts ============================== 2025-12-04T11:13:50.0585521Z platform linux -- Python 3.10.14, pytest-7.3.2, pluggy-1.6.0 -- /opt/conda/envs/py_3.10/bin/python 2025-12-04T11:13:50.0585591Z cachedir: .pytest_cache 2025-12-04T11:13:50.0585897Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow] 2025-12-04T11:13:50.0585986Z rootdir: /var/lib/jenkins/workspace 2025-12-04T11:13:50.0586052Z configfile: pytest.ini 2025-12-04T11:13:50.0586369Z plugins: hypothesis-6.56.4, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-14.0, subtests-0.13.1, xdist-3.3.1, xdoctest-1.3.0, typeguard-4.3.0 2025-12-04T11:13:50.0586517Z collecting ... collected 58 items / 57 deselected / 1 selected 2025-12-04T11:13:50.0587090Z stepcurrent: skipping 6 already run items. Running only test/inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_17_mid_dim_1_in_features_1024_out_features_65_cuda_bfloat16 2025-12-04T11:13:50.0587171Z Running 1 items in this shard 2025-12-04T11:13:50.0587175Z 2025-12-04T11:13:50.0587907Z inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_17_mid_dim_1_in_features_1024_out_features_65_cuda_bfloat16 [W1204 10:32:49.573747346 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0587911Z 2025-12-04T11:13:50.0588215Z [W1204 10:32:58.550429786 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0588218Z 2025-12-04T11:13:50.0588507Z [W1204 10:32:58.550675151 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0588590Z 2025-12-04T11:13:50.0588969Z [W1204 10:32:58.556746885 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0588979Z 2025-12-04T11:13:50.0589273Z [W1204 10:32:58.557337138 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0589276Z 2025-12-04T11:13:50.0589566Z [W1204 10:32:58.557520631 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0589570Z 2025-12-04T11:13:50.0589866Z [W1204 10:32:58.563029263 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0589869Z 2025-12-04T11:13:50.0590156Z [W1204 10:32:58.563555654 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0590162Z 2025-12-04T11:13:50.0590461Z [W1204 10:32:58.563718588 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0590464Z 2025-12-04T11:13:50.0590546Z ('RERUN', {'yellow': True}) [10.9673s] [100%] 2025-12-04T11:13:50.0591282Z inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_17_mid_dim_1_in_features_1024_out_features_65_cuda_bfloat16 [W1204 10:32:59.376733593 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0591286Z 2025-12-04T11:13:50.0591577Z [W1204 10:32:59.377260234 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0591581Z 2025-12-04T11:13:50.0591875Z [W1204 10:32:59.377401027 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0591882Z 2025-12-04T11:13:50.0592173Z [W1204 10:32:59.380375153 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0592176Z 2025-12-04T11:13:50.0592466Z [W1204 10:32:59.380855133 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0592475Z 2025-12-04T11:13:50.0592762Z [W1204 10:32:59.380992036 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0592765Z 2025-12-04T11:13:50.0593053Z [W1204 10:32:59.385602418 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0593056Z 2025-12-04T11:13:50.0593349Z [W1204 10:32:59.386072918 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0593352Z 2025-12-04T11:13:50.0593642Z [W1204 10:32:59.386208921 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0593648Z 2025-12-04T11:13:50.0593733Z ('RERUN', {'yellow': True}) [0.5063s] [100%] 2025-12-04T11:13:50.0594460Z inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_17_mid_dim_1_in_features_1024_out_features_65_cuda_bfloat16 [W1204 10:32:59.879449347 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0594463Z 2025-12-04T11:13:50.0594760Z [W1204 10:32:59.879974439 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0594763Z 2025-12-04T11:13:50.0595053Z [W1204 10:32:59.880163643 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0595056Z 2025-12-04T11:13:50.0595433Z [W1204 10:32:59.883124518 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0595499Z 2025-12-04T11:13:50.0595790Z [W1204 10:32:59.883585418 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0595793Z 2025-12-04T11:13:50.0596082Z [W1204 10:32:59.883720001 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0596085Z 2025-12-04T11:13:50.0596376Z [W1204 10:32:59.888286221 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0596379Z 2025-12-04T11:13:50.0596667Z [W1204 10:32:59.888772272 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0596670Z 2025-12-04T11:13:50.0596965Z [W1204 10:32:59.888907325 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0596968Z 2025-12-04T11:13:50.0597032Z FAILED [0.5028s] [100%] 2025-12-04T11:13:50.0597036Z 2025-12-04T11:13:50.0597125Z ==================================== RERUNS ==================================== 2025-12-04T11:13:50.0597425Z _ TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_1_in_features_1024_out_features_65_cuda_bfloat16 _ 2025-12-04T11:13:50.0597500Z Traceback (most recent call last): 2025-12-04T11:13:50.0597816Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.0597881Z method(*args, **kwargs) 2025-12-04T11:13:50.0598178Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.0598242Z method(*args, **kwargs) 2025-12-04T11:13:50.0598535Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T11:13:50.0598602Z with policy(): 2025-12-04T11:13:50.0598893Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T11:13:50.0598959Z raise RuntimeError(msg) 2025-12-04T11:13:50.0599763Z RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_1_in_features_1024_out_features_65_cuda_bfloat16! Caching allocator allocated memory was 0 and is now reported as 66560 on device 0. CUDA driver allocated memory was 230686720 and is now 274726912. 2025-12-04T11:13:50.0599767Z 2025-12-04T11:13:50.0599896Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.0600429Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_1_in_features_1024_out_features_65_cuda_bfloat16 2025-12-04T11:13:50.0600435Z 2025-12-04T11:13:50.0600596Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.0600730Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.0600829Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.0601377Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('benchmarking.InductorBenchmarker.benchmark_gpu', 2), ('pad_mm_bench', 1), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.0601510Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.0601572Z graph_break [] 2025-12-04T11:13:50.0601699Z ----------------------------- Captured stderr call ----------------------------- 2025-12-04T11:13:50.0602469Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/constant_folding.py:256: UserWarning: Unsupported unwinding pattern: Address not in range (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/profiler/unwind/unwind.cpp:219.) 2025-12-04T11:13:50.0602605Z if out == self.unknown_value: 2025-12-04T11:13:50.0602905Z _ TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_1_in_features_1024_out_features_65_cuda_bfloat16 _ 2025-12-04T11:13:50.0602979Z Traceback (most recent call last): 2025-12-04T11:13:50.0603278Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.0603351Z method(*args, **kwargs) 2025-12-04T11:13:50.0603642Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.0603712Z method(*args, **kwargs) 2025-12-04T11:13:50.0604009Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T11:13:50.0604076Z with policy(): 2025-12-04T11:13:50.0604376Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T11:13:50.0604444Z raise RuntimeError(msg) 2025-12-04T11:13:50.0605263Z RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_1_in_features_1024_out_features_65_cuda_bfloat16! Caching allocator allocated memory was 66560 and is now reported as 133120 on device 0. CUDA driver allocated memory was 274726912 and is now 276824064. 2025-12-04T11:13:50.0605267Z 2025-12-04T11:13:50.0605393Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.0605916Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_1_in_features_1024_out_features_65_cuda_bfloat16 2025-12-04T11:13:50.0605928Z 2025-12-04T11:13:50.0606087Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.0606217Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.0606318Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.0606861Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('benchmarking.InductorBenchmarker.benchmark_gpu', 2), ('pad_mm_bench', 1), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.0606989Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.0607054Z graph_break [] 2025-12-04T11:13:50.0607177Z ----------------------------- Captured stderr call ----------------------------- 2025-12-04T11:13:50.0608033Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/constant_folding.py:256: UserWarning: Unsupported unwinding pattern: Address not in range (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/profiler/unwind/unwind.cpp:219.) 2025-12-04T11:13:50.0608114Z if out == self.unknown_value: 2025-12-04T11:13:50.0608244Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.0608343Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.0608470Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.0609016Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('benchmarking.InductorBenchmarker.benchmark_gpu', 2), ('pad_mm_bench', 1), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.0609078Z graph_break [] 2025-12-04T11:13:50.0609162Z =================================== FAILURES =================================== 2025-12-04T11:13:50.0609460Z _ TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_1_in_features_1024_out_features_65_cuda_bfloat16 _ 2025-12-04T11:13:50.0609648Z Traceback (most recent call last): 2025-12-04T11:13:50.0610015Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.0610088Z method(*args, **kwargs) 2025-12-04T11:13:50.0610379Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.0610449Z method(*args, **kwargs) 2025-12-04T11:13:50.0610741Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T11:13:50.0610807Z with policy(): 2025-12-04T11:13:50.0611107Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T11:13:50.0611173Z raise RuntimeError(msg) 2025-12-04T11:13:50.0612000Z RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_1_in_features_1024_out_features_65_cuda_bfloat16! Caching allocator allocated memory was 133120 and is now reported as 199680 on device 0. CUDA driver allocated memory was 276824064 and is now 278921216. 2025-12-04T11:13:50.0612008Z 2025-12-04T11:13:50.0612134Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.0612656Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_1_in_features_1024_out_features_65_cuda_bfloat16 2025-12-04T11:13:50.0612668Z 2025-12-04T11:13:50.0612824Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.0612950Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.0613047Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.0613597Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('benchmarking.InductorBenchmarker.benchmark_gpu', 2), ('pad_mm_bench', 1), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.0613724Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.0613790Z graph_break [] 2025-12-04T11:13:50.0613914Z ----------------------------- Captured stderr call ----------------------------- 2025-12-04T11:13:50.0614607Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/constant_folding.py:256: UserWarning: Unsupported unwinding pattern: Address not in range (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/profiler/unwind/unwind.cpp:219.) 2025-12-04T11:13:50.0614676Z if out == self.unknown_value: 2025-12-04T11:13:50.0614800Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.0614895Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.0615018Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.0615570Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('benchmarking.InductorBenchmarker.benchmark_gpu', 2), ('pad_mm_bench', 1), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.0615630Z graph_break [] 2025-12-04T11:13:50.0615755Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.0615848Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.0615972Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.0616512Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('benchmarking.InductorBenchmarker.benchmark_gpu', 2), ('pad_mm_bench', 1), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.0616572Z graph_break [] 2025-12-04T11:13:50.0617057Z - generated xml file: /var/lib/jenkins/workspace/test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-ec0356b9a2627168.xml - 2025-12-04T11:13:50.0617320Z =========================== short test summary info ============================ 2025-12-04T11:13:50.0618626Z FAILED [0.5028s] inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_17_mid_dim_1_in_features_1024_out_features_65_cuda_bfloat16 - RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_1_in_features_1024_out_features_65_cuda_bfloat16! Caching allocator allocated memory was 133120 and is now reported as 199680 on device 0. CUDA driver allocated memory was 276824064 and is now 278921216. 2025-12-04T11:13:50.0618630Z 2025-12-04T11:13:50.0618759Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.0619281Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_1_in_features_1024_out_features_65_cuda_bfloat16 2025-12-04T11:13:50.0619290Z 2025-12-04T11:13:50.0619454Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.0619569Z !!!!!!!!!!!!!!!!!!!!!!!!!! stopping after 1 failures !!!!!!!!!!!!!!!!!!!!!!!!!!! 2025-12-04T11:13:50.0619683Z ================== 1 failed, 57 deselected, 2 rerun in 12.00s ================== 2025-12-04T11:13:50.0619747Z Got exit code 1 2025-12-04T11:13:50.0620226Z FAILED CONSISTENTLY: test/inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_17_mid_dim_1_in_features_1024_out_features_65_cuda_bfloat16 2025-12-04T11:13:50.0620478Z Test failed consistently, continuing with the rest of the tests due to continue-through-error being set 2025-12-04T11:13:50.0620744Z W1204 10:33:06.760000 60425 site-packages/torch/_inductor/utils.py:1703] Not enough SMs to use max_autotune_gemm mode 2025-12-04T11:13:50.0621135Z Test results will be stored in test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-c81b585b68f6cf60.xml 2025-12-04T11:13:50.0621236Z ============================= test session starts ============================== 2025-12-04T11:13:50.0621444Z platform linux -- Python 3.10.14, pytest-7.3.2, pluggy-1.6.0 -- /opt/conda/envs/py_3.10/bin/python 2025-12-04T11:13:50.0621513Z cachedir: .pytest_cache 2025-12-04T11:13:50.0621827Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow] 2025-12-04T11:13:50.0621903Z rootdir: /var/lib/jenkins/workspace 2025-12-04T11:13:50.0621976Z configfile: pytest.ini 2025-12-04T11:13:50.0622289Z plugins: hypothesis-6.56.4, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-14.0, subtests-0.13.1, xdist-3.3.1, xdoctest-1.3.0, typeguard-4.3.0 2025-12-04T11:13:50.0622420Z collecting ... collected 58 items / 7 deselected / 51 selected 2025-12-04T11:13:50.0622517Z stepcurrent: skipping 7 already run items. 2025-12-04T11:13:50.0622587Z Running 51 items in this shard 2025-12-04T11:13:50.0622593Z 2025-12-04T11:13:50.0623106Z inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_17_mid_dim_1_in_features_128_out_features_1024_cuda_bfloat16 ('RERUN', {'yellow': True}) [1.9383s] [ 1%] 2025-12-04T11:13:50.0623696Z inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_17_mid_dim_1_in_features_128_out_features_1024_cuda_bfloat16 ('RERUN', {'yellow': True}) [0.4865s] [ 1%] 2025-12-04T11:13:50.0624149Z inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_17_mid_dim_1_in_features_128_out_features_1024_cuda_bfloat16 FAILED [0.4808s] [ 1%] 2025-12-04T11:13:50.0624158Z 2025-12-04T11:13:50.0624242Z ==================================== RERUNS ==================================== 2025-12-04T11:13:50.0624618Z _ TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_1_in_features_128_out_features_1024_cuda_bfloat16 _ 2025-12-04T11:13:50.0624766Z Traceback (most recent call last): 2025-12-04T11:13:50.0625070Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.0625136Z method(*args, **kwargs) 2025-12-04T11:13:50.0625430Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.0625493Z method(*args, **kwargs) 2025-12-04T11:13:50.0625787Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T11:13:50.0625848Z with policy(): 2025-12-04T11:13:50.0626139Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T11:13:50.0626211Z raise RuntimeError(msg) 2025-12-04T11:13:50.0627028Z RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_1_in_features_128_out_features_1024_cuda_bfloat16! Caching allocator allocated memory was 0 and is now reported as 131072 on device 0. CUDA driver allocated memory was 230686720 and is now 274726912. 2025-12-04T11:13:50.0627033Z 2025-12-04T11:13:50.0627165Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.0627689Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_1_in_features_128_out_features_1024_cuda_bfloat16 2025-12-04T11:13:50.0627692Z 2025-12-04T11:13:50.0627851Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.0627985Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.0628081Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.0628443Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.0628574Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.0628634Z graph_break [] 2025-12-04T11:13:50.0628933Z _ TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_1_in_features_128_out_features_1024_cuda_bfloat16 _ 2025-12-04T11:13:50.0629008Z Traceback (most recent call last): 2025-12-04T11:13:50.0629315Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.0629387Z method(*args, **kwargs) 2025-12-04T11:13:50.0629679Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.0629752Z method(*args, **kwargs) 2025-12-04T11:13:50.0630046Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T11:13:50.0630110Z with policy(): 2025-12-04T11:13:50.0630408Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T11:13:50.0630476Z raise RuntimeError(msg) 2025-12-04T11:13:50.0631309Z RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_1_in_features_128_out_features_1024_cuda_bfloat16! Caching allocator allocated memory was 131072 and is now reported as 262144 on device 0. CUDA driver allocated memory was 274726912 and is now 276824064. 2025-12-04T11:13:50.0631313Z 2025-12-04T11:13:50.0631439Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.0631964Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_1_in_features_128_out_features_1024_cuda_bfloat16 2025-12-04T11:13:50.0632048Z 2025-12-04T11:13:50.0632270Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.0632398Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.0632496Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.0632850Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.0632987Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.0633054Z graph_break [] 2025-12-04T11:13:50.0633182Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.0633278Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.0633401Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.0633750Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.0633815Z graph_break [] 2025-12-04T11:13:50.0633903Z =================================== FAILURES =================================== 2025-12-04T11:13:50.0634195Z _ TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_1_in_features_128_out_features_1024_cuda_bfloat16 _ 2025-12-04T11:13:50.0634276Z Traceback (most recent call last): 2025-12-04T11:13:50.0634579Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.0634648Z method(*args, **kwargs) 2025-12-04T11:13:50.0634941Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.0635004Z method(*args, **kwargs) 2025-12-04T11:13:50.0635301Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T11:13:50.0635370Z with policy(): 2025-12-04T11:13:50.0635667Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T11:13:50.0635733Z raise RuntimeError(msg) 2025-12-04T11:13:50.0636550Z RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_1_in_features_128_out_features_1024_cuda_bfloat16! Caching allocator allocated memory was 262144 and is now reported as 393216 on device 0. CUDA driver allocated memory was 276824064 and is now 278921216. 2025-12-04T11:13:50.0636554Z 2025-12-04T11:13:50.0636684Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.0637205Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_1_in_features_128_out_features_1024_cuda_bfloat16 2025-12-04T11:13:50.0637211Z 2025-12-04T11:13:50.0637373Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.0637499Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.0637600Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.0637956Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.0638081Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.0638144Z graph_break [] 2025-12-04T11:13:50.0638267Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.0638357Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.0638481Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.0638991Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.0639052Z graph_break [] 2025-12-04T11:13:50.0639180Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.0639269Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.0639393Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.0639735Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.0639793Z graph_break [] 2025-12-04T11:13:50.0640284Z - generated xml file: /var/lib/jenkins/workspace/test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-c81b585b68f6cf60.xml - 2025-12-04T11:13:50.0640387Z =========================== short test summary info ============================ 2025-12-04T11:13:50.0641695Z FAILED [0.4808s] inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_17_mid_dim_1_in_features_128_out_features_1024_cuda_bfloat16 - RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_1_in_features_128_out_features_1024_cuda_bfloat16! Caching allocator allocated memory was 262144 and is now reported as 393216 on device 0. CUDA driver allocated memory was 276824064 and is now 278921216. 2025-12-04T11:13:50.0641700Z 2025-12-04T11:13:50.0641824Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.0642354Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_1_in_features_128_out_features_1024_cuda_bfloat16 2025-12-04T11:13:50.0642360Z 2025-12-04T11:13:50.0642517Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.0642625Z !!!!!!!!!!!!!!!!!!!!!!!!!! stopping after 1 failures !!!!!!!!!!!!!!!!!!!!!!!!!!! 2025-12-04T11:13:50.0642746Z =================== 1 failed, 7 deselected, 2 rerun in 2.93s =================== 2025-12-04T11:13:50.0642805Z Got exit code 1 2025-12-04T11:13:50.0642875Z Retrying single test... 2025-12-04T11:13:50.0643140Z W1204 10:33:16.859000 60613 site-packages/torch/_inductor/utils.py:1703] Not enough SMs to use max_autotune_gemm mode 2025-12-04T11:13:50.0643530Z Test results will be stored in test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-647efcde8452d0c7.xml 2025-12-04T11:13:50.0643630Z ============================= test session starts ============================== 2025-12-04T11:13:50.0643840Z platform linux -- Python 3.10.14, pytest-7.3.2, pluggy-1.6.0 -- /opt/conda/envs/py_3.10/bin/python 2025-12-04T11:13:50.0643911Z cachedir: .pytest_cache 2025-12-04T11:13:50.0644226Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow] 2025-12-04T11:13:50.0644302Z rootdir: /var/lib/jenkins/workspace 2025-12-04T11:13:50.0644371Z configfile: pytest.ini 2025-12-04T11:13:50.0644690Z plugins: hypothesis-6.56.4, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-14.0, subtests-0.13.1, xdist-3.3.1, xdoctest-1.3.0, typeguard-4.3.0 2025-12-04T11:13:50.0644819Z collecting ... collected 58 items / 57 deselected / 1 selected 2025-12-04T11:13:50.0645398Z stepcurrent: skipping 7 already run items. Running only test/inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_17_mid_dim_1_in_features_128_out_features_1024_cuda_bfloat16 2025-12-04T11:13:50.0645469Z Running 1 items in this shard 2025-12-04T11:13:50.0645473Z 2025-12-04T11:13:50.0646306Z inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_17_mid_dim_1_in_features_128_out_features_1024_cuda_bfloat16 [W1204 10:33:18.204716939 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0646373Z 2025-12-04T11:13:50.0646678Z [W1204 10:33:27.408957231 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0646682Z 2025-12-04T11:13:50.0646977Z [W1204 10:33:27.409185196 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0646981Z 2025-12-04T11:13:50.0647270Z [W1204 10:33:27.414816940 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0647274Z 2025-12-04T11:13:50.0647566Z [W1204 10:33:27.415376422 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0647577Z 2025-12-04T11:13:50.0647870Z [W1204 10:33:27.415554786 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0647873Z 2025-12-04T11:13:50.0648160Z [W1204 10:33:27.421118428 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0648164Z 2025-12-04T11:13:50.0648458Z [W1204 10:33:27.421649570 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0648462Z 2025-12-04T11:13:50.0648751Z [W1204 10:33:27.421805464 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0648754Z 2025-12-04T11:13:50.0648840Z ('RERUN', {'yellow': True}) [11.1600s] [100%] 2025-12-04T11:13:50.0649574Z inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_17_mid_dim_1_in_features_128_out_features_1024_cuda_bfloat16 [W1204 10:33:28.454841194 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0649580Z 2025-12-04T11:13:50.0649876Z [W1204 10:33:28.455384256 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0649880Z 2025-12-04T11:13:50.0650169Z [W1204 10:33:28.455520609 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0650172Z 2025-12-04T11:13:50.0650463Z [W1204 10:33:28.458542575 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0650466Z 2025-12-04T11:13:50.0650894Z [W1204 10:33:28.459117748 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0650905Z 2025-12-04T11:13:50.0651380Z [W1204 10:33:28.459255061 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0651392Z 2025-12-04T11:13:50.0651801Z [W1204 10:33:28.463926393 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0651805Z 2025-12-04T11:13:50.0652096Z [W1204 10:33:28.464403973 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0652100Z 2025-12-04T11:13:50.0652395Z [W1204 10:33:28.464539606 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0652398Z 2025-12-04T11:13:50.0652480Z ('RERUN', {'yellow': True}) [0.4536s] [100%] 2025-12-04T11:13:50.0653314Z inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_17_mid_dim_1_in_features_128_out_features_1024_cuda_bfloat16 [W1204 10:33:28.904048108 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0653381Z 2025-12-04T11:13:50.0653678Z [W1204 10:33:28.904596040 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0653681Z 2025-12-04T11:13:50.0653978Z [W1204 10:33:28.904733032 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0653981Z 2025-12-04T11:13:50.0654269Z [W1204 10:33:28.907703657 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0654272Z 2025-12-04T11:13:50.0654562Z [W1204 10:33:28.908264930 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0654572Z 2025-12-04T11:13:50.0654867Z [W1204 10:33:28.908401723 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0654870Z 2025-12-04T11:13:50.0655161Z [W1204 10:33:28.913085905 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0655165Z 2025-12-04T11:13:50.0655460Z [W1204 10:33:28.913557845 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0655463Z 2025-12-04T11:13:50.0655749Z [W1204 10:33:28.913690148 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0655752Z 2025-12-04T11:13:50.0655825Z FAILED [0.4476s] [100%] 2025-12-04T11:13:50.0655828Z 2025-12-04T11:13:50.0655913Z ==================================== RERUNS ==================================== 2025-12-04T11:13:50.0656226Z _ TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_1_in_features_128_out_features_1024_cuda_bfloat16 _ 2025-12-04T11:13:50.0656314Z Traceback (most recent call last): 2025-12-04T11:13:50.0656626Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.0656698Z method(*args, **kwargs) 2025-12-04T11:13:50.0656996Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.0657061Z method(*args, **kwargs) 2025-12-04T11:13:50.0657359Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T11:13:50.0657422Z with policy(): 2025-12-04T11:13:50.0657730Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T11:13:50.0657797Z raise RuntimeError(msg) 2025-12-04T11:13:50.0658619Z RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_1_in_features_128_out_features_1024_cuda_bfloat16! Caching allocator allocated memory was 0 and is now reported as 131072 on device 0. CUDA driver allocated memory was 230686720 and is now 274726912. 2025-12-04T11:13:50.0658623Z 2025-12-04T11:13:50.0658760Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.0659292Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_1_in_features_128_out_features_1024_cuda_bfloat16 2025-12-04T11:13:50.0659295Z 2025-12-04T11:13:50.0659461Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.0659595Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.0659692Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.0660205Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.0660336Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.0660400Z graph_break [] 2025-12-04T11:13:50.0660527Z ----------------------------- Captured stderr call ----------------------------- 2025-12-04T11:13:50.0661224Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/constant_folding.py:256: UserWarning: Unsupported unwinding pattern: Address not in range (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/profiler/unwind/unwind.cpp:219.) 2025-12-04T11:13:50.0661301Z if out == self.unknown_value: 2025-12-04T11:13:50.0661600Z _ TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_1_in_features_128_out_features_1024_cuda_bfloat16 _ 2025-12-04T11:13:50.0661677Z Traceback (most recent call last): 2025-12-04T11:13:50.0661983Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.0662049Z method(*args, **kwargs) 2025-12-04T11:13:50.0662346Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.0662410Z method(*args, **kwargs) 2025-12-04T11:13:50.0662698Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T11:13:50.0662763Z with policy(): 2025-12-04T11:13:50.0663070Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T11:13:50.0663145Z raise RuntimeError(msg) 2025-12-04T11:13:50.0664060Z RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_1_in_features_128_out_features_1024_cuda_bfloat16! Caching allocator allocated memory was 131072 and is now reported as 262144 on device 0. CUDA driver allocated memory was 274726912 and is now 276824064. 2025-12-04T11:13:50.0664068Z 2025-12-04T11:13:50.0664201Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.0664724Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_1_in_features_128_out_features_1024_cuda_bfloat16 2025-12-04T11:13:50.0664727Z 2025-12-04T11:13:50.0664885Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.0665015Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.0665109Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.0665464Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.0665594Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.0665657Z graph_break [] 2025-12-04T11:13:50.0665788Z ----------------------------- Captured stderr call ----------------------------- 2025-12-04T11:13:50.0666481Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/constant_folding.py:256: UserWarning: Unsupported unwinding pattern: Address not in range (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/profiler/unwind/unwind.cpp:219.) 2025-12-04T11:13:50.0666552Z if out == self.unknown_value: 2025-12-04T11:13:50.0666681Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.0666774Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.0666904Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.0667250Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.0667392Z graph_break [] 2025-12-04T11:13:50.0667546Z =================================== FAILURES =================================== 2025-12-04T11:13:50.0667846Z _ TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_1_in_features_128_out_features_1024_cuda_bfloat16 _ 2025-12-04T11:13:50.0667924Z Traceback (most recent call last): 2025-12-04T11:13:50.0668225Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.0668290Z method(*args, **kwargs) 2025-12-04T11:13:50.0668587Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.0668651Z method(*args, **kwargs) 2025-12-04T11:13:50.0668942Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T11:13:50.0669012Z with policy(): 2025-12-04T11:13:50.0669316Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T11:13:50.0669387Z raise RuntimeError(msg) 2025-12-04T11:13:50.0670210Z RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_1_in_features_128_out_features_1024_cuda_bfloat16! Caching allocator allocated memory was 262144 and is now reported as 393216 on device 0. CUDA driver allocated memory was 276824064 and is now 278921216. 2025-12-04T11:13:50.0670214Z 2025-12-04T11:13:50.0670352Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.0670884Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_1_in_features_128_out_features_1024_cuda_bfloat16 2025-12-04T11:13:50.0670890Z 2025-12-04T11:13:50.0671051Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.0671190Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.0671284Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.0671628Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.0671760Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.0671819Z graph_break [] 2025-12-04T11:13:50.0671948Z ----------------------------- Captured stderr call ----------------------------- 2025-12-04T11:13:50.0672634Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/constant_folding.py:256: UserWarning: Unsupported unwinding pattern: Address not in range (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/profiler/unwind/unwind.cpp:219.) 2025-12-04T11:13:50.0672707Z if out == self.unknown_value: 2025-12-04T11:13:50.0672842Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.0672935Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.0673063Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.0673406Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.0673468Z graph_break [] 2025-12-04T11:13:50.0673598Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.0673690Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.0673814Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.0674160Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.0674319Z graph_break [] 2025-12-04T11:13:50.0674880Z - generated xml file: /var/lib/jenkins/workspace/test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-647efcde8452d0c7.xml - 2025-12-04T11:13:50.0674985Z =========================== short test summary info ============================ 2025-12-04T11:13:50.0676307Z FAILED [0.4476s] inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_17_mid_dim_1_in_features_128_out_features_1024_cuda_bfloat16 - RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_1_in_features_128_out_features_1024_cuda_bfloat16! Caching allocator allocated memory was 262144 and is now reported as 393216 on device 0. CUDA driver allocated memory was 276824064 and is now 278921216. 2025-12-04T11:13:50.0676315Z 2025-12-04T11:13:50.0676439Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.0676964Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_1_in_features_128_out_features_1024_cuda_bfloat16 2025-12-04T11:13:50.0676972Z 2025-12-04T11:13:50.0677130Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.0677233Z !!!!!!!!!!!!!!!!!!!!!!!!!! stopping after 1 failures !!!!!!!!!!!!!!!!!!!!!!!!!!! 2025-12-04T11:13:50.0677356Z ================== 1 failed, 57 deselected, 2 rerun in 12.09s ================== 2025-12-04T11:13:50.0677417Z Got exit code 1 2025-12-04T11:13:50.0677486Z Retrying single test... 2025-12-04T11:13:50.0677756Z W1204 10:33:35.733000 60806 site-packages/torch/_inductor/utils.py:1703] Not enough SMs to use max_autotune_gemm mode 2025-12-04T11:13:50.0678442Z Test results will be stored in test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-abfd2b283c5f3df6.xml 2025-12-04T11:13:50.0678564Z ============================= test session starts ============================== 2025-12-04T11:13:50.0678778Z platform linux -- Python 3.10.14, pytest-7.3.2, pluggy-1.6.0 -- /opt/conda/envs/py_3.10/bin/python 2025-12-04T11:13:50.0678847Z cachedir: .pytest_cache 2025-12-04T11:13:50.0679163Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow] 2025-12-04T11:13:50.0679242Z rootdir: /var/lib/jenkins/workspace 2025-12-04T11:13:50.0679309Z configfile: pytest.ini 2025-12-04T11:13:50.0679636Z plugins: hypothesis-6.56.4, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-14.0, subtests-0.13.1, xdist-3.3.1, xdoctest-1.3.0, typeguard-4.3.0 2025-12-04T11:13:50.0679767Z collecting ... collected 58 items / 57 deselected / 1 selected 2025-12-04T11:13:50.0680349Z stepcurrent: skipping 7 already run items. Running only test/inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_17_mid_dim_1_in_features_128_out_features_1024_cuda_bfloat16 2025-12-04T11:13:50.0680429Z Running 1 items in this shard 2025-12-04T11:13:50.0680434Z 2025-12-04T11:13:50.0681168Z inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_17_mid_dim_1_in_features_128_out_features_1024_cuda_bfloat16 [W1204 10:33:37.077042712 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0681176Z 2025-12-04T11:13:50.0681478Z [W1204 10:33:46.435152978 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0681481Z 2025-12-04T11:13:50.0681775Z [W1204 10:33:46.435386303 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0681786Z 2025-12-04T11:13:50.0682207Z [W1204 10:33:46.441160649 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0682304Z 2025-12-04T11:13:50.0682595Z [W1204 10:33:46.441726121 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0682599Z 2025-12-04T11:13:50.0682892Z [W1204 10:33:46.441897615 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0682895Z 2025-12-04T11:13:50.0683196Z [W1204 10:33:46.447275682 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0683199Z 2025-12-04T11:13:50.0683493Z [W1204 10:33:46.447784963 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0683497Z 2025-12-04T11:13:50.0683784Z [W1204 10:33:46.447936936 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0683791Z 2025-12-04T11:13:50.0683883Z ('RERUN', {'yellow': True}) [11.3121s] [100%] 2025-12-04T11:13:50.0684616Z inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_17_mid_dim_1_in_features_128_out_features_1024_cuda_bfloat16 [W1204 10:33:47.478309690 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0684619Z 2025-12-04T11:13:50.0684909Z [W1204 10:33:47.478843741 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0684918Z 2025-12-04T11:13:50.0685207Z [W1204 10:33:47.478982395 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0685210Z 2025-12-04T11:13:50.0685502Z [W1204 10:33:47.481932739 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0685508Z 2025-12-04T11:13:50.0685801Z [W1204 10:33:47.482502101 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0685804Z 2025-12-04T11:13:50.0686090Z [W1204 10:33:47.482639134 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0686094Z 2025-12-04T11:13:50.0686386Z [W1204 10:33:47.487176273 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0686390Z 2025-12-04T11:13:50.0686677Z [W1204 10:33:47.487641203 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0686681Z 2025-12-04T11:13:50.0686975Z [W1204 10:33:47.487777056 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0686982Z 2025-12-04T11:13:50.0687065Z ('RERUN', {'yellow': True}) [0.4511s] [100%] 2025-12-04T11:13:50.0687796Z inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_17_mid_dim_1_in_features_128_out_features_1024_cuda_bfloat16 [W1204 10:33:47.928969422 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0687800Z 2025-12-04T11:13:50.0688090Z [W1204 10:33:47.929502234 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0688093Z 2025-12-04T11:13:50.0688382Z [W1204 10:33:47.929640177 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0688389Z 2025-12-04T11:13:50.0688676Z [W1204 10:33:47.932546000 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0688752Z 2025-12-04T11:13:50.0689126Z [W1204 10:33:47.933121263 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0689130Z 2025-12-04T11:13:50.0689422Z [W1204 10:33:47.933262826 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0689426Z 2025-12-04T11:13:50.0689721Z [W1204 10:33:47.937780214 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0689725Z 2025-12-04T11:13:50.0690030Z [W1204 10:33:47.938245004 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0690034Z 2025-12-04T11:13:50.0690326Z [W1204 10:33:47.938382687 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0690332Z 2025-12-04T11:13:50.0690402Z FAILED [0.4508s] [100%] 2025-12-04T11:13:50.0690405Z 2025-12-04T11:13:50.0690491Z ==================================== RERUNS ==================================== 2025-12-04T11:13:50.0690792Z _ TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_1_in_features_128_out_features_1024_cuda_bfloat16 _ 2025-12-04T11:13:50.0690873Z Traceback (most recent call last): 2025-12-04T11:13:50.0691183Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.0691253Z method(*args, **kwargs) 2025-12-04T11:13:50.0691548Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.0691612Z method(*args, **kwargs) 2025-12-04T11:13:50.0691908Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T11:13:50.0691973Z with policy(): 2025-12-04T11:13:50.0692271Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T11:13:50.0692344Z raise RuntimeError(msg) 2025-12-04T11:13:50.0693158Z RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_1_in_features_128_out_features_1024_cuda_bfloat16! Caching allocator allocated memory was 0 and is now reported as 131072 on device 0. CUDA driver allocated memory was 230686720 and is now 274726912. 2025-12-04T11:13:50.0693163Z 2025-12-04T11:13:50.0693297Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.0693824Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_1_in_features_128_out_features_1024_cuda_bfloat16 2025-12-04T11:13:50.0693831Z 2025-12-04T11:13:50.0694001Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.0694132Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.0694227Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.0694584Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.0694711Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.0694774Z graph_break [] 2025-12-04T11:13:50.0694908Z ----------------------------- Captured stderr call ----------------------------- 2025-12-04T11:13:50.0695604Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/constant_folding.py:256: UserWarning: Unsupported unwinding pattern: Address not in range (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/profiler/unwind/unwind.cpp:219.) 2025-12-04T11:13:50.0695758Z if out == self.unknown_value: 2025-12-04T11:13:50.0696123Z _ TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_1_in_features_128_out_features_1024_cuda_bfloat16 _ 2025-12-04T11:13:50.0696204Z Traceback (most recent call last): 2025-12-04T11:13:50.0696511Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.0696577Z method(*args, **kwargs) 2025-12-04T11:13:50.0696874Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.0696950Z method(*args, **kwargs) 2025-12-04T11:13:50.0697243Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T11:13:50.0697308Z with policy(): 2025-12-04T11:13:50.0697604Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T11:13:50.0697676Z raise RuntimeError(msg) 2025-12-04T11:13:50.0698511Z RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_1_in_features_128_out_features_1024_cuda_bfloat16! Caching allocator allocated memory was 131072 and is now reported as 262144 on device 0. CUDA driver allocated memory was 274726912 and is now 276824064. 2025-12-04T11:13:50.0698515Z 2025-12-04T11:13:50.0698644Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.0699175Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_1_in_features_128_out_features_1024_cuda_bfloat16 2025-12-04T11:13:50.0699179Z 2025-12-04T11:13:50.0699345Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.0699481Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.0699578Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.0699928Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.0700061Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.0700121Z graph_break [] 2025-12-04T11:13:50.0700253Z ----------------------------- Captured stderr call ----------------------------- 2025-12-04T11:13:50.0700942Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/constant_folding.py:256: UserWarning: Unsupported unwinding pattern: Address not in range (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/profiler/unwind/unwind.cpp:219.) 2025-12-04T11:13:50.0701012Z if out == self.unknown_value: 2025-12-04T11:13:50.0701145Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.0701237Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.0701365Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.0701719Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.0701780Z graph_break [] 2025-12-04T11:13:50.0701871Z =================================== FAILURES =================================== 2025-12-04T11:13:50.0702170Z _ TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_1_in_features_128_out_features_1024_cuda_bfloat16 _ 2025-12-04T11:13:50.0702243Z Traceback (most recent call last): 2025-12-04T11:13:50.0702548Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.0702692Z method(*args, **kwargs) 2025-12-04T11:13:50.0703111Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.0703176Z method(*args, **kwargs) 2025-12-04T11:13:50.0703477Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T11:13:50.0703589Z with policy(): 2025-12-04T11:13:50.0703887Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T11:13:50.0703953Z raise RuntimeError(msg) 2025-12-04T11:13:50.0704784Z RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_1_in_features_128_out_features_1024_cuda_bfloat16! Caching allocator allocated memory was 262144 and is now reported as 393216 on device 0. CUDA driver allocated memory was 276824064 and is now 278921216. 2025-12-04T11:13:50.0704791Z 2025-12-04T11:13:50.0704917Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.0705448Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_1_in_features_128_out_features_1024_cuda_bfloat16 2025-12-04T11:13:50.0705452Z 2025-12-04T11:13:50.0705608Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.0705737Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.0705828Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.0706173Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.0706302Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.0706361Z graph_break [] 2025-12-04T11:13:50.0706487Z ----------------------------- Captured stderr call ----------------------------- 2025-12-04T11:13:50.0707183Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/constant_folding.py:256: UserWarning: Unsupported unwinding pattern: Address not in range (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/profiler/unwind/unwind.cpp:219.) 2025-12-04T11:13:50.0707252Z if out == self.unknown_value: 2025-12-04T11:13:50.0707380Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.0707471Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.0707593Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.0707944Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.0708002Z graph_break [] 2025-12-04T11:13:50.0708132Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.0708235Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.0708362Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.0708709Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.0708768Z graph_break [] 2025-12-04T11:13:50.0709257Z - generated xml file: /var/lib/jenkins/workspace/test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-abfd2b283c5f3df6.xml - 2025-12-04T11:13:50.0709364Z =========================== short test summary info ============================ 2025-12-04T11:13:50.0710740Z FAILED [0.4508s] inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_17_mid_dim_1_in_features_128_out_features_1024_cuda_bfloat16 - RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_1_in_features_128_out_features_1024_cuda_bfloat16! Caching allocator allocated memory was 262144 and is now reported as 393216 on device 0. CUDA driver allocated memory was 276824064 and is now 278921216. 2025-12-04T11:13:50.0710812Z 2025-12-04T11:13:50.0710939Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.0711462Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_1_in_features_128_out_features_1024_cuda_bfloat16 2025-12-04T11:13:50.0711466Z 2025-12-04T11:13:50.0711627Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.0711734Z !!!!!!!!!!!!!!!!!!!!!!!!!! stopping after 1 failures !!!!!!!!!!!!!!!!!!!!!!!!!!! 2025-12-04T11:13:50.0711854Z ================== 1 failed, 57 deselected, 2 rerun in 12.24s ================== 2025-12-04T11:13:50.0711918Z Got exit code 1 2025-12-04T11:13:50.0712402Z FAILED CONSISTENTLY: test/inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_17_mid_dim_1_in_features_128_out_features_1024_cuda_bfloat16 2025-12-04T11:13:50.0712657Z Test failed consistently, continuing with the rest of the tests due to continue-through-error being set 2025-12-04T11:13:50.0712921Z W1204 10:33:54.776000 60999 site-packages/torch/_inductor/utils.py:1703] Not enough SMs to use max_autotune_gemm mode 2025-12-04T11:13:50.0713315Z Test results will be stored in test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-e0f3393878857c02.xml 2025-12-04T11:13:50.0713413Z ============================= test session starts ============================== 2025-12-04T11:13:50.0713622Z platform linux -- Python 3.10.14, pytest-7.3.2, pluggy-1.6.0 -- /opt/conda/envs/py_3.10/bin/python 2025-12-04T11:13:50.0713696Z cachedir: .pytest_cache 2025-12-04T11:13:50.0714009Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow] 2025-12-04T11:13:50.0714085Z rootdir: /var/lib/jenkins/workspace 2025-12-04T11:13:50.0714156Z configfile: pytest.ini 2025-12-04T11:13:50.0714473Z plugins: hypothesis-6.56.4, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-14.0, subtests-0.13.1, xdist-3.3.1, xdoctest-1.3.0, typeguard-4.3.0 2025-12-04T11:13:50.0714607Z collecting ... collected 58 items / 8 deselected / 50 selected 2025-12-04T11:13:50.0714695Z stepcurrent: skipping 8 already run items. 2025-12-04T11:13:50.0714766Z Running 50 items in this shard 2025-12-04T11:13:50.0714769Z 2025-12-04T11:13:50.0715274Z inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_17_mid_dim_1_in_features_128_out_features_64_cuda_bfloat16 ('RERUN', {'yellow': True}) [1.9694s] [ 2%] 2025-12-04T11:13:50.0715766Z inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_17_mid_dim_1_in_features_128_out_features_64_cuda_bfloat16 ('RERUN', {'yellow': True}) [0.5442s] [ 2%] 2025-12-04T11:13:50.0716217Z inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_17_mid_dim_1_in_features_128_out_features_64_cuda_bfloat16 FAILED [0.5380s] [ 2%] 2025-12-04T11:13:50.0716221Z 2025-12-04T11:13:50.0716304Z ==================================== RERUNS ==================================== 2025-12-04T11:13:50.0716604Z _ TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_1_in_features_128_out_features_64_cuda_bfloat16 _ 2025-12-04T11:13:50.0716684Z Traceback (most recent call last): 2025-12-04T11:13:50.0716989Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.0717062Z method(*args, **kwargs) 2025-12-04T11:13:50.0717355Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.0717515Z method(*args, **kwargs) 2025-12-04T11:13:50.0717876Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T11:13:50.0717939Z with policy(): 2025-12-04T11:13:50.0718234Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T11:13:50.0718308Z raise RuntimeError(msg) 2025-12-04T11:13:50.0719106Z RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_1_in_features_128_out_features_64_cuda_bfloat16! Caching allocator allocated memory was 0 and is now reported as 8192 on device 0. CUDA driver allocated memory was 230686720 and is now 274726912. 2025-12-04T11:13:50.0719110Z 2025-12-04T11:13:50.0719241Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.0719773Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_1_in_features_128_out_features_64_cuda_bfloat16 2025-12-04T11:13:50.0719777Z 2025-12-04T11:13:50.0719941Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.0720068Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.0720167Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.0720725Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('benchmarking.InductorBenchmarker.benchmark_gpu', 2), ('pad_mm_bench', 1), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.0720856Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.0720920Z graph_break [] 2025-12-04T11:13:50.0721217Z _ TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_1_in_features_128_out_features_64_cuda_bfloat16 _ 2025-12-04T11:13:50.0721295Z Traceback (most recent call last): 2025-12-04T11:13:50.0721598Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.0721663Z method(*args, **kwargs) 2025-12-04T11:13:50.0721956Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.0722026Z method(*args, **kwargs) 2025-12-04T11:13:50.0722317Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T11:13:50.0722381Z with policy(): 2025-12-04T11:13:50.0722674Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T11:13:50.0722741Z raise RuntimeError(msg) 2025-12-04T11:13:50.0723560Z RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_1_in_features_128_out_features_64_cuda_bfloat16! Caching allocator allocated memory was 8192 and is now reported as 16384 on device 0. CUDA driver allocated memory was 274726912 and is now 276824064. 2025-12-04T11:13:50.0723564Z 2025-12-04T11:13:50.0723701Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.0724229Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_1_in_features_128_out_features_64_cuda_bfloat16 2025-12-04T11:13:50.0724232Z 2025-12-04T11:13:50.0724390Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.0724521Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.0724621Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.0725312Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('benchmarking.InductorBenchmarker.benchmark_gpu', 2), ('pad_mm_bench', 1), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.0725448Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.0725506Z graph_break [] 2025-12-04T11:13:50.0725630Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.0725726Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.0725846Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.0726390Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('benchmarking.InductorBenchmarker.benchmark_gpu', 2), ('pad_mm_bench', 1), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.0726451Z graph_break [] 2025-12-04T11:13:50.0726538Z =================================== FAILURES =================================== 2025-12-04T11:13:50.0726837Z _ TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_1_in_features_128_out_features_64_cuda_bfloat16 _ 2025-12-04T11:13:50.0726911Z Traceback (most recent call last): 2025-12-04T11:13:50.0727215Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.0727281Z method(*args, **kwargs) 2025-12-04T11:13:50.0727576Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.0727644Z method(*args, **kwargs) 2025-12-04T11:13:50.0727936Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T11:13:50.0727995Z with policy(): 2025-12-04T11:13:50.0728292Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T11:13:50.0728362Z raise RuntimeError(msg) 2025-12-04T11:13:50.0729180Z RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_1_in_features_128_out_features_64_cuda_bfloat16! Caching allocator allocated memory was 16384 and is now reported as 24576 on device 0. CUDA driver allocated memory was 276824064 and is now 278921216. 2025-12-04T11:13:50.0729183Z 2025-12-04T11:13:50.0729310Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.0729838Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_1_in_features_128_out_features_64_cuda_bfloat16 2025-12-04T11:13:50.0729846Z 2025-12-04T11:13:50.0730002Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.0730131Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.0730227Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.0730770Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('benchmarking.InductorBenchmarker.benchmark_gpu', 2), ('pad_mm_bench', 1), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.0730904Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.0730971Z graph_break [] 2025-12-04T11:13:50.0731097Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.0731193Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.0731314Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.0731858Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('benchmarking.InductorBenchmarker.benchmark_gpu', 2), ('pad_mm_bench', 1), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.0732003Z graph_break [] 2025-12-04T11:13:50.0732189Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.0732283Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.0732404Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.0732938Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('benchmarking.InductorBenchmarker.benchmark_gpu', 2), ('pad_mm_bench', 1), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.0733004Z graph_break [] 2025-12-04T11:13:50.0733492Z - generated xml file: /var/lib/jenkins/workspace/test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-e0f3393878857c02.xml - 2025-12-04T11:13:50.0733597Z =========================== short test summary info ============================ 2025-12-04T11:13:50.0734901Z FAILED [0.5380s] inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_17_mid_dim_1_in_features_128_out_features_64_cuda_bfloat16 - RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_1_in_features_128_out_features_64_cuda_bfloat16! Caching allocator allocated memory was 16384 and is now reported as 24576 on device 0. CUDA driver allocated memory was 276824064 and is now 278921216. 2025-12-04T11:13:50.0734905Z 2025-12-04T11:13:50.0735035Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.0735554Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_1_in_features_128_out_features_64_cuda_bfloat16 2025-12-04T11:13:50.0735558Z 2025-12-04T11:13:50.0735716Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.0735834Z !!!!!!!!!!!!!!!!!!!!!!!!!! stopping after 1 failures !!!!!!!!!!!!!!!!!!!!!!!!!!! 2025-12-04T11:13:50.0735949Z =================== 1 failed, 8 deselected, 2 rerun in 3.08s =================== 2025-12-04T11:13:50.0736012Z Got exit code 1 2025-12-04T11:13:50.0736079Z Retrying single test... 2025-12-04T11:13:50.0736341Z W1204 10:34:04.817000 61181 site-packages/torch/_inductor/utils.py:1703] Not enough SMs to use max_autotune_gemm mode 2025-12-04T11:13:50.0736735Z Test results will be stored in test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-d5da2e41e9fcecb9.xml 2025-12-04T11:13:50.0736832Z ============================= test session starts ============================== 2025-12-04T11:13:50.0737042Z platform linux -- Python 3.10.14, pytest-7.3.2, pluggy-1.6.0 -- /opt/conda/envs/py_3.10/bin/python 2025-12-04T11:13:50.0737113Z cachedir: .pytest_cache 2025-12-04T11:13:50.0737424Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow] 2025-12-04T11:13:50.0737508Z rootdir: /var/lib/jenkins/workspace 2025-12-04T11:13:50.0737575Z configfile: pytest.ini 2025-12-04T11:13:50.0737893Z plugins: hypothesis-6.56.4, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-14.0, subtests-0.13.1, xdist-3.3.1, xdoctest-1.3.0, typeguard-4.3.0 2025-12-04T11:13:50.0738026Z collecting ... collected 58 items / 57 deselected / 1 selected 2025-12-04T11:13:50.0738596Z stepcurrent: skipping 8 already run items. Running only test/inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_17_mid_dim_1_in_features_128_out_features_64_cuda_bfloat16 2025-12-04T11:13:50.0738673Z Running 1 items in this shard 2025-12-04T11:13:50.0738676Z 2025-12-04T11:13:50.0739486Z inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_17_mid_dim_1_in_features_128_out_features_64_cuda_bfloat16 [W1204 10:34:06.449544254 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0739554Z 2025-12-04T11:13:50.0739857Z [W1204 10:34:15.832689644 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0739864Z 2025-12-04T11:13:50.0740160Z [W1204 10:34:15.832938260 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0740163Z 2025-12-04T11:13:50.0740452Z [W1204 10:34:15.838725447 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0740456Z 2025-12-04T11:13:50.0740750Z [W1204 10:34:15.839287159 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0740754Z 2025-12-04T11:13:50.0741046Z [W1204 10:34:15.839465703 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0741051Z 2025-12-04T11:13:50.0741347Z [W1204 10:34:15.844777430 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0741351Z 2025-12-04T11:13:50.0741640Z [W1204 10:34:15.845288211 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0741644Z 2025-12-04T11:13:50.0741940Z [W1204 10:34:15.845469645 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0741943Z 2025-12-04T11:13:50.0742026Z ('RERUN', {'yellow': True}) [11.3592s] [100%] 2025-12-04T11:13:50.0742767Z inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_17_mid_dim_1_in_features_128_out_features_64_cuda_bfloat16 [W1204 10:34:16.655641848 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0742777Z 2025-12-04T11:13:50.0743071Z [W1204 10:34:16.656185410 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0743074Z 2025-12-04T11:13:50.0743365Z [W1204 10:34:16.656323943 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0743375Z 2025-12-04T11:13:50.0743708Z [W1204 10:34:16.659265468 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0743712Z 2025-12-04T11:13:50.0744001Z [W1204 10:34:16.659730098 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0744004Z 2025-12-04T11:13:50.0744298Z [W1204 10:34:16.659865421 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0744304Z 2025-12-04T11:13:50.0744596Z [W1204 10:34:16.664469662 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0744599Z 2025-12-04T11:13:50.0744895Z [W1204 10:34:16.664952713 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0744898Z 2025-12-04T11:13:50.0745188Z [W1204 10:34:16.665087926 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0745191Z 2025-12-04T11:13:50.0745282Z ('RERUN', {'yellow': True}) [0.5021s] [100%] 2025-12-04T11:13:50.0746014Z inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_17_mid_dim_1_in_features_128_out_features_64_cuda_bfloat16 [W1204 10:34:17.154446952 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0746110Z 2025-12-04T11:13:50.0746471Z [W1204 10:34:17.155013255 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0746475Z 2025-12-04T11:13:50.0746766Z [W1204 10:34:17.155150638 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0746769Z 2025-12-04T11:13:50.0747060Z [W1204 10:34:17.158087072 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0747063Z 2025-12-04T11:13:50.0747358Z [W1204 10:34:17.158551472 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0747361Z 2025-12-04T11:13:50.0747649Z [W1204 10:34:17.158687075 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0747655Z 2025-12-04T11:13:50.0747953Z [W1204 10:34:17.163325947 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0747956Z 2025-12-04T11:13:50.0748244Z [W1204 10:34:17.163798757 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0748248Z 2025-12-04T11:13:50.0748540Z [W1204 10:34:17.163934211 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0748544Z 2025-12-04T11:13:50.0748606Z FAILED [0.4995s] [100%] 2025-12-04T11:13:50.0748610Z 2025-12-04T11:13:50.0748703Z ==================================== RERUNS ==================================== 2025-12-04T11:13:50.0749001Z _ TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_1_in_features_128_out_features_64_cuda_bfloat16 _ 2025-12-04T11:13:50.0749081Z Traceback (most recent call last): 2025-12-04T11:13:50.0749395Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.0749466Z method(*args, **kwargs) 2025-12-04T11:13:50.0749769Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.0749841Z method(*args, **kwargs) 2025-12-04T11:13:50.0750133Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T11:13:50.0750200Z with policy(): 2025-12-04T11:13:50.0750495Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T11:13:50.0750563Z raise RuntimeError(msg) 2025-12-04T11:13:50.0751369Z RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_1_in_features_128_out_features_64_cuda_bfloat16! Caching allocator allocated memory was 0 and is now reported as 8192 on device 0. CUDA driver allocated memory was 230686720 and is now 274726912. 2025-12-04T11:13:50.0751376Z 2025-12-04T11:13:50.0751504Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.0752036Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_1_in_features_128_out_features_64_cuda_bfloat16 2025-12-04T11:13:50.0752040Z 2025-12-04T11:13:50.0752202Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.0752337Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.0752441Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.0753065Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('benchmarking.InductorBenchmarker.benchmark_gpu', 2), ('pad_mm_bench', 1), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.0753274Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.0753342Z graph_break [] 2025-12-04T11:13:50.0753477Z ----------------------------- Captured stderr call ----------------------------- 2025-12-04T11:13:50.0754184Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/constant_folding.py:256: UserWarning: Unsupported unwinding pattern: Address not in range (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/profiler/unwind/unwind.cpp:219.) 2025-12-04T11:13:50.0754256Z if out == self.unknown_value: 2025-12-04T11:13:50.0754565Z _ TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_1_in_features_128_out_features_64_cuda_bfloat16 _ 2025-12-04T11:13:50.0754641Z Traceback (most recent call last): 2025-12-04T11:13:50.0754956Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.0755032Z method(*args, **kwargs) 2025-12-04T11:13:50.0755327Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.0755396Z method(*args, **kwargs) 2025-12-04T11:13:50.0755688Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T11:13:50.0755750Z with policy(): 2025-12-04T11:13:50.0756052Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T11:13:50.0756120Z raise RuntimeError(msg) 2025-12-04T11:13:50.0756932Z RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_1_in_features_128_out_features_64_cuda_bfloat16! Caching allocator allocated memory was 8192 and is now reported as 16384 on device 0. CUDA driver allocated memory was 274726912 and is now 276824064. 2025-12-04T11:13:50.0756944Z 2025-12-04T11:13:50.0757078Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.0757601Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_1_in_features_128_out_features_64_cuda_bfloat16 2025-12-04T11:13:50.0757605Z 2025-12-04T11:13:50.0757769Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.0757896Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.0758001Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.0758549Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('benchmarking.InductorBenchmarker.benchmark_gpu', 2), ('pad_mm_bench', 1), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.0758686Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.0758751Z graph_break [] 2025-12-04T11:13:50.0758880Z ----------------------------- Captured stderr call ----------------------------- 2025-12-04T11:13:50.0759591Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/constant_folding.py:256: UserWarning: Unsupported unwinding pattern: Address not in range (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/profiler/unwind/unwind.cpp:219.) 2025-12-04T11:13:50.0759661Z if out == self.unknown_value: 2025-12-04T11:13:50.0759787Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.0759886Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.0760014Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.0760698Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('benchmarking.InductorBenchmarker.benchmark_gpu', 2), ('pad_mm_bench', 1), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.0760766Z graph_break [] 2025-12-04T11:13:50.0760863Z =================================== FAILURES =================================== 2025-12-04T11:13:50.0761169Z _ TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_1_in_features_128_out_features_64_cuda_bfloat16 _ 2025-12-04T11:13:50.0761247Z Traceback (most recent call last): 2025-12-04T11:13:50.0761551Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.0761625Z method(*args, **kwargs) 2025-12-04T11:13:50.0761920Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.0761997Z method(*args, **kwargs) 2025-12-04T11:13:50.0762293Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T11:13:50.0762357Z with policy(): 2025-12-04T11:13:50.0762658Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T11:13:50.0762727Z raise RuntimeError(msg) 2025-12-04T11:13:50.0763542Z RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_1_in_features_128_out_features_64_cuda_bfloat16! Caching allocator allocated memory was 16384 and is now reported as 24576 on device 0. CUDA driver allocated memory was 276824064 and is now 278921216. 2025-12-04T11:13:50.0763550Z 2025-12-04T11:13:50.0763684Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.0764207Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_1_in_features_128_out_features_64_cuda_bfloat16 2025-12-04T11:13:50.0764217Z 2025-12-04T11:13:50.0764386Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.0764520Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.0764624Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.0765167Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('benchmarking.InductorBenchmarker.benchmark_gpu', 2), ('pad_mm_bench', 1), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.0765298Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.0765366Z graph_break [] 2025-12-04T11:13:50.0765495Z ----------------------------- Captured stderr call ----------------------------- 2025-12-04T11:13:50.0766193Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/constant_folding.py:256: UserWarning: Unsupported unwinding pattern: Address not in range (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/profiler/unwind/unwind.cpp:219.) 2025-12-04T11:13:50.0766267Z if out == self.unknown_value: 2025-12-04T11:13:50.0766394Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.0766494Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.0766622Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.0767163Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('benchmarking.InductorBenchmarker.benchmark_gpu', 2), ('pad_mm_bench', 1), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.0767228Z graph_break [] 2025-12-04T11:13:50.0767354Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.0767531Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.0767751Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.0768299Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('benchmarking.InductorBenchmarker.benchmark_gpu', 2), ('pad_mm_bench', 1), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.0768364Z graph_break [] 2025-12-04T11:13:50.0768859Z - generated xml file: /var/lib/jenkins/workspace/test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-d5da2e41e9fcecb9.xml - 2025-12-04T11:13:50.0768968Z =========================== short test summary info ============================ 2025-12-04T11:13:50.0770276Z FAILED [0.4995s] inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_17_mid_dim_1_in_features_128_out_features_64_cuda_bfloat16 - RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_1_in_features_128_out_features_64_cuda_bfloat16! Caching allocator allocated memory was 16384 and is now reported as 24576 on device 0. CUDA driver allocated memory was 276824064 and is now 278921216. 2025-12-04T11:13:50.0770283Z 2025-12-04T11:13:50.0770417Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.0770937Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_1_in_features_128_out_features_64_cuda_bfloat16 2025-12-04T11:13:50.0770940Z 2025-12-04T11:13:50.0771099Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.0771214Z !!!!!!!!!!!!!!!!!!!!!!!!!! stopping after 1 failures !!!!!!!!!!!!!!!!!!!!!!!!!!! 2025-12-04T11:13:50.0771333Z ================== 1 failed, 57 deselected, 2 rerun in 12.39s ================== 2025-12-04T11:13:50.0771402Z Got exit code 1 2025-12-04T11:13:50.0771470Z Retrying single test... 2025-12-04T11:13:50.0771743Z W1204 10:34:23.965000 61368 site-packages/torch/_inductor/utils.py:1703] Not enough SMs to use max_autotune_gemm mode 2025-12-04T11:13:50.0772140Z Test results will be stored in test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-5fb75fde833c163c.xml 2025-12-04T11:13:50.0772239Z ============================= test session starts ============================== 2025-12-04T11:13:50.0772458Z platform linux -- Python 3.10.14, pytest-7.3.2, pluggy-1.6.0 -- /opt/conda/envs/py_3.10/bin/python 2025-12-04T11:13:50.0772527Z cachedir: .pytest_cache 2025-12-04T11:13:50.0772837Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow] 2025-12-04T11:13:50.0772917Z rootdir: /var/lib/jenkins/workspace 2025-12-04T11:13:50.0772984Z configfile: pytest.ini 2025-12-04T11:13:50.0773304Z plugins: hypothesis-6.56.4, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-14.0, subtests-0.13.1, xdist-3.3.1, xdoctest-1.3.0, typeguard-4.3.0 2025-12-04T11:13:50.0773443Z collecting ... collected 58 items / 57 deselected / 1 selected 2025-12-04T11:13:50.0774017Z stepcurrent: skipping 8 already run items. Running only test/inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_17_mid_dim_1_in_features_128_out_features_64_cuda_bfloat16 2025-12-04T11:13:50.0774092Z Running 1 items in this shard 2025-12-04T11:13:50.0774095Z 2025-12-04T11:13:50.0774830Z inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_17_mid_dim_1_in_features_128_out_features_64_cuda_bfloat16 [W1204 10:34:25.604513453 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0774834Z 2025-12-04T11:13:50.0775137Z [W1204 10:34:34.853036274 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0775218Z 2025-12-04T11:13:50.0775580Z [W1204 10:34:34.853283920 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0775584Z 2025-12-04T11:13:50.0775875Z [W1204 10:34:34.859153139 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0775883Z 2025-12-04T11:13:50.0776171Z [W1204 10:34:34.859732691 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0776174Z 2025-12-04T11:13:50.0776462Z [W1204 10:34:34.859901165 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0776465Z 2025-12-04T11:13:50.0776772Z [W1204 10:34:34.865388146 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0776779Z 2025-12-04T11:13:50.0777072Z [W1204 10:34:34.865925867 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0777076Z 2025-12-04T11:13:50.0777370Z [W1204 10:34:34.866082621 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0777374Z 2025-12-04T11:13:50.0777456Z ('RERUN', {'yellow': True}) [11.2352s] [100%] 2025-12-04T11:13:50.0778358Z inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_17_mid_dim_1_in_features_128_out_features_64_cuda_bfloat16 [W1204 10:34:35.681489964 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0778363Z 2025-12-04T11:13:50.0778656Z [W1204 10:34:35.682053677 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0778663Z 2025-12-04T11:13:50.0778961Z [W1204 10:34:35.682196790 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0778964Z 2025-12-04T11:13:50.0779252Z [W1204 10:34:35.685289668 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0779255Z 2025-12-04T11:13:50.0779551Z [W1204 10:34:35.685770948 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0779562Z 2025-12-04T11:13:50.0779851Z [W1204 10:34:35.685909351 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0779854Z 2025-12-04T11:13:50.0780144Z [W1204 10:34:35.690718387 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0780150Z 2025-12-04T11:13:50.0780447Z [W1204 10:34:35.691203938 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0780450Z 2025-12-04T11:13:50.0780740Z [W1204 10:34:35.691342221 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0780743Z 2025-12-04T11:13:50.0780829Z ('RERUN', {'yellow': True}) [0.5087s] [100%] 2025-12-04T11:13:50.0781564Z inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_17_mid_dim_1_in_features_128_out_features_64_cuda_bfloat16 [W1204 10:34:36.186282917 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0781568Z 2025-12-04T11:13:50.0781867Z [W1204 10:34:36.186863240 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0781984Z 2025-12-04T11:13:50.0782371Z [W1204 10:34:36.187008273 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0782375Z 2025-12-04T11:13:50.0782666Z [W1204 10:34:36.190084521 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0782673Z 2025-12-04T11:13:50.0782963Z [W1204 10:34:36.190577032 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0782966Z 2025-12-04T11:13:50.0783258Z [W1204 10:34:36.190713915 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0783260Z 2025-12-04T11:13:50.0783593Z [W1204 10:34:36.195426548 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0783600Z 2025-12-04T11:13:50.0783890Z [W1204 10:34:36.195907378 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0783893Z 2025-12-04T11:13:50.0784187Z [W1204 10:34:36.196043521 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0784191Z 2025-12-04T11:13:50.0784254Z FAILED [0.5034s] [100%] 2025-12-04T11:13:50.0784257Z 2025-12-04T11:13:50.0784350Z ==================================== RERUNS ==================================== 2025-12-04T11:13:50.0784644Z _ TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_1_in_features_128_out_features_64_cuda_bfloat16 _ 2025-12-04T11:13:50.0784720Z Traceback (most recent call last): 2025-12-04T11:13:50.0785034Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.0785105Z method(*args, **kwargs) 2025-12-04T11:13:50.0785402Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.0785484Z method(*args, **kwargs) 2025-12-04T11:13:50.0785780Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T11:13:50.0785847Z with policy(): 2025-12-04T11:13:50.0786145Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T11:13:50.0786213Z raise RuntimeError(msg) 2025-12-04T11:13:50.0787022Z RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_1_in_features_128_out_features_64_cuda_bfloat16! Caching allocator allocated memory was 0 and is now reported as 8192 on device 0. CUDA driver allocated memory was 230686720 and is now 274726912. 2025-12-04T11:13:50.0787029Z 2025-12-04T11:13:50.0787161Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.0787694Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_1_in_features_128_out_features_64_cuda_bfloat16 2025-12-04T11:13:50.0787698Z 2025-12-04T11:13:50.0787859Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.0787997Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.0788097Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.0788647Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('benchmarking.InductorBenchmarker.benchmark_gpu', 2), ('pad_mm_bench', 1), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.0788781Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.0788919Z graph_break [] 2025-12-04T11:13:50.0789115Z ----------------------------- Captured stderr call ----------------------------- 2025-12-04T11:13:50.0789829Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/constant_folding.py:256: UserWarning: Unsupported unwinding pattern: Address not in range (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/profiler/unwind/unwind.cpp:219.) 2025-12-04T11:13:50.0789900Z if out == self.unknown_value: 2025-12-04T11:13:50.0790202Z _ TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_1_in_features_128_out_features_64_cuda_bfloat16 _ 2025-12-04T11:13:50.0790279Z Traceback (most recent call last): 2025-12-04T11:13:50.0790579Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.0790660Z method(*args, **kwargs) 2025-12-04T11:13:50.0790959Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.0791032Z method(*args, **kwargs) 2025-12-04T11:13:50.0791329Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T11:13:50.0791391Z with policy(): 2025-12-04T11:13:50.0791692Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T11:13:50.0791758Z raise RuntimeError(msg) 2025-12-04T11:13:50.0792577Z RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_1_in_features_128_out_features_64_cuda_bfloat16! Caching allocator allocated memory was 8192 and is now reported as 16384 on device 0. CUDA driver allocated memory was 274726912 and is now 276824064. 2025-12-04T11:13:50.0792581Z 2025-12-04T11:13:50.0792709Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.0793235Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_1_in_features_128_out_features_64_cuda_bfloat16 2025-12-04T11:13:50.0793239Z 2025-12-04T11:13:50.0793400Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.0793530Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.0793631Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.0794177Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('benchmarking.InductorBenchmarker.benchmark_gpu', 2), ('pad_mm_bench', 1), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.0794312Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.0794381Z graph_break [] 2025-12-04T11:13:50.0794509Z ----------------------------- Captured stderr call ----------------------------- 2025-12-04T11:13:50.0795209Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/constant_folding.py:256: UserWarning: Unsupported unwinding pattern: Address not in range (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/profiler/unwind/unwind.cpp:219.) 2025-12-04T11:13:50.0795279Z if out == self.unknown_value: 2025-12-04T11:13:50.0795409Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.0795512Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.0795641Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.0796191Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('benchmarking.InductorBenchmarker.benchmark_gpu', 2), ('pad_mm_bench', 1), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.0796328Z graph_break [] 2025-12-04T11:13:50.0796417Z =================================== FAILURES =================================== 2025-12-04T11:13:50.0796800Z _ TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_1_in_features_128_out_features_64_cuda_bfloat16 _ 2025-12-04T11:13:50.0796878Z Traceback (most recent call last): 2025-12-04T11:13:50.0797178Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.0797252Z method(*args, **kwargs) 2025-12-04T11:13:50.0797547Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.0797617Z method(*args, **kwargs) 2025-12-04T11:13:50.0797910Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T11:13:50.0797972Z with policy(): 2025-12-04T11:13:50.0798273Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T11:13:50.0798347Z raise RuntimeError(msg) 2025-12-04T11:13:50.0799163Z RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_1_in_features_128_out_features_64_cuda_bfloat16! Caching allocator allocated memory was 16384 and is now reported as 24576 on device 0. CUDA driver allocated memory was 276824064 and is now 278921216. 2025-12-04T11:13:50.0799167Z 2025-12-04T11:13:50.0799301Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.0799819Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_1_in_features_128_out_features_64_cuda_bfloat16 2025-12-04T11:13:50.0799828Z 2025-12-04T11:13:50.0799986Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.0800116Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.0800219Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.0800762Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('benchmarking.InductorBenchmarker.benchmark_gpu', 2), ('pad_mm_bench', 1), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.0800890Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.0800953Z graph_break [] 2025-12-04T11:13:50.0801078Z ----------------------------- Captured stderr call ----------------------------- 2025-12-04T11:13:50.0801770Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/constant_folding.py:256: UserWarning: Unsupported unwinding pattern: Address not in range (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/profiler/unwind/unwind.cpp:219.) 2025-12-04T11:13:50.0801843Z if out == self.unknown_value: 2025-12-04T11:13:50.0801967Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.0802067Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.0802190Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.0802737Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('benchmarking.InductorBenchmarker.benchmark_gpu', 2), ('pad_mm_bench', 1), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.0802797Z graph_break [] 2025-12-04T11:13:50.0802936Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.0803036Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.0803179Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.0803795Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('benchmarking.InductorBenchmarker.benchmark_gpu', 2), ('pad_mm_bench', 1), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.0803940Z graph_break [] 2025-12-04T11:13:50.0804430Z - generated xml file: /var/lib/jenkins/workspace/test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-5fb75fde833c163c.xml - 2025-12-04T11:13:50.0804536Z =========================== short test summary info ============================ 2025-12-04T11:13:50.0805822Z FAILED [0.5034s] inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_17_mid_dim_1_in_features_128_out_features_64_cuda_bfloat16 - RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_1_in_features_128_out_features_64_cuda_bfloat16! Caching allocator allocated memory was 16384 and is now reported as 24576 on device 0. CUDA driver allocated memory was 276824064 and is now 278921216. 2025-12-04T11:13:50.0805829Z 2025-12-04T11:13:50.0805964Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.0806483Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_1_in_features_128_out_features_64_cuda_bfloat16 2025-12-04T11:13:50.0806486Z 2025-12-04T11:13:50.0806648Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.0806754Z !!!!!!!!!!!!!!!!!!!!!!!!!! stopping after 1 failures !!!!!!!!!!!!!!!!!!!!!!!!!!! 2025-12-04T11:13:50.0806873Z ================== 1 failed, 57 deselected, 2 rerun in 12.27s ================== 2025-12-04T11:13:50.0806940Z Got exit code 1 2025-12-04T11:13:50.0807417Z FAILED CONSISTENTLY: test/inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_17_mid_dim_1_in_features_128_out_features_64_cuda_bfloat16 2025-12-04T11:13:50.0807665Z Test failed consistently, continuing with the rest of the tests due to continue-through-error being set 2025-12-04T11:13:50.0807938Z W1204 10:34:43.056000 61555 site-packages/torch/_inductor/utils.py:1703] Not enough SMs to use max_autotune_gemm mode 2025-12-04T11:13:50.0808323Z Test results will be stored in test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-2f9606f02b59f87b.xml 2025-12-04T11:13:50.0808427Z ============================= test session starts ============================== 2025-12-04T11:13:50.0808635Z platform linux -- Python 3.10.14, pytest-7.3.2, pluggy-1.6.0 -- /opt/conda/envs/py_3.10/bin/python 2025-12-04T11:13:50.0808706Z cachedir: .pytest_cache 2025-12-04T11:13:50.0809020Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow] 2025-12-04T11:13:50.0809098Z rootdir: /var/lib/jenkins/workspace 2025-12-04T11:13:50.0809182Z configfile: pytest.ini 2025-12-04T11:13:50.0809503Z plugins: hypothesis-6.56.4, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-14.0, subtests-0.13.1, xdist-3.3.1, xdoctest-1.3.0, typeguard-4.3.0 2025-12-04T11:13:50.0809632Z collecting ... collected 58 items / 9 deselected / 49 selected 2025-12-04T11:13:50.0809727Z stepcurrent: skipping 9 already run items. 2025-12-04T11:13:50.0809798Z Running 49 items in this shard 2025-12-04T11:13:50.0809801Z 2025-12-04T11:13:50.0810307Z inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_17_mid_dim_1_in_features_128_out_features_65_cuda_bfloat16 ('RERUN', {'yellow': True}) [1.9668s] [ 2%] 2025-12-04T11:13:50.0810799Z inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_17_mid_dim_1_in_features_128_out_features_65_cuda_bfloat16 ('RERUN', {'yellow': True}) [0.5554s] [ 2%] 2025-12-04T11:13:50.0811245Z inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_17_mid_dim_1_in_features_128_out_features_65_cuda_bfloat16 FAILED [0.5472s] [ 2%] 2025-12-04T11:13:50.0811324Z 2025-12-04T11:13:50.0811481Z ==================================== RERUNS ==================================== 2025-12-04T11:13:50.0811779Z _ TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_1_in_features_128_out_features_65_cuda_bfloat16 _ 2025-12-04T11:13:50.0811859Z Traceback (most recent call last): 2025-12-04T11:13:50.0812168Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.0812248Z method(*args, **kwargs) 2025-12-04T11:13:50.0812552Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.0812619Z method(*args, **kwargs) 2025-12-04T11:13:50.0812916Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T11:13:50.0812979Z with policy(): 2025-12-04T11:13:50.0813277Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T11:13:50.0813350Z raise RuntimeError(msg) 2025-12-04T11:13:50.0814150Z RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_1_in_features_128_out_features_65_cuda_bfloat16! Caching allocator allocated memory was 0 and is now reported as 8704 on device 0. CUDA driver allocated memory was 230686720 and is now 274726912. 2025-12-04T11:13:50.0814154Z 2025-12-04T11:13:50.0814288Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.0814809Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_1_in_features_128_out_features_65_cuda_bfloat16 2025-12-04T11:13:50.0814817Z 2025-12-04T11:13:50.0814974Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.0815118Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.0815216Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.0815772Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('benchmarking.InductorBenchmarker.benchmark_gpu', 2), ('pad_mm_bench', 1), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.0815902Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.0815964Z graph_break [] 2025-12-04T11:13:50.0816262Z _ TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_1_in_features_128_out_features_65_cuda_bfloat16 _ 2025-12-04T11:13:50.0816336Z Traceback (most recent call last): 2025-12-04T11:13:50.0816637Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.0816712Z method(*args, **kwargs) 2025-12-04T11:13:50.0817008Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.0817080Z method(*args, **kwargs) 2025-12-04T11:13:50.0817372Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T11:13:50.0817435Z with policy(): 2025-12-04T11:13:50.0817737Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T11:13:50.0817806Z raise RuntimeError(msg) 2025-12-04T11:13:50.0818619Z RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_1_in_features_128_out_features_65_cuda_bfloat16! Caching allocator allocated memory was 8704 and is now reported as 17408 on device 0. CUDA driver allocated memory was 274726912 and is now 276824064. 2025-12-04T11:13:50.0818768Z 2025-12-04T11:13:50.0818966Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.0819487Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_1_in_features_128_out_features_65_cuda_bfloat16 2025-12-04T11:13:50.0819497Z 2025-12-04T11:13:50.0819657Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.0819784Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.0819890Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.0820433Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('benchmarking.InductorBenchmarker.benchmark_gpu', 2), ('pad_mm_bench', 1), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.0820569Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.0820630Z graph_break [] 2025-12-04T11:13:50.0820761Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.0820858Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.0820980Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.0821519Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('benchmarking.InductorBenchmarker.benchmark_gpu', 2), ('pad_mm_bench', 1), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.0821585Z graph_break [] 2025-12-04T11:13:50.0821670Z =================================== FAILURES =================================== 2025-12-04T11:13:50.0821965Z _ TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_1_in_features_128_out_features_65_cuda_bfloat16 _ 2025-12-04T11:13:50.0826550Z Traceback (most recent call last): 2025-12-04T11:13:50.0826927Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.0827000Z method(*args, **kwargs) 2025-12-04T11:13:50.0827329Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.0827398Z method(*args, **kwargs) 2025-12-04T11:13:50.0827714Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T11:13:50.0827779Z with policy(): 2025-12-04T11:13:50.0828101Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T11:13:50.0828171Z raise RuntimeError(msg) 2025-12-04T11:13:50.0828995Z RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_1_in_features_128_out_features_65_cuda_bfloat16! Caching allocator allocated memory was 17408 and is now reported as 26112 on device 0. CUDA driver allocated memory was 276824064 and is now 278921216. 2025-12-04T11:13:50.0829004Z 2025-12-04T11:13:50.0829140Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.0829677Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_1_in_features_128_out_features_65_cuda_bfloat16 2025-12-04T11:13:50.0829681Z 2025-12-04T11:13:50.0829857Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.0830000Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.0830098Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.0830778Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('benchmarking.InductorBenchmarker.benchmark_gpu', 2), ('pad_mm_bench', 1), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.0830978Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.0831040Z graph_break [] 2025-12-04T11:13:50.0831173Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.0831267Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.0831390Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.0831935Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('benchmarking.InductorBenchmarker.benchmark_gpu', 2), ('pad_mm_bench', 1), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.0831993Z graph_break [] 2025-12-04T11:13:50.0832123Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.0832215Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.0832339Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.0832873Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('benchmarking.InductorBenchmarker.benchmark_gpu', 2), ('pad_mm_bench', 1), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.0832930Z graph_break [] 2025-12-04T11:13:50.0833429Z - generated xml file: /var/lib/jenkins/workspace/test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-2f9606f02b59f87b.xml - 2025-12-04T11:13:50.0833528Z =========================== short test summary info ============================ 2025-12-04T11:13:50.0834814Z FAILED [0.5472s] inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_17_mid_dim_1_in_features_128_out_features_65_cuda_bfloat16 - RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_1_in_features_128_out_features_65_cuda_bfloat16! Caching allocator allocated memory was 17408 and is now reported as 26112 on device 0. CUDA driver allocated memory was 276824064 and is now 278921216. 2025-12-04T11:13:50.0834826Z 2025-12-04T11:13:50.0834954Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.0835472Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_1_in_features_128_out_features_65_cuda_bfloat16 2025-12-04T11:13:50.0835476Z 2025-12-04T11:13:50.0835636Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.0835739Z !!!!!!!!!!!!!!!!!!!!!!!!!! stopping after 1 failures !!!!!!!!!!!!!!!!!!!!!!!!!!! 2025-12-04T11:13:50.0835855Z =================== 1 failed, 9 deselected, 2 rerun in 3.09s =================== 2025-12-04T11:13:50.0835917Z Got exit code 1 2025-12-04T11:13:50.0835983Z Retrying single test... 2025-12-04T11:13:50.0836253Z W1204 10:34:53.060000 61737 site-packages/torch/_inductor/utils.py:1703] Not enough SMs to use max_autotune_gemm mode 2025-12-04T11:13:50.0836638Z Test results will be stored in test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-94458b88fceefe4e.xml 2025-12-04T11:13:50.0836736Z ============================= test session starts ============================== 2025-12-04T11:13:50.0836956Z platform linux -- Python 3.10.14, pytest-7.3.2, pluggy-1.6.0 -- /opt/conda/envs/py_3.10/bin/python 2025-12-04T11:13:50.0837023Z cachedir: .pytest_cache 2025-12-04T11:13:50.0837340Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow] 2025-12-04T11:13:50.0837420Z rootdir: /var/lib/jenkins/workspace 2025-12-04T11:13:50.0837485Z configfile: pytest.ini 2025-12-04T11:13:50.0837951Z plugins: hypothesis-6.56.4, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-14.0, subtests-0.13.1, xdist-3.3.1, xdoctest-1.3.0, typeguard-4.3.0 2025-12-04T11:13:50.0838084Z collecting ... collected 58 items / 57 deselected / 1 selected 2025-12-04T11:13:50.0838659Z stepcurrent: skipping 9 already run items. Running only test/inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_17_mid_dim_1_in_features_128_out_features_65_cuda_bfloat16 2025-12-04T11:13:50.0838730Z Running 1 items in this shard 2025-12-04T11:13:50.0838734Z 2025-12-04T11:13:50.0839470Z inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_17_mid_dim_1_in_features_128_out_features_65_cuda_bfloat16 [W1204 10:34:54.713002107 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0839478Z 2025-12-04T11:13:50.0839789Z [W1204 10:35:03.752537015 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0839796Z 2025-12-04T11:13:50.0840090Z [W1204 10:35:03.752785070 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0840093Z 2025-12-04T11:13:50.0840383Z [W1204 10:35:03.758688320 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0840387Z 2025-12-04T11:13:50.0840670Z [W1204 10:35:03.759262953 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0840674Z 2025-12-04T11:13:50.0840961Z [W1204 10:35:03.759439446 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0840964Z 2025-12-04T11:13:50.0841252Z [W1204 10:35:03.765130402 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0841258Z 2025-12-04T11:13:50.0841550Z [W1204 10:35:03.765779426 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0841554Z 2025-12-04T11:13:50.0841838Z [W1204 10:35:03.765940179 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0841842Z 2025-12-04T11:13:50.0841925Z ('RERUN', {'yellow': True}) [11.0316s] [100%] 2025-12-04T11:13:50.0842648Z inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_17_mid_dim_1_in_features_128_out_features_65_cuda_bfloat16 [W1204 10:35:04.572783263 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0842653Z 2025-12-04T11:13:50.0842938Z [W1204 10:35:04.573331135 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0842946Z 2025-12-04T11:13:50.0843234Z [W1204 10:35:04.573472188 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0843237Z 2025-12-04T11:13:50.0843521Z [W1204 10:35:04.576365311 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0843524Z 2025-12-04T11:13:50.0843809Z [W1204 10:35:04.576839182 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0843813Z 2025-12-04T11:13:50.0844094Z [W1204 10:35:04.576977055 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0844097Z 2025-12-04T11:13:50.0844383Z [W1204 10:35:04.581531655 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0844460Z 2025-12-04T11:13:50.0844810Z [W1204 10:35:04.581998285 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0844814Z 2025-12-04T11:13:50.0845101Z [W1204 10:35:04.582131758 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0845105Z 2025-12-04T11:13:50.0845184Z ('RERUN', {'yellow': True}) [0.4983s] [100%] 2025-12-04T11:13:50.0845903Z inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_17_mid_dim_1_in_features_128_out_features_65_cuda_bfloat16 [W1204 10:35:05.068005813 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0845915Z 2025-12-04T11:13:50.0846209Z [W1204 10:35:05.068587776 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0846215Z 2025-12-04T11:13:50.0846504Z [W1204 10:35:05.068732419 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0846507Z 2025-12-04T11:13:50.0846795Z [W1204 10:35:05.071619202 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0846798Z 2025-12-04T11:13:50.0847081Z [W1204 10:35:05.072080322 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0847084Z 2025-12-04T11:13:50.0847371Z [W1204 10:35:05.072215575 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0847374Z 2025-12-04T11:13:50.0847655Z [W1204 10:35:05.076624432 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0847660Z 2025-12-04T11:13:50.0847951Z [W1204 10:35:05.077086012 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0847955Z 2025-12-04T11:13:50.0848239Z [W1204 10:35:05.077223005 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0848242Z 2025-12-04T11:13:50.0848305Z FAILED [0.4922s] [100%] 2025-12-04T11:13:50.0848308Z 2025-12-04T11:13:50.0848393Z ==================================== RERUNS ==================================== 2025-12-04T11:13:50.0848685Z _ TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_1_in_features_128_out_features_65_cuda_bfloat16 _ 2025-12-04T11:13:50.0848766Z Traceback (most recent call last): 2025-12-04T11:13:50.0849077Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.0849144Z method(*args, **kwargs) 2025-12-04T11:13:50.0849441Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.0849505Z method(*args, **kwargs) 2025-12-04T11:13:50.0849796Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T11:13:50.0849855Z with policy(): 2025-12-04T11:13:50.0850148Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T11:13:50.0850219Z raise RuntimeError(msg) 2025-12-04T11:13:50.0851014Z RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_1_in_features_128_out_features_65_cuda_bfloat16! Caching allocator allocated memory was 0 and is now reported as 8704 on device 0. CUDA driver allocated memory was 230686720 and is now 274726912. 2025-12-04T11:13:50.0851092Z 2025-12-04T11:13:50.0851226Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.0851808Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_1_in_features_128_out_features_65_cuda_bfloat16 2025-12-04T11:13:50.0851813Z 2025-12-04T11:13:50.0851975Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.0852107Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.0852203Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.0852754Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('benchmarking.InductorBenchmarker.benchmark_gpu', 2), ('pad_mm_bench', 1), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.0852886Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.0852949Z graph_break [] 2025-12-04T11:13:50.0853083Z ----------------------------- Captured stderr call ----------------------------- 2025-12-04T11:13:50.0853786Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/constant_folding.py:256: UserWarning: Unsupported unwinding pattern: Address not in range (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/profiler/unwind/unwind.cpp:219.) 2025-12-04T11:13:50.0853862Z if out == self.unknown_value: 2025-12-04T11:13:50.0854160Z _ TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_1_in_features_128_out_features_65_cuda_bfloat16 _ 2025-12-04T11:13:50.0854237Z Traceback (most recent call last): 2025-12-04T11:13:50.0854544Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.0854608Z method(*args, **kwargs) 2025-12-04T11:13:50.0854899Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.0854976Z method(*args, **kwargs) 2025-12-04T11:13:50.0855264Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T11:13:50.0855327Z with policy(): 2025-12-04T11:13:50.0855622Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T11:13:50.0855688Z raise RuntimeError(msg) 2025-12-04T11:13:50.0856498Z RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_1_in_features_128_out_features_65_cuda_bfloat16! Caching allocator allocated memory was 8704 and is now reported as 17408 on device 0. CUDA driver allocated memory was 274726912 and is now 276824064. 2025-12-04T11:13:50.0856502Z 2025-12-04T11:13:50.0856629Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.0857155Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_1_in_features_128_out_features_65_cuda_bfloat16 2025-12-04T11:13:50.0857159Z 2025-12-04T11:13:50.0857316Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.0857444Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.0857536Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.0858082Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('benchmarking.InductorBenchmarker.benchmark_gpu', 2), ('pad_mm_bench', 1), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.0858212Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.0858270Z graph_break [] 2025-12-04T11:13:50.0858488Z ----------------------------- Captured stderr call ----------------------------- 2025-12-04T11:13:50.0859232Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/constant_folding.py:256: UserWarning: Unsupported unwinding pattern: Address not in range (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/profiler/unwind/unwind.cpp:219.) 2025-12-04T11:13:50.0859302Z if out == self.unknown_value: 2025-12-04T11:13:50.0859428Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.0859516Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.0859636Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.0860175Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('benchmarking.InductorBenchmarker.benchmark_gpu', 2), ('pad_mm_bench', 1), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.0860234Z graph_break [] 2025-12-04T11:13:50.0860319Z =================================== FAILURES =================================== 2025-12-04T11:13:50.0860612Z _ TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_1_in_features_128_out_features_65_cuda_bfloat16 _ 2025-12-04T11:13:50.0860684Z Traceback (most recent call last): 2025-12-04T11:13:50.0860978Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.0861042Z method(*args, **kwargs) 2025-12-04T11:13:50.0861331Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.0861393Z method(*args, **kwargs) 2025-12-04T11:13:50.0861682Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T11:13:50.0861755Z with policy(): 2025-12-04T11:13:50.0862056Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T11:13:50.0862123Z raise RuntimeError(msg) 2025-12-04T11:13:50.0862935Z RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_1_in_features_128_out_features_65_cuda_bfloat16! Caching allocator allocated memory was 17408 and is now reported as 26112 on device 0. CUDA driver allocated memory was 276824064 and is now 278921216. 2025-12-04T11:13:50.0862939Z 2025-12-04T11:13:50.0863063Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.0863645Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_1_in_features_128_out_features_65_cuda_bfloat16 2025-12-04T11:13:50.0863649Z 2025-12-04T11:13:50.0863804Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.0863932Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.0864024Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.0864562Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('benchmarking.InductorBenchmarker.benchmark_gpu', 2), ('pad_mm_bench', 1), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.0864687Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.0864745Z graph_break [] 2025-12-04T11:13:50.0864867Z ----------------------------- Captured stderr call ----------------------------- 2025-12-04T11:13:50.0865542Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/constant_folding.py:256: UserWarning: Unsupported unwinding pattern: Address not in range (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/profiler/unwind/unwind.cpp:219.) 2025-12-04T11:13:50.0865690Z if out == self.unknown_value: 2025-12-04T11:13:50.0865815Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.0865966Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.0866089Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.0866629Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('benchmarking.InductorBenchmarker.benchmark_gpu', 2), ('pad_mm_bench', 1), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.0866686Z graph_break [] 2025-12-04T11:13:50.0866808Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.0866895Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.0867013Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.0867553Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('benchmarking.InductorBenchmarker.benchmark_gpu', 2), ('pad_mm_bench', 1), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.0867613Z graph_break [] 2025-12-04T11:13:50.0868104Z - generated xml file: /var/lib/jenkins/workspace/test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-94458b88fceefe4e.xml - 2025-12-04T11:13:50.0868215Z =========================== short test summary info ============================ 2025-12-04T11:13:50.0869506Z FAILED [0.4922s] inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_17_mid_dim_1_in_features_128_out_features_65_cuda_bfloat16 - RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_1_in_features_128_out_features_65_cuda_bfloat16! Caching allocator allocated memory was 17408 and is now reported as 26112 on device 0. CUDA driver allocated memory was 276824064 and is now 278921216. 2025-12-04T11:13:50.0869513Z 2025-12-04T11:13:50.0869640Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.0870155Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_1_in_features_128_out_features_65_cuda_bfloat16 2025-12-04T11:13:50.0870162Z 2025-12-04T11:13:50.0870316Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.0870420Z !!!!!!!!!!!!!!!!!!!!!!!!!! stopping after 1 failures !!!!!!!!!!!!!!!!!!!!!!!!!!! 2025-12-04T11:13:50.0870538Z ================== 1 failed, 57 deselected, 2 rerun in 12.05s ================== 2025-12-04T11:13:50.0870596Z Got exit code 1 2025-12-04T11:13:50.0870659Z Retrying single test... 2025-12-04T11:13:50.0870924Z W1204 10:35:11.848000 61924 site-packages/torch/_inductor/utils.py:1703] Not enough SMs to use max_autotune_gemm mode 2025-12-04T11:13:50.0871313Z Test results will be stored in test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-6787c22d74999180.xml 2025-12-04T11:13:50.0871409Z ============================= test session starts ============================== 2025-12-04T11:13:50.0871616Z platform linux -- Python 3.10.14, pytest-7.3.2, pluggy-1.6.0 -- /opt/conda/envs/py_3.10/bin/python 2025-12-04T11:13:50.0871681Z cachedir: .pytest_cache 2025-12-04T11:13:50.0871989Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow] 2025-12-04T11:13:50.0872064Z rootdir: /var/lib/jenkins/workspace 2025-12-04T11:13:50.0872128Z configfile: pytest.ini 2025-12-04T11:13:50.0872444Z plugins: hypothesis-6.56.4, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-14.0, subtests-0.13.1, xdist-3.3.1, xdoctest-1.3.0, typeguard-4.3.0 2025-12-04T11:13:50.0872572Z collecting ... collected 58 items / 57 deselected / 1 selected 2025-12-04T11:13:50.0873749Z stepcurrent: skipping 9 already run items. Running only test/inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_17_mid_dim_1_in_features_128_out_features_65_cuda_bfloat16 2025-12-04T11:13:50.0873823Z Running 1 items in this shard 2025-12-04T11:13:50.0873827Z 2025-12-04T11:13:50.0874552Z inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_17_mid_dim_1_in_features_128_out_features_65_cuda_bfloat16 [W1204 10:35:13.481009823 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0874559Z 2025-12-04T11:13:50.0874855Z [W1204 10:35:22.633544008 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0874859Z 2025-12-04T11:13:50.0875148Z [W1204 10:35:22.633800353 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0875158Z 2025-12-04T11:13:50.0875446Z [W1204 10:35:22.640159703 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0875450Z 2025-12-04T11:13:50.0875732Z [W1204 10:35:22.640739346 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0875735Z 2025-12-04T11:13:50.0876023Z [W1204 10:35:22.640924780 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0876027Z 2025-12-04T11:13:50.0876309Z [W1204 10:35:22.646322318 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0876313Z 2025-12-04T11:13:50.0876599Z [W1204 10:35:22.646844630 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0876605Z 2025-12-04T11:13:50.0876891Z [W1204 10:35:22.647008193 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0876894Z 2025-12-04T11:13:50.0876978Z ('RERUN', {'yellow': True}) [11.1315s] [100%] 2025-12-04T11:13:50.0877695Z inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_17_mid_dim_1_in_features_128_out_features_65_cuda_bfloat16 [W1204 10:35:23.458215168 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0877699Z 2025-12-04T11:13:50.0878336Z [W1204 10:35:23.458789911 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0878346Z 2025-12-04T11:13:50.0878644Z [W1204 10:35:23.458933314 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0878652Z 2025-12-04T11:13:50.0878941Z [W1204 10:35:23.461937360 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0878945Z 2025-12-04T11:13:50.0879230Z [W1204 10:35:23.462408131 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0879233Z 2025-12-04T11:13:50.0879517Z [W1204 10:35:23.462545224 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0879520Z 2025-12-04T11:13:50.0879807Z [W1204 10:35:23.467116175 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0879810Z 2025-12-04T11:13:50.0880096Z [W1204 10:35:23.467578975 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0880230Z 2025-12-04T11:13:50.0880633Z [W1204 10:35:23.467714828 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0880637Z 2025-12-04T11:13:50.0880718Z ('RERUN', {'yellow': True}) [0.5055s] [100%] 2025-12-04T11:13:50.0881460Z inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_17_mid_dim_1_in_features_128_out_features_65_cuda_bfloat16 [W1204 10:35:24.960421408 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0881464Z 2025-12-04T11:13:50.0881752Z [W1204 10:35:24.961004551 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0881755Z 2025-12-04T11:13:50.0882036Z [W1204 10:35:24.961147624 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0882047Z 2025-12-04T11:13:50.0882333Z [W1204 10:35:24.964083389 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0882337Z 2025-12-04T11:13:50.0882620Z [W1204 10:35:24.964546169 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0882623Z 2025-12-04T11:13:50.0882909Z [W1204 10:35:24.964696972 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0882913Z 2025-12-04T11:13:50.0883196Z [W1204 10:35:24.969196981 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0883199Z 2025-12-04T11:13:50.0883488Z [W1204 10:35:24.969661731 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0883495Z 2025-12-04T11:13:50.0883782Z [W1204 10:35:24.969798244 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0883788Z 2025-12-04T11:13:50.0883851Z FAILED [0.5007s] [100%] 2025-12-04T11:13:50.0883855Z 2025-12-04T11:13:50.0883940Z ==================================== RERUNS ==================================== 2025-12-04T11:13:50.0884236Z _ TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_1_in_features_128_out_features_65_cuda_bfloat16 _ 2025-12-04T11:13:50.0884316Z Traceback (most recent call last): 2025-12-04T11:13:50.0884623Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.0884691Z method(*args, **kwargs) 2025-12-04T11:13:50.0884982Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.0885043Z method(*args, **kwargs) 2025-12-04T11:13:50.0885336Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T11:13:50.0885397Z with policy(): 2025-12-04T11:13:50.0885690Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T11:13:50.0885758Z raise RuntimeError(msg) 2025-12-04T11:13:50.0886551Z RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_1_in_features_128_out_features_65_cuda_bfloat16! Caching allocator allocated memory was 0 and is now reported as 8704 on device 0. CUDA driver allocated memory was 230686720 and is now 274726912. 2025-12-04T11:13:50.0886556Z 2025-12-04T11:13:50.0886687Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.0887206Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_1_in_features_128_out_features_65_cuda_bfloat16 2025-12-04T11:13:50.0887284Z 2025-12-04T11:13:50.0887511Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.0887643Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.0887738Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.0888285Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('benchmarking.InductorBenchmarker.benchmark_gpu', 2), ('pad_mm_bench', 1), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.0888413Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.0888474Z graph_break [] 2025-12-04T11:13:50.0888608Z ----------------------------- Captured stderr call ----------------------------- 2025-12-04T11:13:50.0889309Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/constant_folding.py:256: UserWarning: Unsupported unwinding pattern: Address not in range (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/profiler/unwind/unwind.cpp:219.) 2025-12-04T11:13:50.0889384Z if out == self.unknown_value: 2025-12-04T11:13:50.0889677Z _ TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_1_in_features_128_out_features_65_cuda_bfloat16 _ 2025-12-04T11:13:50.0889750Z Traceback (most recent call last): 2025-12-04T11:13:50.0890049Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.0890111Z method(*args, **kwargs) 2025-12-04T11:13:50.0890401Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.0890461Z method(*args, **kwargs) 2025-12-04T11:13:50.0890748Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T11:13:50.0890816Z with policy(): 2025-12-04T11:13:50.0891111Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T11:13:50.0891178Z raise RuntimeError(msg) 2025-12-04T11:13:50.0891977Z RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_1_in_features_128_out_features_65_cuda_bfloat16! Caching allocator allocated memory was 8704 and is now reported as 17408 on device 0. CUDA driver allocated memory was 274726912 and is now 276824064. 2025-12-04T11:13:50.0891981Z 2025-12-04T11:13:50.0892106Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.0892626Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_1_in_features_128_out_features_65_cuda_bfloat16 2025-12-04T11:13:50.0892633Z 2025-12-04T11:13:50.0892787Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.0892915Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.0893008Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.0893551Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('benchmarking.InductorBenchmarker.benchmark_gpu', 2), ('pad_mm_bench', 1), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.0893680Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.0893739Z graph_break [] 2025-12-04T11:13:50.0893864Z ----------------------------- Captured stderr call ----------------------------- 2025-12-04T11:13:50.0894544Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/constant_folding.py:256: UserWarning: Unsupported unwinding pattern: Address not in range (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/profiler/unwind/unwind.cpp:219.) 2025-12-04T11:13:50.0894751Z if out == self.unknown_value: 2025-12-04T11:13:50.0894879Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.0894968Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.0895092Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.0895628Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('benchmarking.InductorBenchmarker.benchmark_gpu', 2), ('pad_mm_bench', 1), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.0895686Z graph_break [] 2025-12-04T11:13:50.0895772Z =================================== FAILURES =================================== 2025-12-04T11:13:50.0896058Z _ TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_1_in_features_128_out_features_65_cuda_bfloat16 _ 2025-12-04T11:13:50.0896135Z Traceback (most recent call last): 2025-12-04T11:13:50.0896435Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.0896497Z method(*args, **kwargs) 2025-12-04T11:13:50.0896784Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.0896845Z method(*args, **kwargs) 2025-12-04T11:13:50.0897129Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T11:13:50.0897190Z with policy(): 2025-12-04T11:13:50.0897481Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T11:13:50.0897548Z raise RuntimeError(msg) 2025-12-04T11:13:50.0898356Z RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_1_in_features_128_out_features_65_cuda_bfloat16! Caching allocator allocated memory was 17408 and is now reported as 26112 on device 0. CUDA driver allocated memory was 276824064 and is now 278921216. 2025-12-04T11:13:50.0898363Z 2025-12-04T11:13:50.0898489Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.0899005Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_1_in_features_128_out_features_65_cuda_bfloat16 2025-12-04T11:13:50.0899009Z 2025-12-04T11:13:50.0899164Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.0899297Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.0899387Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.0899928Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('benchmarking.InductorBenchmarker.benchmark_gpu', 2), ('pad_mm_bench', 1), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.0900056Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.0900114Z graph_break [] 2025-12-04T11:13:50.0900238Z ----------------------------- Captured stderr call ----------------------------- 2025-12-04T11:13:50.0900929Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/constant_folding.py:256: UserWarning: Unsupported unwinding pattern: Address not in range (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/profiler/unwind/unwind.cpp:219.) 2025-12-04T11:13:50.0900998Z if out == self.unknown_value: 2025-12-04T11:13:50.0901121Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.0901209Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.0901454Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.0902276Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('benchmarking.InductorBenchmarker.benchmark_gpu', 2), ('pad_mm_bench', 1), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.0902343Z graph_break [] 2025-12-04T11:13:50.0902473Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.0902564Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.0902686Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.0903225Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('benchmarking.InductorBenchmarker.benchmark_gpu', 2), ('pad_mm_bench', 1), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.0903283Z graph_break [] 2025-12-04T11:13:50.0903859Z - generated xml file: /var/lib/jenkins/workspace/test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-6787c22d74999180.xml - 2025-12-04T11:13:50.0903962Z =========================== short test summary info ============================ 2025-12-04T11:13:50.0905247Z FAILED [0.5007s] inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_17_mid_dim_1_in_features_128_out_features_65_cuda_bfloat16 - RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_1_in_features_128_out_features_65_cuda_bfloat16! Caching allocator allocated memory was 17408 and is now reported as 26112 on device 0. CUDA driver allocated memory was 276824064 and is now 278921216. 2025-12-04T11:13:50.0905252Z 2025-12-04T11:13:50.0905381Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.0905902Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_1_in_features_128_out_features_65_cuda_bfloat16 2025-12-04T11:13:50.0905911Z 2025-12-04T11:13:50.0906069Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.0906176Z !!!!!!!!!!!!!!!!!!!!!!!!!! stopping after 1 failures !!!!!!!!!!!!!!!!!!!!!!!!!!! 2025-12-04T11:13:50.0906299Z ================== 1 failed, 57 deselected, 2 rerun in 12.16s ================== 2025-12-04T11:13:50.0906359Z Got exit code 1 2025-12-04T11:13:50.0906839Z FAILED CONSISTENTLY: test/inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_17_mid_dim_1_in_features_128_out_features_65_cuda_bfloat16 2025-12-04T11:13:50.0907086Z Test failed consistently, continuing with the rest of the tests due to continue-through-error being set 2025-12-04T11:13:50.0907349Z W1204 10:35:30.807000 62111 site-packages/torch/_inductor/utils.py:1703] Not enough SMs to use max_autotune_gemm mode 2025-12-04T11:13:50.0907745Z Test results will be stored in test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-cdf948359e80bb41.xml 2025-12-04T11:13:50.0907842Z ============================= test session starts ============================== 2025-12-04T11:13:50.0908050Z platform linux -- Python 3.10.14, pytest-7.3.2, pluggy-1.6.0 -- /opt/conda/envs/py_3.10/bin/python 2025-12-04T11:13:50.0908122Z cachedir: .pytest_cache 2025-12-04T11:13:50.0908432Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow] 2025-12-04T11:13:50.0908513Z rootdir: /var/lib/jenkins/workspace 2025-12-04T11:13:50.0908579Z configfile: pytest.ini 2025-12-04T11:13:50.0908895Z plugins: hypothesis-6.56.4, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-14.0, subtests-0.13.1, xdist-3.3.1, xdoctest-1.3.0, typeguard-4.3.0 2025-12-04T11:13:50.0909028Z collecting ... collected 58 items / 10 deselected / 48 selected 2025-12-04T11:13:50.0909200Z stepcurrent: skipping 10 already run items. 2025-12-04T11:13:50.0909273Z Running 48 items in this shard 2025-12-04T11:13:50.0909374Z 2025-12-04T11:13:50.0909883Z inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_17_mid_dim_1_in_features_144_out_features_1024_cuda_bfloat16 ('RERUN', {'yellow': True}) [1.9579s] [ 2%] 2025-12-04T11:13:50.0910375Z inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_17_mid_dim_1_in_features_144_out_features_1024_cuda_bfloat16 ('RERUN', {'yellow': True}) [0.5126s] [ 2%] 2025-12-04T11:13:50.0910828Z inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_17_mid_dim_1_in_features_144_out_features_1024_cuda_bfloat16 FAILED [0.4836s] [ 2%] 2025-12-04T11:13:50.0910833Z 2025-12-04T11:13:50.0910916Z ==================================== RERUNS ==================================== 2025-12-04T11:13:50.0911218Z _ TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_1_in_features_144_out_features_1024_cuda_bfloat16 _ 2025-12-04T11:13:50.0911298Z Traceback (most recent call last): 2025-12-04T11:13:50.0911604Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.0911674Z method(*args, **kwargs) 2025-12-04T11:13:50.0911962Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.0912034Z method(*args, **kwargs) 2025-12-04T11:13:50.0912322Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T11:13:50.0912385Z with policy(): 2025-12-04T11:13:50.0912683Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T11:13:50.0912753Z raise RuntimeError(msg) 2025-12-04T11:13:50.0913562Z RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_1_in_features_144_out_features_1024_cuda_bfloat16! Caching allocator allocated memory was 0 and is now reported as 147456 on device 0. CUDA driver allocated memory was 230686720 and is now 274726912. 2025-12-04T11:13:50.0913576Z 2025-12-04T11:13:50.0913712Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.0914238Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_1_in_features_144_out_features_1024_cuda_bfloat16 2025-12-04T11:13:50.0914242Z 2025-12-04T11:13:50.0914405Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.0914535Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.0914642Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.0914996Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.0915129Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.0915197Z graph_break [] 2025-12-04T11:13:50.0915496Z _ TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_1_in_features_144_out_features_1024_cuda_bfloat16 _ 2025-12-04T11:13:50.0915583Z Traceback (most recent call last): 2025-12-04T11:13:50.0915884Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.0915952Z method(*args, **kwargs) 2025-12-04T11:13:50.0916249Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.0916314Z method(*args, **kwargs) 2025-12-04T11:13:50.0916685Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T11:13:50.0916817Z with policy(): 2025-12-04T11:13:50.0917112Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T11:13:50.0917186Z raise RuntimeError(msg) 2025-12-04T11:13:50.0918015Z RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_1_in_features_144_out_features_1024_cuda_bfloat16! Caching allocator allocated memory was 147456 and is now reported as 294912 on device 0. CUDA driver allocated memory was 274726912 and is now 276824064. 2025-12-04T11:13:50.0918020Z 2025-12-04T11:13:50.0918150Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.0918682Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_1_in_features_144_out_features_1024_cuda_bfloat16 2025-12-04T11:13:50.0918689Z 2025-12-04T11:13:50.0918852Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.0918989Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.0919085Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.0919450Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.0919588Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.0919651Z graph_break [] 2025-12-04T11:13:50.0919789Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.0919878Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.0919999Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.0920353Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.0920413Z graph_break [] 2025-12-04T11:13:50.0920498Z =================================== FAILURES =================================== 2025-12-04T11:13:50.0920798Z _ TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_1_in_features_144_out_features_1024_cuda_bfloat16 _ 2025-12-04T11:13:50.0920871Z Traceback (most recent call last): 2025-12-04T11:13:50.0921174Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.0921238Z method(*args, **kwargs) 2025-12-04T11:13:50.0921526Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.0921596Z method(*args, **kwargs) 2025-12-04T11:13:50.0921884Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T11:13:50.0921947Z with policy(): 2025-12-04T11:13:50.0922242Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T11:13:50.0922309Z raise RuntimeError(msg) 2025-12-04T11:13:50.0923131Z RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_1_in_features_144_out_features_1024_cuda_bfloat16! Caching allocator allocated memory was 294912 and is now reported as 442368 on device 0. CUDA driver allocated memory was 276824064 and is now 278921216. 2025-12-04T11:13:50.0923135Z 2025-12-04T11:13:50.0923260Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.0923785Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_1_in_features_144_out_features_1024_cuda_bfloat16 2025-12-04T11:13:50.0923926Z 2025-12-04T11:13:50.0924087Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.0924216Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.0924319Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.0924661Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.0924786Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.0924847Z graph_break [] 2025-12-04T11:13:50.0924970Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.0925063Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.0925196Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.0925542Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.0925605Z graph_break [] 2025-12-04T11:13:50.0925729Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.0925820Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.0925942Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.0926278Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.0926339Z graph_break [] 2025-12-04T11:13:50.0926826Z - generated xml file: /var/lib/jenkins/workspace/test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-cdf948359e80bb41.xml - 2025-12-04T11:13:50.0926932Z =========================== short test summary info ============================ 2025-12-04T11:13:50.0928230Z FAILED [0.4836s] inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_17_mid_dim_1_in_features_144_out_features_1024_cuda_bfloat16 - RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_1_in_features_144_out_features_1024_cuda_bfloat16! Caching allocator allocated memory was 294912 and is now reported as 442368 on device 0. CUDA driver allocated memory was 276824064 and is now 278921216. 2025-12-04T11:13:50.0928234Z 2025-12-04T11:13:50.0928364Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.0928884Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_1_in_features_144_out_features_1024_cuda_bfloat16 2025-12-04T11:13:50.0928891Z 2025-12-04T11:13:50.0929046Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.0929158Z !!!!!!!!!!!!!!!!!!!!!!!!!! stopping after 1 failures !!!!!!!!!!!!!!!!!!!!!!!!!!! 2025-12-04T11:13:50.0929276Z ================== 1 failed, 10 deselected, 2 rerun in 2.98s =================== 2025-12-04T11:13:50.0929339Z Got exit code 1 2025-12-04T11:13:50.0929412Z Retrying single test... 2025-12-04T11:13:50.0929675Z W1204 10:35:40.903000 62300 site-packages/torch/_inductor/utils.py:1703] Not enough SMs to use max_autotune_gemm mode 2025-12-04T11:13:50.0930063Z Test results will be stored in test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-bd192fd6fb52c26e.xml 2025-12-04T11:13:50.0930159Z ============================= test session starts ============================== 2025-12-04T11:13:50.0930365Z platform linux -- Python 3.10.14, pytest-7.3.2, pluggy-1.6.0 -- /opt/conda/envs/py_3.10/bin/python 2025-12-04T11:13:50.0930513Z cachedir: .pytest_cache 2025-12-04T11:13:50.0931045Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow] 2025-12-04T11:13:50.0931140Z rootdir: /var/lib/jenkins/workspace 2025-12-04T11:13:50.0931208Z configfile: pytest.ini 2025-12-04T11:13:50.0931530Z plugins: hypothesis-6.56.4, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-14.0, subtests-0.13.1, xdist-3.3.1, xdoctest-1.3.0, typeguard-4.3.0 2025-12-04T11:13:50.0931665Z collecting ... collected 58 items / 57 deselected / 1 selected 2025-12-04T11:13:50.0932247Z stepcurrent: skipping 10 already run items. Running only test/inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_17_mid_dim_1_in_features_144_out_features_1024_cuda_bfloat16 2025-12-04T11:13:50.0932323Z Running 1 items in this shard 2025-12-04T11:13:50.0932327Z 2025-12-04T11:13:50.0933063Z inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_17_mid_dim_1_in_features_144_out_features_1024_cuda_bfloat16 [W1204 10:35:42.268949476 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0933071Z 2025-12-04T11:13:50.0933373Z [W1204 10:35:51.318271187 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0933377Z 2025-12-04T11:13:50.0933666Z [W1204 10:35:51.318499212 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0933669Z 2025-12-04T11:13:50.0933955Z [W1204 10:35:51.324192437 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0933959Z 2025-12-04T11:13:50.0934248Z [W1204 10:35:51.324760260 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0934254Z 2025-12-04T11:13:50.0934540Z [W1204 10:35:51.324924534 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0934543Z 2025-12-04T11:13:50.0934832Z [W1204 10:35:51.330205100 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0934835Z 2025-12-04T11:13:50.0935122Z [W1204 10:35:51.330726281 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0935125Z 2025-12-04T11:13:50.0935415Z [W1204 10:35:51.330882965 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0935418Z 2025-12-04T11:13:50.0935510Z ('RERUN', {'yellow': True}) [11.0220s] [100%] 2025-12-04T11:13:50.0936244Z inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_17_mid_dim_1_in_features_144_out_features_1024_cuda_bfloat16 [W1204 10:35:52.365424928 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0936250Z 2025-12-04T11:13:50.0936536Z [W1204 10:35:52.365970030 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0936540Z 2025-12-04T11:13:50.0936827Z [W1204 10:35:52.366112264 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0936834Z 2025-12-04T11:13:50.0937119Z [W1204 10:35:52.369050618 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0937122Z 2025-12-04T11:13:50.0937408Z [W1204 10:35:52.369619021 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0937580Z 2025-12-04T11:13:50.0937938Z [W1204 10:35:52.369755844 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0937941Z 2025-12-04T11:13:50.0938228Z [W1204 10:35:52.374357275 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0938231Z 2025-12-04T11:13:50.0938519Z [W1204 10:35:52.374829516 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0938523Z 2025-12-04T11:13:50.0938807Z [W1204 10:35:52.374963789 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0938810Z 2025-12-04T11:13:50.0938895Z ('RERUN', {'yellow': True}) [0.4539s] [100%] 2025-12-04T11:13:50.0939632Z inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_17_mid_dim_1_in_features_144_out_features_1024_cuda_bfloat16 [W1204 10:35:52.816308184 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0939638Z 2025-12-04T11:13:50.0939934Z [W1204 10:35:52.816854596 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0939937Z 2025-12-04T11:13:50.0940228Z [W1204 10:35:52.816997459 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0940231Z 2025-12-04T11:13:50.0940519Z [W1204 10:35:52.819876892 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0940529Z 2025-12-04T11:13:50.0940815Z [W1204 10:35:52.820464735 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0940821Z 2025-12-04T11:13:50.0941108Z [W1204 10:35:52.820621069 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0941111Z 2025-12-04T11:13:50.0941402Z [W1204 10:35:52.825078947 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0941405Z 2025-12-04T11:13:50.0941690Z [W1204 10:35:52.825540417 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0941693Z 2025-12-04T11:13:50.0941982Z [W1204 10:35:52.825674549 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0941986Z 2025-12-04T11:13:50.0942048Z FAILED [0.4515s] [100%] 2025-12-04T11:13:50.0942052Z 2025-12-04T11:13:50.0942141Z ==================================== RERUNS ==================================== 2025-12-04T11:13:50.0942442Z _ TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_1_in_features_144_out_features_1024_cuda_bfloat16 _ 2025-12-04T11:13:50.0942520Z Traceback (most recent call last): 2025-12-04T11:13:50.0942833Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.0942897Z method(*args, **kwargs) 2025-12-04T11:13:50.0943189Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.0943258Z method(*args, **kwargs) 2025-12-04T11:13:50.0943622Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T11:13:50.0943688Z with policy(): 2025-12-04T11:13:50.0943983Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T11:13:50.0944126Z raise RuntimeError(msg) 2025-12-04T11:13:50.0945000Z RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_1_in_features_144_out_features_1024_cuda_bfloat16! Caching allocator allocated memory was 0 and is now reported as 147456 on device 0. CUDA driver allocated memory was 230686720 and is now 274726912. 2025-12-04T11:13:50.0945005Z 2025-12-04T11:13:50.0945134Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.0945664Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_1_in_features_144_out_features_1024_cuda_bfloat16 2025-12-04T11:13:50.0945668Z 2025-12-04T11:13:50.0945827Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.0945954Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.0946055Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.0946408Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.0946540Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.0946600Z graph_break [] 2025-12-04T11:13:50.0946726Z ----------------------------- Captured stderr call ----------------------------- 2025-12-04T11:13:50.0947419Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/constant_folding.py:256: UserWarning: Unsupported unwinding pattern: Address not in range (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/profiler/unwind/unwind.cpp:219.) 2025-12-04T11:13:50.0947492Z if out == self.unknown_value: 2025-12-04T11:13:50.0947792Z _ TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_1_in_features_144_out_features_1024_cuda_bfloat16 _ 2025-12-04T11:13:50.0947869Z Traceback (most recent call last): 2025-12-04T11:13:50.0948166Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.0948237Z method(*args, **kwargs) 2025-12-04T11:13:50.0948526Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.0948589Z method(*args, **kwargs) 2025-12-04T11:13:50.0948879Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T11:13:50.0948938Z with policy(): 2025-12-04T11:13:50.0949243Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T11:13:50.0949308Z raise RuntimeError(msg) 2025-12-04T11:13:50.0950132Z RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_1_in_features_144_out_features_1024_cuda_bfloat16! Caching allocator allocated memory was 147456 and is now reported as 294912 on device 0. CUDA driver allocated memory was 274726912 and is now 276824064. 2025-12-04T11:13:50.0950143Z 2025-12-04T11:13:50.0950270Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.0950790Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_1_in_features_144_out_features_1024_cuda_bfloat16 2025-12-04T11:13:50.0950794Z 2025-12-04T11:13:50.0950954Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.0951080Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.0951177Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.0951524Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.0951787Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.0951851Z graph_break [] 2025-12-04T11:13:50.0951976Z ----------------------------- Captured stderr call ----------------------------- 2025-12-04T11:13:50.0952667Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/constant_folding.py:256: UserWarning: Unsupported unwinding pattern: Address not in range (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/profiler/unwind/unwind.cpp:219.) 2025-12-04T11:13:50.0952743Z if out == self.unknown_value: 2025-12-04T11:13:50.0952867Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.0952960Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.0953085Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.0953427Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.0953495Z graph_break [] 2025-12-04T11:13:50.0953578Z =================================== FAILURES =================================== 2025-12-04T11:13:50.0953886Z _ TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_1_in_features_144_out_features_1024_cuda_bfloat16 _ 2025-12-04T11:13:50.0953960Z Traceback (most recent call last): 2025-12-04T11:13:50.0954257Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.0954324Z method(*args, **kwargs) 2025-12-04T11:13:50.0954614Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.0954677Z method(*args, **kwargs) 2025-12-04T11:13:50.0954970Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T11:13:50.0955032Z with policy(): 2025-12-04T11:13:50.0955341Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T11:13:50.0955406Z raise RuntimeError(msg) 2025-12-04T11:13:50.0956225Z RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_1_in_features_144_out_features_1024_cuda_bfloat16! Caching allocator allocated memory was 294912 and is now reported as 442368 on device 0. CUDA driver allocated memory was 276824064 and is now 278921216. 2025-12-04T11:13:50.0956234Z 2025-12-04T11:13:50.0956358Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.0956874Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_1_in_features_144_out_features_1024_cuda_bfloat16 2025-12-04T11:13:50.0956881Z 2025-12-04T11:13:50.0957042Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.0957169Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.0957259Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.0957605Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.0957728Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.0957790Z graph_break [] 2025-12-04T11:13:50.0957914Z ----------------------------- Captured stderr call ----------------------------- 2025-12-04T11:13:50.0958599Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/constant_folding.py:256: UserWarning: Unsupported unwinding pattern: Address not in range (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/profiler/unwind/unwind.cpp:219.) 2025-12-04T11:13:50.0958748Z if out == self.unknown_value: 2025-12-04T11:13:50.0958933Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.0959030Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.0959153Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.0959501Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.0959564Z graph_break [] 2025-12-04T11:13:50.0959686Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.0959775Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.0959900Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.0960248Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.0960316Z graph_break [] 2025-12-04T11:13:50.0960808Z - generated xml file: /var/lib/jenkins/workspace/test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-bd192fd6fb52c26e.xml - 2025-12-04T11:13:50.0960907Z =========================== short test summary info ============================ 2025-12-04T11:13:50.0962213Z FAILED [0.4515s] inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_17_mid_dim_1_in_features_144_out_features_1024_cuda_bfloat16 - RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_1_in_features_144_out_features_1024_cuda_bfloat16! Caching allocator allocated memory was 294912 and is now reported as 442368 on device 0. CUDA driver allocated memory was 276824064 and is now 278921216. 2025-12-04T11:13:50.0962220Z 2025-12-04T11:13:50.0962345Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.0962871Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_1_in_features_144_out_features_1024_cuda_bfloat16 2025-12-04T11:13:50.0962875Z 2025-12-04T11:13:50.0963030Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.0963140Z !!!!!!!!!!!!!!!!!!!!!!!!!! stopping after 1 failures !!!!!!!!!!!!!!!!!!!!!!!!!!! 2025-12-04T11:13:50.0963255Z ================== 1 failed, 57 deselected, 2 rerun in 11.95s ================== 2025-12-04T11:13:50.0963315Z Got exit code 1 2025-12-04T11:13:50.0963388Z Retrying single test... 2025-12-04T11:13:50.0963650Z W1204 10:35:59.698000 62493 site-packages/torch/_inductor/utils.py:1703] Not enough SMs to use max_autotune_gemm mode 2025-12-04T11:13:50.0964038Z Test results will be stored in test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-7d48593db7bb6f60.xml 2025-12-04T11:13:50.0964139Z ============================= test session starts ============================== 2025-12-04T11:13:50.0964345Z platform linux -- Python 3.10.14, pytest-7.3.2, pluggy-1.6.0 -- /opt/conda/envs/py_3.10/bin/python 2025-12-04T11:13:50.0964420Z cachedir: .pytest_cache 2025-12-04T11:13:50.0964731Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow] 2025-12-04T11:13:50.0964809Z rootdir: /var/lib/jenkins/workspace 2025-12-04T11:13:50.0964880Z configfile: pytest.ini 2025-12-04T11:13:50.0965195Z plugins: hypothesis-6.56.4, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-14.0, subtests-0.13.1, xdist-3.3.1, xdoctest-1.3.0, typeguard-4.3.0 2025-12-04T11:13:50.0965327Z collecting ... collected 58 items / 57 deselected / 1 selected 2025-12-04T11:13:50.0965988Z stepcurrent: skipping 10 already run items. Running only test/inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_17_mid_dim_1_in_features_144_out_features_1024_cuda_bfloat16 2025-12-04T11:13:50.0966125Z Running 1 items in this shard 2025-12-04T11:13:50.0966129Z 2025-12-04T11:13:50.0966863Z inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_17_mid_dim_1_in_features_144_out_features_1024_cuda_bfloat16 [W1204 10:36:01.064819067 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0966868Z 2025-12-04T11:13:50.0967162Z [W1204 10:36:10.326437227 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0967165Z 2025-12-04T11:13:50.0967461Z [W1204 10:36:10.326685302 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0967467Z 2025-12-04T11:13:50.0967756Z [W1204 10:36:10.332464059 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0967759Z 2025-12-04T11:13:50.0968050Z [W1204 10:36:10.333033602 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0968054Z 2025-12-04T11:13:50.0968336Z [W1204 10:36:10.333192575 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0968339Z 2025-12-04T11:13:50.0968626Z [W1204 10:36:10.338526794 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0968629Z 2025-12-04T11:13:50.0968913Z [W1204 10:36:10.339051865 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0968916Z 2025-12-04T11:13:50.0969208Z [W1204 10:36:10.339217029 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0969214Z 2025-12-04T11:13:50.0969296Z ('RERUN', {'yellow': True}) [11.2411s] [100%] 2025-12-04T11:13:50.0970020Z inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_17_mid_dim_1_in_features_144_out_features_1024_cuda_bfloat16 [W1204 10:36:11.379865428 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0970028Z 2025-12-04T11:13:50.0970314Z [W1204 10:36:11.380428310 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0970318Z 2025-12-04T11:13:50.0970603Z [W1204 10:36:11.380579064 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0970606Z 2025-12-04T11:13:50.0970899Z [W1204 10:36:11.383573020 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0970904Z 2025-12-04T11:13:50.0971188Z [W1204 10:36:11.384152322 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0971192Z 2025-12-04T11:13:50.0971480Z [W1204 10:36:11.384289465 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0971483Z 2025-12-04T11:13:50.0971766Z [W1204 10:36:11.388979489 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0971769Z 2025-12-04T11:13:50.0972058Z [W1204 10:36:11.389459109 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0972061Z 2025-12-04T11:13:50.0972343Z [W1204 10:36:11.389594192 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0972419Z 2025-12-04T11:13:50.0972565Z ('RERUN', {'yellow': True}) [0.4619s] [100%] 2025-12-04T11:13:50.0973291Z inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_17_mid_dim_1_in_features_144_out_features_1024_cuda_bfloat16 [W1204 10:36:11.841550640 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0973294Z 2025-12-04T11:13:50.0973581Z [W1204 10:36:11.842090962 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0973585Z 2025-12-04T11:13:50.0973883Z [W1204 10:36:11.842228165 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0973886Z 2025-12-04T11:13:50.0974176Z [W1204 10:36:11.845124879 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0974179Z 2025-12-04T11:13:50.0974468Z [W1204 10:36:11.845681941 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0974472Z 2025-12-04T11:13:50.0974756Z [W1204 10:36:11.845817635 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0974759Z 2025-12-04T11:13:50.0975044Z [W1204 10:36:11.850375485 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0975048Z 2025-12-04T11:13:50.0975330Z [W1204 10:36:11.850845266 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0975333Z 2025-12-04T11:13:50.0975617Z [W1204 10:36:11.850976478 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.0975623Z 2025-12-04T11:13:50.0975687Z FAILED [0.4581s] [100%] 2025-12-04T11:13:50.0975690Z 2025-12-04T11:13:50.0975775Z ==================================== RERUNS ==================================== 2025-12-04T11:13:50.0976073Z _ TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_1_in_features_144_out_features_1024_cuda_bfloat16 _ 2025-12-04T11:13:50.0976148Z Traceback (most recent call last): 2025-12-04T11:13:50.0976456Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.0976522Z method(*args, **kwargs) 2025-12-04T11:13:50.0976812Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.0976880Z method(*args, **kwargs) 2025-12-04T11:13:50.0977167Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T11:13:50.0977230Z with policy(): 2025-12-04T11:13:50.0977529Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T11:13:50.0977595Z raise RuntimeError(msg) 2025-12-04T11:13:50.0978574Z RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_1_in_features_144_out_features_1024_cuda_bfloat16! Caching allocator allocated memory was 0 and is now reported as 147456 on device 0. CUDA driver allocated memory was 230686720 and is now 274726912. 2025-12-04T11:13:50.0978579Z 2025-12-04T11:13:50.0978707Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.0979236Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_1_in_features_144_out_features_1024_cuda_bfloat16 2025-12-04T11:13:50.0979356Z 2025-12-04T11:13:50.0979606Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.0979735Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.0979831Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.0980179Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.0980308Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.0980368Z graph_break [] 2025-12-04T11:13:50.0980493Z ----------------------------- Captured stderr call ----------------------------- 2025-12-04T11:13:50.0981183Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/constant_folding.py:256: UserWarning: Unsupported unwinding pattern: Address not in range (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/profiler/unwind/unwind.cpp:219.) 2025-12-04T11:13:50.0981260Z if out == self.unknown_value: 2025-12-04T11:13:50.0981554Z _ TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_1_in_features_144_out_features_1024_cuda_bfloat16 _ 2025-12-04T11:13:50.0981631Z Traceback (most recent call last): 2025-12-04T11:13:50.0981924Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.0981991Z method(*args, **kwargs) 2025-12-04T11:13:50.0982278Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.0982343Z method(*args, **kwargs) 2025-12-04T11:13:50.0982634Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T11:13:50.0982693Z with policy(): 2025-12-04T11:13:50.0982998Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T11:13:50.0983072Z raise RuntimeError(msg) 2025-12-04T11:13:50.0983970Z RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_1_in_features_144_out_features_1024_cuda_bfloat16! Caching allocator allocated memory was 147456 and is now reported as 294912 on device 0. CUDA driver allocated memory was 274726912 and is now 276824064. 2025-12-04T11:13:50.0983975Z 2025-12-04T11:13:50.0984111Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.0984630Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_1_in_features_144_out_features_1024_cuda_bfloat16 2025-12-04T11:13:50.0984634Z 2025-12-04T11:13:50.0984795Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.0984926Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.0985024Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.0985378Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.0985506Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.0985570Z graph_break [] 2025-12-04T11:13:50.0985700Z ----------------------------- Captured stderr call ----------------------------- 2025-12-04T11:13:50.0986389Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/constant_folding.py:256: UserWarning: Unsupported unwinding pattern: Address not in range (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/profiler/unwind/unwind.cpp:219.) 2025-12-04T11:13:50.0986465Z if out == self.unknown_value: 2025-12-04T11:13:50.0986668Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.0986840Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.0986972Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.0987318Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.0987381Z graph_break [] 2025-12-04T11:13:50.0987465Z =================================== FAILURES =================================== 2025-12-04T11:13:50.0987762Z _ TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_1_in_features_144_out_features_1024_cuda_bfloat16 _ 2025-12-04T11:13:50.0987852Z Traceback (most recent call last): 2025-12-04T11:13:50.0988158Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.0988230Z method(*args, **kwargs) 2025-12-04T11:13:50.0988523Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.0988585Z method(*args, **kwargs) 2025-12-04T11:13:50.0988878Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T11:13:50.0988937Z with policy(): 2025-12-04T11:13:50.0989234Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T11:13:50.0989308Z raise RuntimeError(msg) 2025-12-04T11:13:50.0990128Z RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_1_in_features_144_out_features_1024_cuda_bfloat16! Caching allocator allocated memory was 294912 and is now reported as 442368 on device 0. CUDA driver allocated memory was 276824064 and is now 278921216. 2025-12-04T11:13:50.0990135Z 2025-12-04T11:13:50.0990265Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.0990790Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_1_in_features_144_out_features_1024_cuda_bfloat16 2025-12-04T11:13:50.0990793Z 2025-12-04T11:13:50.0990953Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.0991082Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.0991174Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.0991523Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.0991648Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.0991713Z graph_break [] 2025-12-04T11:13:50.0991840Z ----------------------------- Captured stderr call ----------------------------- 2025-12-04T11:13:50.0992527Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/constant_folding.py:256: UserWarning: Unsupported unwinding pattern: Address not in range (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/profiler/unwind/unwind.cpp:219.) 2025-12-04T11:13:50.0992601Z if out == self.unknown_value: 2025-12-04T11:13:50.0992724Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.0992815Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.0992941Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.0993283Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.0993346Z graph_break [] 2025-12-04T11:13:50.0993550Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.0993639Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.0993830Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.0994168Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.0994229Z graph_break [] 2025-12-04T11:13:50.0994712Z - generated xml file: /var/lib/jenkins/workspace/test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-7d48593db7bb6f60.xml - 2025-12-04T11:13:50.0994811Z =========================== short test summary info ============================ 2025-12-04T11:13:50.0996128Z FAILED [0.4581s] inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_17_mid_dim_1_in_features_144_out_features_1024_cuda_bfloat16 - RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_1_in_features_144_out_features_1024_cuda_bfloat16! Caching allocator allocated memory was 294912 and is now reported as 442368 on device 0. CUDA driver allocated memory was 276824064 and is now 278921216. 2025-12-04T11:13:50.0996136Z 2025-12-04T11:13:50.0996263Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.0996787Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_1_in_features_144_out_features_1024_cuda_bfloat16 2025-12-04T11:13:50.0996790Z 2025-12-04T11:13:50.0996945Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.0997053Z !!!!!!!!!!!!!!!!!!!!!!!!!! stopping after 1 failures !!!!!!!!!!!!!!!!!!!!!!!!!!! 2025-12-04T11:13:50.0997169Z ================== 1 failed, 57 deselected, 2 rerun in 12.19s ================== 2025-12-04T11:13:50.0997230Z Got exit code 1 2025-12-04T11:13:50.0997709Z FAILED CONSISTENTLY: test/inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_17_mid_dim_1_in_features_144_out_features_1024_cuda_bfloat16 2025-12-04T11:13:50.0997952Z Test failed consistently, continuing with the rest of the tests due to continue-through-error being set 2025-12-04T11:13:50.0998214Z W1204 10:36:18.711000 62686 site-packages/torch/_inductor/utils.py:1703] Not enough SMs to use max_autotune_gemm mode 2025-12-04T11:13:50.0998604Z Test results will be stored in test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-91ed99958e7f9cc9.xml 2025-12-04T11:13:50.0998699Z ============================= test session starts ============================== 2025-12-04T11:13:50.0998916Z platform linux -- Python 3.10.14, pytest-7.3.2, pluggy-1.6.0 -- /opt/conda/envs/py_3.10/bin/python 2025-12-04T11:13:50.0998986Z cachedir: .pytest_cache 2025-12-04T11:13:50.0999294Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow] 2025-12-04T11:13:50.0999376Z rootdir: /var/lib/jenkins/workspace 2025-12-04T11:13:50.0999441Z configfile: pytest.ini 2025-12-04T11:13:50.0999758Z plugins: hypothesis-6.56.4, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-14.0, subtests-0.13.1, xdist-3.3.1, xdoctest-1.3.0, typeguard-4.3.0 2025-12-04T11:13:50.0999886Z collecting ... collected 58 items / 11 deselected / 47 selected 2025-12-04T11:13:50.0999974Z stepcurrent: skipping 11 already run items. 2025-12-04T11:13:50.1000048Z Running 47 items in this shard 2025-12-04T11:13:50.1000051Z 2025-12-04T11:13:50.1000547Z inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_17_mid_dim_1_in_features_144_out_features_64_cuda_bfloat16 ('RERUN', {'yellow': True}) [1.9816s] [ 2%] 2025-12-04T11:13:50.1001112Z inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_17_mid_dim_1_in_features_144_out_features_64_cuda_bfloat16 ('RERUN', {'yellow': True}) [0.5570s] [ 2%] 2025-12-04T11:13:50.1001621Z inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_17_mid_dim_1_in_features_144_out_features_64_cuda_bfloat16 FAILED [0.5437s] [ 2%] 2025-12-04T11:13:50.1001626Z 2025-12-04T11:13:50.1001710Z ==================================== RERUNS ==================================== 2025-12-04T11:13:50.1002006Z _ TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_1_in_features_144_out_features_64_cuda_bfloat16 _ 2025-12-04T11:13:50.1002081Z Traceback (most recent call last): 2025-12-04T11:13:50.1002391Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.1002455Z method(*args, **kwargs) 2025-12-04T11:13:50.1002744Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.1002813Z method(*args, **kwargs) 2025-12-04T11:13:50.1003104Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T11:13:50.1003165Z with policy(): 2025-12-04T11:13:50.1003463Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T11:13:50.1003528Z raise RuntimeError(msg) 2025-12-04T11:13:50.1004322Z RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_1_in_features_144_out_features_64_cuda_bfloat16! Caching allocator allocated memory was 0 and is now reported as 9216 on device 0. CUDA driver allocated memory was 230686720 and is now 274726912. 2025-12-04T11:13:50.1004326Z 2025-12-04T11:13:50.1004452Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.1004980Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_1_in_features_144_out_features_64_cuda_bfloat16 2025-12-04T11:13:50.1004984Z 2025-12-04T11:13:50.1005141Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.1005267Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.1005366Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.1005911Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('benchmarking.InductorBenchmarker.benchmark_gpu', 2), ('pad_mm_bench', 1), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.1006051Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.1006117Z graph_break [] 2025-12-04T11:13:50.1006412Z _ TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_1_in_features_144_out_features_64_cuda_bfloat16 _ 2025-12-04T11:13:50.1006493Z Traceback (most recent call last): 2025-12-04T11:13:50.1006789Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.1006852Z method(*args, **kwargs) 2025-12-04T11:13:50.1007146Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.1007208Z method(*args, **kwargs) 2025-12-04T11:13:50.1007500Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T11:13:50.1007560Z with policy(): 2025-12-04T11:13:50.1007853Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T11:13:50.1007923Z raise RuntimeError(msg) 2025-12-04T11:13:50.1008871Z RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_1_in_features_144_out_features_64_cuda_bfloat16! Caching allocator allocated memory was 9216 and is now reported as 18432 on device 0. CUDA driver allocated memory was 274726912 and is now 276824064. 2025-12-04T11:13:50.1008876Z 2025-12-04T11:13:50.1009009Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.1009530Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_1_in_features_144_out_features_64_cuda_bfloat16 2025-12-04T11:13:50.1009534Z 2025-12-04T11:13:50.1009693Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.1009821Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.1009912Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.1010466Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('benchmarking.InductorBenchmarker.benchmark_gpu', 2), ('pad_mm_bench', 1), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.1010594Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.1010652Z graph_break [] 2025-12-04T11:13:50.1010782Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.1010872Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.1010998Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.1011534Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('benchmarking.InductorBenchmarker.benchmark_gpu', 2), ('pad_mm_bench', 1), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.1011596Z graph_break [] 2025-12-04T11:13:50.1011688Z =================================== FAILURES =================================== 2025-12-04T11:13:50.1011988Z _ TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_1_in_features_144_out_features_64_cuda_bfloat16 _ 2025-12-04T11:13:50.1012067Z Traceback (most recent call last): 2025-12-04T11:13:50.1012368Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.1012432Z method(*args, **kwargs) 2025-12-04T11:13:50.1012728Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.1012791Z method(*args, **kwargs) 2025-12-04T11:13:50.1013081Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T11:13:50.1013148Z with policy(): 2025-12-04T11:13:50.1013441Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T11:13:50.1013516Z raise RuntimeError(msg) 2025-12-04T11:13:50.1014322Z RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_1_in_features_144_out_features_64_cuda_bfloat16! Caching allocator allocated memory was 18432 and is now reported as 27648 on device 0. CUDA driver allocated memory was 276824064 and is now 278921216. 2025-12-04T11:13:50.1014326Z 2025-12-04T11:13:50.1014450Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.1014974Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_1_in_features_144_out_features_64_cuda_bfloat16 2025-12-04T11:13:50.1014977Z 2025-12-04T11:13:50.1015131Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.1015339Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.1015510Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.1016060Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('benchmarking.InductorBenchmarker.benchmark_gpu', 2), ('pad_mm_bench', 1), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.1016185Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.1016244Z graph_break [] 2025-12-04T11:13:50.1016374Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.1016462Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.1016582Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.1017122Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('benchmarking.InductorBenchmarker.benchmark_gpu', 2), ('pad_mm_bench', 1), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.1017186Z graph_break [] 2025-12-04T11:13:50.1017321Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.1017417Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.1017538Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.1018076Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('benchmarking.InductorBenchmarker.benchmark_gpu', 2), ('pad_mm_bench', 1), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.1018135Z graph_break [] 2025-12-04T11:13:50.1018628Z - generated xml file: /var/lib/jenkins/workspace/test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-91ed99958e7f9cc9.xml - 2025-12-04T11:13:50.1018733Z =========================== short test summary info ============================ 2025-12-04T11:13:50.1020019Z FAILED [0.5437s] inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_17_mid_dim_1_in_features_144_out_features_64_cuda_bfloat16 - RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_1_in_features_144_out_features_64_cuda_bfloat16! Caching allocator allocated memory was 18432 and is now reported as 27648 on device 0. CUDA driver allocated memory was 276824064 and is now 278921216. 2025-12-04T11:13:50.1020027Z 2025-12-04T11:13:50.1020152Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.1020670Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_1_in_features_144_out_features_64_cuda_bfloat16 2025-12-04T11:13:50.1020677Z 2025-12-04T11:13:50.1020838Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.1020944Z !!!!!!!!!!!!!!!!!!!!!!!!!! stopping after 1 failures !!!!!!!!!!!!!!!!!!!!!!!!!!! 2025-12-04T11:13:50.1021063Z ================== 1 failed, 11 deselected, 2 rerun in 3.11s =================== 2025-12-04T11:13:50.1021123Z Got exit code 1 2025-12-04T11:13:50.1021189Z Retrying single test... 2025-12-04T11:13:50.1021457Z W1204 10:36:28.824000 62868 site-packages/torch/_inductor/utils.py:1703] Not enough SMs to use max_autotune_gemm mode 2025-12-04T11:13:50.1021841Z Test results will be stored in test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-f4094aba19e671d4.xml 2025-12-04T11:13:50.1021938Z ============================= test session starts ============================== 2025-12-04T11:13:50.1022153Z platform linux -- Python 3.10.14, pytest-7.3.2, pluggy-1.6.0 -- /opt/conda/envs/py_3.10/bin/python 2025-12-04T11:13:50.1022220Z cachedir: .pytest_cache 2025-12-04T11:13:50.1022681Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow] 2025-12-04T11:13:50.1022761Z rootdir: /var/lib/jenkins/workspace 2025-12-04T11:13:50.1022828Z configfile: pytest.ini 2025-12-04T11:13:50.1023149Z plugins: hypothesis-6.56.4, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-14.0, subtests-0.13.1, xdist-3.3.1, xdoctest-1.3.0, typeguard-4.3.0 2025-12-04T11:13:50.1023278Z collecting ... collected 58 items / 57 deselected / 1 selected 2025-12-04T11:13:50.1023902Z stepcurrent: skipping 11 already run items. Running only test/inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_17_mid_dim_1_in_features_144_out_features_64_cuda_bfloat16 2025-12-04T11:13:50.1023974Z Running 1 items in this shard 2025-12-04T11:13:50.1023978Z 2025-12-04T11:13:50.1024706Z inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_17_mid_dim_1_in_features_144_out_features_64_cuda_bfloat16 [W1204 10:36:30.447870230 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1024716Z 2025-12-04T11:13:50.1025012Z [W1204 10:36:39.464274767 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1025016Z 2025-12-04T11:13:50.1025304Z [W1204 10:36:39.464511422 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1025308Z 2025-12-04T11:13:50.1025599Z [W1204 10:36:39.470295480 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1025603Z 2025-12-04T11:13:50.1025888Z [W1204 10:36:39.470875023 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1025894Z 2025-12-04T11:13:50.1026187Z [W1204 10:36:39.471055917 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1026193Z 2025-12-04T11:13:50.1026481Z [W1204 10:36:39.476261162 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1026484Z 2025-12-04T11:13:50.1026777Z [W1204 10:36:39.476783814 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1026780Z 2025-12-04T11:13:50.1027066Z [W1204 10:36:39.476940767 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1027069Z 2025-12-04T11:13:50.1027152Z ('RERUN', {'yellow': True}) [10.9839s] [100%] 2025-12-04T11:13:50.1027879Z inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_17_mid_dim_1_in_features_144_out_features_64_cuda_bfloat16 [W1204 10:36:40.284599875 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1027886Z 2025-12-04T11:13:50.1028172Z [W1204 10:36:40.285159438 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1028176Z 2025-12-04T11:13:50.1028466Z [W1204 10:36:40.285298301 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1028469Z 2025-12-04T11:13:50.1028766Z [W1204 10:36:40.288172685 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1028769Z 2025-12-04T11:13:50.1029066Z [W1204 10:36:40.288633395 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1029069Z 2025-12-04T11:13:50.1029433Z [W1204 10:36:40.288769518 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1029504Z 2025-12-04T11:13:50.1029798Z [W1204 10:36:40.293231426 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1029802Z 2025-12-04T11:13:50.1030089Z [W1204 10:36:40.293689536 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1030092Z 2025-12-04T11:13:50.1030386Z [W1204 10:36:40.293824550 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1030389Z 2025-12-04T11:13:50.1030472Z ('RERUN', {'yellow': True}) [0.4979s] [100%] 2025-12-04T11:13:50.1031188Z inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_17_mid_dim_1_in_features_144_out_features_64_cuda_bfloat16 [W1204 10:36:40.779463391 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1031202Z 2025-12-04T11:13:50.1031491Z [W1204 10:36:40.780037644 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1031495Z 2025-12-04T11:13:50.1031782Z [W1204 10:36:40.780181057 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1031785Z 2025-12-04T11:13:50.1032076Z [W1204 10:36:40.783072301 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1032079Z 2025-12-04T11:13:50.1032368Z [W1204 10:36:40.783520551 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1032371Z 2025-12-04T11:13:50.1032661Z [W1204 10:36:40.783660574 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1032666Z 2025-12-04T11:13:50.1032958Z [W1204 10:36:40.788074952 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1032961Z 2025-12-04T11:13:50.1033255Z [W1204 10:36:40.788527812 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1033258Z 2025-12-04T11:13:50.1033547Z [W1204 10:36:40.788674385 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1033550Z 2025-12-04T11:13:50.1033614Z FAILED [0.4954s] [100%] 2025-12-04T11:13:50.1033622Z 2025-12-04T11:13:50.1033708Z ==================================== RERUNS ==================================== 2025-12-04T11:13:50.1034003Z _ TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_1_in_features_144_out_features_64_cuda_bfloat16 _ 2025-12-04T11:13:50.1034087Z Traceback (most recent call last): 2025-12-04T11:13:50.1034396Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.1034462Z method(*args, **kwargs) 2025-12-04T11:13:50.1034764Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.1034829Z method(*args, **kwargs) 2025-12-04T11:13:50.1035125Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T11:13:50.1035186Z with policy(): 2025-12-04T11:13:50.1035480Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T11:13:50.1035549Z raise RuntimeError(msg) 2025-12-04T11:13:50.1036420Z RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_1_in_features_144_out_features_64_cuda_bfloat16! Caching allocator allocated memory was 0 and is now reported as 9216 on device 0. CUDA driver allocated memory was 230686720 and is now 274726912. 2025-12-04T11:13:50.1036488Z 2025-12-04T11:13:50.1036637Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.1037163Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_1_in_features_144_out_features_64_cuda_bfloat16 2025-12-04T11:13:50.1037166Z 2025-12-04T11:13:50.1037326Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.1037464Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.1037560Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.1038116Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('benchmarking.InductorBenchmarker.benchmark_gpu', 2), ('pad_mm_bench', 1), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.1038249Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.1038309Z graph_break [] 2025-12-04T11:13:50.1038440Z ----------------------------- Captured stderr call ----------------------------- 2025-12-04T11:13:50.1039126Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/constant_folding.py:256: UserWarning: Unsupported unwinding pattern: Address not in range (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/profiler/unwind/unwind.cpp:219.) 2025-12-04T11:13:50.1039201Z if out == self.unknown_value: 2025-12-04T11:13:50.1039490Z _ TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_1_in_features_144_out_features_64_cuda_bfloat16 _ 2025-12-04T11:13:50.1039567Z Traceback (most recent call last): 2025-12-04T11:13:50.1039873Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.1039937Z method(*args, **kwargs) 2025-12-04T11:13:50.1040231Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.1040294Z method(*args, **kwargs) 2025-12-04T11:13:50.1040581Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T11:13:50.1040644Z with policy(): 2025-12-04T11:13:50.1040936Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T11:13:50.1041001Z raise RuntimeError(msg) 2025-12-04T11:13:50.1041813Z RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_1_in_features_144_out_features_64_cuda_bfloat16! Caching allocator allocated memory was 9216 and is now reported as 18432 on device 0. CUDA driver allocated memory was 274726912 and is now 276824064. 2025-12-04T11:13:50.1041819Z 2025-12-04T11:13:50.1041945Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.1042465Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_1_in_features_144_out_features_64_cuda_bfloat16 2025-12-04T11:13:50.1042469Z 2025-12-04T11:13:50.1042625Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.1042756Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.1042850Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.1043392Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('benchmarking.InductorBenchmarker.benchmark_gpu', 2), ('pad_mm_bench', 1), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.1043759Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.1043823Z graph_break [] 2025-12-04T11:13:50.1043948Z ----------------------------- Captured stderr call ----------------------------- 2025-12-04T11:13:50.1044637Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/constant_folding.py:256: UserWarning: Unsupported unwinding pattern: Address not in range (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/profiler/unwind/unwind.cpp:219.) 2025-12-04T11:13:50.1044706Z if out == self.unknown_value: 2025-12-04T11:13:50.1044834Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.1044924Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.1045051Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.1045601Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('benchmarking.InductorBenchmarker.benchmark_gpu', 2), ('pad_mm_bench', 1), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.1045662Z graph_break [] 2025-12-04T11:13:50.1045751Z =================================== FAILURES =================================== 2025-12-04T11:13:50.1046040Z _ TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_1_in_features_144_out_features_64_cuda_bfloat16 _ 2025-12-04T11:13:50.1046112Z Traceback (most recent call last): 2025-12-04T11:13:50.1046414Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.1046479Z method(*args, **kwargs) 2025-12-04T11:13:50.1046773Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.1046839Z method(*args, **kwargs) 2025-12-04T11:13:50.1047129Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T11:13:50.1047196Z with policy(): 2025-12-04T11:13:50.1047489Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T11:13:50.1047558Z raise RuntimeError(msg) 2025-12-04T11:13:50.1048367Z RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_1_in_features_144_out_features_64_cuda_bfloat16! Caching allocator allocated memory was 18432 and is now reported as 27648 on device 0. CUDA driver allocated memory was 276824064 and is now 278921216. 2025-12-04T11:13:50.1048371Z 2025-12-04T11:13:50.1048498Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.1049024Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_1_in_features_144_out_features_64_cuda_bfloat16 2025-12-04T11:13:50.1049033Z 2025-12-04T11:13:50.1049187Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.1049319Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.1049412Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.1049951Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('benchmarking.InductorBenchmarker.benchmark_gpu', 2), ('pad_mm_bench', 1), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.1050082Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.1050142Z graph_break [] 2025-12-04T11:13:50.1050273Z ----------------------------- Captured stderr call ----------------------------- 2025-12-04T11:13:50.1051100Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/constant_folding.py:256: UserWarning: Unsupported unwinding pattern: Address not in range (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/profiler/unwind/unwind.cpp:219.) 2025-12-04T11:13:50.1051173Z if out == self.unknown_value: 2025-12-04T11:13:50.1051302Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.1051393Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.1051516Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.1052059Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('benchmarking.InductorBenchmarker.benchmark_gpu', 2), ('pad_mm_bench', 1), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.1052118Z graph_break [] 2025-12-04T11:13:50.1052254Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.1052353Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.1052478Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.1053017Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('benchmarking.InductorBenchmarker.benchmark_gpu', 2), ('pad_mm_bench', 1), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.1053076Z graph_break [] 2025-12-04T11:13:50.1053565Z - generated xml file: /var/lib/jenkins/workspace/test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-f4094aba19e671d4.xml - 2025-12-04T11:13:50.1053667Z =========================== short test summary info ============================ 2025-12-04T11:13:50.1054957Z FAILED [0.4954s] inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_17_mid_dim_1_in_features_144_out_features_64_cuda_bfloat16 - RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_1_in_features_144_out_features_64_cuda_bfloat16! Caching allocator allocated memory was 18432 and is now reported as 27648 on device 0. CUDA driver allocated memory was 276824064 and is now 278921216. 2025-12-04T11:13:50.1054963Z 2025-12-04T11:13:50.1055088Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.1055603Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_1_in_features_144_out_features_64_cuda_bfloat16 2025-12-04T11:13:50.1055611Z 2025-12-04T11:13:50.1055768Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.1055872Z !!!!!!!!!!!!!!!!!!!!!!!!!! stopping after 1 failures !!!!!!!!!!!!!!!!!!!!!!!!!!! 2025-12-04T11:13:50.1055990Z ================== 1 failed, 57 deselected, 2 rerun in 12.00s ================== 2025-12-04T11:13:50.1056054Z Got exit code 1 2025-12-04T11:13:50.1056121Z Retrying single test... 2025-12-04T11:13:50.1056393Z W1204 10:36:47.608000 63055 site-packages/torch/_inductor/utils.py:1703] Not enough SMs to use max_autotune_gemm mode 2025-12-04T11:13:50.1056777Z Test results will be stored in test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-204edfd0c56893dd.xml 2025-12-04T11:13:50.1056878Z ============================= test session starts ============================== 2025-12-04T11:13:50.1057089Z platform linux -- Python 3.10.14, pytest-7.3.2, pluggy-1.6.0 -- /opt/conda/envs/py_3.10/bin/python 2025-12-04T11:13:50.1057156Z cachedir: .pytest_cache 2025-12-04T11:13:50.1057466Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow] 2025-12-04T11:13:50.1057542Z rootdir: /var/lib/jenkins/workspace 2025-12-04T11:13:50.1057685Z configfile: pytest.ini 2025-12-04T11:13:50.1058067Z plugins: hypothesis-6.56.4, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-14.0, subtests-0.13.1, xdist-3.3.1, xdoctest-1.3.0, typeguard-4.3.0 2025-12-04T11:13:50.1058198Z collecting ... collected 58 items / 57 deselected / 1 selected 2025-12-04T11:13:50.1058769Z stepcurrent: skipping 11 already run items. Running only test/inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_17_mid_dim_1_in_features_144_out_features_64_cuda_bfloat16 2025-12-04T11:13:50.1058841Z Running 1 items in this shard 2025-12-04T11:13:50.1058844Z 2025-12-04T11:13:50.1059576Z inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_17_mid_dim_1_in_features_144_out_features_64_cuda_bfloat16 [W1204 10:36:49.239070214 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1059585Z 2025-12-04T11:13:50.1059885Z [W1204 10:36:58.474596302 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1059891Z 2025-12-04T11:13:50.1060185Z [W1204 10:36:58.474838087 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1060189Z 2025-12-04T11:13:50.1060483Z [W1204 10:36:58.481200408 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1060487Z 2025-12-04T11:13:50.1060772Z [W1204 10:36:58.481766411 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1060775Z 2025-12-04T11:13:50.1061065Z [W1204 10:36:58.481937374 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1061069Z 2025-12-04T11:13:50.1061354Z [W1204 10:36:58.487206920 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1061360Z 2025-12-04T11:13:50.1061653Z [W1204 10:36:58.487720542 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1061657Z 2025-12-04T11:13:50.1061944Z [W1204 10:36:58.487878955 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1061947Z 2025-12-04T11:13:50.1062031Z ('RERUN', {'yellow': True}) [11.2103s] [100%] 2025-12-04T11:13:50.1062748Z inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_17_mid_dim_1_in_features_144_out_features_64_cuda_bfloat16 [W1204 10:36:59.298109513 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1062752Z 2025-12-04T11:13:50.1063043Z [W1204 10:36:59.298668125 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1063055Z 2025-12-04T11:13:50.1063343Z [W1204 10:36:59.298811518 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1063346Z 2025-12-04T11:13:50.1063718Z [W1204 10:36:59.301756773 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1063722Z 2025-12-04T11:13:50.1064010Z [W1204 10:36:59.302220233 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1064014Z 2025-12-04T11:13:50.1064299Z [W1204 10:36:59.302355006 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1064302Z 2025-12-04T11:13:50.1064594Z [W1204 10:36:59.306836525 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1064674Z 2025-12-04T11:13:50.1065024Z [W1204 10:36:59.307295375 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1065028Z 2025-12-04T11:13:50.1065323Z [W1204 10:36:59.307429488 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1065327Z 2025-12-04T11:13:50.1065404Z ('RERUN', {'yellow': True}) [0.5075s] [100%] 2025-12-04T11:13:50.1066123Z inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_17_mid_dim_1_in_features_144_out_features_64_cuda_bfloat16 [W1204 10:36:59.803205485 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1066132Z 2025-12-04T11:13:50.1066423Z [W1204 10:36:59.803748237 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1066429Z 2025-12-04T11:13:50.1066718Z [W1204 10:36:59.803884050 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1066726Z 2025-12-04T11:13:50.1067010Z [W1204 10:36:59.806779654 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1067013Z 2025-12-04T11:13:50.1067298Z [W1204 10:36:59.807232934 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1067301Z 2025-12-04T11:13:50.1067591Z [W1204 10:36:59.807366407 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1067595Z 2025-12-04T11:13:50.1067882Z [W1204 10:36:59.811931268 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1067888Z 2025-12-04T11:13:50.1068185Z [W1204 10:36:59.812397839 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1068188Z 2025-12-04T11:13:50.1068472Z [W1204 10:36:59.812530241 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1068475Z 2025-12-04T11:13:50.1068541Z FAILED [0.5033s] [100%] 2025-12-04T11:13:50.1068545Z 2025-12-04T11:13:50.1068630Z ==================================== RERUNS ==================================== 2025-12-04T11:13:50.1068923Z _ TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_1_in_features_144_out_features_64_cuda_bfloat16 _ 2025-12-04T11:13:50.1069003Z Traceback (most recent call last): 2025-12-04T11:13:50.1069318Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.1069393Z method(*args, **kwargs) 2025-12-04T11:13:50.1069685Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.1069749Z method(*args, **kwargs) 2025-12-04T11:13:50.1070043Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T11:13:50.1070105Z with policy(): 2025-12-04T11:13:50.1070396Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T11:13:50.1070471Z raise RuntimeError(msg) 2025-12-04T11:13:50.1071270Z RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_1_in_features_144_out_features_64_cuda_bfloat16! Caching allocator allocated memory was 0 and is now reported as 9216 on device 0. CUDA driver allocated memory was 230686720 and is now 274726912. 2025-12-04T11:13:50.1071366Z 2025-12-04T11:13:50.1071505Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.1072091Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_1_in_features_144_out_features_64_cuda_bfloat16 2025-12-04T11:13:50.1072095Z 2025-12-04T11:13:50.1072259Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.1072388Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.1072482Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.1073028Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('benchmarking.InductorBenchmarker.benchmark_gpu', 2), ('pad_mm_bench', 1), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.1073159Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.1073219Z graph_break [] 2025-12-04T11:13:50.1073353Z ----------------------------- Captured stderr call ----------------------------- 2025-12-04T11:13:50.1074036Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/constant_folding.py:256: UserWarning: Unsupported unwinding pattern: Address not in range (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/profiler/unwind/unwind.cpp:219.) 2025-12-04T11:13:50.1074113Z if out == self.unknown_value: 2025-12-04T11:13:50.1074403Z _ TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_1_in_features_144_out_features_64_cuda_bfloat16 _ 2025-12-04T11:13:50.1074476Z Traceback (most recent call last): 2025-12-04T11:13:50.1074778Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.1074842Z method(*args, **kwargs) 2025-12-04T11:13:50.1075141Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.1075208Z method(*args, **kwargs) 2025-12-04T11:13:50.1075498Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T11:13:50.1075565Z with policy(): 2025-12-04T11:13:50.1075857Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T11:13:50.1075922Z raise RuntimeError(msg) 2025-12-04T11:13:50.1076732Z RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_1_in_features_144_out_features_64_cuda_bfloat16! Caching allocator allocated memory was 9216 and is now reported as 18432 on device 0. CUDA driver allocated memory was 274726912 and is now 276824064. 2025-12-04T11:13:50.1076736Z 2025-12-04T11:13:50.1076866Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.1077390Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_1_in_features_144_out_features_64_cuda_bfloat16 2025-12-04T11:13:50.1077393Z 2025-12-04T11:13:50.1077561Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.1077694Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.1077975Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.1078631Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('benchmarking.InductorBenchmarker.benchmark_gpu', 2), ('pad_mm_bench', 1), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.1078773Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.1078962Z graph_break [] 2025-12-04T11:13:50.1079096Z ----------------------------- Captured stderr call ----------------------------- 2025-12-04T11:13:50.1079879Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/constant_folding.py:256: UserWarning: Unsupported unwinding pattern: Address not in range (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/profiler/unwind/unwind.cpp:219.) 2025-12-04T11:13:50.1079955Z if out == self.unknown_value: 2025-12-04T11:13:50.1080085Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.1080181Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.1080313Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.1080858Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('benchmarking.InductorBenchmarker.benchmark_gpu', 2), ('pad_mm_bench', 1), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.1080922Z graph_break [] 2025-12-04T11:13:50.1081013Z =================================== FAILURES =================================== 2025-12-04T11:13:50.1081313Z _ TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_1_in_features_144_out_features_64_cuda_bfloat16 _ 2025-12-04T11:13:50.1081389Z Traceback (most recent call last): 2025-12-04T11:13:50.1081695Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.1081762Z method(*args, **kwargs) 2025-12-04T11:13:50.1082059Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.1082124Z method(*args, **kwargs) 2025-12-04T11:13:50.1082416Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T11:13:50.1082482Z with policy(): 2025-12-04T11:13:50.1082779Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T11:13:50.1082852Z raise RuntimeError(msg) 2025-12-04T11:13:50.1083662Z RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_1_in_features_144_out_features_64_cuda_bfloat16! Caching allocator allocated memory was 18432 and is now reported as 27648 on device 0. CUDA driver allocated memory was 276824064 and is now 278921216. 2025-12-04T11:13:50.1083666Z 2025-12-04T11:13:50.1083795Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.1084317Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_1_in_features_144_out_features_64_cuda_bfloat16 2025-12-04T11:13:50.1084321Z 2025-12-04T11:13:50.1084496Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.1084633Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.1084730Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.1085272Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('benchmarking.InductorBenchmarker.benchmark_gpu', 2), ('pad_mm_bench', 1), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.1085404Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.1085463Z graph_break [] 2025-12-04T11:13:50.1085590Z ----------------------------- Captured stderr call ----------------------------- 2025-12-04T11:13:50.1086277Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/constant_folding.py:256: UserWarning: Unsupported unwinding pattern: Address not in range (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/profiler/unwind/unwind.cpp:219.) 2025-12-04T11:13:50.1086427Z if out == self.unknown_value: 2025-12-04T11:13:50.1086620Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.1086714Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.1086841Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.1087379Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('benchmarking.InductorBenchmarker.benchmark_gpu', 2), ('pad_mm_bench', 1), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.1087440Z graph_break [] 2025-12-04T11:13:50.1087566Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.1087658Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.1087780Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.1088324Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('benchmarking.InductorBenchmarker.benchmark_gpu', 2), ('pad_mm_bench', 1), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.1088386Z graph_break [] 2025-12-04T11:13:50.1088879Z - generated xml file: /var/lib/jenkins/workspace/test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-204edfd0c56893dd.xml - 2025-12-04T11:13:50.1088980Z =========================== short test summary info ============================ 2025-12-04T11:13:50.1090294Z FAILED [0.5033s] inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_17_mid_dim_1_in_features_144_out_features_64_cuda_bfloat16 - RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_1_in_features_144_out_features_64_cuda_bfloat16! Caching allocator allocated memory was 18432 and is now reported as 27648 on device 0. CUDA driver allocated memory was 276824064 and is now 278921216. 2025-12-04T11:13:50.1090301Z 2025-12-04T11:13:50.1090433Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.1090954Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_1_in_features_144_out_features_64_cuda_bfloat16 2025-12-04T11:13:50.1090957Z 2025-12-04T11:13:50.1091114Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.1091218Z !!!!!!!!!!!!!!!!!!!!!!!!!! stopping after 1 failures !!!!!!!!!!!!!!!!!!!!!!!!!!! 2025-12-04T11:13:50.1091339Z ================== 1 failed, 57 deselected, 2 rerun in 12.25s ================== 2025-12-04T11:13:50.1091400Z Got exit code 1 2025-12-04T11:13:50.1091876Z FAILED CONSISTENTLY: test/inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_17_mid_dim_1_in_features_144_out_features_64_cuda_bfloat16 2025-12-04T11:13:50.1092133Z Test failed consistently, continuing with the rest of the tests due to continue-through-error being set 2025-12-04T11:13:50.1092395Z W1204 10:37:06.666000 63242 site-packages/torch/_inductor/utils.py:1703] Not enough SMs to use max_autotune_gemm mode 2025-12-04T11:13:50.1092784Z Test results will be stored in test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-5d3545fec05b298e.xml 2025-12-04T11:13:50.1092880Z ============================= test session starts ============================== 2025-12-04T11:13:50.1093088Z platform linux -- Python 3.10.14, pytest-7.3.2, pluggy-1.6.0 -- /opt/conda/envs/py_3.10/bin/python 2025-12-04T11:13:50.1093160Z cachedir: .pytest_cache 2025-12-04T11:13:50.1093463Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow] 2025-12-04T11:13:50.1093545Z rootdir: /var/lib/jenkins/workspace 2025-12-04T11:13:50.1093686Z configfile: pytest.ini 2025-12-04T11:13:50.1094085Z plugins: hypothesis-6.56.4, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-14.0, subtests-0.13.1, xdist-3.3.1, xdoctest-1.3.0, typeguard-4.3.0 2025-12-04T11:13:50.1094220Z collecting ... collected 58 items / 12 deselected / 46 selected 2025-12-04T11:13:50.1094308Z stepcurrent: skipping 12 already run items. 2025-12-04T11:13:50.1094378Z Running 46 items in this shard 2025-12-04T11:13:50.1094385Z 2025-12-04T11:13:50.1094886Z inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_17_mid_dim_1_in_features_144_out_features_65_cuda_bfloat16 ('RERUN', {'yellow': True}) [1.9850s] [ 2%] 2025-12-04T11:13:50.1095371Z inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_17_mid_dim_1_in_features_144_out_features_65_cuda_bfloat16 ('RERUN', {'yellow': True}) [0.5500s] [ 2%] 2025-12-04T11:13:50.1095816Z inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_17_mid_dim_1_in_features_144_out_features_65_cuda_bfloat16 FAILED [0.5289s] [ 2%] 2025-12-04T11:13:50.1095823Z 2025-12-04T11:13:50.1095911Z ==================================== RERUNS ==================================== 2025-12-04T11:13:50.1096269Z _ TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_1_in_features_144_out_features_65_cuda_bfloat16 _ 2025-12-04T11:13:50.1096348Z Traceback (most recent call last): 2025-12-04T11:13:50.1096652Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.1096723Z method(*args, **kwargs) 2025-12-04T11:13:50.1097014Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.1097083Z method(*args, **kwargs) 2025-12-04T11:13:50.1097383Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T11:13:50.1097447Z with policy(): 2025-12-04T11:13:50.1097762Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T11:13:50.1097832Z raise RuntimeError(msg) 2025-12-04T11:13:50.1098632Z RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_1_in_features_144_out_features_65_cuda_bfloat16! Caching allocator allocated memory was 0 and is now reported as 9728 on device 0. CUDA driver allocated memory was 230686720 and is now 274726912. 2025-12-04T11:13:50.1098639Z 2025-12-04T11:13:50.1098764Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.1099288Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_1_in_features_144_out_features_65_cuda_bfloat16 2025-12-04T11:13:50.1099296Z 2025-12-04T11:13:50.1099460Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.1099592Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.1099691Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.1100236Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('benchmarking.InductorBenchmarker.benchmark_gpu', 2), ('pad_mm_bench', 1), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.1100363Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.1100429Z graph_break [] 2025-12-04T11:13:50.1100718Z _ TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_1_in_features_144_out_features_65_cuda_bfloat16 _ 2025-12-04T11:13:50.1100798Z Traceback (most recent call last): 2025-12-04T11:13:50.1101096Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.1101237Z method(*args, **kwargs) 2025-12-04T11:13:50.1101597Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.1101670Z method(*args, **kwargs) 2025-12-04T11:13:50.1101956Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T11:13:50.1102022Z with policy(): 2025-12-04T11:13:50.1102314Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T11:13:50.1102385Z raise RuntimeError(msg) 2025-12-04T11:13:50.1103189Z RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_1_in_features_144_out_features_65_cuda_bfloat16! Caching allocator allocated memory was 9728 and is now reported as 19456 on device 0. CUDA driver allocated memory was 274726912 and is now 276824064. 2025-12-04T11:13:50.1103197Z 2025-12-04T11:13:50.1103325Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.1103888Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_1_in_features_144_out_features_65_cuda_bfloat16 2025-12-04T11:13:50.1103892Z 2025-12-04T11:13:50.1104047Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.1104180Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.1104271Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.1104810Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('benchmarking.InductorBenchmarker.benchmark_gpu', 2), ('pad_mm_bench', 1), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.1104942Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.1105003Z graph_break [] 2025-12-04T11:13:50.1105132Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.1105220Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.1105341Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.1105884Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('benchmarking.InductorBenchmarker.benchmark_gpu', 2), ('pad_mm_bench', 1), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.1105945Z graph_break [] 2025-12-04T11:13:50.1106033Z =================================== FAILURES =================================== 2025-12-04T11:13:50.1106321Z _ TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_1_in_features_144_out_features_65_cuda_bfloat16 _ 2025-12-04T11:13:50.1106397Z Traceback (most recent call last): 2025-12-04T11:13:50.1106702Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.1106767Z method(*args, **kwargs) 2025-12-04T11:13:50.1107058Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.1107125Z method(*args, **kwargs) 2025-12-04T11:13:50.1107424Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T11:13:50.1107488Z with policy(): 2025-12-04T11:13:50.1107778Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T11:13:50.1107845Z raise RuntimeError(msg) 2025-12-04T11:13:50.1108731Z RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_1_in_features_144_out_features_65_cuda_bfloat16! Caching allocator allocated memory was 19456 and is now reported as 29184 on device 0. CUDA driver allocated memory was 276824064 and is now 278921216. 2025-12-04T11:13:50.1108799Z 2025-12-04T11:13:50.1108924Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.1109441Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_1_in_features_144_out_features_65_cuda_bfloat16 2025-12-04T11:13:50.1109446Z 2025-12-04T11:13:50.1109599Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.1109731Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.1109821Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.1110361Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('benchmarking.InductorBenchmarker.benchmark_gpu', 2), ('pad_mm_bench', 1), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.1110491Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.1110550Z graph_break [] 2025-12-04T11:13:50.1110675Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.1110770Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.1110891Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.1111432Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('benchmarking.InductorBenchmarker.benchmark_gpu', 2), ('pad_mm_bench', 1), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.1111490Z graph_break [] 2025-12-04T11:13:50.1111614Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.1111710Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.1111835Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.1112374Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('benchmarking.InductorBenchmarker.benchmark_gpu', 2), ('pad_mm_bench', 1), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.1112435Z graph_break [] 2025-12-04T11:13:50.1112926Z - generated xml file: /var/lib/jenkins/workspace/test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-5d3545fec05b298e.xml - 2025-12-04T11:13:50.1113031Z =========================== short test summary info ============================ 2025-12-04T11:13:50.1114319Z FAILED [0.5289s] inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_17_mid_dim_1_in_features_144_out_features_65_cuda_bfloat16 - RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_1_in_features_144_out_features_65_cuda_bfloat16! Caching allocator allocated memory was 19456 and is now reported as 29184 on device 0. CUDA driver allocated memory was 276824064 and is now 278921216. 2025-12-04T11:13:50.1114327Z 2025-12-04T11:13:50.1114456Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.1114969Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_1_in_features_144_out_features_65_cuda_bfloat16 2025-12-04T11:13:50.1114972Z 2025-12-04T11:13:50.1115132Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.1115235Z !!!!!!!!!!!!!!!!!!!!!!!!!! stopping after 1 failures !!!!!!!!!!!!!!!!!!!!!!!!!!! 2025-12-04T11:13:50.1115351Z ================== 1 failed, 12 deselected, 2 rerun in 3.09s =================== 2025-12-04T11:13:50.1115499Z Got exit code 1 2025-12-04T11:13:50.1115567Z Retrying single test... 2025-12-04T11:13:50.1115896Z W1204 10:37:16.717000 63424 site-packages/torch/_inductor/utils.py:1703] Not enough SMs to use max_autotune_gemm mode 2025-12-04T11:13:50.1116288Z Test results will be stored in test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-c2388e46be16ba24.xml 2025-12-04T11:13:50.1116382Z ============================= test session starts ============================== 2025-12-04T11:13:50.1116593Z platform linux -- Python 3.10.14, pytest-7.3.2, pluggy-1.6.0 -- /opt/conda/envs/py_3.10/bin/python 2025-12-04T11:13:50.1116661Z cachedir: .pytest_cache 2025-12-04T11:13:50.1116968Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow] 2025-12-04T11:13:50.1117049Z rootdir: /var/lib/jenkins/workspace 2025-12-04T11:13:50.1117118Z configfile: pytest.ini 2025-12-04T11:13:50.1117442Z plugins: hypothesis-6.56.4, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-14.0, subtests-0.13.1, xdist-3.3.1, xdoctest-1.3.0, typeguard-4.3.0 2025-12-04T11:13:50.1117572Z collecting ... collected 58 items / 57 deselected / 1 selected 2025-12-04T11:13:50.1118135Z stepcurrent: skipping 12 already run items. Running only test/inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_17_mid_dim_1_in_features_144_out_features_65_cuda_bfloat16 2025-12-04T11:13:50.1118212Z Running 1 items in this shard 2025-12-04T11:13:50.1118216Z 2025-12-04T11:13:50.1118941Z inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_17_mid_dim_1_in_features_144_out_features_65_cuda_bfloat16 [W1204 10:37:18.355156113 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1118946Z 2025-12-04T11:13:50.1119253Z [W1204 10:37:27.653349005 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1119257Z 2025-12-04T11:13:50.1119556Z [W1204 10:37:27.653580730 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1119559Z 2025-12-04T11:13:50.1119855Z [W1204 10:37:27.659276035 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1119858Z 2025-12-04T11:13:50.1120144Z [W1204 10:37:27.659848428 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1120147Z 2025-12-04T11:13:50.1120438Z [W1204 10:37:27.660056202 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1120448Z 2025-12-04T11:13:50.1120733Z [W1204 10:37:27.665384159 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1120739Z 2025-12-04T11:13:50.1121024Z [W1204 10:37:27.665907741 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1121027Z 2025-12-04T11:13:50.1121322Z [W1204 10:37:27.666067894 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1121325Z 2025-12-04T11:13:50.1121407Z ('RERUN', {'yellow': True}) [11.2695s] [100%] 2025-12-04T11:13:50.1122132Z inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_17_mid_dim_1_in_features_144_out_features_65_cuda_bfloat16 [W1204 10:37:28.472818151 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1122136Z 2025-12-04T11:13:50.1122426Z [W1204 10:37:28.473385694 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1122503Z 2025-12-04T11:13:50.1122876Z [W1204 10:37:28.473530867 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1122879Z 2025-12-04T11:13:50.1123164Z [W1204 10:37:28.476448971 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1123167Z 2025-12-04T11:13:50.1123459Z [W1204 10:37:28.476922252 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1123462Z 2025-12-04T11:13:50.1123748Z [W1204 10:37:28.477059064 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1123751Z 2025-12-04T11:13:50.1124036Z [W1204 10:37:28.481667696 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1124055Z 2025-12-04T11:13:50.1124353Z [W1204 10:37:28.482142406 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1124357Z 2025-12-04T11:13:50.1124643Z [W1204 10:37:28.482277039 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1124646Z 2025-12-04T11:13:50.1124733Z ('RERUN', {'yellow': True}) [0.4983s] [100%] 2025-12-04T11:13:50.1125459Z inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_17_mid_dim_1_in_features_144_out_features_65_cuda_bfloat16 [W1204 10:37:29.968001540 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1125463Z 2025-12-04T11:13:50.1125753Z [W1204 10:37:29.968556322 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1125759Z 2025-12-04T11:13:50.1126047Z [W1204 10:37:29.968710835 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1126050Z 2025-12-04T11:13:50.1126344Z [W1204 10:37:29.971671880 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1126347Z 2025-12-04T11:13:50.1126633Z [W1204 10:37:29.972135260 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1126636Z 2025-12-04T11:13:50.1126928Z [W1204 10:37:29.972273424 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1126931Z 2025-12-04T11:13:50.1127214Z [W1204 10:37:29.976768713 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1127220Z 2025-12-04T11:13:50.1127510Z [W1204 10:37:29.977230193 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1127513Z 2025-12-04T11:13:50.1127807Z [W1204 10:37:29.977367216 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1127811Z 2025-12-04T11:13:50.1127874Z FAILED [0.4953s] [100%] 2025-12-04T11:13:50.1127877Z 2025-12-04T11:13:50.1127970Z ==================================== RERUNS ==================================== 2025-12-04T11:13:50.1128261Z _ TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_1_in_features_144_out_features_65_cuda_bfloat16 _ 2025-12-04T11:13:50.1128335Z Traceback (most recent call last): 2025-12-04T11:13:50.1128657Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.1128799Z method(*args, **kwargs) 2025-12-04T11:13:50.1129158Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.1129224Z method(*args, **kwargs) 2025-12-04T11:13:50.1129510Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T11:13:50.1129579Z with policy(): 2025-12-04T11:13:50.1129872Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T11:13:50.1129945Z raise RuntimeError(msg) 2025-12-04T11:13:50.1130739Z RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_1_in_features_144_out_features_65_cuda_bfloat16! Caching allocator allocated memory was 0 and is now reported as 9728 on device 0. CUDA driver allocated memory was 230686720 and is now 274726912. 2025-12-04T11:13:50.1130747Z 2025-12-04T11:13:50.1130879Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.1131400Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_1_in_features_144_out_features_65_cuda_bfloat16 2025-12-04T11:13:50.1131404Z 2025-12-04T11:13:50.1131559Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.1131693Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.1131788Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.1132336Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('benchmarking.InductorBenchmarker.benchmark_gpu', 2), ('pad_mm_bench', 1), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.1132473Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.1132534Z graph_break [] 2025-12-04T11:13:50.1132664Z ----------------------------- Captured stderr call ----------------------------- 2025-12-04T11:13:50.1133351Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/constant_folding.py:256: UserWarning: Unsupported unwinding pattern: Address not in range (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/profiler/unwind/unwind.cpp:219.) 2025-12-04T11:13:50.1133425Z if out == self.unknown_value: 2025-12-04T11:13:50.1133722Z _ TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_1_in_features_144_out_features_65_cuda_bfloat16 _ 2025-12-04T11:13:50.1133796Z Traceback (most recent call last): 2025-12-04T11:13:50.1134099Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.1134164Z method(*args, **kwargs) 2025-12-04T11:13:50.1134454Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.1134524Z method(*args, **kwargs) 2025-12-04T11:13:50.1134811Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T11:13:50.1134871Z with policy(): 2025-12-04T11:13:50.1135169Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T11:13:50.1135236Z raise RuntimeError(msg) 2025-12-04T11:13:50.1136044Z RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_1_in_features_144_out_features_65_cuda_bfloat16! Caching allocator allocated memory was 9728 and is now reported as 19456 on device 0. CUDA driver allocated memory was 274726912 and is now 276824064. 2025-12-04T11:13:50.1136048Z 2025-12-04T11:13:50.1136286Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.1136877Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_1_in_features_144_out_features_65_cuda_bfloat16 2025-12-04T11:13:50.1136882Z 2025-12-04T11:13:50.1137040Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.1137167Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.1137263Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.1137810Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('benchmarking.InductorBenchmarker.benchmark_gpu', 2), ('pad_mm_bench', 1), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.1137954Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.1138019Z graph_break [] 2025-12-04T11:13:50.1138143Z ----------------------------- Captured stderr call ----------------------------- 2025-12-04T11:13:50.1138836Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/constant_folding.py:256: UserWarning: Unsupported unwinding pattern: Address not in range (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/profiler/unwind/unwind.cpp:219.) 2025-12-04T11:13:50.1138907Z if out == self.unknown_value: 2025-12-04T11:13:50.1139033Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.1139137Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.1139260Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.1139801Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('benchmarking.InductorBenchmarker.benchmark_gpu', 2), ('pad_mm_bench', 1), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.1139865Z graph_break [] 2025-12-04T11:13:50.1139952Z =================================== FAILURES =================================== 2025-12-04T11:13:50.1140248Z _ TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_1_in_features_144_out_features_65_cuda_bfloat16 _ 2025-12-04T11:13:50.1140320Z Traceback (most recent call last): 2025-12-04T11:13:50.1140625Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.1140689Z method(*args, **kwargs) 2025-12-04T11:13:50.1140980Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.1141059Z method(*args, **kwargs) 2025-12-04T11:13:50.1141349Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T11:13:50.1141409Z with policy(): 2025-12-04T11:13:50.1141709Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T11:13:50.1141779Z raise RuntimeError(msg) 2025-12-04T11:13:50.1142590Z RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_1_in_features_144_out_features_65_cuda_bfloat16! Caching allocator allocated memory was 19456 and is now reported as 29184 on device 0. CUDA driver allocated memory was 276824064 and is now 278921216. 2025-12-04T11:13:50.1142594Z 2025-12-04T11:13:50.1142719Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.1143242Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_1_in_features_144_out_features_65_cuda_bfloat16 2025-12-04T11:13:50.1143246Z 2025-12-04T11:13:50.1143400Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.1143640Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.1143802Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.1144346Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('benchmarking.InductorBenchmarker.benchmark_gpu', 2), ('pad_mm_bench', 1), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.1144476Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.1144534Z graph_break [] 2025-12-04T11:13:50.1144659Z ----------------------------- Captured stderr call ----------------------------- 2025-12-04T11:13:50.1145350Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/constant_folding.py:256: UserWarning: Unsupported unwinding pattern: Address not in range (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/profiler/unwind/unwind.cpp:219.) 2025-12-04T11:13:50.1145425Z if out == self.unknown_value: 2025-12-04T11:13:50.1145555Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.1145647Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.1145783Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.1146328Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('benchmarking.InductorBenchmarker.benchmark_gpu', 2), ('pad_mm_bench', 1), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.1146389Z graph_break [] 2025-12-04T11:13:50.1146512Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.1146607Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.1146729Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.1147269Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('benchmarking.InductorBenchmarker.benchmark_gpu', 2), ('pad_mm_bench', 1), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.1147332Z graph_break [] 2025-12-04T11:13:50.1147816Z - generated xml file: /var/lib/jenkins/workspace/test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-c2388e46be16ba24.xml - 2025-12-04T11:13:50.1147922Z =========================== short test summary info ============================ 2025-12-04T11:13:50.1149206Z FAILED [0.4953s] inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_17_mid_dim_1_in_features_144_out_features_65_cuda_bfloat16 - RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_1_in_features_144_out_features_65_cuda_bfloat16! Caching allocator allocated memory was 19456 and is now reported as 29184 on device 0. CUDA driver allocated memory was 276824064 and is now 278921216. 2025-12-04T11:13:50.1149213Z 2025-12-04T11:13:50.1149345Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.1149865Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_1_in_features_144_out_features_65_cuda_bfloat16 2025-12-04T11:13:50.1149868Z 2025-12-04T11:13:50.1150026Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.1150131Z !!!!!!!!!!!!!!!!!!!!!!!!!! stopping after 1 failures !!!!!!!!!!!!!!!!!!!!!!!!!!! 2025-12-04T11:13:50.1150245Z ================== 1 failed, 57 deselected, 2 rerun in 12.29s ================== 2025-12-04T11:13:50.1150313Z Got exit code 1 2025-12-04T11:13:50.1150381Z Retrying single test... 2025-12-04T11:13:50.1150652Z W1204 10:37:35.800000 63611 site-packages/torch/_inductor/utils.py:1703] Not enough SMs to use max_autotune_gemm mode 2025-12-04T11:13:50.1151208Z Test results will be stored in test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-e9cf01ceaaab6f89.xml 2025-12-04T11:13:50.1151307Z ============================= test session starts ============================== 2025-12-04T11:13:50.1151518Z platform linux -- Python 3.10.14, pytest-7.3.2, pluggy-1.6.0 -- /opt/conda/envs/py_3.10/bin/python 2025-12-04T11:13:50.1151586Z cachedir: .pytest_cache 2025-12-04T11:13:50.1151888Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow] 2025-12-04T11:13:50.1151970Z rootdir: /var/lib/jenkins/workspace 2025-12-04T11:13:50.1152038Z configfile: pytest.ini 2025-12-04T11:13:50.1152357Z plugins: hypothesis-6.56.4, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-14.0, subtests-0.13.1, xdist-3.3.1, xdoctest-1.3.0, typeguard-4.3.0 2025-12-04T11:13:50.1152487Z collecting ... collected 58 items / 57 deselected / 1 selected 2025-12-04T11:13:50.1153058Z stepcurrent: skipping 12 already run items. Running only test/inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_17_mid_dim_1_in_features_144_out_features_65_cuda_bfloat16 2025-12-04T11:13:50.1153136Z Running 1 items in this shard 2025-12-04T11:13:50.1153140Z 2025-12-04T11:13:50.1153864Z inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_17_mid_dim_1_in_features_144_out_features_65_cuda_bfloat16 [W1204 10:37:37.451977859 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1153868Z 2025-12-04T11:13:50.1154170Z [W1204 10:37:46.724921630 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1154173Z 2025-12-04T11:13:50.1154464Z [W1204 10:37:46.725171075 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1154471Z 2025-12-04T11:13:50.1154763Z [W1204 10:37:46.731059985 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1154767Z 2025-12-04T11:13:50.1155055Z [W1204 10:37:46.731639108 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1155058Z 2025-12-04T11:13:50.1155349Z [W1204 10:37:46.731813852 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1155353Z 2025-12-04T11:13:50.1155642Z [W1204 10:37:46.737178490 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1155646Z 2025-12-04T11:13:50.1155930Z [W1204 10:37:46.737697341 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1155940Z 2025-12-04T11:13:50.1156228Z [W1204 10:37:46.737852164 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1156231Z 2025-12-04T11:13:50.1156311Z ('RERUN', {'yellow': True}) [11.2567s] [100%] 2025-12-04T11:13:50.1157043Z inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_17_mid_dim_1_in_features_144_out_features_65_cuda_bfloat16 [W1204 10:37:47.548127577 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1157048Z 2025-12-04T11:13:50.1157346Z [W1204 10:37:47.548709489 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1157349Z 2025-12-04T11:13:50.1157648Z [W1204 10:37:47.548858313 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1157726Z 2025-12-04T11:13:50.1158162Z [W1204 10:37:47.552013662 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1158165Z 2025-12-04T11:13:50.1158456Z [W1204 10:37:47.552492483 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1158460Z 2025-12-04T11:13:50.1158746Z [W1204 10:37:47.552638166 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1158749Z 2025-12-04T11:13:50.1159041Z [W1204 10:37:47.557543034 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1159044Z 2025-12-04T11:13:50.1159333Z [W1204 10:37:47.558019815 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1159340Z 2025-12-04T11:13:50.1159628Z [W1204 10:37:47.558155788 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1163329Z 2025-12-04T11:13:50.1163459Z ('RERUN', {'yellow': True}) [0.5032s] [100%] 2025-12-04T11:13:50.1164226Z inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_17_mid_dim_1_in_features_144_out_features_65_cuda_bfloat16 [W1204 10:37:48.049479562 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1164231Z 2025-12-04T11:13:50.1164550Z [W1204 10:37:48.050072465 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1164554Z 2025-12-04T11:13:50.1164847Z [W1204 10:37:48.050219668 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1164856Z 2025-12-04T11:13:50.1165151Z [W1204 10:37:48.053402578 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1165179Z 2025-12-04T11:13:50.1165468Z [W1204 10:37:48.053875478 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1165472Z 2025-12-04T11:13:50.1165758Z [W1204 10:37:48.054013181 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1165762Z 2025-12-04T11:13:50.1166053Z [W1204 10:37:48.058905499 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1166057Z 2025-12-04T11:13:50.1166342Z [W1204 10:37:48.059384609 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1166347Z 2025-12-04T11:13:50.1166645Z [W1204 10:37:48.059522352 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1166649Z 2025-12-04T11:13:50.1166716Z FAILED [0.5016s] [100%] 2025-12-04T11:13:50.1166719Z 2025-12-04T11:13:50.1166816Z ==================================== RERUNS ==================================== 2025-12-04T11:13:50.1167120Z _ TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_1_in_features_144_out_features_65_cuda_bfloat16 _ 2025-12-04T11:13:50.1167200Z Traceback (most recent call last): 2025-12-04T11:13:50.1167543Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.1167619Z method(*args, **kwargs) 2025-12-04T11:13:50.1167919Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.1168065Z method(*args, **kwargs) 2025-12-04T11:13:50.1168424Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T11:13:50.1168490Z with policy(): 2025-12-04T11:13:50.1168804Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T11:13:50.1168873Z raise RuntimeError(msg) 2025-12-04T11:13:50.1169685Z RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_1_in_features_144_out_features_65_cuda_bfloat16! Caching allocator allocated memory was 0 and is now reported as 9728 on device 0. CUDA driver allocated memory was 230686720 and is now 274726912. 2025-12-04T11:13:50.1169689Z 2025-12-04T11:13:50.1169825Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.1170353Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_1_in_features_144_out_features_65_cuda_bfloat16 2025-12-04T11:13:50.1170362Z 2025-12-04T11:13:50.1170530Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.1170747Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.1170854Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.1171409Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('benchmarking.InductorBenchmarker.benchmark_gpu', 2), ('pad_mm_bench', 1), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.1171547Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.1171610Z graph_break [] 2025-12-04T11:13:50.1171742Z ----------------------------- Captured stderr call ----------------------------- 2025-12-04T11:13:50.1172460Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/constant_folding.py:256: UserWarning: Unsupported unwinding pattern: Address not in range (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/profiler/unwind/unwind.cpp:219.) 2025-12-04T11:13:50.1172539Z if out == self.unknown_value: 2025-12-04T11:13:50.1172849Z _ TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_1_in_features_144_out_features_65_cuda_bfloat16 _ 2025-12-04T11:13:50.1172929Z Traceback (most recent call last): 2025-12-04T11:13:50.1173239Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.1173312Z method(*args, **kwargs) 2025-12-04T11:13:50.1173606Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.1173673Z method(*args, **kwargs) 2025-12-04T11:13:50.1173969Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T11:13:50.1174034Z with policy(): 2025-12-04T11:13:50.1174336Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T11:13:50.1174406Z raise RuntimeError(msg) 2025-12-04T11:13:50.1175219Z RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_1_in_features_144_out_features_65_cuda_bfloat16! Caching allocator allocated memory was 9728 and is now reported as 19456 on device 0. CUDA driver allocated memory was 274726912 and is now 276824064. 2025-12-04T11:13:50.1175228Z 2025-12-04T11:13:50.1175365Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.1175885Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_1_in_features_144_out_features_65_cuda_bfloat16 2025-12-04T11:13:50.1175935Z 2025-12-04T11:13:50.1176171Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.1176306Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.1176407Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.1176957Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('benchmarking.InductorBenchmarker.benchmark_gpu', 2), ('pad_mm_bench', 1), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.1177089Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.1177155Z graph_break [] 2025-12-04T11:13:50.1177283Z ----------------------------- Captured stderr call ----------------------------- 2025-12-04T11:13:50.1178524Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/constant_folding.py:256: UserWarning: Unsupported unwinding pattern: Address not in range (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/profiler/unwind/unwind.cpp:219.) 2025-12-04T11:13:50.1178616Z if out == self.unknown_value: 2025-12-04T11:13:50.1178746Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.1178972Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.1179103Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.1179649Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('benchmarking.InductorBenchmarker.benchmark_gpu', 2), ('pad_mm_bench', 1), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.1179715Z graph_break [] 2025-12-04T11:13:50.1179802Z =================================== FAILURES =================================== 2025-12-04T11:13:50.1180104Z _ TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_1_in_features_144_out_features_65_cuda_bfloat16 _ 2025-12-04T11:13:50.1180186Z Traceback (most recent call last): 2025-12-04T11:13:50.1180495Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.1180569Z method(*args, **kwargs) 2025-12-04T11:13:50.1180874Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.1180940Z method(*args, **kwargs) 2025-12-04T11:13:50.1181235Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T11:13:50.1181306Z with policy(): 2025-12-04T11:13:50.1181607Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T11:13:50.1181676Z raise RuntimeError(msg) 2025-12-04T11:13:50.1182489Z RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_1_in_features_144_out_features_65_cuda_bfloat16! Caching allocator allocated memory was 19456 and is now reported as 29184 on device 0. CUDA driver allocated memory was 276824064 and is now 278921216. 2025-12-04T11:13:50.1182501Z 2025-12-04T11:13:50.1182633Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.1183153Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_1_in_features_144_out_features_65_cuda_bfloat16 2025-12-04T11:13:50.1183157Z 2025-12-04T11:13:50.1183319Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.1183450Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.1183592Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.1184252Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('benchmarking.InductorBenchmarker.benchmark_gpu', 2), ('pad_mm_bench', 1), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.1184435Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.1184514Z graph_break [] 2025-12-04T11:13:50.1184644Z ----------------------------- Captured stderr call ----------------------------- 2025-12-04T11:13:50.1185339Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/constant_folding.py:256: UserWarning: Unsupported unwinding pattern: Address not in range (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/profiler/unwind/unwind.cpp:219.) 2025-12-04T11:13:50.1185418Z if out == self.unknown_value: 2025-12-04T11:13:50.1185544Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.1185639Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.1185764Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.1186302Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('benchmarking.InductorBenchmarker.benchmark_gpu', 2), ('pad_mm_bench', 1), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.1186417Z graph_break [] 2025-12-04T11:13:50.1186541Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.1186636Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.1186760Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.1187294Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('benchmarking.InductorBenchmarker.benchmark_gpu', 2), ('pad_mm_bench', 1), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.1187360Z graph_break [] 2025-12-04T11:13:50.1187859Z - generated xml file: /var/lib/jenkins/workspace/test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-e9cf01ceaaab6f89.xml - 2025-12-04T11:13:50.1187965Z =========================== short test summary info ============================ 2025-12-04T11:13:50.1189265Z FAILED [0.5016s] inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_17_mid_dim_1_in_features_144_out_features_65_cuda_bfloat16 - RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_1_in_features_144_out_features_65_cuda_bfloat16! Caching allocator allocated memory was 19456 and is now reported as 29184 on device 0. CUDA driver allocated memory was 276824064 and is now 278921216. 2025-12-04T11:13:50.1189270Z 2025-12-04T11:13:50.1189403Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.1189928Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_1_in_features_144_out_features_65_cuda_bfloat16 2025-12-04T11:13:50.1189933Z 2025-12-04T11:13:50.1190093Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.1190205Z !!!!!!!!!!!!!!!!!!!!!!!!!! stopping after 1 failures !!!!!!!!!!!!!!!!!!!!!!!!!!! 2025-12-04T11:13:50.1190322Z ================== 1 failed, 57 deselected, 2 rerun in 12.29s ================== 2025-12-04T11:13:50.1190387Z Got exit code 1 2025-12-04T11:13:50.1190876Z FAILED CONSISTENTLY: test/inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_17_mid_dim_1_in_features_144_out_features_65_cuda_bfloat16 2025-12-04T11:13:50.1191126Z Test failed consistently, continuing with the rest of the tests due to continue-through-error being set 2025-12-04T11:13:50.1191400Z W1204 10:37:54.866000 63798 site-packages/torch/_inductor/utils.py:1703] Not enough SMs to use max_autotune_gemm mode 2025-12-04T11:13:50.1191896Z Test results will be stored in test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-ff104bb89c063280.xml 2025-12-04T11:13:50.1192004Z ============================= test session starts ============================== 2025-12-04T11:13:50.1192219Z platform linux -- Python 3.10.14, pytest-7.3.2, pluggy-1.6.0 -- /opt/conda/envs/py_3.10/bin/python 2025-12-04T11:13:50.1192289Z cachedir: .pytest_cache 2025-12-04T11:13:50.1192604Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow] 2025-12-04T11:13:50.1192686Z rootdir: /var/lib/jenkins/workspace 2025-12-04T11:13:50.1192755Z configfile: pytest.ini 2025-12-04T11:13:50.1193081Z plugins: hypothesis-6.56.4, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-14.0, subtests-0.13.1, xdist-3.3.1, xdoctest-1.3.0, typeguard-4.3.0 2025-12-04T11:13:50.1193214Z collecting ... collected 58 items / 13 deselected / 45 selected 2025-12-04T11:13:50.1193309Z stepcurrent: skipping 13 already run items. 2025-12-04T11:13:50.1193397Z Running 45 items in this shard 2025-12-04T11:13:50.1193401Z 2025-12-04T11:13:50.1193913Z inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_17_mid_dim_8_in_features_1024_out_features_1024_cuda_bfloat16 ('RERUN', {'yellow': True}) [2.0594s] [ 2%] 2025-12-04T11:13:50.1194456Z inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_17_mid_dim_8_in_features_1024_out_features_1024_cuda_bfloat16 ('RERUN', {'yellow': True}) [0.6346s] [ 2%] 2025-12-04T11:13:50.1194903Z inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_17_mid_dim_8_in_features_1024_out_features_1024_cuda_bfloat16 FAILED [0.6362s] [ 2%] 2025-12-04T11:13:50.1194908Z 2025-12-04T11:13:50.1194997Z ==================================== RERUNS ==================================== 2025-12-04T11:13:50.1195302Z _ TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_8_in_features_1024_out_features_1024_cuda_bfloat16 _ 2025-12-04T11:13:50.1195379Z Traceback (most recent call last): 2025-12-04T11:13:50.1195695Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.1195764Z method(*args, **kwargs) 2025-12-04T11:13:50.1196060Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.1196127Z method(*args, **kwargs) 2025-12-04T11:13:50.1196415Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T11:13:50.1196484Z with policy(): 2025-12-04T11:13:50.1196777Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T11:13:50.1196852Z raise RuntimeError(msg) 2025-12-04T11:13:50.1197671Z RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_8_in_features_1024_out_features_1024_cuda_bfloat16! Caching allocator allocated memory was 0 and is now reported as 1048576 on device 0. CUDA driver allocated memory was 230686720 and is now 274726912. 2025-12-04T11:13:50.1197677Z 2025-12-04T11:13:50.1197808Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.1198343Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_8_in_features_1024_out_features_1024_cuda_bfloat16 2025-12-04T11:13:50.1198347Z 2025-12-04T11:13:50.1198506Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.1198644Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.1202399Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.1202910Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.1203061Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.1203125Z graph_break [] 2025-12-04T11:13:50.1203441Z _ TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_8_in_features_1024_out_features_1024_cuda_bfloat16 _ 2025-12-04T11:13:50.1203520Z Traceback (most recent call last): 2025-12-04T11:13:50.1203845Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.1203916Z method(*args, **kwargs) 2025-12-04T11:13:50.1204213Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.1204280Z method(*args, **kwargs) 2025-12-04T11:13:50.1204571Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T11:13:50.1204634Z with policy(): 2025-12-04T11:13:50.1205007Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T11:13:50.1205078Z raise RuntimeError(msg) 2025-12-04T11:13:50.1205925Z RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_8_in_features_1024_out_features_1024_cuda_bfloat16! Caching allocator allocated memory was 1048576 and is now reported as 2097152 on device 0. CUDA driver allocated memory was 274726912 and is now 276824064. 2025-12-04T11:13:50.1205934Z 2025-12-04T11:13:50.1206075Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.1206604Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_8_in_features_1024_out_features_1024_cuda_bfloat16 2025-12-04T11:13:50.1206613Z 2025-12-04T11:13:50.1206782Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.1206923Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.1207020Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.1207375Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.1207507Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.1207569Z graph_break [] 2025-12-04T11:13:50.1207697Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.1207789Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.1207916Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.1208260Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.1208323Z graph_break [] 2025-12-04T11:13:50.1208409Z =================================== FAILURES =================================== 2025-12-04T11:13:50.1208713Z _ TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_8_in_features_1024_out_features_1024_cuda_bfloat16 _ 2025-12-04T11:13:50.1208804Z Traceback (most recent call last): 2025-12-04T11:13:50.1209119Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.1209185Z method(*args, **kwargs) 2025-12-04T11:13:50.1209482Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.1209590Z method(*args, **kwargs) 2025-12-04T11:13:50.1209970Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T11:13:50.1210035Z with policy(): 2025-12-04T11:13:50.1210328Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T11:13:50.1210397Z raise RuntimeError(msg) 2025-12-04T11:13:50.1211231Z RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_8_in_features_1024_out_features_1024_cuda_bfloat16! Caching allocator allocated memory was 2097152 and is now reported as 3145728 on device 0. CUDA driver allocated memory was 276824064 and is now 278921216. 2025-12-04T11:13:50.1211235Z 2025-12-04T11:13:50.1211367Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.1211893Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_8_in_features_1024_out_features_1024_cuda_bfloat16 2025-12-04T11:13:50.1211898Z 2025-12-04T11:13:50.1212098Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.1212233Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.1212325Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.1212677Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.1212800Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.1212858Z graph_break [] 2025-12-04T11:13:50.1212984Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.1213072Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.1213194Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.1213541Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.1213600Z graph_break [] 2025-12-04T11:13:50.1213726Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.1213816Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.1213934Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.1214273Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.1214330Z graph_break [] 2025-12-04T11:13:50.1214824Z - generated xml file: /var/lib/jenkins/workspace/test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-ff104bb89c063280.xml - 2025-12-04T11:13:50.1214929Z =========================== short test summary info ============================ 2025-12-04T11:13:50.1216257Z FAILED [0.6362s] inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_17_mid_dim_8_in_features_1024_out_features_1024_cuda_bfloat16 - RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_8_in_features_1024_out_features_1024_cuda_bfloat16! Caching allocator allocated memory was 2097152 and is now reported as 3145728 on device 0. CUDA driver allocated memory was 276824064 and is now 278921216. 2025-12-04T11:13:50.1216266Z 2025-12-04T11:13:50.1216392Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.1216918Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_8_in_features_1024_out_features_1024_cuda_bfloat16 2025-12-04T11:13:50.1216965Z 2025-12-04T11:13:50.1217192Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.1217298Z !!!!!!!!!!!!!!!!!!!!!!!!!! stopping after 1 failures !!!!!!!!!!!!!!!!!!!!!!!!!!! 2025-12-04T11:13:50.1217417Z ================== 1 failed, 13 deselected, 2 rerun in 3.36s =================== 2025-12-04T11:13:50.1217476Z Got exit code 1 2025-12-04T11:13:50.1217541Z Retrying single test... 2025-12-04T11:13:50.1217811Z W1204 10:38:05.035000 63987 site-packages/torch/_inductor/utils.py:1703] Not enough SMs to use max_autotune_gemm mode 2025-12-04T11:13:50.1218199Z Test results will be stored in test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-61993e92defaa624.xml 2025-12-04T11:13:50.1218296Z ============================= test session starts ============================== 2025-12-04T11:13:50.1218508Z platform linux -- Python 3.10.14, pytest-7.3.2, pluggy-1.6.0 -- /opt/conda/envs/py_3.10/bin/python 2025-12-04T11:13:50.1218575Z cachedir: .pytest_cache 2025-12-04T11:13:50.1218886Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow] 2025-12-04T11:13:50.1219006Z rootdir: /var/lib/jenkins/workspace 2025-12-04T11:13:50.1219069Z configfile: pytest.ini 2025-12-04T11:13:50.1219388Z plugins: hypothesis-6.56.4, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-14.0, subtests-0.13.1, xdist-3.3.1, xdoctest-1.3.0, typeguard-4.3.0 2025-12-04T11:13:50.1219524Z collecting ... collected 58 items / 57 deselected / 1 selected 2025-12-04T11:13:50.1220105Z stepcurrent: skipping 13 already run items. Running only test/inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_17_mid_dim_8_in_features_1024_out_features_1024_cuda_bfloat16 2025-12-04T11:13:50.1220184Z Running 1 items in this shard 2025-12-04T11:13:50.1220188Z 2025-12-04T11:13:50.1220932Z inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_17_mid_dim_8_in_features_1024_out_features_1024_cuda_bfloat16 [W1204 10:38:06.298393246 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1220939Z 2025-12-04T11:13:50.1221233Z [W1204 10:38:15.482259124 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1221237Z 2025-12-04T11:13:50.1221524Z [W1204 10:38:15.482552811 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1221527Z 2025-12-04T11:13:50.1221814Z [W1204 10:38:15.488121894 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1221817Z 2025-12-04T11:13:50.1222103Z [W1204 10:38:15.488678076 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1222108Z 2025-12-04T11:13:50.1222399Z [W1204 10:38:15.488845990 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1222403Z 2025-12-04T11:13:50.1222688Z [W1204 10:38:15.494190047 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1222692Z 2025-12-04T11:13:50.1222977Z [W1204 10:38:15.494710959 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1222980Z 2025-12-04T11:13:50.1223263Z [W1204 10:38:15.494869053 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1223266Z 2025-12-04T11:13:50.1223350Z ('RERUN', {'yellow': True}) [11.2486s] [100%] 2025-12-04T11:13:50.1224240Z inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_17_mid_dim_8_in_features_1024_out_features_1024_cuda_bfloat16 [W1204 10:38:16.860975121 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1224277Z 2025-12-04T11:13:50.1224567Z [W1204 10:38:16.861514233 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1224575Z 2025-12-04T11:13:50.1224858Z [W1204 10:38:16.861651296 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1224862Z 2025-12-04T11:13:50.1225145Z [W1204 10:38:16.864603791 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1225148Z 2025-12-04T11:13:50.1225435Z [W1204 10:38:16.865172864 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1225440Z 2025-12-04T11:13:50.1225727Z [W1204 10:38:16.865308237 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1225770Z 2025-12-04T11:13:50.1226060Z [W1204 10:38:16.869876377 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1226064Z 2025-12-04T11:13:50.1226348Z [W1204 10:38:16.870362618 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1226351Z 2025-12-04T11:13:50.1226637Z [W1204 10:38:16.870502491 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1226641Z 2025-12-04T11:13:50.1226719Z ('RERUN', {'yellow': True}) [0.5983s] [100%] 2025-12-04T11:13:50.1227449Z inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_17_mid_dim_8_in_features_1024_out_features_1024_cuda_bfloat16 [W1204 10:38:17.455095473 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1227460Z 2025-12-04T11:13:50.1227747Z [W1204 10:38:17.455633365 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1227751Z 2025-12-04T11:13:50.1228036Z [W1204 10:38:17.455771238 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1228040Z 2025-12-04T11:13:50.1228331Z [W1204 10:38:17.458730394 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1228334Z 2025-12-04T11:13:50.1228618Z [W1204 10:38:17.459294006 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1228623Z 2025-12-04T11:13:50.1228912Z [W1204 10:38:17.459434209 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1228916Z 2025-12-04T11:13:50.1229198Z [W1204 10:38:17.464052361 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1229201Z 2025-12-04T11:13:50.1229494Z [W1204 10:38:17.464525441 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1229498Z 2025-12-04T11:13:50.1229780Z [W1204 10:38:17.464669845 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1229783Z 2025-12-04T11:13:50.1229846Z FAILED [0.5957s] [100%] 2025-12-04T11:13:50.1229849Z 2025-12-04T11:13:50.1229933Z ==================================== RERUNS ==================================== 2025-12-04T11:13:50.1230334Z _ TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_8_in_features_1024_out_features_1024_cuda_bfloat16 _ 2025-12-04T11:13:50.1230414Z Traceback (most recent call last): 2025-12-04T11:13:50.1230725Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.1230791Z method(*args, **kwargs) 2025-12-04T11:13:50.1231084Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.1231146Z method(*args, **kwargs) 2025-12-04T11:13:50.1231433Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T11:13:50.1231492Z with policy(): 2025-12-04T11:13:50.1231782Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T11:13:50.1231861Z raise RuntimeError(msg) 2025-12-04T11:13:50.1232680Z RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_8_in_features_1024_out_features_1024_cuda_bfloat16! Caching allocator allocated memory was 0 and is now reported as 1048576 on device 0. CUDA driver allocated memory was 230686720 and is now 274726912. 2025-12-04T11:13:50.1232726Z 2025-12-04T11:13:50.1232859Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.1233381Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_8_in_features_1024_out_features_1024_cuda_bfloat16 2025-12-04T11:13:50.1233385Z 2025-12-04T11:13:50.1233544Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.1233671Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.1233768Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.1234117Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.1234247Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.1234305Z graph_break [] 2025-12-04T11:13:50.1234433Z ----------------------------- Captured stderr call ----------------------------- 2025-12-04T11:13:50.1235123Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/constant_folding.py:256: UserWarning: Unsupported unwinding pattern: Address not in range (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/profiler/unwind/unwind.cpp:219.) 2025-12-04T11:13:50.1235193Z if out == self.unknown_value: 2025-12-04T11:13:50.1235489Z _ TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_8_in_features_1024_out_features_1024_cuda_bfloat16 _ 2025-12-04T11:13:50.1235563Z Traceback (most recent call last): 2025-12-04T11:13:50.1235862Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.1235926Z method(*args, **kwargs) 2025-12-04T11:13:50.1236214Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.1236275Z method(*args, **kwargs) 2025-12-04T11:13:50.1236560Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T11:13:50.1236619Z with policy(): 2025-12-04T11:13:50.1236907Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T11:13:50.1236971Z raise RuntimeError(msg) 2025-12-04T11:13:50.1237917Z RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_8_in_features_1024_out_features_1024_cuda_bfloat16! Caching allocator allocated memory was 1048576 and is now reported as 2097152 on device 0. CUDA driver allocated memory was 274726912 and is now 276824064. 2025-12-04T11:13:50.1237956Z 2025-12-04T11:13:50.1238086Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.1238613Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_8_in_features_1024_out_features_1024_cuda_bfloat16 2025-12-04T11:13:50.1238617Z 2025-12-04T11:13:50.1238771Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.1238896Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.1238988Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.1239340Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.1239469Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.1239579Z graph_break [] 2025-12-04T11:13:50.1239705Z ----------------------------- Captured stderr call ----------------------------- 2025-12-04T11:13:50.1240394Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/constant_folding.py:256: UserWarning: Unsupported unwinding pattern: Address not in range (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/profiler/unwind/unwind.cpp:219.) 2025-12-04T11:13:50.1240464Z if out == self.unknown_value: 2025-12-04T11:13:50.1240587Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.1240675Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.1240799Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.1241148Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.1241206Z graph_break [] 2025-12-04T11:13:50.1241291Z =================================== FAILURES =================================== 2025-12-04T11:13:50.1241587Z _ TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_8_in_features_1024_out_features_1024_cuda_bfloat16 _ 2025-12-04T11:13:50.1241658Z Traceback (most recent call last): 2025-12-04T11:13:50.1241953Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.1242017Z method(*args, **kwargs) 2025-12-04T11:13:50.1242304Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.1242370Z method(*args, **kwargs) 2025-12-04T11:13:50.1242657Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T11:13:50.1242720Z with policy(): 2025-12-04T11:13:50.1243009Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T11:13:50.1243074Z raise RuntimeError(msg) 2025-12-04T11:13:50.1243898Z RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_8_in_features_1024_out_features_1024_cuda_bfloat16! Caching allocator allocated memory was 2097152 and is now reported as 3145728 on device 0. CUDA driver allocated memory was 276824064 and is now 278921216. 2025-12-04T11:13:50.1243902Z 2025-12-04T11:13:50.1244026Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.1244547Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_8_in_features_1024_out_features_1024_cuda_bfloat16 2025-12-04T11:13:50.1244594Z 2025-12-04T11:13:50.1244830Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.1244961Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.1245050Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.1245391Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.1245515Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.1245573Z graph_break [] 2025-12-04T11:13:50.1245695Z ----------------------------- Captured stderr call ----------------------------- 2025-12-04T11:13:50.1246385Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/constant_folding.py:256: UserWarning: Unsupported unwinding pattern: Address not in range (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/profiler/unwind/unwind.cpp:219.) 2025-12-04T11:13:50.1246458Z if out == self.unknown_value: 2025-12-04T11:13:50.1246580Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.1246916Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.1247036Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.1247383Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.1247439Z graph_break [] 2025-12-04T11:13:50.1247568Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.1247656Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.1247777Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.1248124Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.1248181Z graph_break [] 2025-12-04T11:13:50.1248673Z - generated xml file: /var/lib/jenkins/workspace/test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-61993e92defaa624.xml - 2025-12-04T11:13:50.1248775Z =========================== short test summary info ============================ 2025-12-04T11:13:50.1250081Z FAILED [0.5957s] inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_17_mid_dim_8_in_features_1024_out_features_1024_cuda_bfloat16 - RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_8_in_features_1024_out_features_1024_cuda_bfloat16! Caching allocator allocated memory was 2097152 and is now reported as 3145728 on device 0. CUDA driver allocated memory was 276824064 and is now 278921216. 2025-12-04T11:13:50.1250090Z 2025-12-04T11:13:50.1250215Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.1250734Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_8_in_features_1024_out_features_1024_cuda_bfloat16 2025-12-04T11:13:50.1250740Z 2025-12-04T11:13:50.1250895Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.1250997Z !!!!!!!!!!!!!!!!!!!!!!!!!! stopping after 1 failures !!!!!!!!!!!!!!!!!!!!!!!!!!! 2025-12-04T11:13:50.1251115Z ================== 1 failed, 57 deselected, 2 rerun in 12.47s ================== 2025-12-04T11:13:50.1251174Z Got exit code 1 2025-12-04T11:13:50.1251240Z Retrying single test... 2025-12-04T11:13:50.1251507Z W1204 10:38:24.276000 64181 site-packages/torch/_inductor/utils.py:1703] Not enough SMs to use max_autotune_gemm mode 2025-12-04T11:13:50.1252003Z Test results will be stored in test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-16a80675dc6db2f9.xml 2025-12-04T11:13:50.1252096Z ============================= test session starts ============================== 2025-12-04T11:13:50.1252309Z platform linux -- Python 3.10.14, pytest-7.3.2, pluggy-1.6.0 -- /opt/conda/envs/py_3.10/bin/python 2025-12-04T11:13:50.1252374Z cachedir: .pytest_cache 2025-12-04T11:13:50.1252682Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow] 2025-12-04T11:13:50.1252756Z rootdir: /var/lib/jenkins/workspace 2025-12-04T11:13:50.1252821Z configfile: pytest.ini 2025-12-04T11:13:50.1253135Z plugins: hypothesis-6.56.4, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-14.0, subtests-0.13.1, xdist-3.3.1, xdoctest-1.3.0, typeguard-4.3.0 2025-12-04T11:13:50.1253264Z collecting ... collected 58 items / 57 deselected / 1 selected 2025-12-04T11:13:50.1253839Z stepcurrent: skipping 13 already run items. Running only test/inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_17_mid_dim_8_in_features_1024_out_features_1024_cuda_bfloat16 2025-12-04T11:13:50.1253954Z Running 1 items in this shard 2025-12-04T11:13:50.1253957Z 2025-12-04T11:13:50.1254692Z inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_17_mid_dim_8_in_features_1024_out_features_1024_cuda_bfloat16 [W1204 10:38:25.534489425 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1254696Z 2025-12-04T11:13:50.1254994Z [W1204 10:38:34.859910216 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1254998Z 2025-12-04T11:13:50.1255284Z [W1204 10:38:34.860234603 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1255289Z 2025-12-04T11:13:50.1255581Z [W1204 10:38:34.865993862 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1255586Z 2025-12-04T11:13:50.1255870Z [W1204 10:38:34.866523064 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1255874Z 2025-12-04T11:13:50.1256156Z [W1204 10:38:34.866690238 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1256160Z 2025-12-04T11:13:50.1256441Z [W1204 10:38:34.872052978 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1256445Z 2025-12-04T11:13:50.1256729Z [W1204 10:38:34.872591960 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1256734Z 2025-12-04T11:13:50.1257020Z [W1204 10:38:34.872750863 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1257025Z 2025-12-04T11:13:50.1257105Z ('RERUN', {'yellow': True}) [11.3783s] [100%] 2025-12-04T11:13:50.1257834Z inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_17_mid_dim_8_in_features_1024_out_features_1024_cuda_bfloat16 [W1204 10:38:36.220119163 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1257839Z 2025-12-04T11:13:50.1258129Z [W1204 10:38:36.220664076 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1258133Z 2025-12-04T11:13:50.1258421Z [W1204 10:38:36.220802559 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1258462Z 2025-12-04T11:13:50.1258811Z [W1204 10:38:36.223752975 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1258817Z 2025-12-04T11:13:50.1259104Z [W1204 10:38:36.224321037 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1259108Z 2025-12-04T11:13:50.1259403Z [W1204 10:38:36.224459200 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1259407Z 2025-12-04T11:13:50.1259697Z [W1204 10:38:36.229022902 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1259701Z 2025-12-04T11:13:50.1259989Z [W1204 10:38:36.229491862 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1259995Z 2025-12-04T11:13:50.1260281Z [W1204 10:38:36.229628735 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1260287Z 2025-12-04T11:13:50.1260423Z ('RERUN', {'yellow': True}) [0.5854s] [100%] 2025-12-04T11:13:50.1261145Z inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_17_mid_dim_8_in_features_1024_out_features_1024_cuda_bfloat16 [W1204 10:38:36.802703604 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1261149Z 2025-12-04T11:13:50.1261437Z [W1204 10:38:36.803240316 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1261440Z 2025-12-04T11:13:50.1261727Z [W1204 10:38:36.803379939 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1261731Z 2025-12-04T11:13:50.1262022Z [W1204 10:38:36.806317734 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1262025Z 2025-12-04T11:13:50.1262310Z [W1204 10:38:36.806881117 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1262313Z 2025-12-04T11:13:50.1262599Z [W1204 10:38:36.807019190 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1262602Z 2025-12-04T11:13:50.1262887Z [W1204 10:38:36.811566531 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1262890Z 2025-12-04T11:13:50.1263175Z [W1204 10:38:36.812036952 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1263179Z 2025-12-04T11:13:50.1263465Z [W1204 10:38:36.812174625 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1263468Z 2025-12-04T11:13:50.1263603Z FAILED [0.5848s] [100%] 2025-12-04T11:13:50.1263609Z 2025-12-04T11:13:50.1263695Z ==================================== RERUNS ==================================== 2025-12-04T11:13:50.1263989Z _ TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_8_in_features_1024_out_features_1024_cuda_bfloat16 _ 2025-12-04T11:13:50.1264063Z Traceback (most recent call last): 2025-12-04T11:13:50.1264369Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.1264433Z method(*args, **kwargs) 2025-12-04T11:13:50.1264724Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.1264830Z method(*args, **kwargs) 2025-12-04T11:13:50.1265184Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T11:13:50.1265248Z with policy(): 2025-12-04T11:13:50.1265541Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T11:13:50.1265616Z raise RuntimeError(msg) 2025-12-04T11:13:50.1266430Z RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_8_in_features_1024_out_features_1024_cuda_bfloat16! Caching allocator allocated memory was 0 and is now reported as 1048576 on device 0. CUDA driver allocated memory was 230686720 and is now 274726912. 2025-12-04T11:13:50.1266434Z 2025-12-04T11:13:50.1266565Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.1267086Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_8_in_features_1024_out_features_1024_cuda_bfloat16 2025-12-04T11:13:50.1267094Z 2025-12-04T11:13:50.1267249Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.1267418Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.1267509Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.1267860Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.1267985Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.1268042Z graph_break [] 2025-12-04T11:13:50.1268169Z ----------------------------- Captured stderr call ----------------------------- 2025-12-04T11:13:50.1268857Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/constant_folding.py:256: UserWarning: Unsupported unwinding pattern: Address not in range (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/profiler/unwind/unwind.cpp:219.) 2025-12-04T11:13:50.1268929Z if out == self.unknown_value: 2025-12-04T11:13:50.1269225Z _ TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_8_in_features_1024_out_features_1024_cuda_bfloat16 _ 2025-12-04T11:13:50.1269297Z Traceback (most recent call last): 2025-12-04T11:13:50.1269593Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.1269656Z method(*args, **kwargs) 2025-12-04T11:13:50.1269942Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.1270008Z method(*args, **kwargs) 2025-12-04T11:13:50.1270292Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T11:13:50.1270354Z with policy(): 2025-12-04T11:13:50.1270645Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T11:13:50.1270710Z raise RuntimeError(msg) 2025-12-04T11:13:50.1271535Z RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_8_in_features_1024_out_features_1024_cuda_bfloat16! Caching allocator allocated memory was 1048576 and is now reported as 2097152 on device 0. CUDA driver allocated memory was 274726912 and is now 276824064. 2025-12-04T11:13:50.1271540Z 2025-12-04T11:13:50.1271663Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.1272183Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_8_in_features_1024_out_features_1024_cuda_bfloat16 2025-12-04T11:13:50.1272228Z 2025-12-04T11:13:50.1272386Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.1272575Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.1272677Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.1273025Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.1273152Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.1273210Z graph_break [] 2025-12-04T11:13:50.1273343Z ----------------------------- Captured stderr call ----------------------------- 2025-12-04T11:13:50.1274027Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/constant_folding.py:256: UserWarning: Unsupported unwinding pattern: Address not in range (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/profiler/unwind/unwind.cpp:219.) 2025-12-04T11:13:50.1274098Z if out == self.unknown_value: 2025-12-04T11:13:50.1274222Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.1274310Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.1274562Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.1274908Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.1274966Z graph_break [] 2025-12-04T11:13:50.1275048Z =================================== FAILURES =================================== 2025-12-04T11:13:50.1275343Z _ TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_8_in_features_1024_out_features_1024_cuda_bfloat16 _ 2025-12-04T11:13:50.1275417Z Traceback (most recent call last): 2025-12-04T11:13:50.1275716Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.1275781Z method(*args, **kwargs) 2025-12-04T11:13:50.1276072Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.1276142Z method(*args, **kwargs) 2025-12-04T11:13:50.1276430Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T11:13:50.1276496Z with policy(): 2025-12-04T11:13:50.1276789Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T11:13:50.1276854Z raise RuntimeError(msg) 2025-12-04T11:13:50.1277682Z RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_8_in_features_1024_out_features_1024_cuda_bfloat16! Caching allocator allocated memory was 2097152 and is now reported as 3145728 on device 0. CUDA driver allocated memory was 276824064 and is now 278921216. 2025-12-04T11:13:50.1277688Z 2025-12-04T11:13:50.1278058Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.1278698Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_8_in_features_1024_out_features_1024_cuda_bfloat16 2025-12-04T11:13:50.1278702Z 2025-12-04T11:13:50.1278868Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.1279000Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.1279101Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.1279457Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.1279594Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.1279750Z graph_break [] 2025-12-04T11:13:50.1279970Z ----------------------------- Captured stderr call ----------------------------- 2025-12-04T11:13:50.1280667Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/constant_folding.py:256: UserWarning: Unsupported unwinding pattern: Address not in range (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/profiler/unwind/unwind.cpp:219.) 2025-12-04T11:13:50.1280738Z if out == self.unknown_value: 2025-12-04T11:13:50.1280870Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.1280964Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.1281090Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.1281440Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.1281500Z graph_break [] 2025-12-04T11:13:50.1281626Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.1281722Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.1281921Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.1282268Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.1282326Z graph_break [] 2025-12-04T11:13:50.1282817Z - generated xml file: /var/lib/jenkins/workspace/test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-16a80675dc6db2f9.xml - 2025-12-04T11:13:50.1282923Z =========================== short test summary info ============================ 2025-12-04T11:13:50.1284233Z FAILED [0.5848s] inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_17_mid_dim_8_in_features_1024_out_features_1024_cuda_bfloat16 - RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_8_in_features_1024_out_features_1024_cuda_bfloat16! Caching allocator allocated memory was 2097152 and is now reported as 3145728 on device 0. CUDA driver allocated memory was 276824064 and is now 278921216. 2025-12-04T11:13:50.1284240Z 2025-12-04T11:13:50.1284369Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.1284898Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_8_in_features_1024_out_features_1024_cuda_bfloat16 2025-12-04T11:13:50.1284902Z 2025-12-04T11:13:50.1285062Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.1285166Z !!!!!!!!!!!!!!!!!!!!!!!!!! stopping after 1 failures !!!!!!!!!!!!!!!!!!!!!!!!!!! 2025-12-04T11:13:50.1285283Z ================== 1 failed, 57 deselected, 2 rerun in 12.57s ================== 2025-12-04T11:13:50.1285348Z Got exit code 1 2025-12-04T11:13:50.1285826Z FAILED CONSISTENTLY: test/inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_17_mid_dim_8_in_features_1024_out_features_1024_cuda_bfloat16 2025-12-04T11:13:50.1286075Z Test failed consistently, continuing with the rest of the tests due to continue-through-error being set 2025-12-04T11:13:50.1286337Z W1204 10:38:43.653000 64375 site-packages/torch/_inductor/utils.py:1703] Not enough SMs to use max_autotune_gemm mode 2025-12-04T11:13:50.1286722Z Test results will be stored in test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-601480a9a7ef82d7.xml 2025-12-04T11:13:50.1286822Z ============================= test session starts ============================== 2025-12-04T11:13:50.1287032Z platform linux -- Python 3.10.14, pytest-7.3.2, pluggy-1.6.0 -- /opt/conda/envs/py_3.10/bin/python 2025-12-04T11:13:50.1287148Z cachedir: .pytest_cache 2025-12-04T11:13:50.1287539Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow] 2025-12-04T11:13:50.1287618Z rootdir: /var/lib/jenkins/workspace 2025-12-04T11:13:50.1287687Z configfile: pytest.ini 2025-12-04T11:13:50.1288002Z plugins: hypothesis-6.56.4, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-14.0, subtests-0.13.1, xdist-3.3.1, xdoctest-1.3.0, typeguard-4.3.0 2025-12-04T11:13:50.1288131Z collecting ... collected 58 items / 14 deselected / 44 selected 2025-12-04T11:13:50.1288223Z stepcurrent: skipping 14 already run items. 2025-12-04T11:13:50.1288293Z Running 44 items in this shard 2025-12-04T11:13:50.1288297Z 2025-12-04T11:13:50.1288798Z inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_17_mid_dim_8_in_features_1024_out_features_64_cuda_bfloat16 ('RERUN', {'yellow': True}) [1.8908s] [ 2%] 2025-12-04T11:13:50.1289312Z inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_17_mid_dim_8_in_features_1024_out_features_64_cuda_bfloat16 ('RERUN', {'yellow': True}) [0.4574s] [ 2%] 2025-12-04T11:13:50.1289813Z inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_17_mid_dim_8_in_features_1024_out_features_64_cuda_bfloat16 FAILED [0.4503s] [ 2%] 2025-12-04T11:13:50.1289821Z 2025-12-04T11:13:50.1289901Z ==================================== RERUNS ==================================== 2025-12-04T11:13:50.1290196Z _ TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_8_in_features_1024_out_features_64_cuda_bfloat16 _ 2025-12-04T11:13:50.1290276Z Traceback (most recent call last): 2025-12-04T11:13:50.1290578Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.1290648Z method(*args, **kwargs) 2025-12-04T11:13:50.1290946Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.1291011Z method(*args, **kwargs) 2025-12-04T11:13:50.1291301Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T11:13:50.1291364Z with policy(): 2025-12-04T11:13:50.1291659Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T11:13:50.1291725Z raise RuntimeError(msg) 2025-12-04T11:13:50.1292525Z RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_8_in_features_1024_out_features_64_cuda_bfloat16! Caching allocator allocated memory was 0 and is now reported as 65536 on device 0. CUDA driver allocated memory was 230686720 and is now 274726912. 2025-12-04T11:13:50.1292533Z 2025-12-04T11:13:50.1292657Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.1293179Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_8_in_features_1024_out_features_64_cuda_bfloat16 2025-12-04T11:13:50.1293186Z 2025-12-04T11:13:50.1293347Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.1293472Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.1293568Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.1293916Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.1294040Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.1294101Z graph_break [] 2025-12-04T11:13:50.1294432Z _ TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_8_in_features_1024_out_features_64_cuda_bfloat16 _ 2025-12-04T11:13:50.1294569Z Traceback (most recent call last): 2025-12-04T11:13:50.1294880Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.1294946Z method(*args, **kwargs) 2025-12-04T11:13:50.1295236Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.1295298Z method(*args, **kwargs) 2025-12-04T11:13:50.1295581Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T11:13:50.1295642Z with policy(): 2025-12-04T11:13:50.1295930Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T11:13:50.1295996Z raise RuntimeError(msg) 2025-12-04T11:13:50.1296809Z RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_8_in_features_1024_out_features_64_cuda_bfloat16! Caching allocator allocated memory was 65536 and is now reported as 131072 on device 0. CUDA driver allocated memory was 274726912 and is now 276824064. 2025-12-04T11:13:50.1296850Z 2025-12-04T11:13:50.1296975Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.1297493Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_8_in_features_1024_out_features_64_cuda_bfloat16 2025-12-04T11:13:50.1297497Z 2025-12-04T11:13:50.1297650Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.1297778Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.1297870Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.1298218Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.1298348Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.1298407Z graph_break [] 2025-12-04T11:13:50.1298531Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.1298620Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.1298739Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.1299078Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.1299134Z graph_break [] 2025-12-04T11:13:50.1299216Z =================================== FAILURES =================================== 2025-12-04T11:13:50.1299516Z _ TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_8_in_features_1024_out_features_64_cuda_bfloat16 _ 2025-12-04T11:13:50.1299590Z Traceback (most recent call last): 2025-12-04T11:13:50.1299889Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.1299953Z method(*args, **kwargs) 2025-12-04T11:13:50.1300240Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.1300306Z method(*args, **kwargs) 2025-12-04T11:13:50.1300590Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T11:13:50.1300648Z with policy(): 2025-12-04T11:13:50.1300938Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T11:13:50.1301048Z raise RuntimeError(msg) 2025-12-04T11:13:50.1301930Z RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_8_in_features_1024_out_features_64_cuda_bfloat16! Caching allocator allocated memory was 131072 and is now reported as 196608 on device 0. CUDA driver allocated memory was 276824064 and is now 278921216. 2025-12-04T11:13:50.1301936Z 2025-12-04T11:13:50.1302060Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.1302575Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_8_in_features_1024_out_features_64_cuda_bfloat16 2025-12-04T11:13:50.1302578Z 2025-12-04T11:13:50.1302733Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.1302856Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.1302952Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.1303295Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.1303466Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.1303585Z graph_break [] 2025-12-04T11:13:50.1303719Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.1303809Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.1303927Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.1304266Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.1304325Z graph_break [] 2025-12-04T11:13:50.1304448Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.1304540Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.1304662Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.1305003Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.1305066Z graph_break [] 2025-12-04T11:13:50.1305552Z - generated xml file: /var/lib/jenkins/workspace/test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-601480a9a7ef82d7.xml - 2025-12-04T11:13:50.1305651Z =========================== short test summary info ============================ 2025-12-04T11:13:50.1306942Z FAILED [0.4503s] inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_17_mid_dim_8_in_features_1024_out_features_64_cuda_bfloat16 - RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_8_in_features_1024_out_features_64_cuda_bfloat16! Caching allocator allocated memory was 131072 and is now reported as 196608 on device 0. CUDA driver allocated memory was 276824064 and is now 278921216. 2025-12-04T11:13:50.1306950Z 2025-12-04T11:13:50.1307076Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.1307591Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_8_in_features_1024_out_features_64_cuda_bfloat16 2025-12-04T11:13:50.1307595Z 2025-12-04T11:13:50.1307747Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.1307854Z !!!!!!!!!!!!!!!!!!!!!!!!!! stopping after 1 failures !!!!!!!!!!!!!!!!!!!!!!!!!!! 2025-12-04T11:13:50.1307970Z ================== 1 failed, 14 deselected, 2 rerun in 2.82s =================== 2025-12-04T11:13:50.1308075Z Got exit code 1 2025-12-04T11:13:50.1308149Z Retrying single test... 2025-12-04T11:13:50.1308477Z W1204 10:38:53.702000 64563 site-packages/torch/_inductor/utils.py:1703] Not enough SMs to use max_autotune_gemm mode 2025-12-04T11:13:50.1308869Z Test results will be stored in test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-fda38496367dc0a9.xml 2025-12-04T11:13:50.1308962Z ============================= test session starts ============================== 2025-12-04T11:13:50.1309169Z platform linux -- Python 3.10.14, pytest-7.3.2, pluggy-1.6.0 -- /opt/conda/envs/py_3.10/bin/python 2025-12-04T11:13:50.1309247Z cachedir: .pytest_cache 2025-12-04T11:13:50.1309552Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow] 2025-12-04T11:13:50.1309632Z rootdir: /var/lib/jenkins/workspace 2025-12-04T11:13:50.1309696Z configfile: pytest.ini 2025-12-04T11:13:50.1310008Z plugins: hypothesis-6.56.4, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-14.0, subtests-0.13.1, xdist-3.3.1, xdoctest-1.3.0, typeguard-4.3.0 2025-12-04T11:13:50.1310142Z collecting ... collected 58 items / 57 deselected / 1 selected 2025-12-04T11:13:50.1310710Z stepcurrent: skipping 14 already run items. Running only test/inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_17_mid_dim_8_in_features_1024_out_features_64_cuda_bfloat16 2025-12-04T11:13:50.1310832Z Running 1 items in this shard 2025-12-04T11:13:50.1310841Z 2025-12-04T11:13:50.1311569Z inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_17_mid_dim_8_in_features_1024_out_features_64_cuda_bfloat16 [W1204 10:38:54.815218367 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1311573Z 2025-12-04T11:13:50.1311872Z [W1204 10:39:04.978473086 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1311881Z 2025-12-04T11:13:50.1312173Z [W1204 10:39:04.978720251 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1312178Z 2025-12-04T11:13:50.1312464Z [W1204 10:39:04.984611712 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1312467Z 2025-12-04T11:13:50.1312758Z [W1204 10:39:04.985169404 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1312761Z 2025-12-04T11:13:50.1313044Z [W1204 10:39:04.985340238 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1313048Z 2025-12-04T11:13:50.1313334Z [W1204 10:39:04.990790968 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1313340Z 2025-12-04T11:13:50.1313626Z [W1204 10:39:04.991344301 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1313631Z 2025-12-04T11:13:50.1313919Z [W1204 10:39:04.991524605 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1313922Z 2025-12-04T11:13:50.1314004Z ('RERUN', {'yellow': True}) [11.0693s] [100%] 2025-12-04T11:13:50.1314727Z inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_17_mid_dim_8_in_features_1024_out_features_64_cuda_bfloat16 [W1204 10:39:05.178369047 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1314735Z 2025-12-04T11:13:50.1315022Z [W1204 10:39:05.178923969 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1315064Z 2025-12-04T11:13:50.1315451Z [W1204 10:39:05.179075242 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1315457Z 2025-12-04T11:13:50.1315751Z [W1204 10:39:05.182172121 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1315754Z 2025-12-04T11:13:50.1316042Z [W1204 10:39:05.182763334 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1316046Z 2025-12-04T11:13:50.1316334Z [W1204 10:39:05.182903117 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1316338Z 2025-12-04T11:13:50.1316621Z [W1204 10:39:05.187569230 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1316625Z 2025-12-04T11:13:50.1316915Z [W1204 10:39:05.188058571 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1316918Z 2025-12-04T11:13:50.1317239Z [W1204 10:39:05.188197084 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1317242Z 2025-12-04T11:13:50.1317324Z ('RERUN', {'yellow': True}) [0.4232s] [100%] 2025-12-04T11:13:50.1318044Z inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_17_mid_dim_8_in_features_1024_out_features_64_cuda_bfloat16 [W1204 10:39:05.597184044 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1318047Z 2025-12-04T11:13:50.1318334Z [W1204 10:39:05.597734626 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1318342Z 2025-12-04T11:13:50.1318628Z [W1204 10:39:05.597878809 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1318633Z 2025-12-04T11:13:50.1318917Z [W1204 10:39:05.600950767 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1318921Z 2025-12-04T11:13:50.1319207Z [W1204 10:39:05.601534280 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1319211Z 2025-12-04T11:13:50.1319497Z [W1204 10:39:05.601673323 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1319500Z 2025-12-04T11:13:50.1319796Z [W1204 10:39:05.606355245 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1319801Z 2025-12-04T11:13:50.1320090Z [W1204 10:39:05.606829426 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1320093Z 2025-12-04T11:13:50.1320382Z [W1204 10:39:05.606966569 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1320385Z 2025-12-04T11:13:50.1320447Z FAILED [0.4176s] [100%] 2025-12-04T11:13:50.1320450Z 2025-12-04T11:13:50.1320533Z ==================================== RERUNS ==================================== 2025-12-04T11:13:50.1320829Z _ TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_8_in_features_1024_out_features_64_cuda_bfloat16 _ 2025-12-04T11:13:50.1320904Z Traceback (most recent call last): 2025-12-04T11:13:50.1321209Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.1321274Z method(*args, **kwargs) 2025-12-04T11:13:50.1321609Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.1321744Z method(*args, **kwargs) 2025-12-04T11:13:50.1322033Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T11:13:50.1322092Z with policy(): 2025-12-04T11:13:50.1322385Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T11:13:50.1322450Z raise RuntimeError(msg) 2025-12-04T11:13:50.1323247Z RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_8_in_features_1024_out_features_64_cuda_bfloat16! Caching allocator allocated memory was 0 and is now reported as 65536 on device 0. CUDA driver allocated memory was 230686720 and is now 274726912. 2025-12-04T11:13:50.1323252Z 2025-12-04T11:13:50.1323378Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.1323903Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_8_in_features_1024_out_features_64_cuda_bfloat16 2025-12-04T11:13:50.1323942Z 2025-12-04T11:13:50.1324102Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.1324234Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.1324331Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.1324682Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.1324814Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.1324873Z graph_break [] 2025-12-04T11:13:50.1325005Z ----------------------------- Captured stderr call ----------------------------- 2025-12-04T11:13:50.1325707Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/constant_folding.py:256: UserWarning: Unsupported unwinding pattern: Address not in range (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/profiler/unwind/unwind.cpp:219.) 2025-12-04T11:13:50.1325779Z if out == self.unknown_value: 2025-12-04T11:13:50.1326080Z _ TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_8_in_features_1024_out_features_64_cuda_bfloat16 _ 2025-12-04T11:13:50.1326153Z Traceback (most recent call last): 2025-12-04T11:13:50.1326448Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.1326514Z method(*args, **kwargs) 2025-12-04T11:13:50.1326801Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.1326864Z method(*args, **kwargs) 2025-12-04T11:13:50.1327156Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T11:13:50.1327215Z with policy(): 2025-12-04T11:13:50.1327512Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T11:13:50.1327578Z raise RuntimeError(msg) 2025-12-04T11:13:50.1328387Z RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_8_in_features_1024_out_features_64_cuda_bfloat16! Caching allocator allocated memory was 65536 and is now reported as 131072 on device 0. CUDA driver allocated memory was 274726912 and is now 276824064. 2025-12-04T11:13:50.1328391Z 2025-12-04T11:13:50.1328537Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.1329058Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_8_in_features_1024_out_features_64_cuda_bfloat16 2025-12-04T11:13:50.1329165Z 2025-12-04T11:13:50.1329332Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.1329462Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.1329556Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.1329911Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.1330039Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.1330099Z graph_break [] 2025-12-04T11:13:50.1330224Z ----------------------------- Captured stderr call ----------------------------- 2025-12-04T11:13:50.1330914Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/constant_folding.py:256: UserWarning: Unsupported unwinding pattern: Address not in range (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/profiler/unwind/unwind.cpp:219.) 2025-12-04T11:13:50.1330990Z if out == self.unknown_value: 2025-12-04T11:13:50.1331160Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.1331256Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.1331382Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.1331727Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.1331787Z graph_break [] 2025-12-04T11:13:50.1331869Z =================================== FAILURES =================================== 2025-12-04T11:13:50.1332163Z _ TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_8_in_features_1024_out_features_64_cuda_bfloat16 _ 2025-12-04T11:13:50.1332244Z Traceback (most recent call last): 2025-12-04T11:13:50.1332544Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.1332617Z method(*args, **kwargs) 2025-12-04T11:13:50.1332910Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.1332975Z method(*args, **kwargs) 2025-12-04T11:13:50.1333263Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T11:13:50.1333321Z with policy(): 2025-12-04T11:13:50.1333613Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T11:13:50.1333682Z raise RuntimeError(msg) 2025-12-04T11:13:50.1334496Z RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_8_in_features_1024_out_features_64_cuda_bfloat16! Caching allocator allocated memory was 131072 and is now reported as 196608 on device 0. CUDA driver allocated memory was 276824064 and is now 278921216. 2025-12-04T11:13:50.1334503Z 2025-12-04T11:13:50.1334638Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.1335156Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_8_in_features_1024_out_features_64_cuda_bfloat16 2025-12-04T11:13:50.1335160Z 2025-12-04T11:13:50.1335321Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.1335448Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.1335539Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.1335885Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.1336127Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.1336194Z graph_break [] 2025-12-04T11:13:50.1336319Z ----------------------------- Captured stderr call ----------------------------- 2025-12-04T11:13:50.1337005Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/constant_folding.py:256: UserWarning: Unsupported unwinding pattern: Address not in range (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/profiler/unwind/unwind.cpp:219.) 2025-12-04T11:13:50.1337076Z if out == self.unknown_value: 2025-12-04T11:13:50.1337199Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.1337295Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.1337418Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.1337761Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.1337823Z graph_break [] 2025-12-04T11:13:50.1337995Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.1338083Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.1338207Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.1338547Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.1338607Z graph_break [] 2025-12-04T11:13:50.1339094Z - generated xml file: /var/lib/jenkins/workspace/test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-fda38496367dc0a9.xml - 2025-12-04T11:13:50.1339193Z =========================== short test summary info ============================ 2025-12-04T11:13:50.1340496Z FAILED [0.4176s] inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_17_mid_dim_8_in_features_1024_out_features_64_cuda_bfloat16 - RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_8_in_features_1024_out_features_64_cuda_bfloat16! Caching allocator allocated memory was 131072 and is now reported as 196608 on device 0. CUDA driver allocated memory was 276824064 and is now 278921216. 2025-12-04T11:13:50.1340503Z 2025-12-04T11:13:50.1340626Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.1341148Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_8_in_features_1024_out_features_64_cuda_bfloat16 2025-12-04T11:13:50.1341152Z 2025-12-04T11:13:50.1341309Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.1341418Z !!!!!!!!!!!!!!!!!!!!!!!!!! stopping after 1 failures !!!!!!!!!!!!!!!!!!!!!!!!!!! 2025-12-04T11:13:50.1341533Z ================== 1 failed, 57 deselected, 2 rerun in 11.93s ================== 2025-12-04T11:13:50.1341593Z Got exit code 1 2025-12-04T11:13:50.1341664Z Retrying single test... 2025-12-04T11:13:50.1341929Z W1204 10:39:12.455000 64756 site-packages/torch/_inductor/utils.py:1703] Not enough SMs to use max_autotune_gemm mode 2025-12-04T11:13:50.1342316Z Test results will be stored in test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-13c7a98252f889c4.xml 2025-12-04T11:13:50.1342416Z ============================= test session starts ============================== 2025-12-04T11:13:50.1342624Z platform linux -- Python 3.10.14, pytest-7.3.2, pluggy-1.6.0 -- /opt/conda/envs/py_3.10/bin/python 2025-12-04T11:13:50.1342691Z cachedir: .pytest_cache 2025-12-04T11:13:50.1343056Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow] 2025-12-04T11:13:50.1343199Z rootdir: /var/lib/jenkins/workspace 2025-12-04T11:13:50.1343270Z configfile: pytest.ini 2025-12-04T11:13:50.1343654Z plugins: hypothesis-6.56.4, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-14.0, subtests-0.13.1, xdist-3.3.1, xdoctest-1.3.0, typeguard-4.3.0 2025-12-04T11:13:50.1343788Z collecting ... collected 58 items / 57 deselected / 1 selected 2025-12-04T11:13:50.1344358Z stepcurrent: skipping 14 already run items. Running only test/inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_17_mid_dim_8_in_features_1024_out_features_64_cuda_bfloat16 2025-12-04T11:13:50.1344428Z Running 1 items in this shard 2025-12-04T11:13:50.1344432Z 2025-12-04T11:13:50.1345164Z inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_17_mid_dim_8_in_features_1024_out_features_64_cuda_bfloat16 [W1204 10:39:13.555244134 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1345169Z 2025-12-04T11:13:50.1345470Z [W1204 10:39:22.657897292 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1345515Z 2025-12-04T11:13:50.1345814Z [W1204 10:39:22.658133467 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1345818Z 2025-12-04T11:13:50.1346106Z [W1204 10:39:22.663941257 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1346110Z 2025-12-04T11:13:50.1346397Z [W1204 10:39:22.664502589 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1346400Z 2025-12-04T11:13:50.1346686Z [W1204 10:39:22.664694354 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1346692Z 2025-12-04T11:13:50.1346981Z [W1204 10:39:22.670094774 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1346986Z 2025-12-04T11:13:50.1347270Z [W1204 10:39:22.670650346 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1347273Z 2025-12-04T11:13:50.1347560Z [W1204 10:39:22.670822520 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1347566Z 2025-12-04T11:13:50.1347645Z ('RERUN', {'yellow': True}) [11.0014s] [100%] 2025-12-04T11:13:50.1348363Z inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_17_mid_dim_8_in_features_1024_out_features_64_cuda_bfloat16 [W1204 10:39:23.861431691 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1348370Z 2025-12-04T11:13:50.1348662Z [W1204 10:39:23.861974553 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1348667Z 2025-12-04T11:13:50.1348950Z [W1204 10:39:23.862114776 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1348954Z 2025-12-04T11:13:50.1349241Z [W1204 10:39:23.865093522 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1349245Z 2025-12-04T11:13:50.1349528Z [W1204 10:39:23.865665114 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1349532Z 2025-12-04T11:13:50.1349822Z [W1204 10:39:23.865806097 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1349866Z 2025-12-04T11:13:50.1350217Z [W1204 10:39:23.870390458 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1350223Z 2025-12-04T11:13:50.1350511Z [W1204 10:39:23.870871309 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1350514Z 2025-12-04T11:13:50.1350801Z [W1204 10:39:23.871009272 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1350805Z 2025-12-04T11:13:50.1350892Z ('RERUN', {'yellow': True}) [0.4217s] [100%] 2025-12-04T11:13:50.1351617Z inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_17_mid_dim_8_in_features_1024_out_features_64_cuda_bfloat16 [W1204 10:39:24.278138114 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1351625Z 2025-12-04T11:13:50.1351912Z [W1204 10:39:24.278689446 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1351951Z 2025-12-04T11:13:50.1352242Z [W1204 10:39:24.278832249 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1352245Z 2025-12-04T11:13:50.1352529Z [W1204 10:39:24.281825135 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1352533Z 2025-12-04T11:13:50.1352820Z [W1204 10:39:24.282403648 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1352823Z 2025-12-04T11:13:50.1353108Z [W1204 10:39:24.282542740 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1353112Z 2025-12-04T11:13:50.1353404Z [W1204 10:39:24.287130781 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1353408Z 2025-12-04T11:13:50.1353693Z [W1204 10:39:24.287601842 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1353696Z 2025-12-04T11:13:50.1353980Z [W1204 10:39:24.287744535 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1353985Z 2025-12-04T11:13:50.1354047Z FAILED [0.4169s] [100%] 2025-12-04T11:13:50.1354050Z 2025-12-04T11:13:50.1354138Z ==================================== RERUNS ==================================== 2025-12-04T11:13:50.1354436Z _ TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_8_in_features_1024_out_features_64_cuda_bfloat16 _ 2025-12-04T11:13:50.1354513Z Traceback (most recent call last): 2025-12-04T11:13:50.1354820Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.1354890Z method(*args, **kwargs) 2025-12-04T11:13:50.1355188Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.1355256Z method(*args, **kwargs) 2025-12-04T11:13:50.1355548Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T11:13:50.1355608Z with policy(): 2025-12-04T11:13:50.1355902Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T11:13:50.1355971Z raise RuntimeError(msg) 2025-12-04T11:13:50.1356848Z RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_8_in_features_1024_out_features_64_cuda_bfloat16! Caching allocator allocated memory was 0 and is now reported as 65536 on device 0. CUDA driver allocated memory was 230686720 and is now 274726912. 2025-12-04T11:13:50.1356886Z 2025-12-04T11:13:50.1357021Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.1357542Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_8_in_features_1024_out_features_64_cuda_bfloat16 2025-12-04T11:13:50.1357549Z 2025-12-04T11:13:50.1357709Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.1357838Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.1357937Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.1358287Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.1358422Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.1358521Z graph_break [] 2025-12-04T11:13:50.1358645Z ----------------------------- Captured stderr call ----------------------------- 2025-12-04T11:13:50.1359344Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/constant_folding.py:256: UserWarning: Unsupported unwinding pattern: Address not in range (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/profiler/unwind/unwind.cpp:219.) 2025-12-04T11:13:50.1359415Z if out == self.unknown_value: 2025-12-04T11:13:50.1359703Z _ TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_8_in_features_1024_out_features_64_cuda_bfloat16 _ 2025-12-04T11:13:50.1359778Z Traceback (most recent call last): 2025-12-04T11:13:50.1360073Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.1360141Z method(*args, **kwargs) 2025-12-04T11:13:50.1360436Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.1360499Z method(*args, **kwargs) 2025-12-04T11:13:50.1360786Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T11:13:50.1360843Z with policy(): 2025-12-04T11:13:50.1361133Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T11:13:50.1361201Z raise RuntimeError(msg) 2025-12-04T11:13:50.1362012Z RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_8_in_features_1024_out_features_64_cuda_bfloat16! Caching allocator allocated memory was 65536 and is now reported as 131072 on device 0. CUDA driver allocated memory was 274726912 and is now 276824064. 2025-12-04T11:13:50.1362018Z 2025-12-04T11:13:50.1362149Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.1362665Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_8_in_features_1024_out_features_64_cuda_bfloat16 2025-12-04T11:13:50.1362669Z 2025-12-04T11:13:50.1362836Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.1362964Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.1363056Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.1363406Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.1363580Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.1363638Z graph_break [] 2025-12-04T11:13:50.1363832Z ----------------------------- Captured stderr call ----------------------------- 2025-12-04T11:13:50.1364518Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/constant_folding.py:256: UserWarning: Unsupported unwinding pattern: Address not in range (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/profiler/unwind/unwind.cpp:219.) 2025-12-04T11:13:50.1364590Z if out == self.unknown_value: 2025-12-04T11:13:50.1364713Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.1364804Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.1364930Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.1365273Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.1365334Z graph_break [] 2025-12-04T11:13:50.1365420Z =================================== FAILURES =================================== 2025-12-04T11:13:50.1365710Z _ TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_8_in_features_1024_out_features_64_cuda_bfloat16 _ 2025-12-04T11:13:50.1365839Z Traceback (most recent call last): 2025-12-04T11:13:50.1366134Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.1366199Z method(*args, **kwargs) 2025-12-04T11:13:50.1366488Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.1366550Z method(*args, **kwargs) 2025-12-04T11:13:50.1366838Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T11:13:50.1366903Z with policy(): 2025-12-04T11:13:50.1367195Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T11:13:50.1367263Z raise RuntimeError(msg) 2025-12-04T11:13:50.1368084Z RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_8_in_features_1024_out_features_64_cuda_bfloat16! Caching allocator allocated memory was 131072 and is now reported as 196608 on device 0. CUDA driver allocated memory was 276824064 and is now 278921216. 2025-12-04T11:13:50.1368088Z 2025-12-04T11:13:50.1368225Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.1368742Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_8_in_features_1024_out_features_64_cuda_bfloat16 2025-12-04T11:13:50.1368746Z 2025-12-04T11:13:50.1368906Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.1369032Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.1369120Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.1369469Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.1369593Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.1369650Z graph_break [] 2025-12-04T11:13:50.1369773Z ----------------------------- Captured stderr call ----------------------------- 2025-12-04T11:13:50.1370454Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/constant_folding.py:256: UserWarning: Unsupported unwinding pattern: Address not in range (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/profiler/unwind/unwind.cpp:219.) 2025-12-04T11:13:50.1370571Z if out == self.unknown_value: 2025-12-04T11:13:50.1370691Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.1370846Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.1370974Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.1371314Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.1371376Z graph_break [] 2025-12-04T11:13:50.1371497Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.1371583Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.1371707Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.1372044Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.1372104Z graph_break [] 2025-12-04T11:13:50.1372599Z - generated xml file: /var/lib/jenkins/workspace/test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-13c7a98252f889c4.xml - 2025-12-04T11:13:50.1372736Z =========================== short test summary info ============================ 2025-12-04T11:13:50.1374025Z FAILED [0.4169s] inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_17_mid_dim_8_in_features_1024_out_features_64_cuda_bfloat16 - RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_8_in_features_1024_out_features_64_cuda_bfloat16! Caching allocator allocated memory was 131072 and is now reported as 196608 on device 0. CUDA driver allocated memory was 276824064 and is now 278921216. 2025-12-04T11:13:50.1374029Z 2025-12-04T11:13:50.1374152Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.1374678Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_8_in_features_1024_out_features_64_cuda_bfloat16 2025-12-04T11:13:50.1374683Z 2025-12-04T11:13:50.1374838Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.1374942Z !!!!!!!!!!!!!!!!!!!!!!!!!! stopping after 1 failures !!!!!!!!!!!!!!!!!!!!!!!!!!! 2025-12-04T11:13:50.1375073Z ================== 1 failed, 57 deselected, 2 rerun in 11.86s ================== 2025-12-04T11:13:50.1375131Z Got exit code 1 2025-12-04T11:13:50.1375606Z FAILED CONSISTENTLY: test/inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_17_mid_dim_8_in_features_1024_out_features_64_cuda_bfloat16 2025-12-04T11:13:50.1375851Z Test failed consistently, continuing with the rest of the tests due to continue-through-error being set 2025-12-04T11:13:50.1376116Z W1204 10:39:31.156000 64949 site-packages/torch/_inductor/utils.py:1703] Not enough SMs to use max_autotune_gemm mode 2025-12-04T11:13:50.1376505Z Test results will be stored in test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-d1b821ab0fc5c966.xml 2025-12-04T11:13:50.1376602Z ============================= test session starts ============================== 2025-12-04T11:13:50.1376812Z platform linux -- Python 3.10.14, pytest-7.3.2, pluggy-1.6.0 -- /opt/conda/envs/py_3.10/bin/python 2025-12-04T11:13:50.1376879Z cachedir: .pytest_cache 2025-12-04T11:13:50.1377183Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow] 2025-12-04T11:13:50.1377262Z rootdir: /var/lib/jenkins/workspace 2025-12-04T11:13:50.1377328Z configfile: pytest.ini 2025-12-04T11:13:50.1377641Z plugins: hypothesis-6.56.4, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-14.0, subtests-0.13.1, xdist-3.3.1, xdoctest-1.3.0, typeguard-4.3.0 2025-12-04T11:13:50.1377984Z collecting ... collected 58 items / 15 deselected / 43 selected 2025-12-04T11:13:50.1378182Z stepcurrent: skipping 15 already run items. 2025-12-04T11:13:50.1378258Z Running 43 items in this shard 2025-12-04T11:13:50.1378264Z 2025-12-04T11:13:50.1378763Z inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_17_mid_dim_8_in_features_1024_out_features_65_cuda_bfloat16 ('RERUN', {'yellow': True}) [1.9695s] [ 2%] 2025-12-04T11:13:50.1379249Z inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_17_mid_dim_8_in_features_1024_out_features_65_cuda_bfloat16 ('RERUN', {'yellow': True}) [0.5571s] [ 2%] 2025-12-04T11:13:50.1379704Z inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_17_mid_dim_8_in_features_1024_out_features_65_cuda_bfloat16 FAILED [0.5566s] [ 2%] 2025-12-04T11:13:50.1379708Z 2025-12-04T11:13:50.1379793Z ==================================== RERUNS ==================================== 2025-12-04T11:13:50.1380098Z _ TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_8_in_features_1024_out_features_65_cuda_bfloat16 _ 2025-12-04T11:13:50.1380224Z Traceback (most recent call last): 2025-12-04T11:13:50.1380532Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.1380596Z method(*args, **kwargs) 2025-12-04T11:13:50.1380885Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.1380952Z method(*args, **kwargs) 2025-12-04T11:13:50.1381242Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T11:13:50.1381300Z with policy(): 2025-12-04T11:13:50.1381596Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T11:13:50.1381666Z raise RuntimeError(msg) 2025-12-04T11:13:50.1382471Z RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_8_in_features_1024_out_features_65_cuda_bfloat16! Caching allocator allocated memory was 0 and is now reported as 66560 on device 0. CUDA driver allocated memory was 230686720 and is now 274726912. 2025-12-04T11:13:50.1382477Z 2025-12-04T11:13:50.1382602Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.1383123Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_8_in_features_1024_out_features_65_cuda_bfloat16 2025-12-04T11:13:50.1383131Z 2025-12-04T11:13:50.1383288Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.1383415Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.1383564Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.1384139Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('benchmarking.InductorBenchmarker.benchmark_gpu', 2), ('pad_mm_bench', 1), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.1384272Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.1384331Z graph_break [] 2025-12-04T11:13:50.1384625Z _ TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_8_in_features_1024_out_features_65_cuda_bfloat16 _ 2025-12-04T11:13:50.1384704Z Traceback (most recent call last): 2025-12-04T11:13:50.1385003Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.1385069Z method(*args, **kwargs) 2025-12-04T11:13:50.1385422Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.1385630Z method(*args, **kwargs) 2025-12-04T11:13:50.1385921Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T11:13:50.1385981Z with policy(): 2025-12-04T11:13:50.1386271Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T11:13:50.1386340Z raise RuntimeError(msg) 2025-12-04T11:13:50.1387153Z RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_8_in_features_1024_out_features_65_cuda_bfloat16! Caching allocator allocated memory was 66560 and is now reported as 133120 on device 0. CUDA driver allocated memory was 274726912 and is now 276824064. 2025-12-04T11:13:50.1387157Z 2025-12-04T11:13:50.1387284Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.1387808Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_8_in_features_1024_out_features_65_cuda_bfloat16 2025-12-04T11:13:50.1387850Z 2025-12-04T11:13:50.1388006Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.1388140Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.1388232Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.1388783Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('benchmarking.InductorBenchmarker.benchmark_gpu', 2), ('pad_mm_bench', 1), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.1388911Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.1388971Z graph_break [] 2025-12-04T11:13:50.1389100Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.1389192Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.1389320Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.1389864Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('benchmarking.InductorBenchmarker.benchmark_gpu', 2), ('pad_mm_bench', 1), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.1389921Z graph_break [] 2025-12-04T11:13:50.1390011Z =================================== FAILURES =================================== 2025-12-04T11:13:50.1390303Z _ TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_8_in_features_1024_out_features_65_cuda_bfloat16 _ 2025-12-04T11:13:50.1390376Z Traceback (most recent call last): 2025-12-04T11:13:50.1390681Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.1390749Z method(*args, **kwargs) 2025-12-04T11:13:50.1391046Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.1391111Z method(*args, **kwargs) 2025-12-04T11:13:50.1391396Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T11:13:50.1391469Z with policy(): 2025-12-04T11:13:50.1391764Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T11:13:50.1391830Z raise RuntimeError(msg) 2025-12-04T11:13:50.1392650Z RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_8_in_features_1024_out_features_65_cuda_bfloat16! Caching allocator allocated memory was 133120 and is now reported as 199680 on device 0. CUDA driver allocated memory was 276824064 and is now 278921216. 2025-12-04T11:13:50.1392697Z 2025-12-04T11:13:50.1392908Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.1393432Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_8_in_features_1024_out_features_65_cuda_bfloat16 2025-12-04T11:13:50.1393436Z 2025-12-04T11:13:50.1393591Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.1393720Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.1393809Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.1394349Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('benchmarking.InductorBenchmarker.benchmark_gpu', 2), ('pad_mm_bench', 1), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.1394480Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.1394541Z graph_break [] 2025-12-04T11:13:50.1394668Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.1394793Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.1394910Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.1395450Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('benchmarking.InductorBenchmarker.benchmark_gpu', 2), ('pad_mm_bench', 1), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.1395518Z graph_break [] 2025-12-04T11:13:50.1395647Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.1395740Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.1395861Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.1396405Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('benchmarking.InductorBenchmarker.benchmark_gpu', 2), ('pad_mm_bench', 1), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.1396466Z graph_break [] 2025-12-04T11:13:50.1396958Z - generated xml file: /var/lib/jenkins/workspace/test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-d1b821ab0fc5c966.xml - 2025-12-04T11:13:50.1397064Z =========================== short test summary info ============================ 2025-12-04T11:13:50.1398355Z FAILED [0.5566s] inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_17_mid_dim_8_in_features_1024_out_features_65_cuda_bfloat16 - RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_8_in_features_1024_out_features_65_cuda_bfloat16! Caching allocator allocated memory was 133120 and is now reported as 199680 on device 0. CUDA driver allocated memory was 276824064 and is now 278921216. 2025-12-04T11:13:50.1398366Z 2025-12-04T11:13:50.1398492Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.1399010Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_8_in_features_1024_out_features_65_cuda_bfloat16 2025-12-04T11:13:50.1399014Z 2025-12-04T11:13:50.1399174Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.1399279Z !!!!!!!!!!!!!!!!!!!!!!!!!! stopping after 1 failures !!!!!!!!!!!!!!!!!!!!!!!!!!! 2025-12-04T11:13:50.1399393Z ================== 1 failed, 15 deselected, 2 rerun in 3.11s =================== 2025-12-04T11:13:50.1399457Z Got exit code 1 2025-12-04T11:13:50.1399521Z Retrying single test... 2025-12-04T11:13:50.1399838Z W1204 10:39:41.255000 65138 site-packages/torch/_inductor/utils.py:1703] Not enough SMs to use max_autotune_gemm mode 2025-12-04T11:13:50.1400290Z Test results will be stored in test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-d70416164f95a17a.xml 2025-12-04T11:13:50.1400388Z ============================= test session starts ============================== 2025-12-04T11:13:50.1400598Z platform linux -- Python 3.10.14, pytest-7.3.2, pluggy-1.6.0 -- /opt/conda/envs/py_3.10/bin/python 2025-12-04T11:13:50.1400665Z cachedir: .pytest_cache 2025-12-04T11:13:50.1400972Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow] 2025-12-04T11:13:50.1401049Z rootdir: /var/lib/jenkins/workspace 2025-12-04T11:13:50.1401115Z configfile: pytest.ini 2025-12-04T11:13:50.1401432Z plugins: hypothesis-6.56.4, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-14.0, subtests-0.13.1, xdist-3.3.1, xdoctest-1.3.0, typeguard-4.3.0 2025-12-04T11:13:50.1401562Z collecting ... collected 58 items / 57 deselected / 1 selected 2025-12-04T11:13:50.1402135Z stepcurrent: skipping 15 already run items. Running only test/inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_17_mid_dim_8_in_features_1024_out_features_65_cuda_bfloat16 2025-12-04T11:13:50.1402258Z Running 1 items in this shard 2025-12-04T11:13:50.1402262Z 2025-12-04T11:13:50.1402990Z inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_17_mid_dim_8_in_features_1024_out_features_65_cuda_bfloat16 [W1204 10:39:42.905089510 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1402994Z 2025-12-04T11:13:50.1403304Z [W1204 10:39:52.148483347 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1403310Z 2025-12-04T11:13:50.1403604Z [W1204 10:39:52.148734612 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1403607Z 2025-12-04T11:13:50.1403898Z [W1204 10:39:52.154651453 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1403903Z 2025-12-04T11:13:50.1404194Z [W1204 10:39:52.155213695 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1404197Z 2025-12-04T11:13:50.1404487Z [W1204 10:39:52.155390099 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1404490Z 2025-12-04T11:13:50.1404773Z [W1204 10:39:52.160718816 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1404776Z 2025-12-04T11:13:50.1405063Z [W1204 10:39:52.161231307 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1405069Z 2025-12-04T11:13:50.1405355Z [W1204 10:39:52.161395461 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1405360Z 2025-12-04T11:13:50.1405443Z ('RERUN', {'yellow': True}) [11.2367s] [100%] 2025-12-04T11:13:50.1406169Z inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_17_mid_dim_8_in_features_1024_out_features_65_cuda_bfloat16 [W1204 10:39:53.977671383 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1406173Z 2025-12-04T11:13:50.1406464Z [W1204 10:39:53.978207435 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1406467Z 2025-12-04T11:13:50.1406797Z [W1204 10:39:53.978347428 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1406864Z 2025-12-04T11:13:50.1407151Z [W1204 10:39:53.981387135 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1407156Z 2025-12-04T11:13:50.1407445Z [W1204 10:39:53.981864676 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1407448Z 2025-12-04T11:13:50.1407732Z [W1204 10:39:53.982003709 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1407735Z 2025-12-04T11:13:50.1408028Z [W1204 10:39:53.986654362 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1408033Z 2025-12-04T11:13:50.1408324Z [W1204 10:39:53.987130252 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1408330Z 2025-12-04T11:13:50.1408618Z [W1204 10:39:53.987266546 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1408663Z 2025-12-04T11:13:50.1408744Z ('RERUN', {'yellow': True}) [0.5105s] [100%] 2025-12-04T11:13:50.1409463Z inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_17_mid_dim_8_in_features_1024_out_features_65_cuda_bfloat16 [W1204 10:39:53.485046286 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1409467Z 2025-12-04T11:13:50.1409760Z [W1204 10:39:53.485583338 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1409763Z 2025-12-04T11:13:50.1410051Z [W1204 10:39:53.485724411 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1410054Z 2025-12-04T11:13:50.1410345Z [W1204 10:39:53.488739017 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1410350Z 2025-12-04T11:13:50.1410637Z [W1204 10:39:53.489214187 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1410640Z 2025-12-04T11:13:50.1410929Z [W1204 10:39:53.489353100 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1410932Z 2025-12-04T11:13:50.1411217Z [W1204 10:39:53.494058574 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1411220Z 2025-12-04T11:13:50.1411509Z [W1204 10:39:53.494546275 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1411513Z 2025-12-04T11:13:50.1411800Z [W1204 10:39:53.494681958 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1411805Z 2025-12-04T11:13:50.1411865Z FAILED [0.5078s] [100%] 2025-12-04T11:13:50.1411868Z 2025-12-04T11:13:50.1411959Z ==================================== RERUNS ==================================== 2025-12-04T11:13:50.1412251Z _ TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_8_in_features_1024_out_features_65_cuda_bfloat16 _ 2025-12-04T11:13:50.1412328Z Traceback (most recent call last): 2025-12-04T11:13:50.1412633Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.1412697Z method(*args, **kwargs) 2025-12-04T11:13:50.1412993Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.1413096Z method(*args, **kwargs) 2025-12-04T11:13:50.1413447Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T11:13:50.1413520Z with policy(): 2025-12-04T11:13:50.1413816Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T11:13:50.1413884Z raise RuntimeError(msg) 2025-12-04T11:13:50.1414683Z RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_8_in_features_1024_out_features_65_cuda_bfloat16! Caching allocator allocated memory was 0 and is now reported as 66560 on device 0. CUDA driver allocated memory was 230686720 and is now 274726912. 2025-12-04T11:13:50.1414687Z 2025-12-04T11:13:50.1414818Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.1415339Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_8_in_features_1024_out_features_65_cuda_bfloat16 2025-12-04T11:13:50.1415378Z 2025-12-04T11:13:50.1415536Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.1415670Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.1415763Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.1416310Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('benchmarking.InductorBenchmarker.benchmark_gpu', 2), ('pad_mm_bench', 1), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.1416446Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.1416506Z graph_break [] 2025-12-04T11:13:50.1416635Z ----------------------------- Captured stderr call ----------------------------- 2025-12-04T11:13:50.1417324Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/constant_folding.py:256: UserWarning: Unsupported unwinding pattern: Address not in range (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/profiler/unwind/unwind.cpp:219.) 2025-12-04T11:13:50.1417399Z if out == self.unknown_value: 2025-12-04T11:13:50.1417690Z _ TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_8_in_features_1024_out_features_65_cuda_bfloat16 _ 2025-12-04T11:13:50.1417767Z Traceback (most recent call last): 2025-12-04T11:13:50.1418062Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.1418125Z method(*args, **kwargs) 2025-12-04T11:13:50.1418413Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.1418481Z method(*args, **kwargs) 2025-12-04T11:13:50.1418769Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T11:13:50.1418833Z with policy(): 2025-12-04T11:13:50.1419126Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T11:13:50.1419201Z raise RuntimeError(msg) 2025-12-04T11:13:50.1420011Z RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_8_in_features_1024_out_features_65_cuda_bfloat16! Caching allocator allocated memory was 66560 and is now reported as 133120 on device 0. CUDA driver allocated memory was 274726912 and is now 276824064. 2025-12-04T11:13:50.1420015Z 2025-12-04T11:13:50.1420142Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.1420663Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_8_in_features_1024_out_features_65_cuda_bfloat16 2025-12-04T11:13:50.1420789Z 2025-12-04T11:13:50.1420946Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.1421072Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.1421168Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.1421711Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('benchmarking.InductorBenchmarker.benchmark_gpu', 2), ('pad_mm_bench', 1), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.1421843Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.1421902Z graph_break [] 2025-12-04T11:13:50.1422025Z ----------------------------- Captured stderr call ----------------------------- 2025-12-04T11:13:50.1422718Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/constant_folding.py:256: UserWarning: Unsupported unwinding pattern: Address not in range (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/profiler/unwind/unwind.cpp:219.) 2025-12-04T11:13:50.1422834Z if out == self.unknown_value: 2025-12-04T11:13:50.1422962Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.1423051Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.1423174Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.1423763Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('benchmarking.InductorBenchmarker.benchmark_gpu', 2), ('pad_mm_bench', 1), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.1423825Z graph_break [] 2025-12-04T11:13:50.1423912Z =================================== FAILURES =================================== 2025-12-04T11:13:50.1424206Z _ TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_8_in_features_1024_out_features_65_cuda_bfloat16 _ 2025-12-04T11:13:50.1424280Z Traceback (most recent call last): 2025-12-04T11:13:50.1424581Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.1424645Z method(*args, **kwargs) 2025-12-04T11:13:50.1424932Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.1425003Z method(*args, **kwargs) 2025-12-04T11:13:50.1425290Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T11:13:50.1425355Z with policy(): 2025-12-04T11:13:50.1425644Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T11:13:50.1425711Z raise RuntimeError(msg) 2025-12-04T11:13:50.1426530Z RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_8_in_features_1024_out_features_65_cuda_bfloat16! Caching allocator allocated memory was 133120 and is now reported as 199680 on device 0. CUDA driver allocated memory was 276824064 and is now 278921216. 2025-12-04T11:13:50.1426535Z 2025-12-04T11:13:50.1426661Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.1427180Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_8_in_features_1024_out_features_65_cuda_bfloat16 2025-12-04T11:13:50.1427184Z 2025-12-04T11:13:50.1427337Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.1427469Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.1427606Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.1428212Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('benchmarking.InductorBenchmarker.benchmark_gpu', 2), ('pad_mm_bench', 1), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.1428342Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.1428400Z graph_break [] 2025-12-04T11:13:50.1428523Z ----------------------------- Captured stderr call ----------------------------- 2025-12-04T11:13:50.1429207Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/constant_folding.py:256: UserWarning: Unsupported unwinding pattern: Address not in range (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/profiler/unwind/unwind.cpp:219.) 2025-12-04T11:13:50.1429277Z if out == self.unknown_value: 2025-12-04T11:13:50.1429405Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.1429497Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.1429622Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.1430202Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('benchmarking.InductorBenchmarker.benchmark_gpu', 2), ('pad_mm_bench', 1), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.1430260Z graph_break [] 2025-12-04T11:13:50.1430385Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.1430474Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.1430596Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.1431132Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('benchmarking.InductorBenchmarker.benchmark_gpu', 2), ('pad_mm_bench', 1), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.1431193Z graph_break [] 2025-12-04T11:13:50.1431682Z - generated xml file: /var/lib/jenkins/workspace/test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-d70416164f95a17a.xml - 2025-12-04T11:13:50.1431784Z =========================== short test summary info ============================ 2025-12-04T11:13:50.1433081Z FAILED [0.5078s] inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_17_mid_dim_8_in_features_1024_out_features_65_cuda_bfloat16 - RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_8_in_features_1024_out_features_65_cuda_bfloat16! Caching allocator allocated memory was 133120 and is now reported as 199680 on device 0. CUDA driver allocated memory was 276824064 and is now 278921216. 2025-12-04T11:13:50.1433092Z 2025-12-04T11:13:50.1433217Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.1433739Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_8_in_features_1024_out_features_65_cuda_bfloat16 2025-12-04T11:13:50.1433744Z 2025-12-04T11:13:50.1433900Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.1434003Z !!!!!!!!!!!!!!!!!!!!!!!!!! stopping after 1 failures !!!!!!!!!!!!!!!!!!!!!!!!!!! 2025-12-04T11:13:50.1434119Z ================== 1 failed, 57 deselected, 2 rerun in 12.28s ================== 2025-12-04T11:13:50.1434179Z Got exit code 1 2025-12-04T11:13:50.1434243Z Retrying single test... 2025-12-04T11:13:50.1434509Z W1204 10:40:00.656000 65332 site-packages/torch/_inductor/utils.py:1703] Not enough SMs to use max_autotune_gemm mode 2025-12-04T11:13:50.1434890Z Test results will be stored in test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-a1832326f6932688.xml 2025-12-04T11:13:50.1435090Z ============================= test session starts ============================== 2025-12-04T11:13:50.1435299Z platform linux -- Python 3.10.14, pytest-7.3.2, pluggy-1.6.0 -- /opt/conda/envs/py_3.10/bin/python 2025-12-04T11:13:50.1435368Z cachedir: .pytest_cache 2025-12-04T11:13:50.1435675Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow] 2025-12-04T11:13:50.1435749Z rootdir: /var/lib/jenkins/workspace 2025-12-04T11:13:50.1435815Z configfile: pytest.ini 2025-12-04T11:13:50.1436135Z plugins: hypothesis-6.56.4, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-14.0, subtests-0.13.1, xdist-3.3.1, xdoctest-1.3.0, typeguard-4.3.0 2025-12-04T11:13:50.1436263Z collecting ... collected 58 items / 57 deselected / 1 selected 2025-12-04T11:13:50.1436835Z stepcurrent: skipping 15 already run items. Running only test/inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_17_mid_dim_8_in_features_1024_out_features_65_cuda_bfloat16 2025-12-04T11:13:50.1436908Z Running 1 items in this shard 2025-12-04T11:13:50.1436912Z 2025-12-04T11:13:50.1437676Z inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_17_mid_dim_8_in_features_1024_out_features_65_cuda_bfloat16 [W1204 10:40:02.291157508 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1437684Z 2025-12-04T11:13:50.1437981Z [W1204 10:40:11.347210771 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1437985Z 2025-12-04T11:13:50.1438273Z [W1204 10:40:11.347451666 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1438277Z 2025-12-04T11:13:50.1438567Z [W1204 10:40:11.353404278 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1438571Z 2025-12-04T11:13:50.1438862Z [W1204 10:40:11.353967660 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1438867Z 2025-12-04T11:13:50.1439153Z [W1204 10:40:11.354151024 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1439156Z 2025-12-04T11:13:50.1439447Z [W1204 10:40:11.359568334 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1439450Z 2025-12-04T11:13:50.1439739Z [W1204 10:40:11.360110566 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1439742Z 2025-12-04T11:13:50.1440029Z [W1204 10:40:11.360277150 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1440034Z 2025-12-04T11:13:50.1440116Z ('RERUN', {'yellow': True}) [11.0374s] [100%] 2025-12-04T11:13:50.1440843Z inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_17_mid_dim_8_in_features_1024_out_features_65_cuda_bfloat16 [W1204 10:40:12.174229126 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1440849Z 2025-12-04T11:13:50.1441138Z [W1204 10:40:12.174752178 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1441141Z 2025-12-04T11:13:50.1441429Z [W1204 10:40:12.174892361 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1441433Z 2025-12-04T11:13:50.1441718Z [W1204 10:40:12.177848197 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1441763Z 2025-12-04T11:13:50.1442118Z [W1204 10:40:12.178312887 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1442123Z 2025-12-04T11:13:50.1442407Z [W1204 10:40:12.178447770 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1442410Z 2025-12-04T11:13:50.1442696Z [W1204 10:40:12.183014381 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1442700Z 2025-12-04T11:13:50.1442984Z [W1204 10:40:12.183481971 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1442987Z 2025-12-04T11:13:50.1443276Z [W1204 10:40:12.183618204 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1443281Z 2025-12-04T11:13:50.1443360Z ('RERUN', {'yellow': True}) [0.5065s] [100%] 2025-12-04T11:13:50.1444078Z inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_17_mid_dim_8_in_features_1024_out_features_65_cuda_bfloat16 [W1204 10:40:12.680338819 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1444122Z 2025-12-04T11:13:50.1444410Z [W1204 10:40:12.680871571 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1444414Z 2025-12-04T11:13:50.1444699Z [W1204 10:40:12.681014594 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1444702Z 2025-12-04T11:13:50.1444988Z [W1204 10:40:12.683942339 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1444993Z 2025-12-04T11:13:50.1445281Z [W1204 10:40:12.684406399 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1445287Z 2025-12-04T11:13:50.1445576Z [W1204 10:40:12.684543092 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1445579Z 2025-12-04T11:13:50.1445862Z [W1204 10:40:12.689067882 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1445866Z 2025-12-04T11:13:50.1446165Z [W1204 10:40:12.689538323 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1446168Z 2025-12-04T11:13:50.1446453Z [W1204 10:40:12.689673596 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1446458Z 2025-12-04T11:13:50.1446525Z FAILED [0.5067s] [100%] 2025-12-04T11:13:50.1446531Z 2025-12-04T11:13:50.1446616Z ==================================== RERUNS ==================================== 2025-12-04T11:13:50.1446909Z _ TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_8_in_features_1024_out_features_65_cuda_bfloat16 _ 2025-12-04T11:13:50.1446987Z Traceback (most recent call last): 2025-12-04T11:13:50.1447291Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.1447356Z method(*args, **kwargs) 2025-12-04T11:13:50.1447652Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.1447716Z method(*args, **kwargs) 2025-12-04T11:13:50.1448009Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T11:13:50.1448126Z with policy(): 2025-12-04T11:13:50.1448483Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T11:13:50.1448558Z raise RuntimeError(msg) 2025-12-04T11:13:50.1449364Z RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_8_in_features_1024_out_features_65_cuda_bfloat16! Caching allocator allocated memory was 0 and is now reported as 66560 on device 0. CUDA driver allocated memory was 230686720 and is now 274726912. 2025-12-04T11:13:50.1449368Z 2025-12-04T11:13:50.1449499Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.1450019Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_8_in_features_1024_out_features_65_cuda_bfloat16 2025-12-04T11:13:50.1450024Z 2025-12-04T11:13:50.1450179Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.1450313Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.1450442Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.1450990Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('benchmarking.InductorBenchmarker.benchmark_gpu', 2), ('pad_mm_bench', 1), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.1451120Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.1451180Z graph_break [] 2025-12-04T11:13:50.1451307Z ----------------------------- Captured stderr call ----------------------------- 2025-12-04T11:13:50.1451998Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/constant_folding.py:256: UserWarning: Unsupported unwinding pattern: Address not in range (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/profiler/unwind/unwind.cpp:219.) 2025-12-04T11:13:50.1452077Z if out == self.unknown_value: 2025-12-04T11:13:50.1452368Z _ TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_8_in_features_1024_out_features_65_cuda_bfloat16 _ 2025-12-04T11:13:50.1452443Z Traceback (most recent call last): 2025-12-04T11:13:50.1452746Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.1452810Z method(*args, **kwargs) 2025-12-04T11:13:50.1453103Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.1453166Z method(*args, **kwargs) 2025-12-04T11:13:50.1453452Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T11:13:50.1453513Z with policy(): 2025-12-04T11:13:50.1453805Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T11:13:50.1453876Z raise RuntimeError(msg) 2025-12-04T11:13:50.1454703Z RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_8_in_features_1024_out_features_65_cuda_bfloat16! Caching allocator allocated memory was 66560 and is now reported as 133120 on device 0. CUDA driver allocated memory was 274726912 and is now 276824064. 2025-12-04T11:13:50.1454708Z 2025-12-04T11:13:50.1454834Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.1455358Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_8_in_features_1024_out_features_65_cuda_bfloat16 2025-12-04T11:13:50.1455361Z 2025-12-04T11:13:50.1455515Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.1455690Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.1455853Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.1456398Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('benchmarking.InductorBenchmarker.benchmark_gpu', 2), ('pad_mm_bench', 1), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.1456527Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.1456586Z graph_break [] 2025-12-04T11:13:50.1456712Z ----------------------------- Captured stderr call ----------------------------- 2025-12-04T11:13:50.1457393Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/constant_folding.py:256: UserWarning: Unsupported unwinding pattern: Address not in range (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/profiler/unwind/unwind.cpp:219.) 2025-12-04T11:13:50.1457462Z if out == self.unknown_value: 2025-12-04T11:13:50.1457588Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.1457677Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.1457839Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.1458378Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('benchmarking.InductorBenchmarker.benchmark_gpu', 2), ('pad_mm_bench', 1), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.1458437Z graph_break [] 2025-12-04T11:13:50.1458524Z =================================== FAILURES =================================== 2025-12-04T11:13:50.1458813Z _ TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_8_in_features_1024_out_features_65_cuda_bfloat16 _ 2025-12-04T11:13:50.1458887Z Traceback (most recent call last): 2025-12-04T11:13:50.1459186Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.1459252Z method(*args, **kwargs) 2025-12-04T11:13:50.1459544Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.1459612Z method(*args, **kwargs) 2025-12-04T11:13:50.1459896Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T11:13:50.1459968Z with policy(): 2025-12-04T11:13:50.1460265Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T11:13:50.1460332Z raise RuntimeError(msg) 2025-12-04T11:13:50.1461150Z RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_8_in_features_1024_out_features_65_cuda_bfloat16! Caching allocator allocated memory was 133120 and is now reported as 199680 on device 0. CUDA driver allocated memory was 276824064 and is now 278921216. 2025-12-04T11:13:50.1461156Z 2025-12-04T11:13:50.1461281Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.1461805Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_8_in_features_1024_out_features_65_cuda_bfloat16 2025-12-04T11:13:50.1461808Z 2025-12-04T11:13:50.1461965Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.1462098Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.1462188Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.1462726Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('benchmarking.InductorBenchmarker.benchmark_gpu', 2), ('pad_mm_bench', 1), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.1462961Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.1463020Z graph_break [] 2025-12-04T11:13:50.1463150Z ----------------------------- Captured stderr call ----------------------------- 2025-12-04T11:13:50.1463911Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/constant_folding.py:256: UserWarning: Unsupported unwinding pattern: Address not in range (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/profiler/unwind/unwind.cpp:219.) 2025-12-04T11:13:50.1463981Z if out == self.unknown_value: 2025-12-04T11:13:50.1464110Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.1464198Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.1464320Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.1464867Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('benchmarking.InductorBenchmarker.benchmark_gpu', 2), ('pad_mm_bench', 1), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.1464965Z graph_break [] 2025-12-04T11:13:50.1465093Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.1465181Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.1465304Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.1465846Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('benchmarking.InductorBenchmarker.benchmark_gpu', 2), ('pad_mm_bench', 1), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.1465904Z graph_break [] 2025-12-04T11:13:50.1466391Z - generated xml file: /var/lib/jenkins/workspace/test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-a1832326f6932688.xml - 2025-12-04T11:13:50.1466494Z =========================== short test summary info ============================ 2025-12-04T11:13:50.1467786Z FAILED [0.5067s] inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_17_mid_dim_8_in_features_1024_out_features_65_cuda_bfloat16 - RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_8_in_features_1024_out_features_65_cuda_bfloat16! Caching allocator allocated memory was 133120 and is now reported as 199680 on device 0. CUDA driver allocated memory was 276824064 and is now 278921216. 2025-12-04T11:13:50.1467791Z 2025-12-04T11:13:50.1467914Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.1468431Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_8_in_features_1024_out_features_65_cuda_bfloat16 2025-12-04T11:13:50.1468440Z 2025-12-04T11:13:50.1468596Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.1468699Z !!!!!!!!!!!!!!!!!!!!!!!!!! stopping after 1 failures !!!!!!!!!!!!!!!!!!!!!!!!!!! 2025-12-04T11:13:50.1468818Z ================== 1 failed, 57 deselected, 2 rerun in 12.08s ================== 2025-12-04T11:13:50.1468878Z Got exit code 1 2025-12-04T11:13:50.1469349Z FAILED CONSISTENTLY: test/inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_17_mid_dim_8_in_features_1024_out_features_65_cuda_bfloat16 2025-12-04T11:13:50.1469594Z Test failed consistently, continuing with the rest of the tests due to continue-through-error being set 2025-12-04T11:13:50.1469868Z W1204 10:40:19.482000 65526 site-packages/torch/_inductor/utils.py:1703] Not enough SMs to use max_autotune_gemm mode 2025-12-04T11:13:50.1470250Z Test results will be stored in test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-1965a836c7535ef2.xml 2025-12-04T11:13:50.1470466Z ============================= test session starts ============================== 2025-12-04T11:13:50.1470674Z platform linux -- Python 3.10.14, pytest-7.3.2, pluggy-1.6.0 -- /opt/conda/envs/py_3.10/bin/python 2025-12-04T11:13:50.1470746Z cachedir: .pytest_cache 2025-12-04T11:13:50.1471051Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow] 2025-12-04T11:13:50.1471131Z rootdir: /var/lib/jenkins/workspace 2025-12-04T11:13:50.1471198Z configfile: pytest.ini 2025-12-04T11:13:50.1471515Z plugins: hypothesis-6.56.4, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-14.0, subtests-0.13.1, xdist-3.3.1, xdoctest-1.3.0, typeguard-4.3.0 2025-12-04T11:13:50.1471648Z collecting ... collected 58 items / 16 deselected / 42 selected 2025-12-04T11:13:50.1471735Z stepcurrent: skipping 16 already run items. 2025-12-04T11:13:50.1471808Z Running 42 items in this shard 2025-12-04T11:13:50.1471811Z 2025-12-04T11:13:50.1472321Z inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_17_mid_dim_8_in_features_128_out_features_1024_cuda_bfloat16 ('RERUN', {'yellow': True}) [1.9288s] [ 2%] 2025-12-04T11:13:50.1472868Z inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_17_mid_dim_8_in_features_128_out_features_1024_cuda_bfloat16 ('RERUN', {'yellow': True}) [0.4896s] [ 2%] 2025-12-04T11:13:50.1473324Z inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_17_mid_dim_8_in_features_128_out_features_1024_cuda_bfloat16 FAILED [0.4720s] [ 2%] 2025-12-04T11:13:50.1473328Z 2025-12-04T11:13:50.1473412Z ==================================== RERUNS ==================================== 2025-12-04T11:13:50.1473708Z _ TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_8_in_features_128_out_features_1024_cuda_bfloat16 _ 2025-12-04T11:13:50.1473785Z Traceback (most recent call last): 2025-12-04T11:13:50.1474091Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.1474163Z method(*args, **kwargs) 2025-12-04T11:13:50.1474454Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.1474517Z method(*args, **kwargs) 2025-12-04T11:13:50.1474807Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T11:13:50.1474868Z with policy(): 2025-12-04T11:13:50.1475163Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T11:13:50.1475229Z raise RuntimeError(msg) 2025-12-04T11:13:50.1476036Z RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_8_in_features_128_out_features_1024_cuda_bfloat16! Caching allocator allocated memory was 0 and is now reported as 131072 on device 0. CUDA driver allocated memory was 230686720 and is now 274726912. 2025-12-04T11:13:50.1476046Z 2025-12-04T11:13:50.1476171Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.1476693Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_8_in_features_128_out_features_1024_cuda_bfloat16 2025-12-04T11:13:50.1476698Z 2025-12-04T11:13:50.1476857Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.1476985Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.1477085Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.1477433Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.1477663Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.1477730Z graph_break [] 2025-12-04T11:13:50.1478335Z _ TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_8_in_features_128_out_features_1024_cuda_bfloat16 _ 2025-12-04T11:13:50.1478421Z Traceback (most recent call last): 2025-12-04T11:13:50.1478733Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.1478797Z method(*args, **kwargs) 2025-12-04T11:13:50.1479091Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.1479154Z method(*args, **kwargs) 2025-12-04T11:13:50.1479452Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T11:13:50.1479521Z with policy(): 2025-12-04T11:13:50.1479816Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T11:13:50.1479965Z raise RuntimeError(msg) 2025-12-04T11:13:50.1480793Z RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_8_in_features_128_out_features_1024_cuda_bfloat16! Caching allocator allocated memory was 131072 and is now reported as 262144 on device 0. CUDA driver allocated memory was 274726912 and is now 276824064. 2025-12-04T11:13:50.1480799Z 2025-12-04T11:13:50.1480929Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.1481455Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_8_in_features_128_out_features_1024_cuda_bfloat16 2025-12-04T11:13:50.1481461Z 2025-12-04T11:13:50.1481623Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.1481754Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.1481848Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.1482199Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.1482327Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.1482386Z graph_break [] 2025-12-04T11:13:50.1482514Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.1482605Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.1482729Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.1483075Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.1483134Z graph_break [] 2025-12-04T11:13:50.1483217Z =================================== FAILURES =================================== 2025-12-04T11:13:50.1483514Z _ TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_8_in_features_128_out_features_1024_cuda_bfloat16 _ 2025-12-04T11:13:50.1483588Z Traceback (most recent call last): 2025-12-04T11:13:50.1483898Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.1483971Z method(*args, **kwargs) 2025-12-04T11:13:50.1484263Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.1484330Z method(*args, **kwargs) 2025-12-04T11:13:50.1484617Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T11:13:50.1484740Z with policy(): 2025-12-04T11:13:50.1485135Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T11:13:50.1485204Z raise RuntimeError(msg) 2025-12-04T11:13:50.1486039Z RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_8_in_features_128_out_features_1024_cuda_bfloat16! Caching allocator allocated memory was 262144 and is now reported as 393216 on device 0. CUDA driver allocated memory was 276824064 and is now 278921216. 2025-12-04T11:13:50.1486043Z 2025-12-04T11:13:50.1486170Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.1486692Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_8_in_features_128_out_features_1024_cuda_bfloat16 2025-12-04T11:13:50.1486698Z 2025-12-04T11:13:50.1486856Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.1486981Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.1487115Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.1487460Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.1487586Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.1487645Z graph_break [] 2025-12-04T11:13:50.1487769Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.1487864Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.1487983Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.1488327Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.1488399Z graph_break [] 2025-12-04T11:13:50.1488526Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.1488618Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.1488741Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.1489077Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.1489139Z graph_break [] 2025-12-04T11:13:50.1489633Z - generated xml file: /var/lib/jenkins/workspace/test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-1965a836c7535ef2.xml - 2025-12-04T11:13:50.1489733Z =========================== short test summary info ============================ 2025-12-04T11:13:50.1491061Z FAILED [0.4720s] inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_17_mid_dim_8_in_features_128_out_features_1024_cuda_bfloat16 - RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_8_in_features_128_out_features_1024_cuda_bfloat16! Caching allocator allocated memory was 262144 and is now reported as 393216 on device 0. CUDA driver allocated memory was 276824064 and is now 278921216. 2025-12-04T11:13:50.1491068Z 2025-12-04T11:13:50.1491194Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.1491716Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_8_in_features_128_out_features_1024_cuda_bfloat16 2025-12-04T11:13:50.1491720Z 2025-12-04T11:13:50.1491919Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.1492093Z !!!!!!!!!!!!!!!!!!!!!!!!!! stopping after 1 failures !!!!!!!!!!!!!!!!!!!!!!!!!!! 2025-12-04T11:13:50.1492210Z ================== 1 failed, 16 deselected, 2 rerun in 2.91s =================== 2025-12-04T11:13:50.1492272Z Got exit code 1 2025-12-04T11:13:50.1492344Z Retrying single test... 2025-12-04T11:13:50.1492607Z W1204 10:40:29.608000 65714 site-packages/torch/_inductor/utils.py:1703] Not enough SMs to use max_autotune_gemm mode 2025-12-04T11:13:50.1492990Z Test results will be stored in test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-12f95c060627756e.xml 2025-12-04T11:13:50.1493085Z ============================= test session starts ============================== 2025-12-04T11:13:50.1493290Z platform linux -- Python 3.10.14, pytest-7.3.2, pluggy-1.6.0 -- /opt/conda/envs/py_3.10/bin/python 2025-12-04T11:13:50.1493362Z cachedir: .pytest_cache 2025-12-04T11:13:50.1493665Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow] 2025-12-04T11:13:50.1493749Z rootdir: /var/lib/jenkins/workspace 2025-12-04T11:13:50.1493818Z configfile: pytest.ini 2025-12-04T11:13:50.1494185Z plugins: hypothesis-6.56.4, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-14.0, subtests-0.13.1, xdist-3.3.1, xdoctest-1.3.0, typeguard-4.3.0 2025-12-04T11:13:50.1494318Z collecting ... collected 58 items / 57 deselected / 1 selected 2025-12-04T11:13:50.1494891Z stepcurrent: skipping 16 already run items. Running only test/inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_17_mid_dim_8_in_features_128_out_features_1024_cuda_bfloat16 2025-12-04T11:13:50.1494963Z Running 1 items in this shard 2025-12-04T11:13:50.1494970Z 2025-12-04T11:13:50.1495705Z inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_17_mid_dim_8_in_features_128_out_features_1024_cuda_bfloat16 [W1204 10:40:30.740600575 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1495711Z 2025-12-04T11:13:50.1496012Z [W1204 10:40:39.808772795 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1496020Z 2025-12-04T11:13:50.1496309Z [W1204 10:40:39.809008131 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1496312Z 2025-12-04T11:13:50.1496599Z [W1204 10:40:39.814736387 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1496602Z 2025-12-04T11:13:50.1496892Z [W1204 10:40:39.815292220 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1496895Z 2025-12-04T11:13:50.1497181Z [W1204 10:40:39.815458333 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1497186Z 2025-12-04T11:13:50.1497476Z [W1204 10:40:39.820848993 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1497481Z 2025-12-04T11:13:50.1497768Z [W1204 10:40:39.821368594 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1497771Z 2025-12-04T11:13:50.1498057Z [W1204 10:40:39.821525508 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1498060Z 2025-12-04T11:13:50.1498142Z ('RERUN', {'yellow': True}) [11.0006s] [100%] 2025-12-04T11:13:50.1498868Z inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_17_mid_dim_8_in_features_128_out_features_1024_cuda_bfloat16 [W1204 10:40:41.056114526 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1499076Z 2025-12-04T11:13:50.1499379Z [W1204 10:40:41.056680078 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1499384Z 2025-12-04T11:13:50.1499730Z [W1204 10:40:41.056825351 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1499733Z 2025-12-04T11:13:50.1500042Z [W1204 10:40:41.059824818 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1500046Z 2025-12-04T11:13:50.1500334Z [W1204 10:40:41.060429611 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1500336Z 2025-12-04T11:13:50.1500626Z [W1204 10:40:41.060580385 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1500631Z 2025-12-04T11:13:50.1500918Z [W1204 10:40:41.065268879 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1500968Z 2025-12-04T11:13:50.1501261Z [W1204 10:40:41.065745550 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1501264Z 2025-12-04T11:13:50.1501546Z [W1204 10:40:41.065883943 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1501549Z 2025-12-04T11:13:50.1501634Z ('RERUN', {'yellow': True}) [0.4663s] [100%] 2025-12-04T11:13:50.1502356Z inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_17_mid_dim_8_in_features_128_out_features_1024_cuda_bfloat16 [W1204 10:40:41.514758692 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1502364Z 2025-12-04T11:13:50.1502653Z [W1204 10:40:41.515303154 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1502661Z 2025-12-04T11:13:50.1502945Z [W1204 10:40:41.515445187 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1502949Z 2025-12-04T11:13:50.1503235Z [W1204 10:40:41.518424853 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1503238Z 2025-12-04T11:13:50.1503567Z [W1204 10:40:41.518995416 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1503571Z 2025-12-04T11:13:50.1503857Z [W1204 10:40:41.519134229 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1503862Z 2025-12-04T11:13:50.1504153Z [W1204 10:40:41.523735842 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1504159Z 2025-12-04T11:13:50.1504446Z [W1204 10:40:41.524218332 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1504449Z 2025-12-04T11:13:50.1504752Z [W1204 10:40:41.524354666 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1504755Z 2025-12-04T11:13:50.1504817Z FAILED [0.4546s] [100%] 2025-12-04T11:13:50.1504820Z 2025-12-04T11:13:50.1504904Z ==================================== RERUNS ==================================== 2025-12-04T11:13:50.1505203Z _ TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_8_in_features_128_out_features_1024_cuda_bfloat16 _ 2025-12-04T11:13:50.1505323Z Traceback (most recent call last): 2025-12-04T11:13:50.1505694Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.1505763Z method(*args, **kwargs) 2025-12-04T11:13:50.1506056Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.1506125Z method(*args, **kwargs) 2025-12-04T11:13:50.1506412Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T11:13:50.1506474Z with policy(): 2025-12-04T11:13:50.1506767Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T11:13:50.1506833Z raise RuntimeError(msg) 2025-12-04T11:13:50.1507641Z RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_8_in_features_128_out_features_1024_cuda_bfloat16! Caching allocator allocated memory was 0 and is now reported as 131072 on device 0. CUDA driver allocated memory was 230686720 and is now 274726912. 2025-12-04T11:13:50.1507682Z 2025-12-04T11:13:50.1507812Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.1508339Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_8_in_features_128_out_features_1024_cuda_bfloat16 2025-12-04T11:13:50.1508343Z 2025-12-04T11:13:50.1508499Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.1508626Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.1508724Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.1509075Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.1509213Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.1509274Z graph_break [] 2025-12-04T11:13:50.1509403Z ----------------------------- Captured stderr call ----------------------------- 2025-12-04T11:13:50.1510097Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/constant_folding.py:256: UserWarning: Unsupported unwinding pattern: Address not in range (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/profiler/unwind/unwind.cpp:219.) 2025-12-04T11:13:50.1510168Z if out == self.unknown_value: 2025-12-04T11:13:50.1510464Z _ TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_8_in_features_128_out_features_1024_cuda_bfloat16 _ 2025-12-04T11:13:50.1510538Z Traceback (most recent call last): 2025-12-04T11:13:50.1510831Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.1510900Z method(*args, **kwargs) 2025-12-04T11:13:50.1511204Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.1511272Z method(*args, **kwargs) 2025-12-04T11:13:50.1511564Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T11:13:50.1511623Z with policy(): 2025-12-04T11:13:50.1511918Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T11:13:50.1511985Z raise RuntimeError(msg) 2025-12-04T11:13:50.1512806Z RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_8_in_features_128_out_features_1024_cuda_bfloat16! Caching allocator allocated memory was 131072 and is now reported as 262144 on device 0. CUDA driver allocated memory was 274726912 and is now 276824064. 2025-12-04T11:13:50.1512851Z 2025-12-04T11:13:50.1513048Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.1513570Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_8_in_features_128_out_features_1024_cuda_bfloat16 2025-12-04T11:13:50.1513573Z 2025-12-04T11:13:50.1513732Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.1513858Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.1513950Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.1514303Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.1514431Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.1514494Z graph_break [] 2025-12-04T11:13:50.1514621Z ----------------------------- Captured stderr call ----------------------------- 2025-12-04T11:13:50.1515365Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/constant_folding.py:256: UserWarning: Unsupported unwinding pattern: Address not in range (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/profiler/unwind/unwind.cpp:219.) 2025-12-04T11:13:50.1515441Z if out == self.unknown_value: 2025-12-04T11:13:50.1515562Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.1515657Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.1515780Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.1516127Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.1516190Z graph_break [] 2025-12-04T11:13:50.1516276Z =================================== FAILURES =================================== 2025-12-04T11:13:50.1516571Z _ TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_8_in_features_128_out_features_1024_cuda_bfloat16 _ 2025-12-04T11:13:50.1516651Z Traceback (most recent call last): 2025-12-04T11:13:50.1516947Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.1517016Z method(*args, **kwargs) 2025-12-04T11:13:50.1517305Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.1517371Z method(*args, **kwargs) 2025-12-04T11:13:50.1517665Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T11:13:50.1517726Z with policy(): 2025-12-04T11:13:50.1518020Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T11:13:50.1518090Z raise RuntimeError(msg) 2025-12-04T11:13:50.1518916Z RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_8_in_features_128_out_features_1024_cuda_bfloat16! Caching allocator allocated memory was 262144 and is now reported as 393216 on device 0. CUDA driver allocated memory was 276824064 and is now 278921216. 2025-12-04T11:13:50.1518920Z 2025-12-04T11:13:50.1519049Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.1519574Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_8_in_features_128_out_features_1024_cuda_bfloat16 2025-12-04T11:13:50.1519578Z 2025-12-04T11:13:50.1519783Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.1519989Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.1520083Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.1520434Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.1520555Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.1520615Z graph_break [] 2025-12-04T11:13:50.1520737Z ----------------------------- Captured stderr call ----------------------------- 2025-12-04T11:13:50.1521421Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/constant_folding.py:256: UserWarning: Unsupported unwinding pattern: Address not in range (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/profiler/unwind/unwind.cpp:219.) 2025-12-04T11:13:50.1521496Z if out == self.unknown_value: 2025-12-04T11:13:50.1521621Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.1521714Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.1521879Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.1522219Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.1522285Z graph_break [] 2025-12-04T11:13:50.1522406Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.1522493Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.1522616Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.1522955Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.1523016Z graph_break [] 2025-12-04T11:13:50.1523500Z - generated xml file: /var/lib/jenkins/workspace/test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-12f95c060627756e.xml - 2025-12-04T11:13:50.1523602Z =========================== short test summary info ============================ 2025-12-04T11:13:50.1524903Z FAILED [0.4546s] inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_17_mid_dim_8_in_features_128_out_features_1024_cuda_bfloat16 - RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_8_in_features_128_out_features_1024_cuda_bfloat16! Caching allocator allocated memory was 262144 and is now reported as 393216 on device 0. CUDA driver allocated memory was 276824064 and is now 278921216. 2025-12-04T11:13:50.1524908Z 2025-12-04T11:13:50.1525029Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.1525555Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_8_in_features_128_out_features_1024_cuda_bfloat16 2025-12-04T11:13:50.1525560Z 2025-12-04T11:13:50.1525714Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.1525819Z !!!!!!!!!!!!!!!!!!!!!!!!!! stopping after 1 failures !!!!!!!!!!!!!!!!!!!!!!!!!!! 2025-12-04T11:13:50.1525935Z ================== 1 failed, 57 deselected, 2 rerun in 11.95s ================== 2025-12-04T11:13:50.1525993Z Got exit code 1 2025-12-04T11:13:50.1526062Z Retrying single test... 2025-12-04T11:13:50.1526335Z W1204 10:40:48.336000 65907 site-packages/torch/_inductor/utils.py:1703] Not enough SMs to use max_autotune_gemm mode 2025-12-04T11:13:50.1526721Z Test results will be stored in test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-c82b9fa27197518a.xml 2025-12-04T11:13:50.1526860Z ============================= test session starts ============================== 2025-12-04T11:13:50.1527149Z platform linux -- Python 3.10.14, pytest-7.3.2, pluggy-1.6.0 -- /opt/conda/envs/py_3.10/bin/python 2025-12-04T11:13:50.1527224Z cachedir: .pytest_cache 2025-12-04T11:13:50.1527526Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow] 2025-12-04T11:13:50.1527601Z rootdir: /var/lib/jenkins/workspace 2025-12-04T11:13:50.1527671Z configfile: pytest.ini 2025-12-04T11:13:50.1527985Z plugins: hypothesis-6.56.4, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-14.0, subtests-0.13.1, xdist-3.3.1, xdoctest-1.3.0, typeguard-4.3.0 2025-12-04T11:13:50.1528113Z collecting ... collected 58 items / 57 deselected / 1 selected 2025-12-04T11:13:50.1528695Z stepcurrent: skipping 16 already run items. Running only test/inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_17_mid_dim_8_in_features_128_out_features_1024_cuda_bfloat16 2025-12-04T11:13:50.1528768Z Running 1 items in this shard 2025-12-04T11:13:50.1528774Z 2025-12-04T11:13:50.1529509Z inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_17_mid_dim_8_in_features_128_out_features_1024_cuda_bfloat16 [W1204 10:40:49.451001619 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1529550Z 2025-12-04T11:13:50.1529847Z [W1204 10:40:58.322716622 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1529851Z 2025-12-04T11:13:50.1530141Z [W1204 10:40:58.322960357 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1530144Z 2025-12-04T11:13:50.1530432Z [W1204 10:40:58.328598092 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1530437Z 2025-12-04T11:13:50.1530727Z [W1204 10:40:58.329151974 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1530732Z 2025-12-04T11:13:50.1531017Z [W1204 10:40:58.329318408 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1531021Z 2025-12-04T11:13:50.1531308Z [W1204 10:40:58.334722009 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1531312Z 2025-12-04T11:13:50.1531597Z [W1204 10:40:58.335248460 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1531600Z 2025-12-04T11:13:50.1531884Z [W1204 10:40:58.335407624 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1531893Z 2025-12-04T11:13:50.1531974Z ('RERUN', {'yellow': True}) [10.7768s] [100%] 2025-12-04T11:13:50.1532700Z inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_17_mid_dim_8_in_features_128_out_features_1024_cuda_bfloat16 [W1204 10:40:59.552993309 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1532705Z 2025-12-04T11:13:50.1532994Z [W1204 10:40:59.553531561 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1532997Z 2025-12-04T11:13:50.1533283Z [W1204 10:40:59.553674624 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1533287Z 2025-12-04T11:13:50.1533573Z [W1204 10:40:59.556629029 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1533616Z 2025-12-04T11:13:50.1533965Z [W1204 10:40:59.557197712 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1533971Z 2025-12-04T11:13:50.1534261Z [W1204 10:40:59.557334565 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1534264Z 2025-12-04T11:13:50.1534548Z [W1204 10:40:59.561911626 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1534552Z 2025-12-04T11:13:50.1534839Z [W1204 10:40:59.562386577 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1534842Z 2025-12-04T11:13:50.1535131Z [W1204 10:40:59.562524070 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1535136Z 2025-12-04T11:13:50.1535214Z ('RERUN', {'yellow': True}) [0.4600s] [100%] 2025-12-04T11:13:50.1535940Z inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_17_mid_dim_8_in_features_128_out_features_1024_cuda_bfloat16 [W1204 10:41:00.005226705 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1535988Z 2025-12-04T11:13:50.1536276Z [W1204 10:41:00.005764527 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1536280Z 2025-12-04T11:13:50.1536569Z [W1204 10:41:00.005906320 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1536572Z 2025-12-04T11:13:50.1536855Z [W1204 10:41:00.008849405 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1536861Z 2025-12-04T11:13:50.1537153Z [W1204 10:41:00.009411018 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1537158Z 2025-12-04T11:13:50.1537444Z [W1204 10:41:00.009551250 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1537448Z 2025-12-04T11:13:50.1537737Z [W1204 10:41:00.014084362 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1537740Z 2025-12-04T11:13:50.1538025Z [W1204 10:41:00.014556622 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1538028Z 2025-12-04T11:13:50.1538314Z [W1204 10:41:00.014697095 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1538324Z 2025-12-04T11:13:50.1538386Z FAILED [0.4500s] [100%] 2025-12-04T11:13:50.1538390Z 2025-12-04T11:13:50.1538477Z ==================================== RERUNS ==================================== 2025-12-04T11:13:50.1538775Z _ TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_8_in_features_128_out_features_1024_cuda_bfloat16 _ 2025-12-04T11:13:50.1538848Z Traceback (most recent call last): 2025-12-04T11:13:50.1539149Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.1539224Z method(*args, **kwargs) 2025-12-04T11:13:50.1539516Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.1539583Z method(*args, **kwargs) 2025-12-04T11:13:50.1539869Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T11:13:50.1539972Z with policy(): 2025-12-04T11:13:50.1540333Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T11:13:50.1540403Z raise RuntimeError(msg) 2025-12-04T11:13:50.1541208Z RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_8_in_features_128_out_features_1024_cuda_bfloat16! Caching allocator allocated memory was 0 and is now reported as 131072 on device 0. CUDA driver allocated memory was 230686720 and is now 274726912. 2025-12-04T11:13:50.1541212Z 2025-12-04T11:13:50.1541339Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.1541864Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_8_in_features_128_out_features_1024_cuda_bfloat16 2025-12-04T11:13:50.1541873Z 2025-12-04T11:13:50.1542030Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.1542166Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.1542302Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.1542650Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.1542776Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.1542839Z graph_break [] 2025-12-04T11:13:50.1542961Z ----------------------------- Captured stderr call ----------------------------- 2025-12-04T11:13:50.1543729Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/constant_folding.py:256: UserWarning: Unsupported unwinding pattern: Address not in range (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/profiler/unwind/unwind.cpp:219.) 2025-12-04T11:13:50.1543803Z if out == self.unknown_value: 2025-12-04T11:13:50.1544100Z _ TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_8_in_features_128_out_features_1024_cuda_bfloat16 _ 2025-12-04T11:13:50.1544181Z Traceback (most recent call last): 2025-12-04T11:13:50.1544473Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.1544539Z method(*args, **kwargs) 2025-12-04T11:13:50.1544826Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.1544892Z method(*args, **kwargs) 2025-12-04T11:13:50.1545185Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T11:13:50.1545248Z with policy(): 2025-12-04T11:13:50.1545539Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T11:13:50.1545612Z raise RuntimeError(msg) 2025-12-04T11:13:50.1546431Z RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_8_in_features_128_out_features_1024_cuda_bfloat16! Caching allocator allocated memory was 131072 and is now reported as 262144 on device 0. CUDA driver allocated memory was 274726912 and is now 276824064. 2025-12-04T11:13:50.1546436Z 2025-12-04T11:13:50.1546565Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.1547083Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_8_in_features_128_out_features_1024_cuda_bfloat16 2025-12-04T11:13:50.1547086Z 2025-12-04T11:13:50.1547246Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.1547371Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.1547508Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.1547921Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.1548049Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.1548107Z graph_break [] 2025-12-04T11:13:50.1548234Z ----------------------------- Captured stderr call ----------------------------- 2025-12-04T11:13:50.1548919Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/constant_folding.py:256: UserWarning: Unsupported unwinding pattern: Address not in range (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/profiler/unwind/unwind.cpp:219.) 2025-12-04T11:13:50.1548990Z if out == self.unknown_value: 2025-12-04T11:13:50.1549114Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.1549206Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.1549348Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.1549689Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.1549798Z graph_break [] 2025-12-04T11:13:50.1549882Z =================================== FAILURES =================================== 2025-12-04T11:13:50.1550174Z _ TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_8_in_features_128_out_features_1024_cuda_bfloat16 _ 2025-12-04T11:13:50.1550252Z Traceback (most recent call last): 2025-12-04T11:13:50.1550547Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.1550611Z method(*args, **kwargs) 2025-12-04T11:13:50.1550905Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.1550969Z method(*args, **kwargs) 2025-12-04T11:13:50.1551260Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T11:13:50.1551321Z with policy(): 2025-12-04T11:13:50.1551612Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T11:13:50.1551681Z raise RuntimeError(msg) 2025-12-04T11:13:50.1552494Z RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_8_in_features_128_out_features_1024_cuda_bfloat16! Caching allocator allocated memory was 262144 and is now reported as 393216 on device 0. CUDA driver allocated memory was 276824064 and is now 278921216. 2025-12-04T11:13:50.1552498Z 2025-12-04T11:13:50.1552624Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.1553145Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_8_in_features_128_out_features_1024_cuda_bfloat16 2025-12-04T11:13:50.1553150Z 2025-12-04T11:13:50.1553308Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.1553433Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.1553523Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.1553870Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.1553993Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.1554051Z graph_break [] 2025-12-04T11:13:50.1554177Z ----------------------------- Captured stderr call ----------------------------- 2025-12-04T11:13:50.1554996Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/constant_folding.py:256: UserWarning: Unsupported unwinding pattern: Address not in range (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/profiler/unwind/unwind.cpp:219.) 2025-12-04T11:13:50.1555079Z if out == self.unknown_value: 2025-12-04T11:13:50.1555203Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.1555294Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.1555418Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.1555757Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.1555818Z graph_break [] 2025-12-04T11:13:50.1555940Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.1556030Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.1556157Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.1556497Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.1556594Z graph_break [] 2025-12-04T11:13:50.1557083Z - generated xml file: /var/lib/jenkins/workspace/test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-c82b9fa27197518a.xml - 2025-12-04T11:13:50.1557187Z =========================== short test summary info ============================ 2025-12-04T11:13:50.1558488Z FAILED [0.4500s] inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_17_mid_dim_8_in_features_128_out_features_1024_cuda_bfloat16 - RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_8_in_features_128_out_features_1024_cuda_bfloat16! Caching allocator allocated memory was 262144 and is now reported as 393216 on device 0. CUDA driver allocated memory was 276824064 and is now 278921216. 2025-12-04T11:13:50.1558495Z 2025-12-04T11:13:50.1558619Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.1559143Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_8_in_features_128_out_features_1024_cuda_bfloat16 2025-12-04T11:13:50.1559147Z 2025-12-04T11:13:50.1559300Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.1559406Z !!!!!!!!!!!!!!!!!!!!!!!!!! stopping after 1 failures !!!!!!!!!!!!!!!!!!!!!!!!!!! 2025-12-04T11:13:50.1559525Z ================== 1 failed, 57 deselected, 2 rerun in 11.71s ================== 2025-12-04T11:13:50.1559584Z Got exit code 1 2025-12-04T11:13:50.1560068Z FAILED CONSISTENTLY: test/inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_17_mid_dim_8_in_features_128_out_features_1024_cuda_bfloat16 2025-12-04T11:13:50.1560310Z Test failed consistently, continuing with the rest of the tests due to continue-through-error being set 2025-12-04T11:13:50.1560574Z W1204 10:41:06.831000 66100 site-packages/torch/_inductor/utils.py:1703] Not enough SMs to use max_autotune_gemm mode 2025-12-04T11:13:50.1560961Z Test results will be stored in test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-36ce77ac1c8c9763.xml 2025-12-04T11:13:50.1561056Z ============================= test session starts ============================== 2025-12-04T11:13:50.1561486Z platform linux -- Python 3.10.14, pytest-7.3.2, pluggy-1.6.0 -- /opt/conda/envs/py_3.10/bin/python 2025-12-04T11:13:50.1568314Z cachedir: .pytest_cache 2025-12-04T11:13:50.1568685Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow] 2025-12-04T11:13:50.1568848Z rootdir: /var/lib/jenkins/workspace 2025-12-04T11:13:50.1568986Z configfile: pytest.ini 2025-12-04T11:13:50.1569331Z plugins: hypothesis-6.56.4, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-14.0, subtests-0.13.1, xdist-3.3.1, xdoctest-1.3.0, typeguard-4.3.0 2025-12-04T11:13:50.1569476Z collecting ... collected 58 items / 17 deselected / 41 selected 2025-12-04T11:13:50.1569567Z stepcurrent: skipping 17 already run items. 2025-12-04T11:13:50.1569641Z Running 41 items in this shard 2025-12-04T11:13:50.1569646Z 2025-12-04T11:13:50.1570166Z inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_17_mid_dim_8_in_features_128_out_features_64_cuda_bfloat16 ('RERUN', {'yellow': True}) [1.8795s] [ 2%] 2025-12-04T11:13:50.1570661Z inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_17_mid_dim_8_in_features_128_out_features_64_cuda_bfloat16 ('RERUN', {'yellow': True}) [0.4467s] [ 2%] 2025-12-04T11:13:50.1571115Z inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_17_mid_dim_8_in_features_128_out_features_64_cuda_bfloat16 FAILED [0.4427s] [ 2%] 2025-12-04T11:13:50.1571164Z 2025-12-04T11:13:50.1571256Z ==================================== RERUNS ==================================== 2025-12-04T11:13:50.1571552Z _ TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_8_in_features_128_out_features_64_cuda_bfloat16 _ 2025-12-04T11:13:50.1571633Z Traceback (most recent call last): 2025-12-04T11:13:50.1571944Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.1572016Z method(*args, **kwargs) 2025-12-04T11:13:50.1572306Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.1572371Z method(*args, **kwargs) 2025-12-04T11:13:50.1572659Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T11:13:50.1572721Z with policy(): 2025-12-04T11:13:50.1573015Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T11:13:50.1573084Z raise RuntimeError(msg) 2025-12-04T11:13:50.1573879Z RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_8_in_features_128_out_features_64_cuda_bfloat16! Caching allocator allocated memory was 0 and is now reported as 8192 on device 0. CUDA driver allocated memory was 230686720 and is now 274726912. 2025-12-04T11:13:50.1573884Z 2025-12-04T11:13:50.1574017Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.1574538Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_8_in_features_128_out_features_64_cuda_bfloat16 2025-12-04T11:13:50.1574546Z 2025-12-04T11:13:50.1574711Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.1574848Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.1574947Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.1575301Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.1575428Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.1575491Z graph_break [] 2025-12-04T11:13:50.1575790Z _ TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_8_in_features_128_out_features_64_cuda_bfloat16 _ 2025-12-04T11:13:50.1575866Z Traceback (most recent call last): 2025-12-04T11:13:50.1576279Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.1576345Z method(*args, **kwargs) 2025-12-04T11:13:50.1576636Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.1576700Z method(*args, **kwargs) 2025-12-04T11:13:50.1576984Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T11:13:50.1577046Z with policy(): 2025-12-04T11:13:50.1577335Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T11:13:50.1577400Z raise RuntimeError(msg) 2025-12-04T11:13:50.1578419Z RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_8_in_features_128_out_features_64_cuda_bfloat16! Caching allocator allocated memory was 8192 and is now reported as 16384 on device 0. CUDA driver allocated memory was 274726912 and is now 276824064. 2025-12-04T11:13:50.1578426Z 2025-12-04T11:13:50.1578638Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.1579162Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_8_in_features_128_out_features_64_cuda_bfloat16 2025-12-04T11:13:50.1579167Z 2025-12-04T11:13:50.1579339Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.1579472Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.1579570Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.1579917Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.1580051Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.1580113Z graph_break [] 2025-12-04T11:13:50.1580242Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.1580338Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.1580460Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.1580802Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.1580865Z graph_break [] 2025-12-04T11:13:50.1580949Z =================================== FAILURES =================================== 2025-12-04T11:13:50.1581247Z _ TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_8_in_features_128_out_features_64_cuda_bfloat16 _ 2025-12-04T11:13:50.1581322Z Traceback (most recent call last): 2025-12-04T11:13:50.1581635Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.1581703Z method(*args, **kwargs) 2025-12-04T11:13:50.1581994Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.1582060Z method(*args, **kwargs) 2025-12-04T11:13:50.1582344Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T11:13:50.1582403Z with policy(): 2025-12-04T11:13:50.1582696Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T11:13:50.1582761Z raise RuntimeError(msg) 2025-12-04T11:13:50.1583728Z RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_8_in_features_128_out_features_64_cuda_bfloat16! Caching allocator allocated memory was 16384 and is now reported as 24576 on device 0. CUDA driver allocated memory was 276824064 and is now 278921216. 2025-12-04T11:13:50.1583777Z 2025-12-04T11:13:50.1583921Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.1584445Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_8_in_features_128_out_features_64_cuda_bfloat16 2025-12-04T11:13:50.1584450Z 2025-12-04T11:13:50.1584613Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.1584740Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.1584834Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.1585182Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.1585310Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.1585373Z graph_break [] 2025-12-04T11:13:50.1585495Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.1585619Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.1585740Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.1586090Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.1586152Z graph_break [] 2025-12-04T11:13:50.1586273Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.1586358Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.1586479Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.1586820Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.1586880Z graph_break [] 2025-12-04T11:13:50.1587376Z - generated xml file: /var/lib/jenkins/workspace/test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-36ce77ac1c8c9763.xml - 2025-12-04T11:13:50.1587476Z =========================== short test summary info ============================ 2025-12-04T11:13:50.1588757Z FAILED [0.4427s] inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_17_mid_dim_8_in_features_128_out_features_64_cuda_bfloat16 - RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_8_in_features_128_out_features_64_cuda_bfloat16! Caching allocator allocated memory was 16384 and is now reported as 24576 on device 0. CUDA driver allocated memory was 276824064 and is now 278921216. 2025-12-04T11:13:50.1588763Z 2025-12-04T11:13:50.1588891Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.1589415Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_8_in_features_128_out_features_64_cuda_bfloat16 2025-12-04T11:13:50.1589421Z 2025-12-04T11:13:50.1589576Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.1589682Z !!!!!!!!!!!!!!!!!!!!!!!!!! stopping after 1 failures !!!!!!!!!!!!!!!!!!!!!!!!!!! 2025-12-04T11:13:50.1589799Z ================== 1 failed, 17 deselected, 2 rerun in 2.79s =================== 2025-12-04T11:13:50.1589857Z Got exit code 1 2025-12-04T11:13:50.1589926Z Retrying single test... 2025-12-04T11:13:50.1590187Z W1204 10:41:16.894000 66281 site-packages/torch/_inductor/utils.py:1703] Not enough SMs to use max_autotune_gemm mode 2025-12-04T11:13:50.1590697Z Test results will be stored in test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-e4f38ecbc39decbb.xml 2025-12-04T11:13:50.1590799Z ============================= test session starts ============================== 2025-12-04T11:13:50.1591010Z platform linux -- Python 3.10.14, pytest-7.3.2, pluggy-1.6.0 -- /opt/conda/envs/py_3.10/bin/python 2025-12-04T11:13:50.1591080Z cachedir: .pytest_cache 2025-12-04T11:13:50.1591386Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow] 2025-12-04T11:13:50.1591462Z rootdir: /var/lib/jenkins/workspace 2025-12-04T11:13:50.1591531Z configfile: pytest.ini 2025-12-04T11:13:50.1591849Z plugins: hypothesis-6.56.4, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-14.0, subtests-0.13.1, xdist-3.3.1, xdoctest-1.3.0, typeguard-4.3.0 2025-12-04T11:13:50.1591979Z collecting ... collected 58 items / 57 deselected / 1 selected 2025-12-04T11:13:50.1592555Z stepcurrent: skipping 17 already run items. Running only test/inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_17_mid_dim_8_in_features_128_out_features_64_cuda_bfloat16 2025-12-04T11:13:50.1592672Z Running 1 items in this shard 2025-12-04T11:13:50.1592676Z 2025-12-04T11:13:50.1593404Z inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_17_mid_dim_8_in_features_128_out_features_64_cuda_bfloat16 [W1204 10:41:18.994594825 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1593408Z 2025-12-04T11:13:50.1593706Z [W1204 10:41:27.963103615 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1593709Z 2025-12-04T11:13:50.1594002Z [W1204 10:41:27.963342450 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1594007Z 2025-12-04T11:13:50.1594295Z [W1204 10:41:27.968987895 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1594300Z 2025-12-04T11:13:50.1594587Z [W1204 10:41:27.969563018 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1594590Z 2025-12-04T11:13:50.1594872Z [W1204 10:41:27.969734182 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1594875Z 2025-12-04T11:13:50.1595162Z [W1204 10:41:27.975305475 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1595165Z 2025-12-04T11:13:50.1595448Z [W1204 10:41:27.975869667 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1595453Z 2025-12-04T11:13:50.1595745Z [W1204 10:41:27.976044361 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1595750Z 2025-12-04T11:13:50.1595834Z ('RERUN', {'yellow': True}) [10.8683s] [100%] 2025-12-04T11:13:50.1596551Z inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_17_mid_dim_8_in_features_128_out_features_64_cuda_bfloat16 [W1204 10:41:28.169647916 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1596556Z 2025-12-04T11:13:50.1596846Z [W1204 10:41:28.170253799 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1596849Z 2025-12-04T11:13:50.1597133Z [W1204 10:41:28.170403282 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1597175Z 2025-12-04T11:13:50.1597523Z [W1204 10:41:28.173433279 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1597529Z 2025-12-04T11:13:50.1597813Z [W1204 10:41:28.174026912 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1597817Z 2025-12-04T11:13:50.1598104Z [W1204 10:41:28.174164975 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1598108Z 2025-12-04T11:13:50.1598393Z [W1204 10:41:28.178791718 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1598396Z 2025-12-04T11:13:50.1598678Z [W1204 10:41:28.179269398 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1598685Z 2025-12-04T11:13:50.1598971Z [W1204 10:41:28.179405081 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1598974Z 2025-12-04T11:13:50.1599084Z ('RERUN', {'yellow': True}) [0.4258s] [100%] 2025-12-04T11:13:50.1599808Z inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_17_mid_dim_8_in_features_128_out_features_64_cuda_bfloat16 [W1204 10:41:28.589236105 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1599812Z 2025-12-04T11:13:50.1600096Z [W1204 10:41:28.589816338 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1600099Z 2025-12-04T11:13:50.1600388Z [W1204 10:41:28.589966651 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1600393Z 2025-12-04T11:13:50.1600677Z [W1204 10:41:28.592983878 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1600681Z 2025-12-04T11:13:50.1600969Z [W1204 10:41:28.593569940 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1600972Z 2025-12-04T11:13:50.1601255Z [W1204 10:41:28.593713534 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1601258Z 2025-12-04T11:13:50.1601545Z [W1204 10:41:28.598351585 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1601548Z 2025-12-04T11:13:50.1601833Z [W1204 10:41:28.598825926 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1601838Z 2025-12-04T11:13:50.1602123Z [W1204 10:41:28.598967169 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1602129Z 2025-12-04T11:13:50.1602201Z FAILED [0.4181s] [100%] 2025-12-04T11:13:50.1602206Z 2025-12-04T11:13:50.1602292Z ==================================== RERUNS ==================================== 2025-12-04T11:13:50.1602588Z _ TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_8_in_features_128_out_features_64_cuda_bfloat16 _ 2025-12-04T11:13:50.1602666Z Traceback (most recent call last): 2025-12-04T11:13:50.1602973Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.1603041Z method(*args, **kwargs) 2025-12-04T11:13:50.1603332Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.1603397Z method(*args, **kwargs) 2025-12-04T11:13:50.1603793Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T11:13:50.1603855Z with policy(): 2025-12-04T11:13:50.1604150Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T11:13:50.1604214Z raise RuntimeError(msg) 2025-12-04T11:13:50.1605004Z RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_8_in_features_128_out_features_64_cuda_bfloat16! Caching allocator allocated memory was 0 and is now reported as 8192 on device 0. CUDA driver allocated memory was 230686720 and is now 274726912. 2025-12-04T11:13:50.1605010Z 2025-12-04T11:13:50.1605139Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.1605654Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_8_in_features_128_out_features_64_cuda_bfloat16 2025-12-04T11:13:50.1605659Z 2025-12-04T11:13:50.1605818Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.1605983Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.1606081Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.1606425Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.1606552Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.1606612Z graph_break [] 2025-12-04T11:13:50.1606735Z ----------------------------- Captured stderr call ----------------------------- 2025-12-04T11:13:50.1607427Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/constant_folding.py:256: UserWarning: Unsupported unwinding pattern: Address not in range (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/profiler/unwind/unwind.cpp:219.) 2025-12-04T11:13:50.1607503Z if out == self.unknown_value: 2025-12-04T11:13:50.1607795Z _ TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_8_in_features_128_out_features_64_cuda_bfloat16 _ 2025-12-04T11:13:50.1607872Z Traceback (most recent call last): 2025-12-04T11:13:50.1608168Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.1608231Z method(*args, **kwargs) 2025-12-04T11:13:50.1608521Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.1608582Z method(*args, **kwargs) 2025-12-04T11:13:50.1608871Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T11:13:50.1608932Z with policy(): 2025-12-04T11:13:50.1609222Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T11:13:50.1609292Z raise RuntimeError(msg) 2025-12-04T11:13:50.1610090Z RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_8_in_features_128_out_features_64_cuda_bfloat16! Caching allocator allocated memory was 8192 and is now reported as 16384 on device 0. CUDA driver allocated memory was 274726912 and is now 276824064. 2025-12-04T11:13:50.1610094Z 2025-12-04T11:13:50.1610229Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.1610749Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_8_in_features_128_out_features_64_cuda_bfloat16 2025-12-04T11:13:50.1610753Z 2025-12-04T11:13:50.1610950Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.1611141Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.1611235Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.1611587Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.1611717Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.1611774Z graph_break [] 2025-12-04T11:13:50.1611901Z ----------------------------- Captured stderr call ----------------------------- 2025-12-04T11:13:50.1612590Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/constant_folding.py:256: UserWarning: Unsupported unwinding pattern: Address not in range (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/profiler/unwind/unwind.cpp:219.) 2025-12-04T11:13:50.1612664Z if out == self.unknown_value: 2025-12-04T11:13:50.1612785Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.1612879Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.1613039Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.1613378Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.1613439Z graph_break [] 2025-12-04T11:13:50.1613521Z =================================== FAILURES =================================== 2025-12-04T11:13:50.1613807Z _ TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_8_in_features_128_out_features_64_cuda_bfloat16 _ 2025-12-04T11:13:50.1613884Z Traceback (most recent call last): 2025-12-04T11:13:50.1614180Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.1614243Z method(*args, **kwargs) 2025-12-04T11:13:50.1614537Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.1614602Z method(*args, **kwargs) 2025-12-04T11:13:50.1614888Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T11:13:50.1614945Z with policy(): 2025-12-04T11:13:50.1615233Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T11:13:50.1615301Z raise RuntimeError(msg) 2025-12-04T11:13:50.1616102Z RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_8_in_features_128_out_features_64_cuda_bfloat16! Caching allocator allocated memory was 16384 and is now reported as 24576 on device 0. CUDA driver allocated memory was 276824064 and is now 278921216. 2025-12-04T11:13:50.1616108Z 2025-12-04T11:13:50.1616237Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.1616754Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_8_in_features_128_out_features_64_cuda_bfloat16 2025-12-04T11:13:50.1616760Z 2025-12-04T11:13:50.1616914Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.1617037Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.1617127Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.1617470Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.1617591Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.1617690Z graph_break [] 2025-12-04T11:13:50.1617896Z ----------------------------- Captured stderr call ----------------------------- 2025-12-04T11:13:50.1618584Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/constant_folding.py:256: UserWarning: Unsupported unwinding pattern: Address not in range (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/profiler/unwind/unwind.cpp:219.) 2025-12-04T11:13:50.1618657Z if out == self.unknown_value: 2025-12-04T11:13:50.1618778Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.1618866Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.1618987Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.1619334Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.1619392Z graph_break [] 2025-12-04T11:13:50.1619517Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.1619606Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.1619728Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.1620239Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.1620298Z graph_break [] 2025-12-04T11:13:50.1620787Z - generated xml file: /var/lib/jenkins/workspace/test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-e4f38ecbc39decbb.xml - 2025-12-04T11:13:50.1620886Z =========================== short test summary info ============================ 2025-12-04T11:13:50.1622172Z FAILED [0.4181s] inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_17_mid_dim_8_in_features_128_out_features_64_cuda_bfloat16 - RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_8_in_features_128_out_features_64_cuda_bfloat16! Caching allocator allocated memory was 16384 and is now reported as 24576 on device 0. CUDA driver allocated memory was 276824064 and is now 278921216. 2025-12-04T11:13:50.1622179Z 2025-12-04T11:13:50.1622301Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.1622818Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_8_in_features_128_out_features_64_cuda_bfloat16 2025-12-04T11:13:50.1622822Z 2025-12-04T11:13:50.1622976Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.1623078Z !!!!!!!!!!!!!!!!!!!!!!!!!! stopping after 1 failures !!!!!!!!!!!!!!!!!!!!!!!!!!! 2025-12-04T11:13:50.1623196Z ================== 1 failed, 57 deselected, 2 rerun in 11.74s ================== 2025-12-04T11:13:50.1623255Z Got exit code 1 2025-12-04T11:13:50.1623320Z Retrying single test... 2025-12-04T11:13:50.1623633Z W1204 10:41:35.448000 66467 site-packages/torch/_inductor/utils.py:1703] Not enough SMs to use max_autotune_gemm mode 2025-12-04T11:13:50.1624019Z Test results will be stored in test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-e20c92a1eaab2de9.xml 2025-12-04T11:13:50.1624116Z ============================= test session starts ============================== 2025-12-04T11:13:50.1624324Z platform linux -- Python 3.10.14, pytest-7.3.2, pluggy-1.6.0 -- /opt/conda/envs/py_3.10/bin/python 2025-12-04T11:13:50.1624390Z cachedir: .pytest_cache 2025-12-04T11:13:50.1624695Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow] 2025-12-04T11:13:50.1624768Z rootdir: /var/lib/jenkins/workspace 2025-12-04T11:13:50.1624877Z configfile: pytest.ini 2025-12-04T11:13:50.1625253Z plugins: hypothesis-6.56.4, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-14.0, subtests-0.13.1, xdist-3.3.1, xdoctest-1.3.0, typeguard-4.3.0 2025-12-04T11:13:50.1625383Z collecting ... collected 58 items / 57 deselected / 1 selected 2025-12-04T11:13:50.1625953Z stepcurrent: skipping 17 already run items. Running only test/inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_17_mid_dim_8_in_features_128_out_features_64_cuda_bfloat16 2025-12-04T11:13:50.1626021Z Running 1 items in this shard 2025-12-04T11:13:50.1626025Z 2025-12-04T11:13:50.1626753Z inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_17_mid_dim_8_in_features_128_out_features_64_cuda_bfloat16 [W1204 10:41:36.530889097 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1626757Z 2025-12-04T11:13:50.1627056Z [W1204 10:41:45.599974026 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1627062Z 2025-12-04T11:13:50.1627350Z [W1204 10:41:45.600273632 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1627387Z 2025-12-04T11:13:50.1627673Z [W1204 10:41:45.605872546 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1627677Z 2025-12-04T11:13:50.1627963Z [W1204 10:41:45.606409528 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1627969Z 2025-12-04T11:13:50.1628252Z [W1204 10:41:45.606577132 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1628255Z 2025-12-04T11:13:50.1628536Z [W1204 10:41:45.611954751 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1628541Z 2025-12-04T11:13:50.1628833Z [W1204 10:41:45.612516043 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1628838Z 2025-12-04T11:13:50.1629122Z [W1204 10:41:45.612697787 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1629126Z 2025-12-04T11:13:50.1629212Z ('RERUN', {'yellow': True}) [10.9465s] [100%] 2025-12-04T11:13:50.1629928Z inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_17_mid_dim_8_in_features_128_out_features_64_cuda_bfloat16 [W1204 10:41:46.800241290 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1629932Z 2025-12-04T11:13:50.1630220Z [W1204 10:41:46.800817823 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1630225Z 2025-12-04T11:13:50.1630512Z [W1204 10:41:46.800955496 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1630517Z 2025-12-04T11:13:50.1630819Z [W1204 10:41:46.803975503 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1630822Z 2025-12-04T11:13:50.1631108Z [W1204 10:41:46.804552966 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1631112Z 2025-12-04T11:13:50.1631398Z [W1204 10:41:46.804698619 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1631401Z 2025-12-04T11:13:50.1631685Z [W1204 10:41:46.809332721 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1631727Z 2025-12-04T11:13:50.1632071Z [W1204 10:41:46.809810642 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1632077Z 2025-12-04T11:13:50.1632366Z [W1204 10:41:46.809946825 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1632369Z 2025-12-04T11:13:50.1632446Z ('RERUN', {'yellow': True}) [0.4292s] [100%] 2025-12-04T11:13:50.1633162Z inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_17_mid_dim_8_in_features_128_out_features_64_cuda_bfloat16 [W1204 10:41:47.223790940 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1633167Z 2025-12-04T11:13:50.1633451Z [W1204 10:41:47.224358813 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1633456Z 2025-12-04T11:13:50.1633745Z [W1204 10:41:47.224498446 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1633804Z 2025-12-04T11:13:50.1634089Z [W1204 10:41:47.227478172 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1634092Z 2025-12-04T11:13:50.1634375Z [W1204 10:41:47.228048224 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1634381Z 2025-12-04T11:13:50.1634663Z [W1204 10:41:47.228185487 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1634666Z 2025-12-04T11:13:50.1634952Z [W1204 10:41:47.232792279 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1634957Z 2025-12-04T11:13:50.1635247Z [W1204 10:41:47.233271750 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1635252Z 2025-12-04T11:13:50.1635535Z [W1204 10:41:47.233407802 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1635538Z 2025-12-04T11:13:50.1635602Z FAILED [0.4199s] [100%] 2025-12-04T11:13:50.1635605Z 2025-12-04T11:13:50.1635688Z ==================================== RERUNS ==================================== 2025-12-04T11:13:50.1635977Z _ TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_8_in_features_128_out_features_64_cuda_bfloat16 _ 2025-12-04T11:13:50.1636054Z Traceback (most recent call last): 2025-12-04T11:13:50.1636356Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.1636436Z method(*args, **kwargs) 2025-12-04T11:13:50.1636731Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.1636795Z method(*args, **kwargs) 2025-12-04T11:13:50.1637082Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T11:13:50.1637141Z with policy(): 2025-12-04T11:13:50.1637431Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T11:13:50.1637496Z raise RuntimeError(msg) 2025-12-04T11:13:50.1638285Z RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_8_in_features_128_out_features_64_cuda_bfloat16! Caching allocator allocated memory was 0 and is now reported as 8192 on device 0. CUDA driver allocated memory was 230686720 and is now 274726912. 2025-12-04T11:13:50.1638330Z 2025-12-04T11:13:50.1638524Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.1639041Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_8_in_features_128_out_features_64_cuda_bfloat16 2025-12-04T11:13:50.1639047Z 2025-12-04T11:13:50.1639205Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.1639333Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.1639425Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.1639773Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.1639899Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.1639960Z graph_break [] 2025-12-04T11:13:50.1640085Z ----------------------------- Captured stderr call ----------------------------- 2025-12-04T11:13:50.1640771Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/constant_folding.py:256: UserWarning: Unsupported unwinding pattern: Address not in range (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/profiler/unwind/unwind.cpp:219.) 2025-12-04T11:13:50.1640880Z if out == self.unknown_value: 2025-12-04T11:13:50.1641167Z _ TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_8_in_features_128_out_features_64_cuda_bfloat16 _ 2025-12-04T11:13:50.1641241Z Traceback (most recent call last): 2025-12-04T11:13:50.1641539Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.1641602Z method(*args, **kwargs) 2025-12-04T11:13:50.1641891Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.1641955Z method(*args, **kwargs) 2025-12-04T11:13:50.1642244Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T11:13:50.1642307Z with policy(): 2025-12-04T11:13:50.1642595Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T11:13:50.1642663Z raise RuntimeError(msg) 2025-12-04T11:13:50.1643466Z RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_8_in_features_128_out_features_64_cuda_bfloat16! Caching allocator allocated memory was 8192 and is now reported as 16384 on device 0. CUDA driver allocated memory was 274726912 and is now 276824064. 2025-12-04T11:13:50.1643470Z 2025-12-04T11:13:50.1643597Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.1644118Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_8_in_features_128_out_features_64_cuda_bfloat16 2025-12-04T11:13:50.1644123Z 2025-12-04T11:13:50.1644278Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.1644411Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.1644504Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.1644850Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.1644979Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.1645038Z graph_break [] 2025-12-04T11:13:50.1645162Z ----------------------------- Captured stderr call ----------------------------- 2025-12-04T11:13:50.1645968Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/constant_folding.py:256: UserWarning: Unsupported unwinding pattern: Address not in range (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/profiler/unwind/unwind.cpp:219.) 2025-12-04T11:13:50.1646040Z if out == self.unknown_value: 2025-12-04T11:13:50.1646166Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.1646254Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.1646383Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.1646724Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.1646783Z graph_break [] 2025-12-04T11:13:50.1646868Z =================================== FAILURES =================================== 2025-12-04T11:13:50.1647154Z _ TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_8_in_features_128_out_features_64_cuda_bfloat16 _ 2025-12-04T11:13:50.1647229Z Traceback (most recent call last): 2025-12-04T11:13:50.1647527Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.1647638Z method(*args, **kwargs) 2025-12-04T11:13:50.1647930Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.1647995Z method(*args, **kwargs) 2025-12-04T11:13:50.1648283Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T11:13:50.1648348Z with policy(): 2025-12-04T11:13:50.1648636Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T11:13:50.1648704Z raise RuntimeError(msg) 2025-12-04T11:13:50.1649508Z RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_8_in_features_128_out_features_64_cuda_bfloat16! Caching allocator allocated memory was 16384 and is now reported as 24576 on device 0. CUDA driver allocated memory was 276824064 and is now 278921216. 2025-12-04T11:13:50.1649513Z 2025-12-04T11:13:50.1649638Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.1650160Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_8_in_features_128_out_features_64_cuda_bfloat16 2025-12-04T11:13:50.1650164Z 2025-12-04T11:13:50.1650317Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.1650446Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.1650536Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.1650883Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.1651012Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.1651071Z graph_break [] 2025-12-04T11:13:50.1651199Z ----------------------------- Captured stderr call ----------------------------- 2025-12-04T11:13:50.1651886Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/constant_folding.py:256: UserWarning: Unsupported unwinding pattern: Address not in range (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/profiler/unwind/unwind.cpp:219.) 2025-12-04T11:13:50.1651955Z if out == self.unknown_value: 2025-12-04T11:13:50.1652083Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.1652170Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.1652334Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.1652740Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.1652803Z graph_break [] 2025-12-04T11:13:50.1652930Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.1653018Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.1653141Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.1653481Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.1653537Z graph_break [] 2025-12-04T11:13:50.1654025Z - generated xml file: /var/lib/jenkins/workspace/test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-e20c92a1eaab2de9.xml - 2025-12-04T11:13:50.1654126Z =========================== short test summary info ============================ 2025-12-04T11:13:50.1655414Z FAILED [0.4199s] inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_17_mid_dim_8_in_features_128_out_features_64_cuda_bfloat16 - RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_8_in_features_128_out_features_64_cuda_bfloat16! Caching allocator allocated memory was 16384 and is now reported as 24576 on device 0. CUDA driver allocated memory was 276824064 and is now 278921216. 2025-12-04T11:13:50.1655457Z 2025-12-04T11:13:50.1655581Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.1656097Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_8_in_features_128_out_features_64_cuda_bfloat16 2025-12-04T11:13:50.1656105Z 2025-12-04T11:13:50.1656260Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.1656366Z !!!!!!!!!!!!!!!!!!!!!!!!!! stopping after 1 failures !!!!!!!!!!!!!!!!!!!!!!!!!!! 2025-12-04T11:13:50.1656490Z ================== 1 failed, 57 deselected, 2 rerun in 11.82s ================== 2025-12-04T11:13:50.1656550Z Got exit code 1 2025-12-04T11:13:50.1657023Z FAILED CONSISTENTLY: test/inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_17_mid_dim_8_in_features_128_out_features_64_cuda_bfloat16 2025-12-04T11:13:50.1657263Z Test failed consistently, continuing with the rest of the tests due to continue-through-error being set 2025-12-04T11:13:50.1657528Z W1204 10:41:54.090000 66653 site-packages/torch/_inductor/utils.py:1703] Not enough SMs to use max_autotune_gemm mode 2025-12-04T11:13:50.1657911Z Test results will be stored in test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-2048f1d3bc205458.xml 2025-12-04T11:13:50.1658010Z ============================= test session starts ============================== 2025-12-04T11:13:50.1658229Z platform linux -- Python 3.10.14, pytest-7.3.2, pluggy-1.6.0 -- /opt/conda/envs/py_3.10/bin/python 2025-12-04T11:13:50.1658297Z cachedir: .pytest_cache 2025-12-04T11:13:50.1658600Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow] 2025-12-04T11:13:50.1658680Z rootdir: /var/lib/jenkins/workspace 2025-12-04T11:13:50.1658744Z configfile: pytest.ini 2025-12-04T11:13:50.1659069Z plugins: hypothesis-6.56.4, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-14.0, subtests-0.13.1, xdist-3.3.1, xdoctest-1.3.0, typeguard-4.3.0 2025-12-04T11:13:50.1659200Z collecting ... collected 58 items / 18 deselected / 40 selected 2025-12-04T11:13:50.1659289Z stepcurrent: skipping 18 already run items. 2025-12-04T11:13:50.1659364Z Running 40 items in this shard 2025-12-04T11:13:50.1659409Z 2025-12-04T11:13:50.1659985Z inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_17_mid_dim_8_in_features_128_out_features_65_cuda_bfloat16 ('RERUN', {'yellow': True}) [1.9069s] [ 2%] 2025-12-04T11:13:50.1660477Z inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_17_mid_dim_8_in_features_128_out_features_65_cuda_bfloat16 ('RERUN', {'yellow': True}) [0.4654s] [ 2%] 2025-12-04T11:13:50.1660918Z inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_17_mid_dim_8_in_features_128_out_features_65_cuda_bfloat16 FAILED [0.4635s] [ 2%] 2025-12-04T11:13:50.1660921Z 2025-12-04T11:13:50.1661007Z ==================================== RERUNS ==================================== 2025-12-04T11:13:50.1661299Z _ TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_8_in_features_128_out_features_65_cuda_bfloat16 _ 2025-12-04T11:13:50.1661376Z Traceback (most recent call last): 2025-12-04T11:13:50.1661696Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.1661761Z method(*args, **kwargs) 2025-12-04T11:13:50.1662088Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.1662154Z method(*args, **kwargs) 2025-12-04T11:13:50.1662439Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T11:13:50.1662501Z with policy(): 2025-12-04T11:13:50.1662792Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T11:13:50.1662856Z raise RuntimeError(msg) 2025-12-04T11:13:50.1663697Z RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_8_in_features_128_out_features_65_cuda_bfloat16! Caching allocator allocated memory was 0 and is now reported as 8704 on device 0. CUDA driver allocated memory was 230686720 and is now 274726912. 2025-12-04T11:13:50.1663705Z 2025-12-04T11:13:50.1663832Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.1664351Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_8_in_features_128_out_features_65_cuda_bfloat16 2025-12-04T11:13:50.1664355Z 2025-12-04T11:13:50.1664510Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.1664636Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.1664742Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.1665091Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.1665223Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.1665282Z graph_break [] 2025-12-04T11:13:50.1665573Z _ TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_8_in_features_128_out_features_65_cuda_bfloat16 _ 2025-12-04T11:13:50.1665651Z Traceback (most recent call last): 2025-12-04T11:13:50.1665943Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.1666008Z method(*args, **kwargs) 2025-12-04T11:13:50.1666296Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.1666357Z method(*args, **kwargs) 2025-12-04T11:13:50.1666646Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T11:13:50.1666749Z with policy(): 2025-12-04T11:13:50.1667101Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T11:13:50.1667171Z raise RuntimeError(msg) 2025-12-04T11:13:50.1667972Z RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_8_in_features_128_out_features_65_cuda_bfloat16! Caching allocator allocated memory was 8704 and is now reported as 17408 on device 0. CUDA driver allocated memory was 274726912 and is now 276824064. 2025-12-04T11:13:50.1667976Z 2025-12-04T11:13:50.1668101Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.1668616Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_8_in_features_128_out_features_65_cuda_bfloat16 2025-12-04T11:13:50.1668619Z 2025-12-04T11:13:50.1668775Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.1668902Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.1668992Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.1669394Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.1669519Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.1669577Z graph_break [] 2025-12-04T11:13:50.1669704Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.1669792Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.1669914Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.1670256Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.1670316Z graph_break [] 2025-12-04T11:13:50.1670405Z =================================== FAILURES =================================== 2025-12-04T11:13:50.1670696Z _ TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_8_in_features_128_out_features_65_cuda_bfloat16 _ 2025-12-04T11:13:50.1670766Z Traceback (most recent call last): 2025-12-04T11:13:50.1671064Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.1671136Z method(*args, **kwargs) 2025-12-04T11:13:50.1671426Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.1671488Z method(*args, **kwargs) 2025-12-04T11:13:50.1671776Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T11:13:50.1671839Z with policy(): 2025-12-04T11:13:50.1672131Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T11:13:50.1672199Z raise RuntimeError(msg) 2025-12-04T11:13:50.1673003Z RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_8_in_features_128_out_features_65_cuda_bfloat16! Caching allocator allocated memory was 17408 and is now reported as 26112 on device 0. CUDA driver allocated memory was 276824064 and is now 278921216. 2025-12-04T11:13:50.1673007Z 2025-12-04T11:13:50.1673130Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.1673647Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_8_in_features_128_out_features_65_cuda_bfloat16 2025-12-04T11:13:50.1673692Z 2025-12-04T11:13:50.1673847Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.1674039Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.1674131Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.1674472Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.1674597Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.1674655Z graph_break [] 2025-12-04T11:13:50.1674781Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.1674869Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.1674988Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.1675329Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.1675389Z graph_break [] 2025-12-04T11:13:50.1675514Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.1675644Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.1675764Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.1676104Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.1676160Z graph_break [] 2025-12-04T11:13:50.1676642Z - generated xml file: /var/lib/jenkins/workspace/test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-2048f1d3bc205458.xml - 2025-12-04T11:13:50.1676743Z =========================== short test summary info ============================ 2025-12-04T11:13:50.1678661Z FAILED [0.4635s] inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_17_mid_dim_8_in_features_128_out_features_65_cuda_bfloat16 - RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_8_in_features_128_out_features_65_cuda_bfloat16! Caching allocator allocated memory was 17408 and is now reported as 26112 on device 0. CUDA driver allocated memory was 276824064 and is now 278921216. 2025-12-04T11:13:50.1678687Z 2025-12-04T11:13:50.1678909Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.1679598Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_8_in_features_128_out_features_65_cuda_bfloat16 2025-12-04T11:13:50.1679604Z 2025-12-04T11:13:50.1679826Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.1679970Z !!!!!!!!!!!!!!!!!!!!!!!!!! stopping after 1 failures !!!!!!!!!!!!!!!!!!!!!!!!!!! 2025-12-04T11:13:50.1680128Z ================== 1 failed, 18 deselected, 2 rerun in 2.86s =================== 2025-12-04T11:13:50.1680210Z Got exit code 1 2025-12-04T11:13:50.1680294Z Retrying single test... 2025-12-04T11:13:50.1680648Z W1204 10:42:04.141000 66834 site-packages/torch/_inductor/utils.py:1703] Not enough SMs to use max_autotune_gemm mode 2025-12-04T11:13:50.1681139Z Test results will be stored in test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-d670b011b4f3da72.xml 2025-12-04T11:13:50.1681267Z ============================= test session starts ============================== 2025-12-04T11:13:50.1681540Z platform linux -- Python 3.10.14, pytest-7.3.2, pluggy-1.6.0 -- /opt/conda/envs/py_3.10/bin/python 2025-12-04T11:13:50.1681625Z cachedir: .pytest_cache 2025-12-04T11:13:50.1682015Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow] 2025-12-04T11:13:50.1682278Z rootdir: /var/lib/jenkins/workspace 2025-12-04T11:13:50.1682359Z configfile: pytest.ini 2025-12-04T11:13:50.1682901Z plugins: hypothesis-6.56.4, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-14.0, subtests-0.13.1, xdist-3.3.1, xdoctest-1.3.0, typeguard-4.3.0 2025-12-04T11:13:50.1683080Z collecting ... collected 58 items / 57 deselected / 1 selected 2025-12-04T11:13:50.1683827Z stepcurrent: skipping 18 already run items. Running only test/inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_17_mid_dim_8_in_features_128_out_features_65_cuda_bfloat16 2025-12-04T11:13:50.1683921Z Running 1 items in this shard 2025-12-04T11:13:50.1683927Z 2025-12-04T11:13:50.1684841Z inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_17_mid_dim_8_in_features_128_out_features_65_cuda_bfloat16 [W1204 10:42:05.467706206 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1684849Z 2025-12-04T11:13:50.1685235Z [W1204 10:42:14.637719544 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1685305Z 2025-12-04T11:13:50.1685666Z [W1204 10:42:14.637951099 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1685670Z 2025-12-04T11:13:50.1686029Z [W1204 10:42:14.644044264 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1686033Z 2025-12-04T11:13:50.1686388Z [W1204 10:42:14.644607866 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1686392Z 2025-12-04T11:13:50.1686749Z [W1204 10:42:14.644787020 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1686755Z 2025-12-04T11:13:50.1687110Z [W1204 10:42:14.650160508 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1687116Z 2025-12-04T11:13:50.1687471Z [W1204 10:42:14.650694050 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1687480Z 2025-12-04T11:13:50.1687834Z [W1204 10:42:14.650880354 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1687839Z 2025-12-04T11:13:50.1687941Z ('RERUN', {'yellow': True}) [11.1080s] [100%] 2025-12-04T11:13:50.1688838Z inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_17_mid_dim_8_in_features_128_out_features_65_cuda_bfloat16 [W1204 10:42:15.651143450 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1688845Z 2025-12-04T11:13:50.1689203Z [W1204 10:42:15.651728753 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1689209Z 2025-12-04T11:13:50.1689571Z [W1204 10:42:15.651867026 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1689574Z 2025-12-04T11:13:50.1689858Z [W1204 10:42:15.654854752 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1689862Z 2025-12-04T11:13:50.1690148Z [W1204 10:42:15.655425015 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1690152Z 2025-12-04T11:13:50.1690435Z [W1204 10:42:15.655564318 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1690487Z 2025-12-04T11:13:50.1690849Z [W1204 10:42:15.660168180 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1690855Z 2025-12-04T11:13:50.1691139Z [W1204 10:42:15.660654480 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1691142Z 2025-12-04T11:13:50.1691424Z [W1204 10:42:15.660791394 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1691427Z 2025-12-04T11:13:50.1691510Z ('RERUN', {'yellow': True}) [0.4211s] [100%] 2025-12-04T11:13:50.1692221Z inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_17_mid_dim_8_in_features_128_out_features_65_cuda_bfloat16 [W1204 10:42:16.076886032 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1692226Z 2025-12-04T11:13:50.1692515Z [W1204 10:42:16.077460664 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1692553Z 2025-12-04T11:13:50.1692836Z [W1204 10:42:16.077607148 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1692840Z 2025-12-04T11:13:50.1693124Z [W1204 10:42:16.080737877 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1693127Z 2025-12-04T11:13:50.1693410Z [W1204 10:42:16.081320760 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1693412Z 2025-12-04T11:13:50.1693696Z [W1204 10:42:16.081456493 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1693701Z 2025-12-04T11:13:50.1693986Z [W1204 10:42:16.086274949 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1693989Z 2025-12-04T11:13:50.1694270Z [W1204 10:42:16.086746129 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1694276Z 2025-12-04T11:13:50.1694557Z [W1204 10:42:16.086878162 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1694560Z 2025-12-04T11:13:50.1694622Z FAILED [0.4220s] [100%] 2025-12-04T11:13:50.1694625Z 2025-12-04T11:13:50.1694714Z ==================================== RERUNS ==================================== 2025-12-04T11:13:50.1695013Z _ TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_8_in_features_128_out_features_65_cuda_bfloat16 _ 2025-12-04T11:13:50.1695092Z Traceback (most recent call last): 2025-12-04T11:13:50.1695428Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.1695500Z method(*args, **kwargs) 2025-12-04T11:13:50.1695812Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.1695876Z method(*args, **kwargs) 2025-12-04T11:13:50.1696175Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T11:13:50.1696242Z with policy(): 2025-12-04T11:13:50.1696536Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T11:13:50.1696605Z raise RuntimeError(msg) 2025-12-04T11:13:50.1697506Z RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_8_in_features_128_out_features_65_cuda_bfloat16! Caching allocator allocated memory was 0 and is now reported as 8704 on device 0. CUDA driver allocated memory was 230686720 and is now 274726912. 2025-12-04T11:13:50.1697546Z 2025-12-04T11:13:50.1697682Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.1698224Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_8_in_features_128_out_features_65_cuda_bfloat16 2025-12-04T11:13:50.1698228Z 2025-12-04T11:13:50.1698395Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.1698536Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.1698634Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.1698985Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.1699119Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.1699183Z graph_break [] 2025-12-04T11:13:50.1699320Z ----------------------------- Captured stderr call ----------------------------- 2025-12-04T11:13:50.1700066Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/constant_folding.py:256: UserWarning: Unsupported unwinding pattern: Address not in range (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/profiler/unwind/unwind.cpp:219.) 2025-12-04T11:13:50.1700137Z if out == self.unknown_value: 2025-12-04T11:13:50.1700436Z _ TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_8_in_features_128_out_features_65_cuda_bfloat16 _ 2025-12-04T11:13:50.1700512Z Traceback (most recent call last): 2025-12-04T11:13:50.1700821Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.1700887Z method(*args, **kwargs) 2025-12-04T11:13:50.1701177Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.1701248Z method(*args, **kwargs) 2025-12-04T11:13:50.1701549Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T11:13:50.1701610Z with policy(): 2025-12-04T11:13:50.1701915Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T11:13:50.1701980Z raise RuntimeError(msg) 2025-12-04T11:13:50.1702784Z RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_8_in_features_128_out_features_65_cuda_bfloat16! Caching allocator allocated memory was 8704 and is now reported as 17408 on device 0. CUDA driver allocated memory was 274726912 and is now 276824064. 2025-12-04T11:13:50.1702790Z 2025-12-04T11:13:50.1702916Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.1703442Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_8_in_features_128_out_features_65_cuda_bfloat16 2025-12-04T11:13:50.1703448Z 2025-12-04T11:13:50.1703698Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.1703826Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.1703923Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.1704275Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.1704405Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.1704513Z graph_break [] 2025-12-04T11:13:50.1704637Z ----------------------------- Captured stderr call ----------------------------- 2025-12-04T11:13:50.1705417Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/constant_folding.py:256: UserWarning: Unsupported unwinding pattern: Address not in range (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/profiler/unwind/unwind.cpp:219.) 2025-12-04T11:13:50.1705493Z if out == self.unknown_value: 2025-12-04T11:13:50.1705629Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.1705724Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.1705846Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.1706192Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.1706250Z graph_break [] 2025-12-04T11:13:50.1706337Z =================================== FAILURES =================================== 2025-12-04T11:13:50.1706632Z _ TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_8_in_features_128_out_features_65_cuda_bfloat16 _ 2025-12-04T11:13:50.1706749Z Traceback (most recent call last): 2025-12-04T11:13:50.1707049Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.1707113Z method(*args, **kwargs) 2025-12-04T11:13:50.1707400Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.1707475Z method(*args, **kwargs) 2025-12-04T11:13:50.1707765Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T11:13:50.1707824Z with policy(): 2025-12-04T11:13:50.1708125Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T11:13:50.1708192Z raise RuntimeError(msg) 2025-12-04T11:13:50.1709009Z RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_8_in_features_128_out_features_65_cuda_bfloat16! Caching allocator allocated memory was 17408 and is now reported as 26112 on device 0. CUDA driver allocated memory was 276824064 and is now 278921216. 2025-12-04T11:13:50.1709015Z 2025-12-04T11:13:50.1709142Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.1709671Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_8_in_features_128_out_features_65_cuda_bfloat16 2025-12-04T11:13:50.1709679Z 2025-12-04T11:13:50.1709834Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.1709960Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.1710055Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.1710404Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.1710528Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.1710586Z graph_break [] 2025-12-04T11:13:50.1710707Z ----------------------------- Captured stderr call ----------------------------- 2025-12-04T11:13:50.1711399Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/constant_folding.py:256: UserWarning: Unsupported unwinding pattern: Address not in range (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/profiler/unwind/unwind.cpp:219.) 2025-12-04T11:13:50.1711468Z if out == self.unknown_value: 2025-12-04T11:13:50.1711591Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.1711729Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.1711915Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.1712260Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.1712319Z graph_break [] 2025-12-04T11:13:50.1712440Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.1712541Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.1712665Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.1713005Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.1713065Z graph_break [] 2025-12-04T11:13:50.1713563Z - generated xml file: /var/lib/jenkins/workspace/test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-d670b011b4f3da72.xml - 2025-12-04T11:13:50.1713669Z =========================== short test summary info ============================ 2025-12-04T11:13:50.1714991Z FAILED [0.4220s] inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_17_mid_dim_8_in_features_128_out_features_65_cuda_bfloat16 - RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_8_in_features_128_out_features_65_cuda_bfloat16! Caching allocator allocated memory was 17408 and is now reported as 26112 on device 0. CUDA driver allocated memory was 276824064 and is now 278921216. 2025-12-04T11:13:50.1714996Z 2025-12-04T11:13:50.1715123Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.1715644Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_8_in_features_128_out_features_65_cuda_bfloat16 2025-12-04T11:13:50.1715653Z 2025-12-04T11:13:50.1715810Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.1715914Z !!!!!!!!!!!!!!!!!!!!!!!!!! stopping after 1 failures !!!!!!!!!!!!!!!!!!!!!!!!!!! 2025-12-04T11:13:50.1716038Z ================== 1 failed, 57 deselected, 2 rerun in 11.98s ================== 2025-12-04T11:13:50.1716100Z Got exit code 1 2025-12-04T11:13:50.1716165Z Retrying single test... 2025-12-04T11:13:50.1716426Z W1204 10:42:22.880000 67020 site-packages/torch/_inductor/utils.py:1703] Not enough SMs to use max_autotune_gemm mode 2025-12-04T11:13:50.1716820Z Test results will be stored in test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-56e0b2ecffd4aa2f.xml 2025-12-04T11:13:50.1716913Z ============================= test session starts ============================== 2025-12-04T11:13:50.1717124Z platform linux -- Python 3.10.14, pytest-7.3.2, pluggy-1.6.0 -- /opt/conda/envs/py_3.10/bin/python 2025-12-04T11:13:50.1717191Z cachedir: .pytest_cache 2025-12-04T11:13:50.1717494Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow] 2025-12-04T11:13:50.1717574Z rootdir: /var/lib/jenkins/workspace 2025-12-04T11:13:50.1717639Z configfile: pytest.ini 2025-12-04T11:13:50.1717955Z plugins: hypothesis-6.56.4, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-14.0, subtests-0.13.1, xdist-3.3.1, xdoctest-1.3.0, typeguard-4.3.0 2025-12-04T11:13:50.1718088Z collecting ... collected 58 items / 57 deselected / 1 selected 2025-12-04T11:13:50.1718651Z stepcurrent: skipping 18 already run items. Running only test/inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_17_mid_dim_8_in_features_128_out_features_65_cuda_bfloat16 2025-12-04T11:13:50.1718725Z Running 1 items in this shard 2025-12-04T11:13:50.1718770Z 2025-12-04T11:13:50.1719582Z inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_17_mid_dim_8_in_features_128_out_features_65_cuda_bfloat16 [W1204 10:42:24.190722628 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1719589Z 2025-12-04T11:13:50.1719892Z [W1204 10:42:33.248848789 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1719896Z 2025-12-04T11:13:50.1720185Z [W1204 10:42:33.249091615 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1720188Z 2025-12-04T11:13:50.1720473Z [W1204 10:42:33.254907712 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1720480Z 2025-12-04T11:13:50.1720767Z [W1204 10:42:33.255470124 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1720773Z 2025-12-04T11:13:50.1721057Z [W1204 10:42:33.255634368 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1721096Z 2025-12-04T11:13:50.1721385Z [W1204 10:42:33.261322092 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1721389Z 2025-12-04T11:13:50.1721672Z [W1204 10:42:33.261849023 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1721675Z 2025-12-04T11:13:50.1721961Z [W1204 10:42:33.262038388 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1721964Z 2025-12-04T11:13:50.1722044Z ('RERUN', {'yellow': True}) [10.9804s] [100%] 2025-12-04T11:13:50.1722765Z inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_17_mid_dim_8_in_features_128_out_features_65_cuda_bfloat16 [W1204 10:42:34.256840961 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1722770Z 2025-12-04T11:13:50.1723058Z [W1204 10:42:34.257418513 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1723062Z 2025-12-04T11:13:50.1723350Z [W1204 10:42:34.257558056 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1723353Z 2025-12-04T11:13:50.1723639Z [W1204 10:42:34.260523831 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1723642Z 2025-12-04T11:13:50.1723925Z [W1204 10:42:34.261114664 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1723934Z 2025-12-04T11:13:50.1724221Z [W1204 10:42:34.261251017 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1724226Z 2025-12-04T11:13:50.1724511Z [W1204 10:42:34.265802467 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1724513Z 2025-12-04T11:13:50.1724799Z [W1204 10:42:34.266270587 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1724802Z 2025-12-04T11:13:50.1725084Z [W1204 10:42:34.266407850 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1725087Z 2025-12-04T11:13:50.1725166Z ('RERUN', {'yellow': True}) [0.4107s] [100%] 2025-12-04T11:13:50.1725988Z inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_17_mid_dim_8_in_features_128_out_features_65_cuda_bfloat16 [W1204 10:42:34.667482306 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1725994Z 2025-12-04T11:13:50.1726286Z [W1204 10:42:34.668047469 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1726289Z 2025-12-04T11:13:50.1726573Z [W1204 10:42:34.668186582 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1726576Z 2025-12-04T11:13:50.1726865Z [W1204 10:42:34.671190258 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1726868Z 2025-12-04T11:13:50.1727152Z [W1204 10:42:34.671759520 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1727157Z 2025-12-04T11:13:50.1727442Z [W1204 10:42:34.671896523 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1727480Z 2025-12-04T11:13:50.1727782Z [W1204 10:42:34.676527354 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1727785Z 2025-12-04T11:13:50.1728069Z [W1204 10:42:34.677003314 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1728072Z 2025-12-04T11:13:50.1728359Z [W1204 10:42:34.677137807 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1728363Z 2025-12-04T11:13:50.1728425Z FAILED [0.4138s] [100%] 2025-12-04T11:13:50.1728429Z 2025-12-04T11:13:50.1728517Z ==================================== RERUNS ==================================== 2025-12-04T11:13:50.1728816Z _ TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_8_in_features_128_out_features_65_cuda_bfloat16 _ 2025-12-04T11:13:50.1728895Z Traceback (most recent call last): 2025-12-04T11:13:50.1729203Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.1729277Z method(*args, **kwargs) 2025-12-04T11:13:50.1729570Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.1729633Z method(*args, **kwargs) 2025-12-04T11:13:50.1729919Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T11:13:50.1729980Z with policy(): 2025-12-04T11:13:50.1730270Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T11:13:50.1730341Z raise RuntimeError(msg) 2025-12-04T11:13:50.1731139Z RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_8_in_features_128_out_features_65_cuda_bfloat16! Caching allocator allocated memory was 0 and is now reported as 8704 on device 0. CUDA driver allocated memory was 230686720 and is now 274726912. 2025-12-04T11:13:50.1731145Z 2025-12-04T11:13:50.1731272Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.1731793Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_8_in_features_128_out_features_65_cuda_bfloat16 2025-12-04T11:13:50.1731797Z 2025-12-04T11:13:50.1731957Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.1732090Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.1732235Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.1732672Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.1732805Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.1732863Z graph_break [] 2025-12-04T11:13:50.1732988Z ----------------------------- Captured stderr call ----------------------------- 2025-12-04T11:13:50.1733681Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/constant_folding.py:256: UserWarning: Unsupported unwinding pattern: Address not in range (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/profiler/unwind/unwind.cpp:219.) 2025-12-04T11:13:50.1733751Z if out == self.unknown_value: 2025-12-04T11:13:50.1734044Z _ TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_8_in_features_128_out_features_65_cuda_bfloat16 _ 2025-12-04T11:13:50.1734118Z Traceback (most recent call last): 2025-12-04T11:13:50.1734414Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.1734517Z method(*args, **kwargs) 2025-12-04T11:13:50.1734805Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.1734869Z method(*args, **kwargs) 2025-12-04T11:13:50.1735156Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T11:13:50.1735213Z with policy(): 2025-12-04T11:13:50.1735506Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T11:13:50.1735571Z raise RuntimeError(msg) 2025-12-04T11:13:50.1736375Z RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_8_in_features_128_out_features_65_cuda_bfloat16! Caching allocator allocated memory was 8704 and is now reported as 17408 on device 0. CUDA driver allocated memory was 274726912 and is now 276824064. 2025-12-04T11:13:50.1736390Z 2025-12-04T11:13:50.1736521Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.1737035Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_8_in_features_128_out_features_65_cuda_bfloat16 2025-12-04T11:13:50.1737039Z 2025-12-04T11:13:50.1737198Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.1737321Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.1737415Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.1737767Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.1737893Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.1737957Z graph_break [] 2025-12-04T11:13:50.1738078Z ----------------------------- Captured stderr call ----------------------------- 2025-12-04T11:13:50.1738768Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/constant_folding.py:256: UserWarning: Unsupported unwinding pattern: Address not in range (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/profiler/unwind/unwind.cpp:219.) 2025-12-04T11:13:50.1738836Z if out == self.unknown_value: 2025-12-04T11:13:50.1738959Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.1739051Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.1739170Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.1739619Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.1739682Z graph_break [] 2025-12-04T11:13:50.1739764Z =================================== FAILURES =================================== 2025-12-04T11:13:50.1740056Z _ TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_8_in_features_128_out_features_65_cuda_bfloat16 _ 2025-12-04T11:13:50.1740126Z Traceback (most recent call last): 2025-12-04T11:13:50.1740419Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.1740483Z method(*args, **kwargs) 2025-12-04T11:13:50.1740774Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.1740835Z method(*args, **kwargs) 2025-12-04T11:13:50.1741125Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T11:13:50.1741186Z with policy(): 2025-12-04T11:13:50.1741478Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T11:13:50.1741580Z raise RuntimeError(msg) 2025-12-04T11:13:50.1742388Z RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_8_in_features_128_out_features_65_cuda_bfloat16! Caching allocator allocated memory was 17408 and is now reported as 26112 on device 0. CUDA driver allocated memory was 276824064 and is now 278921216. 2025-12-04T11:13:50.1742395Z 2025-12-04T11:13:50.1742517Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.1743030Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_8_in_features_128_out_features_65_cuda_bfloat16 2025-12-04T11:13:50.1743036Z 2025-12-04T11:13:50.1743196Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.1743320Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.1743411Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.1743836Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.1743960Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.1744022Z graph_break [] 2025-12-04T11:13:50.1744144Z ----------------------------- Captured stderr call ----------------------------- 2025-12-04T11:13:50.1744827Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/constant_folding.py:256: UserWarning: Unsupported unwinding pattern: Address not in range (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/profiler/unwind/unwind.cpp:219.) 2025-12-04T11:13:50.1744904Z if out == self.unknown_value: 2025-12-04T11:13:50.1745027Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.1745120Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.1745239Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.1745596Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.1745658Z graph_break [] 2025-12-04T11:13:50.1745778Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.1745867Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.1745988Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.1746449Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.1746511Z graph_break [] 2025-12-04T11:13:50.1746998Z - generated xml file: /var/lib/jenkins/workspace/test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-56e0b2ecffd4aa2f.xml - 2025-12-04T11:13:50.1747102Z =========================== short test summary info ============================ 2025-12-04T11:13:50.1748384Z FAILED [0.4138s] inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_17_mid_dim_8_in_features_128_out_features_65_cuda_bfloat16 - RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_8_in_features_128_out_features_65_cuda_bfloat16! Caching allocator allocated memory was 17408 and is now reported as 26112 on device 0. CUDA driver allocated memory was 276824064 and is now 278921216. 2025-12-04T11:13:50.1748390Z 2025-12-04T11:13:50.1748515Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.1749034Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_8_in_features_128_out_features_65_cuda_bfloat16 2025-12-04T11:13:50.1749084Z 2025-12-04T11:13:50.1749236Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.1749341Z !!!!!!!!!!!!!!!!!!!!!!!!!! stopping after 1 failures !!!!!!!!!!!!!!!!!!!!!!!!!!! 2025-12-04T11:13:50.1749456Z ================== 1 failed, 57 deselected, 2 rerun in 11.83s ================== 2025-12-04T11:13:50.1749514Z Got exit code 1 2025-12-04T11:13:50.1749995Z FAILED CONSISTENTLY: test/inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_17_mid_dim_8_in_features_128_out_features_65_cuda_bfloat16 2025-12-04T11:13:50.1750239Z Test failed consistently, continuing with the rest of the tests due to continue-through-error being set 2025-12-04T11:13:50.1750506Z W1204 10:42:41.534000 67206 site-packages/torch/_inductor/utils.py:1703] Not enough SMs to use max_autotune_gemm mode 2025-12-04T11:13:50.1750889Z Test results will be stored in test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-1e9198c3a728666b.xml 2025-12-04T11:13:50.1750985Z ============================= test session starts ============================== 2025-12-04T11:13:50.1751199Z platform linux -- Python 3.10.14, pytest-7.3.2, pluggy-1.6.0 -- /opt/conda/envs/py_3.10/bin/python 2025-12-04T11:13:50.1751266Z cachedir: .pytest_cache 2025-12-04T11:13:50.1751571Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow] 2025-12-04T11:13:50.1751647Z rootdir: /var/lib/jenkins/workspace 2025-12-04T11:13:50.1751712Z configfile: pytest.ini 2025-12-04T11:13:50.1752030Z plugins: hypothesis-6.56.4, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-14.0, subtests-0.13.1, xdist-3.3.1, xdoctest-1.3.0, typeguard-4.3.0 2025-12-04T11:13:50.1752156Z collecting ... collected 58 items / 19 deselected / 39 selected 2025-12-04T11:13:50.1752250Z stepcurrent: skipping 19 already run items. 2025-12-04T11:13:50.1752331Z Running 39 items in this shard 2025-12-04T11:13:50.1752335Z 2025-12-04T11:13:50.1752835Z inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_17_mid_dim_8_in_features_144_out_features_1024_cuda_bfloat16 ('RERUN', {'yellow': True}) [1.9217s] [ 2%] 2025-12-04T11:13:50.1753326Z inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_17_mid_dim_8_in_features_144_out_features_1024_cuda_bfloat16 ('RERUN', {'yellow': True}) [0.5026s] [ 2%] 2025-12-04T11:13:50.1753769Z inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_17_mid_dim_8_in_features_144_out_features_1024_cuda_bfloat16 FAILED [0.5133s] [ 2%] 2025-12-04T11:13:50.1753814Z 2025-12-04T11:13:50.1753965Z ==================================== RERUNS ==================================== 2025-12-04T11:13:50.1754261Z _ TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_8_in_features_144_out_features_1024_cuda_bfloat16 _ 2025-12-04T11:13:50.1754333Z Traceback (most recent call last): 2025-12-04T11:13:50.1754640Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.1754705Z method(*args, **kwargs) 2025-12-04T11:13:50.1754997Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.1755061Z method(*args, **kwargs) 2025-12-04T11:13:50.1755346Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T11:13:50.1755413Z with policy(): 2025-12-04T11:13:50.1755717Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T11:13:50.1755825Z raise RuntimeError(msg) 2025-12-04T11:13:50.1756644Z RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_8_in_features_144_out_features_1024_cuda_bfloat16! Caching allocator allocated memory was 0 and is now reported as 147456 on device 0. CUDA driver allocated memory was 230686720 and is now 274726912. 2025-12-04T11:13:50.1756648Z 2025-12-04T11:13:50.1756777Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.1757305Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_8_in_features_144_out_features_1024_cuda_bfloat16 2025-12-04T11:13:50.1757311Z 2025-12-04T11:13:50.1757469Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.1757602Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.1757708Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.1758060Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.1758194Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.1758253Z graph_break [] 2025-12-04T11:13:50.1758545Z _ TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_8_in_features_144_out_features_1024_cuda_bfloat16 _ 2025-12-04T11:13:50.1758624Z Traceback (most recent call last): 2025-12-04T11:13:50.1758933Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.1759002Z method(*args, **kwargs) 2025-12-04T11:13:50.1759296Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.1759368Z method(*args, **kwargs) 2025-12-04T11:13:50.1759661Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T11:13:50.1759720Z with policy(): 2025-12-04T11:13:50.1760017Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T11:13:50.1760083Z raise RuntimeError(msg) 2025-12-04T11:13:50.1760903Z RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_8_in_features_144_out_features_1024_cuda_bfloat16! Caching allocator allocated memory was 147456 and is now reported as 294912 on device 0. CUDA driver allocated memory was 274726912 and is now 276824064. 2025-12-04T11:13:50.1760968Z 2025-12-04T11:13:50.1761097Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.1761687Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_8_in_features_144_out_features_1024_cuda_bfloat16 2025-12-04T11:13:50.1761693Z 2025-12-04T11:13:50.1761850Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.1761978Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.1762070Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.1762417Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.1762543Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.1762605Z graph_break [] 2025-12-04T11:13:50.1762728Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.1762818Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.1762986Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.1763330Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.1763387Z graph_break [] 2025-12-04T11:13:50.1763474Z =================================== FAILURES =================================== 2025-12-04T11:13:50.1763763Z _ TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_8_in_features_144_out_features_1024_cuda_bfloat16 _ 2025-12-04T11:13:50.1763839Z Traceback (most recent call last): 2025-12-04T11:13:50.1764140Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.1764206Z method(*args, **kwargs) 2025-12-04T11:13:50.1764510Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.1764575Z method(*args, **kwargs) 2025-12-04T11:13:50.1764871Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T11:13:50.1764933Z with policy(): 2025-12-04T11:13:50.1765225Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T11:13:50.1765294Z raise RuntimeError(msg) 2025-12-04T11:13:50.1766113Z RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_8_in_features_144_out_features_1024_cuda_bfloat16! Caching allocator allocated memory was 294912 and is now reported as 442368 on device 0. CUDA driver allocated memory was 276824064 and is now 278921216. 2025-12-04T11:13:50.1766119Z 2025-12-04T11:13:50.1766252Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.1766776Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_8_in_features_144_out_features_1024_cuda_bfloat16 2025-12-04T11:13:50.1766781Z 2025-12-04T11:13:50.1766939Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.1767066Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.1767156Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.1767505Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.1767627Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.1767731Z graph_break [] 2025-12-04T11:13:50.1767925Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.1768015Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.1768135Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.1768481Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.1768540Z graph_break [] 2025-12-04T11:13:50.1768664Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.1768750Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.1768871Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.1769209Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.1769270Z graph_break [] 2025-12-04T11:13:50.1769772Z - generated xml file: /var/lib/jenkins/workspace/test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-1e9198c3a728666b.xml - 2025-12-04T11:13:50.1769921Z =========================== short test summary info ============================ 2025-12-04T11:13:50.1771216Z FAILED [0.5133s] inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_17_mid_dim_8_in_features_144_out_features_1024_cuda_bfloat16 - RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_8_in_features_144_out_features_1024_cuda_bfloat16! Caching allocator allocated memory was 294912 and is now reported as 442368 on device 0. CUDA driver allocated memory was 276824064 and is now 278921216. 2025-12-04T11:13:50.1771223Z 2025-12-04T11:13:50.1771349Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.1771872Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_8_in_features_144_out_features_1024_cuda_bfloat16 2025-12-04T11:13:50.1771877Z 2025-12-04T11:13:50.1772036Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.1772139Z !!!!!!!!!!!!!!!!!!!!!!!!!! stopping after 1 failures !!!!!!!!!!!!!!!!!!!!!!!!!!! 2025-12-04T11:13:50.1772260Z ================== 1 failed, 19 deselected, 2 rerun in 2.96s =================== 2025-12-04T11:13:50.1772319Z Got exit code 1 2025-12-04T11:13:50.1772383Z Retrying single test... 2025-12-04T11:13:50.1772647Z W1204 10:42:51.588000 67395 site-packages/torch/_inductor/utils.py:1703] Not enough SMs to use max_autotune_gemm mode 2025-12-04T11:13:50.1773031Z Test results will be stored in test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-c1c9525c22bede8d.xml 2025-12-04T11:13:50.1773130Z ============================= test session starts ============================== 2025-12-04T11:13:50.1773342Z platform linux -- Python 3.10.14, pytest-7.3.2, pluggy-1.6.0 -- /opt/conda/envs/py_3.10/bin/python 2025-12-04T11:13:50.1773409Z cachedir: .pytest_cache 2025-12-04T11:13:50.1773717Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow] 2025-12-04T11:13:50.1773793Z rootdir: /var/lib/jenkins/workspace 2025-12-04T11:13:50.1773855Z configfile: pytest.ini 2025-12-04T11:13:50.1774172Z plugins: hypothesis-6.56.4, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-14.0, subtests-0.13.1, xdist-3.3.1, xdoctest-1.3.0, typeguard-4.3.0 2025-12-04T11:13:50.1774299Z collecting ... collected 58 items / 57 deselected / 1 selected 2025-12-04T11:13:50.1774870Z stepcurrent: skipping 19 already run items. Running only test/inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_17_mid_dim_8_in_features_144_out_features_1024_cuda_bfloat16 2025-12-04T11:13:50.1775054Z Running 1 items in this shard 2025-12-04T11:13:50.1775058Z 2025-12-04T11:13:50.1775789Z inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_17_mid_dim_8_in_features_144_out_features_1024_cuda_bfloat16 [W1204 10:42:52.718799791 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1775798Z 2025-12-04T11:13:50.1776100Z [W1204 10:43:02.963419878 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1776104Z 2025-12-04T11:13:50.1776391Z [W1204 10:43:02.963657733 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1776395Z 2025-12-04T11:13:50.1776685Z [W1204 10:43:02.969362969 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1776690Z 2025-12-04T11:13:50.1776976Z [W1204 10:43:02.969922751 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1777014Z 2025-12-04T11:13:50.1777302Z [W1204 10:43:02.970152336 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1777305Z 2025-12-04T11:13:50.1777588Z [W1204 10:43:02.975584446 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1777592Z 2025-12-04T11:13:50.1778065Z [W1204 10:43:02.976115398 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1778069Z 2025-12-04T11:13:50.1778359Z [W1204 10:43:02.976272592 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1778365Z 2025-12-04T11:13:50.1778453Z ('RERUN', {'yellow': True}) [11.1712s] [100%] 2025-12-04T11:13:50.1779181Z inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_17_mid_dim_8_in_features_144_out_features_1024_cuda_bfloat16 [W1204 10:43:03.203427188 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1779187Z 2025-12-04T11:13:50.1779476Z [W1204 10:43:03.203973590 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1779480Z 2025-12-04T11:13:50.1779770Z [W1204 10:43:03.204115883 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1779773Z 2025-12-04T11:13:50.1780060Z [W1204 10:43:03.207132540 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1780064Z 2025-12-04T11:13:50.1780356Z [W1204 10:43:03.207709713 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1780360Z 2025-12-04T11:13:50.1780644Z [W1204 10:43:03.207846836 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1780647Z 2025-12-04T11:13:50.1780933Z [W1204 10:43:03.212482828 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1780936Z 2025-12-04T11:13:50.1781219Z [W1204 10:43:03.212972539 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1781223Z 2025-12-04T11:13:50.1781511Z [W1204 10:43:03.213108712 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1781592Z 2025-12-04T11:13:50.1781771Z ('RERUN', {'yellow': True}) [0.4626s] [100%] 2025-12-04T11:13:50.1782497Z inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_17_mid_dim_8_in_features_144_out_features_1024_cuda_bfloat16 [W1204 10:43:03.661609267 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1782505Z 2025-12-04T11:13:50.1782792Z [W1204 10:43:03.662160330 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1782796Z 2025-12-04T11:13:50.1783079Z [W1204 10:43:03.662302993 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1783082Z 2025-12-04T11:13:50.1783371Z [W1204 10:43:03.665265218 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1783376Z 2025-12-04T11:13:50.1783705Z [W1204 10:43:03.665843151 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1783779Z 2025-12-04T11:13:50.1784067Z [W1204 10:43:03.665982354 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1784071Z 2025-12-04T11:13:50.1784355Z [W1204 10:43:03.670635857 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1784358Z 2025-12-04T11:13:50.1784645Z [W1204 10:43:03.671115537 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1784648Z 2025-12-04T11:13:50.1784932Z [W1204 10:43:03.671256471 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1784937Z 2025-12-04T11:13:50.1784998Z FAILED [0.4596s] [100%] 2025-12-04T11:13:50.1785003Z 2025-12-04T11:13:50.1785090Z ==================================== RERUNS ==================================== 2025-12-04T11:13:50.1785384Z _ TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_8_in_features_144_out_features_1024_cuda_bfloat16 _ 2025-12-04T11:13:50.1785461Z Traceback (most recent call last): 2025-12-04T11:13:50.1785785Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.1785854Z method(*args, **kwargs) 2025-12-04T11:13:50.1786151Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.1786215Z method(*args, **kwargs) 2025-12-04T11:13:50.1786504Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T11:13:50.1786568Z with policy(): 2025-12-04T11:13:50.1786863Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T11:13:50.1786932Z raise RuntimeError(msg) 2025-12-04T11:13:50.1787734Z RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_8_in_features_144_out_features_1024_cuda_bfloat16! Caching allocator allocated memory was 0 and is now reported as 147456 on device 0. CUDA driver allocated memory was 230686720 and is now 274726912. 2025-12-04T11:13:50.1787738Z 2025-12-04T11:13:50.1787869Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.1788390Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_8_in_features_144_out_features_1024_cuda_bfloat16 2025-12-04T11:13:50.1788440Z 2025-12-04T11:13:50.1788600Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.1788798Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.1788894Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.1789247Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.1789374Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.1789437Z graph_break [] 2025-12-04T11:13:50.1789565Z ----------------------------- Captured stderr call ----------------------------- 2025-12-04T11:13:50.1790258Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/constant_folding.py:256: UserWarning: Unsupported unwinding pattern: Address not in range (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/profiler/unwind/unwind.cpp:219.) 2025-12-04T11:13:50.1790332Z if out == self.unknown_value: 2025-12-04T11:13:50.1790626Z _ TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_8_in_features_144_out_features_1024_cuda_bfloat16 _ 2025-12-04T11:13:50.1790735Z Traceback (most recent call last): 2025-12-04T11:13:50.1791038Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.1791102Z method(*args, **kwargs) 2025-12-04T11:13:50.1791390Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.1791458Z method(*args, **kwargs) 2025-12-04T11:13:50.1791744Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T11:13:50.1791807Z with policy(): 2025-12-04T11:13:50.1792101Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T11:13:50.1792168Z raise RuntimeError(msg) 2025-12-04T11:13:50.1792993Z RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_8_in_features_144_out_features_1024_cuda_bfloat16! Caching allocator allocated memory was 147456 and is now reported as 294912 on device 0. CUDA driver allocated memory was 274726912 and is now 276824064. 2025-12-04T11:13:50.1792998Z 2025-12-04T11:13:50.1793123Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.1793642Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_8_in_features_144_out_features_1024_cuda_bfloat16 2025-12-04T11:13:50.1793646Z 2025-12-04T11:13:50.1793799Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.1793929Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.1794026Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.1794379Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.1794510Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.1794568Z graph_break [] 2025-12-04T11:13:50.1794693Z ----------------------------- Captured stderr call ----------------------------- 2025-12-04T11:13:50.1795402Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/constant_folding.py:256: UserWarning: Unsupported unwinding pattern: Address not in range (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/profiler/unwind/unwind.cpp:219.) 2025-12-04T11:13:50.1795472Z if out == self.unknown_value: 2025-12-04T11:13:50.1795596Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.1795730Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.1795919Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.1796271Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.1796331Z graph_break [] 2025-12-04T11:13:50.1796419Z =================================== FAILURES =================================== 2025-12-04T11:13:50.1796711Z _ TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_8_in_features_144_out_features_1024_cuda_bfloat16 _ 2025-12-04T11:13:50.1796782Z Traceback (most recent call last): 2025-12-04T11:13:50.1797094Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.1797158Z method(*args, **kwargs) 2025-12-04T11:13:50.1797451Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.1797520Z method(*args, **kwargs) 2025-12-04T11:13:50.1797805Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T11:13:50.1797905Z with policy(): 2025-12-04T11:13:50.1798195Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T11:13:50.1798260Z raise RuntimeError(msg) 2025-12-04T11:13:50.1799082Z RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_8_in_features_144_out_features_1024_cuda_bfloat16! Caching allocator allocated memory was 294912 and is now reported as 442368 on device 0. CUDA driver allocated memory was 276824064 and is now 278921216. 2025-12-04T11:13:50.1799086Z 2025-12-04T11:13:50.1799210Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.1799739Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_8_in_features_144_out_features_1024_cuda_bfloat16 2025-12-04T11:13:50.1799744Z 2025-12-04T11:13:50.1799908Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.1800034Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.1800127Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.1800467Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.1800591Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.1800650Z graph_break [] 2025-12-04T11:13:50.1800771Z ----------------------------- Captured stderr call ----------------------------- 2025-12-04T11:13:50.1801468Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/constant_folding.py:256: UserWarning: Unsupported unwinding pattern: Address not in range (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/profiler/unwind/unwind.cpp:219.) 2025-12-04T11:13:50.1801537Z if out == self.unknown_value: 2025-12-04T11:13:50.1801660Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.1801750Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.1801871Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.1802212Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.1802270Z graph_break [] 2025-12-04T11:13:50.1802392Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.1802530Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.1802719Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.1803065Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.1803125Z graph_break [] 2025-12-04T11:13:50.1803614Z - generated xml file: /var/lib/jenkins/workspace/test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-c1c9525c22bede8d.xml - 2025-12-04T11:13:50.1803728Z =========================== short test summary info ============================ 2025-12-04T11:13:50.1805023Z FAILED [0.4596s] inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_17_mid_dim_8_in_features_144_out_features_1024_cuda_bfloat16 - RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_8_in_features_144_out_features_1024_cuda_bfloat16! Caching allocator allocated memory was 294912 and is now reported as 442368 on device 0. CUDA driver allocated memory was 276824064 and is now 278921216. 2025-12-04T11:13:50.1805065Z 2025-12-04T11:13:50.1805195Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.1805716Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_8_in_features_144_out_features_1024_cuda_bfloat16 2025-12-04T11:13:50.1805720Z 2025-12-04T11:13:50.1805876Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.1805978Z !!!!!!!!!!!!!!!!!!!!!!!!!! stopping after 1 failures !!!!!!!!!!!!!!!!!!!!!!!!!!! 2025-12-04T11:13:50.1806092Z ================== 1 failed, 57 deselected, 2 rerun in 12.12s ================== 2025-12-04T11:13:50.1806155Z Got exit code 1 2025-12-04T11:13:50.1806220Z Retrying single test... 2025-12-04T11:13:50.1806486Z W1204 10:43:10.536000 67588 site-packages/torch/_inductor/utils.py:1703] Not enough SMs to use max_autotune_gemm mode 2025-12-04T11:13:50.1806872Z Test results will be stored in test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-cc4ec46989be7746.xml 2025-12-04T11:13:50.1806965Z ============================= test session starts ============================== 2025-12-04T11:13:50.1807177Z platform linux -- Python 3.10.14, pytest-7.3.2, pluggy-1.6.0 -- /opt/conda/envs/py_3.10/bin/python 2025-12-04T11:13:50.1807242Z cachedir: .pytest_cache 2025-12-04T11:13:50.1807543Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow] 2025-12-04T11:13:50.1807621Z rootdir: /var/lib/jenkins/workspace 2025-12-04T11:13:50.1807687Z configfile: pytest.ini 2025-12-04T11:13:50.1808006Z plugins: hypothesis-6.56.4, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-14.0, subtests-0.13.1, xdist-3.3.1, xdoctest-1.3.0, typeguard-4.3.0 2025-12-04T11:13:50.1808138Z collecting ... collected 58 items / 57 deselected / 1 selected 2025-12-04T11:13:50.1808705Z stepcurrent: skipping 19 already run items. Running only test/inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_17_mid_dim_8_in_features_144_out_features_1024_cuda_bfloat16 2025-12-04T11:13:50.1808782Z Running 1 items in this shard 2025-12-04T11:13:50.1808786Z 2025-12-04T11:13:50.1809519Z inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_17_mid_dim_8_in_features_144_out_features_1024_cuda_bfloat16 [W1204 10:43:11.659814201 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1809523Z 2025-12-04T11:13:50.1809822Z [W1204 10:43:20.784696996 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1809868Z 2025-12-04T11:13:50.1810240Z [W1204 10:43:20.784934211 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1810246Z 2025-12-04T11:13:50.1810536Z [W1204 10:43:20.790655588 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1810540Z 2025-12-04T11:13:50.1810825Z [W1204 10:43:20.791203950 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1810828Z 2025-12-04T11:13:50.1811126Z [W1204 10:43:20.791377324 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1811129Z 2025-12-04T11:13:50.1811420Z [W1204 10:43:20.796791522 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1811425Z 2025-12-04T11:13:50.1811711Z [W1204 10:43:20.797320684 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1811759Z 2025-12-04T11:13:50.1812043Z [W1204 10:43:20.797473768 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1812047Z 2025-12-04T11:13:50.1812128Z ('RERUN', {'yellow': True}) [11.0446s] [100%] 2025-12-04T11:13:50.1812856Z inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_17_mid_dim_8_in_features_144_out_features_1024_cuda_bfloat16 [W1204 10:43:22.019192376 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1812859Z 2025-12-04T11:13:50.1813146Z [W1204 10:43:22.019744168 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1813151Z 2025-12-04T11:13:50.1813442Z [W1204 10:43:22.019881951 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1813446Z 2025-12-04T11:13:50.1813730Z [W1204 10:43:22.022917128 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1813733Z 2025-12-04T11:13:50.1814023Z [W1204 10:43:22.023495330 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1814026Z 2025-12-04T11:13:50.1814308Z [W1204 10:43:22.023632943 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1814311Z 2025-12-04T11:13:50.1814599Z [W1204 10:43:22.028257175 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1814603Z 2025-12-04T11:13:50.1814890Z [W1204 10:43:22.028733886 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1814893Z 2025-12-04T11:13:50.1815178Z [W1204 10:43:22.028868998 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1815181Z 2025-12-04T11:13:50.1815263Z ('RERUN', {'yellow': True}) [0.4567s] [100%] 2025-12-04T11:13:50.1815987Z inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_17_mid_dim_8_in_features_144_out_features_1024_cuda_bfloat16 [W1204 10:43:22.470720743 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1815991Z 2025-12-04T11:13:50.1816283Z [W1204 10:43:22.471265105 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1816326Z 2025-12-04T11:13:50.1816678Z [W1204 10:43:22.471407838 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1816682Z 2025-12-04T11:13:50.1816971Z [W1204 10:43:22.474366533 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1816975Z 2025-12-04T11:13:50.1817262Z [W1204 10:43:22.474932016 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1817265Z 2025-12-04T11:13:50.1817554Z [W1204 10:43:22.475069579 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1817557Z 2025-12-04T11:13:50.1817842Z [W1204 10:43:22.479640149 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1817847Z 2025-12-04T11:13:50.1818133Z [W1204 10:43:22.480126530 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1818139Z 2025-12-04T11:13:50.1818460Z [W1204 10:43:22.480267353 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1818464Z 2025-12-04T11:13:50.1818525Z FAILED [0.4518s] [100%] 2025-12-04T11:13:50.1818529Z 2025-12-04T11:13:50.1818624Z ==================================== RERUNS ==================================== 2025-12-04T11:13:50.1818919Z _ TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_8_in_features_144_out_features_1024_cuda_bfloat16 _ 2025-12-04T11:13:50.1818996Z Traceback (most recent call last): 2025-12-04T11:13:50.1819302Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.1819365Z method(*args, **kwargs) 2025-12-04T11:13:50.1819661Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.1819727Z method(*args, **kwargs) 2025-12-04T11:13:50.1820020Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T11:13:50.1820082Z with policy(): 2025-12-04T11:13:50.1820376Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T11:13:50.1820443Z raise RuntimeError(msg) 2025-12-04T11:13:50.1821248Z RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_8_in_features_144_out_features_1024_cuda_bfloat16! Caching allocator allocated memory was 0 and is now reported as 147456 on device 0. CUDA driver allocated memory was 230686720 and is now 274726912. 2025-12-04T11:13:50.1821254Z 2025-12-04T11:13:50.1821380Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.1821909Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_8_in_features_144_out_features_1024_cuda_bfloat16 2025-12-04T11:13:50.1821914Z 2025-12-04T11:13:50.1822070Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.1822200Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.1822294Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.1822640Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.1822781Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.1822840Z graph_break [] 2025-12-04T11:13:50.1823019Z ----------------------------- Captured stderr call ----------------------------- 2025-12-04T11:13:50.1823825Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/constant_folding.py:256: UserWarning: Unsupported unwinding pattern: Address not in range (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/profiler/unwind/unwind.cpp:219.) 2025-12-04T11:13:50.1823898Z if out == self.unknown_value: 2025-12-04T11:13:50.1824190Z _ TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_8_in_features_144_out_features_1024_cuda_bfloat16 _ 2025-12-04T11:13:50.1824264Z Traceback (most recent call last): 2025-12-04T11:13:50.1824561Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.1824624Z method(*args, **kwargs) 2025-12-04T11:13:50.1824915Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.1824982Z method(*args, **kwargs) 2025-12-04T11:13:50.1825272Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T11:13:50.1825330Z with policy(): 2025-12-04T11:13:50.1825663Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T11:13:50.1825727Z raise RuntimeError(msg) 2025-12-04T11:13:50.1826544Z RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_8_in_features_144_out_features_1024_cuda_bfloat16! Caching allocator allocated memory was 147456 and is now reported as 294912 on device 0. CUDA driver allocated memory was 274726912 and is now 276824064. 2025-12-04T11:13:50.1826548Z 2025-12-04T11:13:50.1826671Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.1827195Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_8_in_features_144_out_features_1024_cuda_bfloat16 2025-12-04T11:13:50.1827199Z 2025-12-04T11:13:50.1827355Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.1827478Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.1827573Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.1827919Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.1828045Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.1828104Z graph_break [] 2025-12-04T11:13:50.1828228Z ----------------------------- Captured stderr call ----------------------------- 2025-12-04T11:13:50.1828925Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/constant_folding.py:256: UserWarning: Unsupported unwinding pattern: Address not in range (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/profiler/unwind/unwind.cpp:219.) 2025-12-04T11:13:50.1828994Z if out == self.unknown_value: 2025-12-04T11:13:50.1829118Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.1829212Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.1829334Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.1829678Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.1829738Z graph_break [] 2025-12-04T11:13:50.1829821Z =================================== FAILURES =================================== 2025-12-04T11:13:50.1830114Z _ TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_8_in_features_144_out_features_1024_cuda_bfloat16 _ 2025-12-04T11:13:50.1830228Z Traceback (most recent call last): 2025-12-04T11:13:50.1830598Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.1830665Z method(*args, **kwargs) 2025-12-04T11:13:50.1830956Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.1831021Z method(*args, **kwargs) 2025-12-04T11:13:50.1831313Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T11:13:50.1831372Z with policy(): 2025-12-04T11:13:50.1831665Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T11:13:50.1831732Z raise RuntimeError(msg) 2025-12-04T11:13:50.1832562Z RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_8_in_features_144_out_features_1024_cuda_bfloat16! Caching allocator allocated memory was 294912 and is now reported as 442368 on device 0. CUDA driver allocated memory was 276824064 and is now 278921216. 2025-12-04T11:13:50.1832608Z 2025-12-04T11:13:50.1832735Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.1833255Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_8_in_features_144_out_features_1024_cuda_bfloat16 2025-12-04T11:13:50.1833259Z 2025-12-04T11:13:50.1833413Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.1833537Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.1833631Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.1833972Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.1834102Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.1834162Z graph_break [] 2025-12-04T11:13:50.1834283Z ----------------------------- Captured stderr call ----------------------------- 2025-12-04T11:13:50.1834975Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/constant_folding.py:256: UserWarning: Unsupported unwinding pattern: Address not in range (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/profiler/unwind/unwind.cpp:219.) 2025-12-04T11:13:50.1835044Z if out == self.unknown_value: 2025-12-04T11:13:50.1835165Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.1835260Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.1835381Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.1835733Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.1835796Z graph_break [] 2025-12-04T11:13:50.1835993Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.1836157Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.1836310Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.1836684Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.1836820Z graph_break [] 2025-12-04T11:13:50.1837471Z - generated xml file: /var/lib/jenkins/workspace/test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-cc4ec46989be7746.xml - 2025-12-04T11:13:50.1837688Z =========================== short test summary info ============================ 2025-12-04T11:13:50.1839119Z FAILED [0.4518s] inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_17_mid_dim_8_in_features_144_out_features_1024_cuda_bfloat16 - RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_8_in_features_144_out_features_1024_cuda_bfloat16! Caching allocator allocated memory was 294912 and is now reported as 442368 on device 0. CUDA driver allocated memory was 276824064 and is now 278921216. 2025-12-04T11:13:50.1839126Z 2025-12-04T11:13:50.1839299Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.1839906Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_8_in_features_144_out_features_1024_cuda_bfloat16 2025-12-04T11:13:50.1839910Z 2025-12-04T11:13:50.1840089Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.1840297Z !!!!!!!!!!!!!!!!!!!!!!!!!! stopping after 1 failures !!!!!!!!!!!!!!!!!!!!!!!!!!! 2025-12-04T11:13:50.1840497Z ================== 1 failed, 57 deselected, 2 rerun in 11.98s ================== 2025-12-04T11:13:50.1840631Z Got exit code 1 2025-12-04T11:13:50.1841192Z FAILED CONSISTENTLY: test/inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_17_mid_dim_8_in_features_144_out_features_1024_cuda_bfloat16 2025-12-04T11:13:50.1841470Z Test failed consistently, continuing with the rest of the tests due to continue-through-error being set 2025-12-04T11:13:50.1841746Z W1204 10:43:29.295000 67781 site-packages/torch/_inductor/utils.py:1703] Not enough SMs to use max_autotune_gemm mode 2025-12-04T11:13:50.1842272Z Test results will be stored in test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-3f42debbb8d01db8.xml 2025-12-04T11:13:50.1842402Z ============================= test session starts ============================== 2025-12-04T11:13:50.1842683Z platform linux -- Python 3.10.14, pytest-7.3.2, pluggy-1.6.0 -- /opt/conda/envs/py_3.10/bin/python 2025-12-04T11:13:50.1842801Z cachedir: .pytest_cache 2025-12-04T11:13:50.1843139Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow] 2025-12-04T11:13:50.1843321Z rootdir: /var/lib/jenkins/workspace 2025-12-04T11:13:50.1843440Z configfile: pytest.ini 2025-12-04T11:13:50.1843835Z plugins: hypothesis-6.56.4, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-14.0, subtests-0.13.1, xdist-3.3.1, xdoctest-1.3.0, typeguard-4.3.0 2025-12-04T11:13:50.1844016Z collecting ... collected 58 items / 20 deselected / 38 selected 2025-12-04T11:13:50.1844137Z stepcurrent: skipping 20 already run items. 2025-12-04T11:13:50.1844265Z Running 38 items in this shard 2025-12-04T11:13:50.1844269Z 2025-12-04T11:13:50.1844853Z inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_17_mid_dim_8_in_features_144_out_features_64_cuda_bfloat16 ('RERUN', {'yellow': True}) [1.8690s] [ 2%] 2025-12-04T11:13:50.1845423Z inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_17_mid_dim_8_in_features_144_out_features_64_cuda_bfloat16 ('RERUN', {'yellow': True}) [0.4578s] [ 2%] 2025-12-04T11:13:50.1845898Z inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_17_mid_dim_8_in_features_144_out_features_64_cuda_bfloat16 FAILED [0.4581s] [ 2%] 2025-12-04T11:13:50.1845903Z 2025-12-04T11:13:50.1846034Z ==================================== RERUNS ==================================== 2025-12-04T11:13:50.1846395Z _ TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_8_in_features_144_out_features_64_cuda_bfloat16 _ 2025-12-04T11:13:50.1846488Z Traceback (most recent call last): 2025-12-04T11:13:50.1846969Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.1847159Z method(*args, **kwargs) 2025-12-04T11:13:50.1847486Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.1847622Z method(*args, **kwargs) 2025-12-04T11:13:50.1847945Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T11:13:50.1848120Z with policy(): 2025-12-04T11:13:50.1848479Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T11:13:50.1848581Z raise RuntimeError(msg) 2025-12-04T11:13:50.1849441Z RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_8_in_features_144_out_features_64_cuda_bfloat16! Caching allocator allocated memory was 0 and is now reported as 9216 on device 0. CUDA driver allocated memory was 230686720 and is now 274726912. 2025-12-04T11:13:50.1849451Z 2025-12-04T11:13:50.1849607Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.1850216Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_8_in_features_144_out_features_64_cuda_bfloat16 2025-12-04T11:13:50.1850220Z 2025-12-04T11:13:50.1850460Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.1850688Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.1850814Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.1851195Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.1851388Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.1851469Z graph_break [] 2025-12-04T11:13:50.1851853Z _ TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_8_in_features_144_out_features_64_cuda_bfloat16 _ 2025-12-04T11:13:50.1852012Z Traceback (most recent call last): 2025-12-04T11:13:50.1852341Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.1852471Z method(*args, **kwargs) 2025-12-04T11:13:50.1852802Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.1852886Z method(*args, **kwargs) 2025-12-04T11:13:50.1853322Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T11:13:50.1853413Z with policy(): 2025-12-04T11:13:50.1853777Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T11:13:50.1853887Z raise RuntimeError(msg) 2025-12-04T11:13:50.1854723Z RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_8_in_features_144_out_features_64_cuda_bfloat16! Caching allocator allocated memory was 9216 and is now reported as 18432 on device 0. CUDA driver allocated memory was 274726912 and is now 276824064. 2025-12-04T11:13:50.1854729Z 2025-12-04T11:13:50.1854993Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.1855554Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_8_in_features_144_out_features_64_cuda_bfloat16 2025-12-04T11:13:50.1855559Z 2025-12-04T11:13:50.1855783Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.1855988Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.1856195Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.1856595Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.1856807Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.1856944Z graph_break [] 2025-12-04T11:13:50.1857102Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.1857243Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.1857441Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.1857805Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.1857994Z graph_break [] 2025-12-04T11:13:50.1858114Z =================================== FAILURES =================================== 2025-12-04T11:13:50.1858438Z _ TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_8_in_features_144_out_features_64_cuda_bfloat16 _ 2025-12-04T11:13:50.1858637Z Traceback (most recent call last): 2025-12-04T11:13:50.1858971Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.1859136Z method(*args, **kwargs) 2025-12-04T11:13:50.1859475Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.1859577Z method(*args, **kwargs) 2025-12-04T11:13:50.1859949Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T11:13:50.1860038Z with policy(): 2025-12-04T11:13:50.1860365Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T11:13:50.1860533Z raise RuntimeError(msg) 2025-12-04T11:13:50.1861387Z RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_8_in_features_144_out_features_64_cuda_bfloat16! Caching allocator allocated memory was 18432 and is now reported as 27648 on device 0. CUDA driver allocated memory was 276824064 and is now 278921216. 2025-12-04T11:13:50.1861393Z 2025-12-04T11:13:50.1861601Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.1862149Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_8_in_features_144_out_features_64_cuda_bfloat16 2025-12-04T11:13:50.1862153Z 2025-12-04T11:13:50.1862373Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.1862515Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.1862688Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.1863127Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.1863286Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.1863408Z graph_break [] 2025-12-04T11:13:50.1863613Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.1863721Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.1863987Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.1864358Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.1864504Z graph_break [] 2025-12-04T11:13:50.1864759Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.1864879Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.1865123Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.1865511Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.1865602Z graph_break [] 2025-12-04T11:13:50.1866160Z - generated xml file: /var/lib/jenkins/workspace/test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-3f42debbb8d01db8.xml - 2025-12-04T11:13:50.1866290Z =========================== short test summary info ============================ 2025-12-04T11:13:50.1867642Z FAILED [0.4581s] inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_17_mid_dim_8_in_features_144_out_features_64_cuda_bfloat16 - RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_8_in_features_144_out_features_64_cuda_bfloat16! Caching allocator allocated memory was 18432 and is now reported as 27648 on device 0. CUDA driver allocated memory was 276824064 and is now 278921216. 2025-12-04T11:13:50.1867693Z 2025-12-04T11:13:50.1867909Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.1868511Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_8_in_features_144_out_features_64_cuda_bfloat16 2025-12-04T11:13:50.1868515Z 2025-12-04T11:13:50.1868704Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.1868840Z !!!!!!!!!!!!!!!!!!!!!!!!!! stopping after 1 failures !!!!!!!!!!!!!!!!!!!!!!!!!!! 2025-12-04T11:13:50.1869040Z ================== 1 failed, 20 deselected, 2 rerun in 2.81s =================== 2025-12-04T11:13:50.1869118Z Got exit code 1 2025-12-04T11:13:50.1869315Z Retrying single test... 2025-12-04T11:13:50.1869618Z W1204 10:43:39.357000 67962 site-packages/torch/_inductor/utils.py:1703] Not enough SMs to use max_autotune_gemm mode 2025-12-04T11:13:50.1870034Z Test results will be stored in test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-92851bf92613c0d7.xml 2025-12-04T11:13:50.1870208Z ============================= test session starts ============================== 2025-12-04T11:13:50.1870450Z platform linux -- Python 3.10.14, pytest-7.3.2, pluggy-1.6.0 -- /opt/conda/envs/py_3.10/bin/python 2025-12-04T11:13:50.1870617Z cachedir: .pytest_cache 2025-12-04T11:13:50.1870971Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow] 2025-12-04T11:13:50.1871081Z rootdir: /var/lib/jenkins/workspace 2025-12-04T11:13:50.1871213Z configfile: pytest.ini 2025-12-04T11:13:50.1871583Z plugins: hypothesis-6.56.4, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-14.0, subtests-0.13.1, xdist-3.3.1, xdoctest-1.3.0, typeguard-4.3.0 2025-12-04T11:13:50.1871746Z collecting ... collected 58 items / 57 deselected / 1 selected 2025-12-04T11:13:50.1872418Z stepcurrent: skipping 20 already run items. Running only test/inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_17_mid_dim_8_in_features_144_out_features_64_cuda_bfloat16 2025-12-04T11:13:50.1872538Z Running 1 items in this shard 2025-12-04T11:13:50.1872542Z 2025-12-04T11:13:50.1873335Z inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_17_mid_dim_8_in_features_144_out_features_64_cuda_bfloat16 [W1204 10:43:40.463416481 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1873383Z 2025-12-04T11:13:50.1873804Z [W1204 10:43:49.694154796 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1873813Z 2025-12-04T11:13:50.1874173Z [W1204 10:43:49.694392512 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1874178Z 2025-12-04T11:13:50.1874483Z [W1204 10:43:49.700195589 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1874487Z 2025-12-04T11:13:50.1874912Z [W1204 10:43:49.700764242 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1874917Z 2025-12-04T11:13:50.1875254Z [W1204 10:43:49.700930085 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1875259Z 2025-12-04T11:13:50.1875618Z [W1204 10:43:49.706294894 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1875622Z 2025-12-04T11:13:50.1875979Z [W1204 10:43:49.706845866 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1875982Z 2025-12-04T11:13:50.1876299Z [W1204 10:43:49.707021510 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1876323Z 2025-12-04T11:13:50.1876494Z ('RERUN', {'yellow': True}) [11.1250s] [100%] 2025-12-04T11:13:50.1877283Z inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_17_mid_dim_8_in_features_144_out_features_64_cuda_bfloat16 [W1204 10:43:50.889627947 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1877289Z 2025-12-04T11:13:50.1877651Z [W1204 10:43:50.890242961 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1877654Z 2025-12-04T11:13:50.1878144Z [W1204 10:43:50.890393104 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1878148Z 2025-12-04T11:13:50.1878512Z [W1204 10:43:50.893362160 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1878516Z 2025-12-04T11:13:50.1878825Z [W1204 10:43:50.893939793 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1878828Z 2025-12-04T11:13:50.1879287Z [W1204 10:43:50.894078296 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1879300Z 2025-12-04T11:13:50.1879620Z [W1204 10:43:50.898619817 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1879627Z 2025-12-04T11:13:50.1879979Z [W1204 10:43:50.899088747 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1879984Z 2025-12-04T11:13:50.1880303Z [W1204 10:43:50.899226330 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1880307Z 2025-12-04T11:13:50.1880420Z ('RERUN', {'yellow': True}) [0.4212s] [100%] 2025-12-04T11:13:50.1881252Z inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_17_mid_dim_8_in_features_144_out_features_64_cuda_bfloat16 [W1204 10:43:51.302405466 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1881352Z 2025-12-04T11:13:50.1881799Z [W1204 10:43:51.302993099 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1881803Z 2025-12-04T11:13:50.1882166Z [W1204 10:43:51.303135962 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1882171Z 2025-12-04T11:13:50.1882490Z [W1204 10:43:51.306097707 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1882494Z 2025-12-04T11:13:50.1882862Z [W1204 10:43:51.306662050 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1882865Z 2025-12-04T11:13:50.1883168Z [W1204 10:43:51.306797523 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1883172Z 2025-12-04T11:13:50.1883598Z [W1204 10:43:51.311364623 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1883605Z 2025-12-04T11:13:50.1883926Z [W1204 10:43:51.311840494 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1883984Z 2025-12-04T11:13:50.1884338Z [W1204 10:43:51.311974377 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1884341Z 2025-12-04T11:13:50.1884454Z FAILED [0.4107s] [100%] 2025-12-04T11:13:50.1884458Z 2025-12-04T11:13:50.1884577Z ==================================== RERUNS ==================================== 2025-12-04T11:13:50.1884973Z _ TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_8_in_features_144_out_features_64_cuda_bfloat16 _ 2025-12-04T11:13:50.1885097Z Traceback (most recent call last): 2025-12-04T11:13:50.1885485Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.1885591Z method(*args, **kwargs) 2025-12-04T11:13:50.1885934Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.1886050Z method(*args, **kwargs) 2025-12-04T11:13:50.1886422Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T11:13:50.1886564Z with policy(): 2025-12-04T11:13:50.1886890Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T11:13:50.1887004Z raise RuntimeError(msg) 2025-12-04T11:13:50.1887870Z RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_8_in_features_144_out_features_64_cuda_bfloat16! Caching allocator allocated memory was 0 and is now reported as 9216 on device 0. CUDA driver allocated memory was 230686720 and is now 274726912. 2025-12-04T11:13:50.1887876Z 2025-12-04T11:13:50.1888029Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.1888681Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_8_in_features_144_out_features_64_cuda_bfloat16 2025-12-04T11:13:50.1888685Z 2025-12-04T11:13:50.1888879Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.1889041Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.1889220Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.1889605Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.1889883Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.1889996Z graph_break [] 2025-12-04T11:13:50.1890243Z ----------------------------- Captured stderr call ----------------------------- 2025-12-04T11:13:50.1891021Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/constant_folding.py:256: UserWarning: Unsupported unwinding pattern: Address not in range (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/profiler/unwind/unwind.cpp:219.) 2025-12-04T11:13:50.1891128Z if out == self.unknown_value: 2025-12-04T11:13:50.1891485Z _ TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_8_in_features_144_out_features_64_cuda_bfloat16 _ 2025-12-04T11:13:50.1891650Z Traceback (most recent call last): 2025-12-04T11:13:50.1892015Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.1892147Z method(*args, **kwargs) 2025-12-04T11:13:50.1892471Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.1892606Z method(*args, **kwargs) 2025-12-04T11:13:50.1892913Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T11:13:50.1893104Z with policy(): 2025-12-04T11:13:50.1893497Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T11:13:50.1893598Z raise RuntimeError(msg) 2025-12-04T11:13:50.1894431Z RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_8_in_features_144_out_features_64_cuda_bfloat16! Caching allocator allocated memory was 9216 and is now reported as 18432 on device 0. CUDA driver allocated memory was 274726912 and is now 276824064. 2025-12-04T11:13:50.1894472Z 2025-12-04T11:13:50.1894631Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.1895169Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_8_in_features_144_out_features_64_cuda_bfloat16 2025-12-04T11:13:50.1895282Z 2025-12-04T11:13:50.1895490Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.1895649Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.1895811Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.1896188Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.1896349Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.1896531Z graph_break [] 2025-12-04T11:13:50.1896715Z ----------------------------- Captured stderr call ----------------------------- 2025-12-04T11:13:50.1897476Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/constant_folding.py:256: UserWarning: Unsupported unwinding pattern: Address not in range (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/profiler/unwind/unwind.cpp:219.) 2025-12-04T11:13:50.1897578Z if out == self.unknown_value: 2025-12-04T11:13:50.1897735Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.1897895Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.1898102Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.1898531Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.1898623Z graph_break [] 2025-12-04T11:13:50.1898738Z =================================== FAILURES =================================== 2025-12-04T11:13:50.1899218Z _ TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_8_in_features_144_out_features_64_cuda_bfloat16 _ 2025-12-04T11:13:50.1899309Z Traceback (most recent call last): 2025-12-04T11:13:50.1899776Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.1899927Z method(*args, **kwargs) 2025-12-04T11:13:50.1900251Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.1900396Z method(*args, **kwargs) 2025-12-04T11:13:50.1900719Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T11:13:50.1900795Z with policy(): 2025-12-04T11:13:50.1901228Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T11:13:50.1901327Z raise RuntimeError(msg) 2025-12-04T11:13:50.1902217Z RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_8_in_features_144_out_features_64_cuda_bfloat16! Caching allocator allocated memory was 18432 and is now reported as 27648 on device 0. CUDA driver allocated memory was 276824064 and is now 278921216. 2025-12-04T11:13:50.1902263Z 2025-12-04T11:13:50.1902422Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.1902971Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_8_in_features_144_out_features_64_cuda_bfloat16 2025-12-04T11:13:50.1902994Z 2025-12-04T11:13:50.1903246Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.1903423Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.1903666Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.1904049Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.1904241Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.1904317Z graph_break [] 2025-12-04T11:13:50.1904528Z ----------------------------- Captured stderr call ----------------------------- 2025-12-04T11:13:50.1905315Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/constant_folding.py:256: UserWarning: Unsupported unwinding pattern: Address not in range (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/profiler/unwind/unwind.cpp:219.) 2025-12-04T11:13:50.1905417Z if out == self.unknown_value: 2025-12-04T11:13:50.1905573Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.1905816Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.1905960Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.1906436Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.1906546Z graph_break [] 2025-12-04T11:13:50.1906705Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.1906862Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.1907014Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.1907454Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.1907576Z graph_break [] 2025-12-04T11:13:50.1908090Z - generated xml file: /var/lib/jenkins/workspace/test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-92851bf92613c0d7.xml - 2025-12-04T11:13:50.1908373Z =========================== short test summary info ============================ 2025-12-04T11:13:50.1909695Z FAILED [0.4107s] inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_17_mid_dim_8_in_features_144_out_features_64_cuda_bfloat16 - RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_8_in_features_144_out_features_64_cuda_bfloat16! Caching allocator allocated memory was 18432 and is now reported as 27648 on device 0. CUDA driver allocated memory was 276824064 and is now 278921216. 2025-12-04T11:13:50.1909701Z 2025-12-04T11:13:50.1909892Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.1910509Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_8_in_features_144_out_features_64_cuda_bfloat16 2025-12-04T11:13:50.1910515Z 2025-12-04T11:13:50.1910755Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.1910932Z !!!!!!!!!!!!!!!!!!!!!!!!!! stopping after 1 failures !!!!!!!!!!!!!!!!!!!!!!!!!!! 2025-12-04T11:13:50.1911082Z ================== 1 failed, 57 deselected, 2 rerun in 11.98s ================== 2025-12-04T11:13:50.1911223Z Got exit code 1 2025-12-04T11:13:50.1911306Z Retrying single test... 2025-12-04T11:13:50.1911701Z W1204 10:43:58.152000 68148 site-packages/torch/_inductor/utils.py:1703] Not enough SMs to use max_autotune_gemm mode 2025-12-04T11:13:50.1912117Z Test results will be stored in test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-3610b3094a676974.xml 2025-12-04T11:13:50.1912243Z ============================= test session starts ============================== 2025-12-04T11:13:50.1912591Z platform linux -- Python 3.10.14, pytest-7.3.2, pluggy-1.6.0 -- /opt/conda/envs/py_3.10/bin/python 2025-12-04T11:13:50.1912710Z cachedir: .pytest_cache 2025-12-04T11:13:50.1913136Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow] 2025-12-04T11:13:50.1913265Z rootdir: /var/lib/jenkins/workspace 2025-12-04T11:13:50.1913365Z configfile: pytest.ini 2025-12-04T11:13:50.1913749Z plugins: hypothesis-6.56.4, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-14.0, subtests-0.13.1, xdist-3.3.1, xdoctest-1.3.0, typeguard-4.3.0 2025-12-04T11:13:50.1913927Z collecting ... collected 58 items / 57 deselected / 1 selected 2025-12-04T11:13:50.1914529Z stepcurrent: skipping 20 already run items. Running only test/inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_17_mid_dim_8_in_features_144_out_features_64_cuda_bfloat16 2025-12-04T11:13:50.1914706Z Running 1 items in this shard 2025-12-04T11:13:50.1914711Z 2025-12-04T11:13:50.1915491Z inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_17_mid_dim_8_in_features_144_out_features_64_cuda_bfloat16 [W1204 10:43:59.238927335 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1915497Z 2025-12-04T11:13:50.1915862Z [W1204 10:44:08.453856707 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1915865Z 2025-12-04T11:13:50.1916186Z [W1204 10:44:08.454087642 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1916190Z 2025-12-04T11:13:50.1916571Z [W1204 10:44:08.459796318 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1916575Z 2025-12-04T11:13:50.1916881Z [W1204 10:44:08.460367201 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1916929Z 2025-12-04T11:13:50.1917419Z [W1204 10:44:08.460547595 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1917425Z 2025-12-04T11:13:50.1917744Z [W1204 10:44:08.465804891 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1917748Z 2025-12-04T11:13:50.1918118Z [W1204 10:44:08.466355224 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1918121Z 2025-12-04T11:13:50.1918443Z [W1204 10:44:08.466532948 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1918446Z 2025-12-04T11:13:50.1918559Z ('RERUN', {'yellow': True}) [11.0967s] [100%] 2025-12-04T11:13:50.1919398Z inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_17_mid_dim_8_in_features_144_out_features_64_cuda_bfloat16 [W1204 10:44:09.651407998 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1919440Z 2025-12-04T11:13:50.1919855Z [W1204 10:44:09.651983471 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1919859Z 2025-12-04T11:13:50.1920228Z [W1204 10:44:09.652124314 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1920232Z 2025-12-04T11:13:50.1920546Z [W1204 10:44:09.655070099 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1920550Z 2025-12-04T11:13:50.1920902Z [W1204 10:44:09.655647632 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1920908Z 2025-12-04T11:13:50.1921211Z [W1204 10:44:09.655784435 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1921216Z 2025-12-04T11:13:50.1921655Z [W1204 10:44:09.660347136 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1921659Z 2025-12-04T11:13:50.1921975Z [W1204 10:44:09.660827127 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1921978Z 2025-12-04T11:13:50.1922333Z [W1204 10:44:09.660961210 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1922336Z 2025-12-04T11:13:50.1922447Z ('RERUN', {'yellow': True}) [0.4260s] [100%] 2025-12-04T11:13:50.1923201Z inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_17_mid_dim_8_in_features_144_out_features_64_cuda_bfloat16 [W1204 10:44:10.074893336 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1923229Z 2025-12-04T11:13:50.1923630Z [W1204 10:44:10.075471899 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1923634Z 2025-12-04T11:13:50.1923989Z [W1204 10:44:10.075612792 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1924033Z 2025-12-04T11:13:50.1924359Z [W1204 10:44:10.078560627 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1924363Z 2025-12-04T11:13:50.1924680Z [W1204 10:44:10.079128829 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1924727Z 2025-12-04T11:13:50.1925160Z [W1204 10:44:10.079267052 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1925165Z 2025-12-04T11:13:50.1925487Z [W1204 10:44:10.083874774 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1925491Z 2025-12-04T11:13:50.1925926Z [W1204 10:44:10.084343595 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1925930Z 2025-12-04T11:13:50.1926247Z [W1204 10:44:10.084480398 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1926250Z 2025-12-04T11:13:50.1926379Z FAILED [0.4195s] [100%] 2025-12-04T11:13:50.1926382Z 2025-12-04T11:13:50.1926503Z ==================================== RERUNS ==================================== 2025-12-04T11:13:50.1926855Z _ TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_8_in_features_144_out_features_64_cuda_bfloat16 _ 2025-12-04T11:13:50.1927114Z Traceback (most recent call last): 2025-12-04T11:13:50.1927515Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.1927652Z method(*args, **kwargs) 2025-12-04T11:13:50.1927975Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.1928086Z method(*args, **kwargs) 2025-12-04T11:13:50.1928424Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T11:13:50.1928565Z with policy(): 2025-12-04T11:13:50.1928905Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T11:13:50.1929052Z raise RuntimeError(msg) 2025-12-04T11:13:50.1929889Z RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_8_in_features_144_out_features_64_cuda_bfloat16! Caching allocator allocated memory was 0 and is now reported as 9216 on device 0. CUDA driver allocated memory was 230686720 and is now 274726912. 2025-12-04T11:13:50.1929895Z 2025-12-04T11:13:50.1930112Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.1930651Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_8_in_features_144_out_features_64_cuda_bfloat16 2025-12-04T11:13:50.1930655Z 2025-12-04T11:13:50.1930944Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.1931110Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.1931241Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.1931682Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.1931848Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.1932010Z graph_break [] 2025-12-04T11:13:50.1932191Z ----------------------------- Captured stderr call ----------------------------- 2025-12-04T11:13:50.1932918Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/constant_folding.py:256: UserWarning: Unsupported unwinding pattern: Address not in range (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/profiler/unwind/unwind.cpp:219.) 2025-12-04T11:13:50.1933082Z if out == self.unknown_value: 2025-12-04T11:13:50.1933409Z _ TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_8_in_features_144_out_features_64_cuda_bfloat16 _ 2025-12-04T11:13:50.1933657Z Traceback (most recent call last): 2025-12-04T11:13:50.1934108Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.1934224Z method(*args, **kwargs) 2025-12-04T11:13:50.1934603Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.1934698Z method(*args, **kwargs) 2025-12-04T11:13:50.1935018Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T11:13:50.1935140Z with policy(): 2025-12-04T11:13:50.1935519Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T11:13:50.1935689Z raise RuntimeError(msg) 2025-12-04T11:13:50.1936527Z RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_8_in_features_144_out_features_64_cuda_bfloat16! Caching allocator allocated memory was 9216 and is now reported as 18432 on device 0. CUDA driver allocated memory was 274726912 and is now 276824064. 2025-12-04T11:13:50.1936572Z 2025-12-04T11:13:50.1936773Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.1937328Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_8_in_features_144_out_features_64_cuda_bfloat16 2025-12-04T11:13:50.1937332Z 2025-12-04T11:13:50.1937506Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.1937785Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.1937911Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.1938325Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.1938497Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.1938590Z graph_break [] 2025-12-04T11:13:50.1938833Z ----------------------------- Captured stderr call ----------------------------- 2025-12-04T11:13:50.1939576Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/constant_folding.py:256: UserWarning: Unsupported unwinding pattern: Address not in range (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/profiler/unwind/unwind.cpp:219.) 2025-12-04T11:13:50.1939712Z if out == self.unknown_value: 2025-12-04T11:13:50.1939943Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.1940080Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.1940257Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.1940704Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.1940848Z graph_break [] 2025-12-04T11:13:50.1940966Z =================================== FAILURES =================================== 2025-12-04T11:13:50.1941285Z _ TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_8_in_features_144_out_features_64_cuda_bfloat16 _ 2025-12-04T11:13:50.1941426Z Traceback (most recent call last): 2025-12-04T11:13:50.1941756Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.1941906Z method(*args, **kwargs) 2025-12-04T11:13:50.1942276Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.1942370Z method(*args, **kwargs) 2025-12-04T11:13:50.1942726Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T11:13:50.1942888Z with policy(): 2025-12-04T11:13:50.1943266Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T11:13:50.1943496Z raise RuntimeError(msg) 2025-12-04T11:13:50.1944401Z RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_8_in_features_144_out_features_64_cuda_bfloat16! Caching allocator allocated memory was 18432 and is now reported as 27648 on device 0. CUDA driver allocated memory was 276824064 and is now 278921216. 2025-12-04T11:13:50.1944407Z 2025-12-04T11:13:50.1944627Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.1945190Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_8_in_features_144_out_features_64_cuda_bfloat16 2025-12-04T11:13:50.1945196Z 2025-12-04T11:13:50.1945411Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.1945627Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.1945889Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.1946316Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.1946474Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.1946563Z graph_break [] 2025-12-04T11:13:50.1946719Z ----------------------------- Captured stderr call ----------------------------- 2025-12-04T11:13:50.1947517Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/constant_folding.py:256: UserWarning: Unsupported unwinding pattern: Address not in range (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/profiler/unwind/unwind.cpp:219.) 2025-12-04T11:13:50.1951062Z if out == self.unknown_value: 2025-12-04T11:13:50.1951234Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.1951349Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.1951490Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.1951862Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.1951929Z graph_break [] 2025-12-04T11:13:50.1952075Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.1952177Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.1952311Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.1952658Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.1952738Z graph_break [] 2025-12-04T11:13:50.1953240Z - generated xml file: /var/lib/jenkins/workspace/test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-3610b3094a676974.xml - 2025-12-04T11:13:50.1953349Z =========================== short test summary info ============================ 2025-12-04T11:13:50.1954648Z FAILED [0.4195s] inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_17_mid_dim_8_in_features_144_out_features_64_cuda_bfloat16 - RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_8_in_features_144_out_features_64_cuda_bfloat16! Caching allocator allocated memory was 18432 and is now reported as 27648 on device 0. CUDA driver allocated memory was 276824064 and is now 278921216. 2025-12-04T11:13:50.1954736Z 2025-12-04T11:13:50.1954873Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.1955475Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_8_in_features_144_out_features_64_cuda_bfloat16 2025-12-04T11:13:50.1955482Z 2025-12-04T11:13:50.1955652Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.1955764Z !!!!!!!!!!!!!!!!!!!!!!!!!! stopping after 1 failures !!!!!!!!!!!!!!!!!!!!!!!!!!! 2025-12-04T11:13:50.1955886Z ================== 1 failed, 57 deselected, 2 rerun in 11.97s ================== 2025-12-04T11:13:50.1955947Z Got exit code 1 2025-12-04T11:13:50.1956425Z FAILED CONSISTENTLY: test/inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_17_mid_dim_8_in_features_144_out_features_64_cuda_bfloat16 2025-12-04T11:13:50.1956673Z Test failed consistently, continuing with the rest of the tests due to continue-through-error being set 2025-12-04T11:13:50.1956944Z W1204 10:44:16.862000 68334 site-packages/torch/_inductor/utils.py:1703] Not enough SMs to use max_autotune_gemm mode 2025-12-04T11:13:50.1957383Z Test results will be stored in test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-dba8bfd8733d5e09.xml 2025-12-04T11:13:50.1957484Z ============================= test session starts ============================== 2025-12-04T11:13:50.1957696Z platform linux -- Python 3.10.14, pytest-7.3.2, pluggy-1.6.0 -- /opt/conda/envs/py_3.10/bin/python 2025-12-04T11:13:50.1957765Z cachedir: .pytest_cache 2025-12-04T11:13:50.1958077Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow] 2025-12-04T11:13:50.1958156Z rootdir: /var/lib/jenkins/workspace 2025-12-04T11:13:50.1958222Z configfile: pytest.ini 2025-12-04T11:13:50.1958541Z plugins: hypothesis-6.56.4, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-14.0, subtests-0.13.1, xdist-3.3.1, xdoctest-1.3.0, typeguard-4.3.0 2025-12-04T11:13:50.1958676Z collecting ... collected 58 items / 21 deselected / 37 selected 2025-12-04T11:13:50.1958766Z stepcurrent: skipping 21 already run items. 2025-12-04T11:13:50.1958849Z Running 37 items in this shard 2025-12-04T11:13:50.1958853Z 2025-12-04T11:13:50.1959362Z inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_17_mid_dim_8_in_features_144_out_features_65_cuda_bfloat16 ('RERUN', {'yellow': True}) [1.9707s] [ 2%] 2025-12-04T11:13:50.1959850Z inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_17_mid_dim_8_in_features_144_out_features_65_cuda_bfloat16 ('RERUN', {'yellow': True}) [0.5583s] [ 2%] 2025-12-04T11:13:50.1960289Z inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_17_mid_dim_8_in_features_144_out_features_65_cuda_bfloat16 FAILED [0.5698s] [ 2%] 2025-12-04T11:13:50.1960294Z 2025-12-04T11:13:50.1960386Z ==================================== RERUNS ==================================== 2025-12-04T11:13:50.1960679Z _ TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_8_in_features_144_out_features_65_cuda_bfloat16 _ 2025-12-04T11:13:50.1960756Z Traceback (most recent call last): 2025-12-04T11:13:50.1961069Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.1961137Z method(*args, **kwargs) 2025-12-04T11:13:50.1961428Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.1961493Z method(*args, **kwargs) 2025-12-04T11:13:50.1961779Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T11:13:50.1961884Z with policy(): 2025-12-04T11:13:50.1962259Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T11:13:50.1962326Z raise RuntimeError(msg) 2025-12-04T11:13:50.1963123Z RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_8_in_features_144_out_features_65_cuda_bfloat16! Caching allocator allocated memory was 0 and is now reported as 9728 on device 0. CUDA driver allocated memory was 230686720 and is now 274726912. 2025-12-04T11:13:50.1963128Z 2025-12-04T11:13:50.1963259Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.1963778Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_8_in_features_144_out_features_65_cuda_bfloat16 2025-12-04T11:13:50.1963783Z 2025-12-04T11:13:50.1963941Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.1964080Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.1964178Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.1964758Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('benchmarking.InductorBenchmarker.benchmark_gpu', 2), ('pad_mm_bench', 1), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.1964892Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.1964952Z graph_break [] 2025-12-04T11:13:50.1965242Z _ TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_8_in_features_144_out_features_65_cuda_bfloat16 _ 2025-12-04T11:13:50.1965320Z Traceback (most recent call last): 2025-12-04T11:13:50.1965615Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.1965683Z method(*args, **kwargs) 2025-12-04T11:13:50.1965970Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.1966034Z method(*args, **kwargs) 2025-12-04T11:13:50.1966323Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T11:13:50.1966382Z with policy(): 2025-12-04T11:13:50.1966672Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T11:13:50.1966740Z raise RuntimeError(msg) 2025-12-04T11:13:50.1967540Z RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_8_in_features_144_out_features_65_cuda_bfloat16! Caching allocator allocated memory was 9728 and is now reported as 19456 on device 0. CUDA driver allocated memory was 274726912 and is now 276824064. 2025-12-04T11:13:50.1967547Z 2025-12-04T11:13:50.1967676Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.1968195Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_8_in_features_144_out_features_65_cuda_bfloat16 2025-12-04T11:13:50.1968200Z 2025-12-04T11:13:50.1968364Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.1968494Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.1968597Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.1969144Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('benchmarking.InductorBenchmarker.benchmark_gpu', 2), ('pad_mm_bench', 1), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.1969312Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.1969375Z graph_break [] 2025-12-04T11:13:50.1969567Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.1969657Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.1969778Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.1970309Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('benchmarking.InductorBenchmarker.benchmark_gpu', 2), ('pad_mm_bench', 1), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.1970368Z graph_break [] 2025-12-04T11:13:50.1970452Z =================================== FAILURES =================================== 2025-12-04T11:13:50.1970737Z _ TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_8_in_features_144_out_features_65_cuda_bfloat16 _ 2025-12-04T11:13:50.1970812Z Traceback (most recent call last): 2025-12-04T11:13:50.1971112Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.1971176Z method(*args, **kwargs) 2025-12-04T11:13:50.1971503Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.1971565Z method(*args, **kwargs) 2025-12-04T11:13:50.1971853Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T11:13:50.1971912Z with policy(): 2025-12-04T11:13:50.1972199Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T11:13:50.1972269Z raise RuntimeError(msg) 2025-12-04T11:13:50.1973071Z RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_8_in_features_144_out_features_65_cuda_bfloat16! Caching allocator allocated memory was 19456 and is now reported as 29184 on device 0. CUDA driver allocated memory was 276824064 and is now 278921216. 2025-12-04T11:13:50.1973077Z 2025-12-04T11:13:50.1973205Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.1973715Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_8_in_features_144_out_features_65_cuda_bfloat16 2025-12-04T11:13:50.1973719Z 2025-12-04T11:13:50.1973873Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.1973999Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.1974089Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.1974627Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('benchmarking.InductorBenchmarker.benchmark_gpu', 2), ('pad_mm_bench', 1), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.1974754Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.1974813Z graph_break [] 2025-12-04T11:13:50.1974942Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.1975030Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.1975153Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.1975687Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('benchmarking.InductorBenchmarker.benchmark_gpu', 2), ('pad_mm_bench', 1), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.1975750Z graph_break [] 2025-12-04T11:13:50.1975874Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.1976004Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.1976123Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.1976721Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('benchmarking.InductorBenchmarker.benchmark_gpu', 2), ('pad_mm_bench', 1), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.1976781Z graph_break [] 2025-12-04T11:13:50.1977276Z - generated xml file: /var/lib/jenkins/workspace/test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-dba8bfd8733d5e09.xml - 2025-12-04T11:13:50.1977374Z =========================== short test summary info ============================ 2025-12-04T11:13:50.1978872Z FAILED [0.5698s] inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_17_mid_dim_8_in_features_144_out_features_65_cuda_bfloat16 - RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_8_in_features_144_out_features_65_cuda_bfloat16! Caching allocator allocated memory was 19456 and is now reported as 29184 on device 0. CUDA driver allocated memory was 276824064 and is now 278921216. 2025-12-04T11:13:50.1978954Z 2025-12-04T11:13:50.1979090Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.1979621Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_8_in_features_144_out_features_65_cuda_bfloat16 2025-12-04T11:13:50.1979628Z 2025-12-04T11:13:50.1979785Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.1979890Z !!!!!!!!!!!!!!!!!!!!!!!!!! stopping after 1 failures !!!!!!!!!!!!!!!!!!!!!!!!!!! 2025-12-04T11:13:50.1980008Z ================== 1 failed, 21 deselected, 2 rerun in 3.12s =================== 2025-12-04T11:13:50.1980069Z Got exit code 1 2025-12-04T11:13:50.1980134Z Retrying single test... 2025-12-04T11:13:50.1980405Z W1204 10:44:26.864000 68516 site-packages/torch/_inductor/utils.py:1703] Not enough SMs to use max_autotune_gemm mode 2025-12-04T11:13:50.1980791Z Test results will be stored in test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-6c64eb8af833b17a.xml 2025-12-04T11:13:50.1980887Z ============================= test session starts ============================== 2025-12-04T11:13:50.1981095Z platform linux -- Python 3.10.14, pytest-7.3.2, pluggy-1.6.0 -- /opt/conda/envs/py_3.10/bin/python 2025-12-04T11:13:50.1981162Z cachedir: .pytest_cache 2025-12-04T11:13:50.1981471Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow] 2025-12-04T11:13:50.1981550Z rootdir: /var/lib/jenkins/workspace 2025-12-04T11:13:50.1981616Z configfile: pytest.ini 2025-12-04T11:13:50.1981932Z plugins: hypothesis-6.56.4, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-14.0, subtests-0.13.1, xdist-3.3.1, xdoctest-1.3.0, typeguard-4.3.0 2025-12-04T11:13:50.1982066Z collecting ... collected 58 items / 57 deselected / 1 selected 2025-12-04T11:13:50.1982640Z stepcurrent: skipping 21 already run items. Running only test/inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_17_mid_dim_8_in_features_144_out_features_65_cuda_bfloat16 2025-12-04T11:13:50.1982710Z Running 1 items in this shard 2025-12-04T11:13:50.1982714Z 2025-12-04T11:13:50.1983443Z inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_17_mid_dim_8_in_features_144_out_features_65_cuda_bfloat16 [W1204 10:44:28.518745028 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1983447Z 2025-12-04T11:13:50.1983816Z [W1204 10:44:37.664722588 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1983881Z 2025-12-04T11:13:50.1984266Z [W1204 10:44:37.664964554 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1984275Z 2025-12-04T11:13:50.1984563Z [W1204 10:44:37.670818733 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1984567Z 2025-12-04T11:13:50.1984849Z [W1204 10:44:37.671383096 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1984853Z 2025-12-04T11:13:50.1985141Z [W1204 10:44:37.671565720 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1985145Z 2025-12-04T11:13:50.1985428Z [W1204 10:44:37.676967629 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1985433Z 2025-12-04T11:13:50.1985723Z [W1204 10:44:37.677487351 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1985762Z 2025-12-04T11:13:50.1986043Z [W1204 10:44:37.677646124 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1986046Z 2025-12-04T11:13:50.1986132Z ('RERUN', {'yellow': True}) [11.1326s] [100%] 2025-12-04T11:13:50.1986847Z inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_17_mid_dim_8_in_features_144_out_features_65_cuda_bfloat16 [W1204 10:44:38.484100080 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1986851Z 2025-12-04T11:13:50.1987138Z [W1204 10:44:38.484673783 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1987144Z 2025-12-04T11:13:50.1987433Z [W1204 10:44:38.484813656 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1987438Z 2025-12-04T11:13:50.1987723Z [W1204 10:44:38.487878584 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1987726Z 2025-12-04T11:13:50.1988010Z [W1204 10:44:38.488348594 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1988014Z 2025-12-04T11:13:50.1988297Z [W1204 10:44:38.488485347 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1988300Z 2025-12-04T11:13:50.1988585Z [W1204 10:44:38.493357655 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1988589Z 2025-12-04T11:13:50.1988889Z [W1204 10:44:38.493843376 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1988894Z 2025-12-04T11:13:50.1989184Z [W1204 10:44:38.493979868 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1989187Z 2025-12-04T11:13:50.1989267Z ('RERUN', {'yellow': True}) [0.5011s] [100%] 2025-12-04T11:13:50.1990258Z inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_17_mid_dim_8_in_features_144_out_features_65_cuda_bfloat16 [W1204 10:44:39.981923079 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1990266Z 2025-12-04T11:13:50.1990605Z [W1204 10:44:39.982485492 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1990687Z 2025-12-04T11:13:50.1991050Z [W1204 10:44:39.982623235 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1991059Z 2025-12-04T11:13:50.1991345Z [W1204 10:44:39.985723663 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1991348Z 2025-12-04T11:13:50.1991632Z [W1204 10:44:39.986190764 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1991636Z 2025-12-04T11:13:50.1991924Z [W1204 10:44:39.986327117 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1991927Z 2025-12-04T11:13:50.1992211Z [W1204 10:44:39.991080062 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1992217Z 2025-12-04T11:13:50.1992504Z [W1204 10:44:39.991561112 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1992508Z 2025-12-04T11:13:50.1992832Z [W1204 10:44:39.991698896 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.1992835Z 2025-12-04T11:13:50.1992903Z FAILED [0.4983s] [100%] 2025-12-04T11:13:50.1992906Z 2025-12-04T11:13:50.1992993Z ==================================== RERUNS ==================================== 2025-12-04T11:13:50.1993288Z _ TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_8_in_features_144_out_features_65_cuda_bfloat16 _ 2025-12-04T11:13:50.1993369Z Traceback (most recent call last): 2025-12-04T11:13:50.1993682Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.1993753Z method(*args, **kwargs) 2025-12-04T11:13:50.1994134Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.1994249Z method(*args, **kwargs) 2025-12-04T11:13:50.1994722Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T11:13:50.1994791Z with policy(): 2025-12-04T11:13:50.1995084Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T11:13:50.1995206Z raise RuntimeError(msg) 2025-12-04T11:13:50.1996152Z RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_8_in_features_144_out_features_65_cuda_bfloat16! Caching allocator allocated memory was 0 and is now reported as 9728 on device 0. CUDA driver allocated memory was 230686720 and is now 274726912. 2025-12-04T11:13:50.1996165Z 2025-12-04T11:13:50.1996352Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.1996880Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_8_in_features_144_out_features_65_cuda_bfloat16 2025-12-04T11:13:50.1996886Z 2025-12-04T11:13:50.1997049Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.1997181Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.1997277Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.1997829Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('benchmarking.InductorBenchmarker.benchmark_gpu', 2), ('pad_mm_bench', 1), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.1997958Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.1998095Z graph_break [] 2025-12-04T11:13:50.1998292Z ----------------------------- Captured stderr call ----------------------------- 2025-12-04T11:13:50.1998988Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/constant_folding.py:256: UserWarning: Unsupported unwinding pattern: Address not in range (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/profiler/unwind/unwind.cpp:219.) 2025-12-04T11:13:50.1999066Z if out == self.unknown_value: 2025-12-04T11:13:50.1999358Z _ TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_8_in_features_144_out_features_65_cuda_bfloat16 _ 2025-12-04T11:13:50.1999433Z Traceback (most recent call last): 2025-12-04T11:13:50.1999735Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.1999801Z method(*args, **kwargs) 2025-12-04T11:13:50.2000090Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.2000164Z method(*args, **kwargs) 2025-12-04T11:13:50.2000457Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T11:13:50.2000896Z with policy(): 2025-12-04T11:13:50.2001193Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T11:13:50.2001265Z raise RuntimeError(msg) 2025-12-04T11:13:50.2002077Z RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_8_in_features_144_out_features_65_cuda_bfloat16! Caching allocator allocated memory was 9728 and is now reported as 19456 on device 0. CUDA driver allocated memory was 274726912 and is now 276824064. 2025-12-04T11:13:50.2002081Z 2025-12-04T11:13:50.2002209Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.2002737Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_8_in_features_144_out_features_65_cuda_bfloat16 2025-12-04T11:13:50.2002742Z 2025-12-04T11:13:50.2002902Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.2003030Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.2003125Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.2003668Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('benchmarking.InductorBenchmarker.benchmark_gpu', 2), ('pad_mm_bench', 1), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.2003798Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.2003858Z graph_break [] 2025-12-04T11:13:50.2003984Z ----------------------------- Captured stderr call ----------------------------- 2025-12-04T11:13:50.2004677Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/constant_folding.py:256: UserWarning: Unsupported unwinding pattern: Address not in range (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/profiler/unwind/unwind.cpp:219.) 2025-12-04T11:13:50.2004751Z if out == self.unknown_value: 2025-12-04T11:13:50.2004877Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.2004970Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.2005097Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.2005634Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('benchmarking.InductorBenchmarker.benchmark_gpu', 2), ('pad_mm_bench', 1), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.2005694Z graph_break [] 2025-12-04T11:13:50.2005837Z =================================== FAILURES =================================== 2025-12-04T11:13:50.2006195Z _ TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_8_in_features_144_out_features_65_cuda_bfloat16 _ 2025-12-04T11:13:50.2006278Z Traceback (most recent call last): 2025-12-04T11:13:50.2006575Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.2006640Z method(*args, **kwargs) 2025-12-04T11:13:50.2006938Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.2007001Z method(*args, **kwargs) 2025-12-04T11:13:50.2007288Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T11:13:50.2007349Z with policy(): 2025-12-04T11:13:50.2007643Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T11:13:50.2007713Z raise RuntimeError(msg) 2025-12-04T11:13:50.2008535Z RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_8_in_features_144_out_features_65_cuda_bfloat16! Caching allocator allocated memory was 19456 and is now reported as 29184 on device 0. CUDA driver allocated memory was 276824064 and is now 278921216. 2025-12-04T11:13:50.2008578Z 2025-12-04T11:13:50.2008709Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.2009229Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_8_in_features_144_out_features_65_cuda_bfloat16 2025-12-04T11:13:50.2009233Z 2025-12-04T11:13:50.2009389Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.2009525Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.2009617Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.2010157Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('benchmarking.InductorBenchmarker.benchmark_gpu', 2), ('pad_mm_bench', 1), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.2010289Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.2010348Z graph_break [] 2025-12-04T11:13:50.2010475Z ----------------------------- Captured stderr call ----------------------------- 2025-12-04T11:13:50.2011161Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/constant_folding.py:256: UserWarning: Unsupported unwinding pattern: Address not in range (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/profiler/unwind/unwind.cpp:219.) 2025-12-04T11:13:50.2011230Z if out == self.unknown_value: 2025-12-04T11:13:50.2011357Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.2011448Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.2011570Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.2012108Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('benchmarking.InductorBenchmarker.benchmark_gpu', 2), ('pad_mm_bench', 1), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.2012165Z graph_break [] 2025-12-04T11:13:50.2012291Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.2012377Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.2012499Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.2013034Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('benchmarking.InductorBenchmarker.benchmark_gpu', 2), ('pad_mm_bench', 1), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.2013200Z graph_break [] 2025-12-04T11:13:50.2013701Z - generated xml file: /var/lib/jenkins/workspace/test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-6c64eb8af833b17a.xml - 2025-12-04T11:13:50.2013805Z =========================== short test summary info ============================ 2025-12-04T11:13:50.2015092Z FAILED [0.4983s] inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_17_mid_dim_8_in_features_144_out_features_65_cuda_bfloat16 - RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_8_in_features_144_out_features_65_cuda_bfloat16! Caching allocator allocated memory was 19456 and is now reported as 29184 on device 0. CUDA driver allocated memory was 276824064 and is now 278921216. 2025-12-04T11:13:50.2015098Z 2025-12-04T11:13:50.2015221Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.2015742Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_8_in_features_144_out_features_65_cuda_bfloat16 2025-12-04T11:13:50.2015801Z 2025-12-04T11:13:50.2015958Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.2016061Z !!!!!!!!!!!!!!!!!!!!!!!!!! stopping after 1 failures !!!!!!!!!!!!!!!!!!!!!!!!!!! 2025-12-04T11:13:50.2016178Z ================== 1 failed, 57 deselected, 2 rerun in 12.16s ================== 2025-12-04T11:13:50.2016238Z Got exit code 1 2025-12-04T11:13:50.2016306Z Retrying single test... 2025-12-04T11:13:50.2016570Z W1204 10:44:45.825000 68703 site-packages/torch/_inductor/utils.py:1703] Not enough SMs to use max_autotune_gemm mode 2025-12-04T11:13:50.2016957Z Test results will be stored in test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-f878b685b7c3b00a.xml 2025-12-04T11:13:50.2017059Z ============================= test session starts ============================== 2025-12-04T11:13:50.2017270Z platform linux -- Python 3.10.14, pytest-7.3.2, pluggy-1.6.0 -- /opt/conda/envs/py_3.10/bin/python 2025-12-04T11:13:50.2017338Z cachedir: .pytest_cache 2025-12-04T11:13:50.2017656Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow] 2025-12-04T11:13:50.2017734Z rootdir: /var/lib/jenkins/workspace 2025-12-04T11:13:50.2017804Z configfile: pytest.ini 2025-12-04T11:13:50.2018122Z plugins: hypothesis-6.56.4, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-14.0, subtests-0.13.1, xdist-3.3.1, xdoctest-1.3.0, typeguard-4.3.0 2025-12-04T11:13:50.2018249Z collecting ... collected 58 items / 57 deselected / 1 selected 2025-12-04T11:13:50.2018825Z stepcurrent: skipping 21 already run items. Running only test/inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_17_mid_dim_8_in_features_144_out_features_65_cuda_bfloat16 2025-12-04T11:13:50.2018897Z Running 1 items in this shard 2025-12-04T11:13:50.2018903Z 2025-12-04T11:13:50.2019640Z inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_17_mid_dim_8_in_features_144_out_features_65_cuda_bfloat16 [W1204 10:44:47.474318849 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2019644Z 2025-12-04T11:13:50.2019942Z [W1204 10:44:56.565634139 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2019946Z 2025-12-04T11:13:50.2020239Z [W1204 10:44:56.565870874 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2020285Z 2025-12-04T11:13:50.2020634Z [W1204 10:44:56.571728823 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2020638Z 2025-12-04T11:13:50.2020925Z [W1204 10:44:56.572299406 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2020931Z 2025-12-04T11:13:50.2021214Z [W1204 10:44:56.572481870 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2021218Z 2025-12-04T11:13:50.2021500Z [W1204 10:44:56.577747496 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2021503Z 2025-12-04T11:13:50.2021790Z [W1204 10:44:56.578268398 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2021795Z 2025-12-04T11:13:50.2022082Z [W1204 10:44:56.578428231 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2022086Z 2025-12-04T11:13:50.2022168Z ('RERUN', {'yellow': True}) [11.0847s] [100%] 2025-12-04T11:13:50.2022924Z inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_17_mid_dim_8_in_features_144_out_features_65_cuda_bfloat16 [W1204 10:44:57.393228319 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2022929Z 2025-12-04T11:13:50.2023222Z [W1204 10:44:57.393778071 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2023226Z 2025-12-04T11:13:50.2023570Z [W1204 10:44:57.393914434 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2023577Z 2025-12-04T11:13:50.2023886Z [W1204 10:44:57.396836679 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2023892Z 2025-12-04T11:13:50.2024177Z [W1204 10:44:57.397295399 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2024183Z 2025-12-04T11:13:50.2024465Z [W1204 10:44:57.397431452 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2024469Z 2025-12-04T11:13:50.2024758Z [W1204 10:44:57.402089355 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2024761Z 2025-12-04T11:13:50.2025045Z [W1204 10:44:57.402562965 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2025048Z 2025-12-04T11:13:50.2025338Z [W1204 10:44:57.402696628 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2025344Z 2025-12-04T11:13:50.2025422Z ('RERUN', {'yellow': True}) [0.5031s] [100%] 2025-12-04T11:13:50.2026142Z inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_17_mid_dim_8_in_features_144_out_features_65_cuda_bfloat16 [W1204 10:44:57.893520263 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2026146Z 2025-12-04T11:13:50.2026432Z [W1204 10:44:57.894081545 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2026436Z 2025-12-04T11:13:50.2026724Z [W1204 10:44:57.894222938 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2026727Z 2025-12-04T11:13:50.2027055Z [W1204 10:44:57.897163153 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2027123Z 2025-12-04T11:13:50.2027412Z [W1204 10:44:57.897626363 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2027421Z 2025-12-04T11:13:50.2027705Z [W1204 10:44:57.897766757 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2027708Z 2025-12-04T11:13:50.2027993Z [W1204 10:44:57.902426210 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2027997Z 2025-12-04T11:13:50.2028284Z [W1204 10:44:57.902901490 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2028287Z 2025-12-04T11:13:50.2028572Z [W1204 10:44:57.903036513 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2028576Z 2025-12-04T11:13:50.2028643Z FAILED [0.5000s] [100%] 2025-12-04T11:13:50.2028647Z 2025-12-04T11:13:50.2028766Z ==================================== RERUNS ==================================== 2025-12-04T11:13:50.2029062Z _ TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_8_in_features_144_out_features_65_cuda_bfloat16 _ 2025-12-04T11:13:50.2029142Z Traceback (most recent call last): 2025-12-04T11:13:50.2029461Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.2029531Z method(*args, **kwargs) 2025-12-04T11:13:50.2029824Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.2029887Z method(*args, **kwargs) 2025-12-04T11:13:50.2030177Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T11:13:50.2030237Z with policy(): 2025-12-04T11:13:50.2030533Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T11:13:50.2030602Z raise RuntimeError(msg) 2025-12-04T11:13:50.2031392Z RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_8_in_features_144_out_features_65_cuda_bfloat16! Caching allocator allocated memory was 0 and is now reported as 9728 on device 0. CUDA driver allocated memory was 230686720 and is now 274726912. 2025-12-04T11:13:50.2031396Z 2025-12-04T11:13:50.2031528Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.2032046Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_8_in_features_144_out_features_65_cuda_bfloat16 2025-12-04T11:13:50.2032051Z 2025-12-04T11:13:50.2032215Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.2032352Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.2032450Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.2033002Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('benchmarking.InductorBenchmarker.benchmark_gpu', 2), ('pad_mm_bench', 1), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.2033130Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.2033192Z graph_break [] 2025-12-04T11:13:50.2033319Z ----------------------------- Captured stderr call ----------------------------- 2025-12-04T11:13:50.2034087Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/constant_folding.py:256: UserWarning: Unsupported unwinding pattern: Address not in range (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/profiler/unwind/unwind.cpp:219.) 2025-12-04T11:13:50.2034197Z if out == self.unknown_value: 2025-12-04T11:13:50.2034488Z _ TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_8_in_features_144_out_features_65_cuda_bfloat16 _ 2025-12-04T11:13:50.2034566Z Traceback (most recent call last): 2025-12-04T11:13:50.2034860Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.2034925Z method(*args, **kwargs) 2025-12-04T11:13:50.2035216Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.2035276Z method(*args, **kwargs) 2025-12-04T11:13:50.2035562Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T11:13:50.2035627Z with policy(): 2025-12-04T11:13:50.2035920Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T11:13:50.2036029Z raise RuntimeError(msg) 2025-12-04T11:13:50.2036829Z RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_8_in_features_144_out_features_65_cuda_bfloat16! Caching allocator allocated memory was 9728 and is now reported as 19456 on device 0. CUDA driver allocated memory was 274726912 and is now 276824064. 2025-12-04T11:13:50.2036834Z 2025-12-04T11:13:50.2036959Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.2037473Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_8_in_features_144_out_features_65_cuda_bfloat16 2025-12-04T11:13:50.2037479Z 2025-12-04T11:13:50.2037635Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.2037765Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.2037858Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.2038402Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('benchmarking.InductorBenchmarker.benchmark_gpu', 2), ('pad_mm_bench', 1), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.2038527Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.2038586Z graph_break [] 2025-12-04T11:13:50.2038712Z ----------------------------- Captured stderr call ----------------------------- 2025-12-04T11:13:50.2039395Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/constant_folding.py:256: UserWarning: Unsupported unwinding pattern: Address not in range (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/profiler/unwind/unwind.cpp:219.) 2025-12-04T11:13:50.2039470Z if out == self.unknown_value: 2025-12-04T11:13:50.2039596Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.2039687Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.2039810Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.2040347Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('benchmarking.InductorBenchmarker.benchmark_gpu', 2), ('pad_mm_bench', 1), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.2040407Z graph_break [] 2025-12-04T11:13:50.2040498Z =================================== FAILURES =================================== 2025-12-04T11:13:50.2040788Z _ TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_8_in_features_144_out_features_65_cuda_bfloat16 _ 2025-12-04T11:13:50.2040908Z Traceback (most recent call last): 2025-12-04T11:13:50.2041370Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.2041437Z method(*args, **kwargs) 2025-12-04T11:13:50.2041729Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.2041792Z method(*args, **kwargs) 2025-12-04T11:13:50.2042079Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T11:13:50.2042141Z with policy(): 2025-12-04T11:13:50.2042429Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T11:13:50.2042496Z raise RuntimeError(msg) 2025-12-04T11:13:50.2043303Z RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_8_in_features_144_out_features_65_cuda_bfloat16! Caching allocator allocated memory was 19456 and is now reported as 29184 on device 0. CUDA driver allocated memory was 276824064 and is now 278921216. 2025-12-04T11:13:50.2043343Z 2025-12-04T11:13:50.2043472Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.2043991Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_8_in_features_144_out_features_65_cuda_bfloat16 2025-12-04T11:13:50.2043994Z 2025-12-04T11:13:50.2044155Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.2044279Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.2044369Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.2044911Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('benchmarking.InductorBenchmarker.benchmark_gpu', 2), ('pad_mm_bench', 1), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.2045039Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.2045103Z graph_break [] 2025-12-04T11:13:50.2045226Z ----------------------------- Captured stderr call ----------------------------- 2025-12-04T11:13:50.2045909Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/constant_folding.py:256: UserWarning: Unsupported unwinding pattern: Address not in range (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/profiler/unwind/unwind.cpp:219.) 2025-12-04T11:13:50.2045995Z if out == self.unknown_value: 2025-12-04T11:13:50.2046121Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.2046216Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.2046337Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.2046877Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('benchmarking.InductorBenchmarker.benchmark_gpu', 2), ('pad_mm_bench', 1), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.2046944Z graph_break [] 2025-12-04T11:13:50.2047066Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.2047156Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.2047276Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.2047810Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('benchmarking.InductorBenchmarker.benchmark_gpu', 2), ('pad_mm_bench', 1), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.2047872Z graph_break [] 2025-12-04T11:13:50.2048357Z - generated xml file: /var/lib/jenkins/workspace/test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-f878b685b7c3b00a.xml - 2025-12-04T11:13:50.2048572Z =========================== short test summary info ============================ 2025-12-04T11:13:50.2049866Z FAILED [0.5000s] inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_17_mid_dim_8_in_features_144_out_features_65_cuda_bfloat16 - RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_8_in_features_144_out_features_65_cuda_bfloat16! Caching allocator allocated memory was 19456 and is now reported as 29184 on device 0. CUDA driver allocated memory was 276824064 and is now 278921216. 2025-12-04T11:13:50.2049870Z 2025-12-04T11:13:50.2049998Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.2050513Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_17_mid_dim_8_in_features_144_out_features_65_cuda_bfloat16 2025-12-04T11:13:50.2050518Z 2025-12-04T11:13:50.2050675Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.2050829Z !!!!!!!!!!!!!!!!!!!!!!!!!! stopping after 1 failures !!!!!!!!!!!!!!!!!!!!!!!!!!! 2025-12-04T11:13:50.2050948Z ================== 1 failed, 57 deselected, 2 rerun in 12.11s ================== 2025-12-04T11:13:50.2051012Z Got exit code 1 2025-12-04T11:13:50.2051478Z FAILED CONSISTENTLY: test/inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_17_mid_dim_8_in_features_144_out_features_65_cuda_bfloat16 2025-12-04T11:13:50.2051721Z Test failed consistently, continuing with the rest of the tests due to continue-through-error being set 2025-12-04T11:13:50.2051988Z W1204 10:45:04.754000 68890 site-packages/torch/_inductor/utils.py:1703] Not enough SMs to use max_autotune_gemm mode 2025-12-04T11:13:50.2052374Z Test results will be stored in test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-232061e660a2ff26.xml 2025-12-04T11:13:50.2052473Z ============================= test session starts ============================== 2025-12-04T11:13:50.2052678Z platform linux -- Python 3.10.14, pytest-7.3.2, pluggy-1.6.0 -- /opt/conda/envs/py_3.10/bin/python 2025-12-04T11:13:50.2052744Z cachedir: .pytest_cache 2025-12-04T11:13:50.2053052Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow] 2025-12-04T11:13:50.2053129Z rootdir: /var/lib/jenkins/workspace 2025-12-04T11:13:50.2053194Z configfile: pytest.ini 2025-12-04T11:13:50.2053511Z plugins: hypothesis-6.56.4, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-14.0, subtests-0.13.1, xdist-3.3.1, xdoctest-1.3.0, typeguard-4.3.0 2025-12-04T11:13:50.2053638Z collecting ... collected 58 items / 22 deselected / 36 selected 2025-12-04T11:13:50.2053730Z stepcurrent: skipping 22 already run items. 2025-12-04T11:13:50.2053799Z Running 36 items in this shard 2025-12-04T11:13:50.2053802Z 2025-12-04T11:13:50.2054305Z inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_1_mid_dim_1_in_features_1024_out_features_1024_cuda_bfloat16 ('RERUN', {'yellow': True}) [2.0590s] [ 2%] 2025-12-04T11:13:50.2054802Z inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_1_mid_dim_1_in_features_1024_out_features_1024_cuda_bfloat16 ('RERUN', {'yellow': True}) [0.6194s] [ 2%] 2025-12-04T11:13:50.2055249Z inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_1_mid_dim_1_in_features_1024_out_features_1024_cuda_bfloat16 FAILED [0.6199s] [ 2%] 2025-12-04T11:13:50.2055253Z 2025-12-04T11:13:50.2055343Z ==================================== RERUNS ==================================== 2025-12-04T11:13:50.2055637Z _ TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_1_in_features_1024_out_features_1024_cuda_bfloat16 _ 2025-12-04T11:13:50.2055828Z Traceback (most recent call last): 2025-12-04T11:13:50.2056140Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.2056207Z method(*args, **kwargs) 2025-12-04T11:13:50.2056498Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.2056561Z method(*args, **kwargs) 2025-12-04T11:13:50.2056845Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T11:13:50.2056909Z with policy(): 2025-12-04T11:13:50.2057197Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T11:13:50.2057266Z raise RuntimeError(msg) 2025-12-04T11:13:50.2058075Z RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_1_in_features_1024_out_features_1024_cuda_bfloat16! Caching allocator allocated memory was 0 and is now reported as 1048576 on device 0. CUDA driver allocated memory was 230686720 and is now 274726912. 2025-12-04T11:13:50.2058114Z 2025-12-04T11:13:50.2058243Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.2058770Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_1_in_features_1024_out_features_1024_cuda_bfloat16 2025-12-04T11:13:50.2058773Z 2025-12-04T11:13:50.2058930Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.2059061Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.2059153Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.2059504Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.2059648Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.2059713Z graph_break [] 2025-12-04T11:13:50.2060007Z _ TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_1_in_features_1024_out_features_1024_cuda_bfloat16 _ 2025-12-04T11:13:50.2060082Z Traceback (most recent call last): 2025-12-04T11:13:50.2060379Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.2060447Z method(*args, **kwargs) 2025-12-04T11:13:50.2060735Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.2060798Z method(*args, **kwargs) 2025-12-04T11:13:50.2061094Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T11:13:50.2061153Z with policy(): 2025-12-04T11:13:50.2061450Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T11:13:50.2061518Z raise RuntimeError(msg) 2025-12-04T11:13:50.2062336Z RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_1_in_features_1024_out_features_1024_cuda_bfloat16! Caching allocator allocated memory was 1048576 and is now reported as 2097152 on device 0. CUDA driver allocated memory was 274726912 and is now 276824064. 2025-12-04T11:13:50.2062345Z 2025-12-04T11:13:50.2062471Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.2062991Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_1_in_features_1024_out_features_1024_cuda_bfloat16 2025-12-04T11:13:50.2063036Z 2025-12-04T11:13:50.2063264Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.2063394Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.2063488Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.2063894Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.2064024Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.2064090Z graph_break [] 2025-12-04T11:13:50.2064218Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.2064305Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.2064430Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.2064776Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.2064882Z graph_break [] 2025-12-04T11:13:50.2064968Z =================================== FAILURES =================================== 2025-12-04T11:13:50.2065261Z _ TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_1_in_features_1024_out_features_1024_cuda_bfloat16 _ 2025-12-04T11:13:50.2065338Z Traceback (most recent call last): 2025-12-04T11:13:50.2065648Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.2065713Z method(*args, **kwargs) 2025-12-04T11:13:50.2066006Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.2066069Z method(*args, **kwargs) 2025-12-04T11:13:50.2066361Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T11:13:50.2066422Z with policy(): 2025-12-04T11:13:50.2066711Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T11:13:50.2066783Z raise RuntimeError(msg) 2025-12-04T11:13:50.2067600Z RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_1_in_features_1024_out_features_1024_cuda_bfloat16! Caching allocator allocated memory was 2097152 and is now reported as 3145728 on device 0. CUDA driver allocated memory was 276824064 and is now 278921216. 2025-12-04T11:13:50.2067604Z 2025-12-04T11:13:50.2067732Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.2068245Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_1_in_features_1024_out_features_1024_cuda_bfloat16 2025-12-04T11:13:50.2068251Z 2025-12-04T11:13:50.2068411Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.2068543Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.2068636Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.2068979Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.2069110Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.2069177Z graph_break [] 2025-12-04T11:13:50.2069303Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.2069392Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.2069516Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.2069987Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.2070051Z graph_break [] 2025-12-04T11:13:50.2070178Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.2070265Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.2070384Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.2070729Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.2070790Z graph_break [] 2025-12-04T11:13:50.2071275Z - generated xml file: /var/lib/jenkins/workspace/test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-232061e660a2ff26.xml - 2025-12-04T11:13:50.2071375Z =========================== short test summary info ============================ 2025-12-04T11:13:50.2072670Z FAILED [0.6199s] inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_1_mid_dim_1_in_features_1024_out_features_1024_cuda_bfloat16 - RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_1_in_features_1024_out_features_1024_cuda_bfloat16! Caching allocator allocated memory was 2097152 and is now reported as 3145728 on device 0. CUDA driver allocated memory was 276824064 and is now 278921216. 2025-12-04T11:13:50.2072715Z 2025-12-04T11:13:50.2072842Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.2073364Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_1_in_features_1024_out_features_1024_cuda_bfloat16 2025-12-04T11:13:50.2073373Z 2025-12-04T11:13:50.2073528Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.2073635Z !!!!!!!!!!!!!!!!!!!!!!!!!! stopping after 1 failures !!!!!!!!!!!!!!!!!!!!!!!!!!! 2025-12-04T11:13:50.2073764Z ================== 1 failed, 22 deselected, 2 rerun in 3.32s =================== 2025-12-04T11:13:50.2073824Z Got exit code 1 2025-12-04T11:13:50.2073889Z Retrying single test... 2025-12-04T11:13:50.2074158Z W1204 10:45:14.966000 69079 site-packages/torch/_inductor/utils.py:1703] Not enough SMs to use max_autotune_gemm mode 2025-12-04T11:13:50.2074544Z Test results will be stored in test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-fe4ea69fd2c99307.xml 2025-12-04T11:13:50.2074644Z ============================= test session starts ============================== 2025-12-04T11:13:50.2074849Z platform linux -- Python 3.10.14, pytest-7.3.2, pluggy-1.6.0 -- /opt/conda/envs/py_3.10/bin/python 2025-12-04T11:13:50.2074917Z cachedir: .pytest_cache 2025-12-04T11:13:50.2075228Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow] 2025-12-04T11:13:50.2075305Z rootdir: /var/lib/jenkins/workspace 2025-12-04T11:13:50.2075373Z configfile: pytest.ini 2025-12-04T11:13:50.2075692Z plugins: hypothesis-6.56.4, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-14.0, subtests-0.13.1, xdist-3.3.1, xdoctest-1.3.0, typeguard-4.3.0 2025-12-04T11:13:50.2075820Z collecting ... collected 58 items / 57 deselected / 1 selected 2025-12-04T11:13:50.2076391Z stepcurrent: skipping 22 already run items. Running only test/inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_1_mid_dim_1_in_features_1024_out_features_1024_cuda_bfloat16 2025-12-04T11:13:50.2076463Z Running 1 items in this shard 2025-12-04T11:13:50.2076467Z 2025-12-04T11:13:50.2077268Z inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_1_mid_dim_1_in_features_1024_out_features_1024_cuda_bfloat16 [W1204 10:45:16.240907211 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2077311Z 2025-12-04T11:13:50.2077615Z [W1204 10:45:25.453595551 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2077619Z 2025-12-04T11:13:50.2078082Z [W1204 10:45:25.453829116 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2078087Z 2025-12-04T11:13:50.2078381Z [W1204 10:45:25.459458260 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2078384Z 2025-12-04T11:13:50.2078671Z [W1204 10:45:25.459988751 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2078677Z 2025-12-04T11:13:50.2078972Z [W1204 10:45:25.460198276 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2078976Z 2025-12-04T11:13:50.2079340Z [W1204 10:45:25.465516712 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2079344Z 2025-12-04T11:13:50.2079633Z [W1204 10:45:25.466032573 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2079636Z 2025-12-04T11:13:50.2079919Z [W1204 10:45:25.466191427 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2079923Z 2025-12-04T11:13:50.2080009Z ('RERUN', {'yellow': True}) [11.2804s] [100%] 2025-12-04T11:13:50.2080742Z inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_1_mid_dim_1_in_features_1024_out_features_1024_cuda_bfloat16 [W1204 10:45:26.822865170 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2080749Z 2025-12-04T11:13:50.2081040Z [W1204 10:45:26.823417902 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2081047Z 2025-12-04T11:13:50.2081337Z [W1204 10:45:26.823562566 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2081340Z 2025-12-04T11:13:50.2081626Z [W1204 10:45:26.826517270 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2081629Z 2025-12-04T11:13:50.2081919Z [W1204 10:45:26.827082362 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2081924Z 2025-12-04T11:13:50.2082213Z [W1204 10:45:26.827219465 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2082216Z 2025-12-04T11:13:50.2082505Z [W1204 10:45:26.831826076 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2082510Z 2025-12-04T11:13:50.2082794Z [W1204 10:45:26.832297366 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2082797Z 2025-12-04T11:13:50.2083099Z [W1204 10:45:26.832434019 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2083102Z 2025-12-04T11:13:50.2083183Z ('RERUN', {'yellow': True}) [0.5919s] [100%] 2025-12-04T11:13:50.2084007Z inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_1_mid_dim_1_in_features_1024_out_features_1024_cuda_bfloat16 [W1204 10:45:27.406390120 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2084072Z 2025-12-04T11:13:50.2084367Z [W1204 10:45:27.406920152 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2084371Z 2025-12-04T11:13:50.2084658Z [W1204 10:45:27.407058225 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2084661Z 2025-12-04T11:13:50.2084950Z [W1204 10:45:27.409960828 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2084954Z 2025-12-04T11:13:50.2085237Z [W1204 10:45:27.410543251 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2085240Z 2025-12-04T11:13:50.2085532Z [W1204 10:45:27.410685514 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2085537Z 2025-12-04T11:13:50.2085823Z [W1204 10:45:27.415227163 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2085863Z 2025-12-04T11:13:50.2086153Z [W1204 10:45:27.415691723 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2086156Z 2025-12-04T11:13:50.2086442Z [W1204 10:45:27.415828486 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2086446Z 2025-12-04T11:13:50.2086513Z FAILED [0.5832s] [100%] 2025-12-04T11:13:50.2086516Z 2025-12-04T11:13:50.2086600Z ==================================== RERUNS ==================================== 2025-12-04T11:13:50.2086894Z _ TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_1_in_features_1024_out_features_1024_cuda_bfloat16 _ 2025-12-04T11:13:50.2086979Z Traceback (most recent call last): 2025-12-04T11:13:50.2087280Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.2087348Z method(*args, **kwargs) 2025-12-04T11:13:50.2087641Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.2087705Z method(*args, **kwargs) 2025-12-04T11:13:50.2087994Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T11:13:50.2088054Z with policy(): 2025-12-04T11:13:50.2088344Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T11:13:50.2088417Z raise RuntimeError(msg) 2025-12-04T11:13:50.2089228Z RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_1_in_features_1024_out_features_1024_cuda_bfloat16! Caching allocator allocated memory was 0 and is now reported as 1048576 on device 0. CUDA driver allocated memory was 230686720 and is now 274726912. 2025-12-04T11:13:50.2089234Z 2025-12-04T11:13:50.2089367Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.2089884Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_1_in_features_1024_out_features_1024_cuda_bfloat16 2025-12-04T11:13:50.2089888Z 2025-12-04T11:13:50.2090049Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.2090179Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.2090274Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.2090738Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.2090869Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.2090931Z graph_break [] 2025-12-04T11:13:50.2091060Z ----------------------------- Captured stderr call ----------------------------- 2025-12-04T11:13:50.2091746Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/constant_folding.py:256: UserWarning: Unsupported unwinding pattern: Address not in range (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/profiler/unwind/unwind.cpp:219.) 2025-12-04T11:13:50.2091821Z if out == self.unknown_value: 2025-12-04T11:13:50.2092111Z _ TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_1_in_features_1024_out_features_1024_cuda_bfloat16 _ 2025-12-04T11:13:50.2092184Z Traceback (most recent call last): 2025-12-04T11:13:50.2092487Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.2092552Z method(*args, **kwargs) 2025-12-04T11:13:50.2092885Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.2092955Z method(*args, **kwargs) 2025-12-04T11:13:50.2093244Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T11:13:50.2093310Z with policy(): 2025-12-04T11:13:50.2093598Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T11:13:50.2093666Z raise RuntimeError(msg) 2025-12-04T11:13:50.2094497Z RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_1_in_features_1024_out_features_1024_cuda_bfloat16! Caching allocator allocated memory was 1048576 and is now reported as 2097152 on device 0. CUDA driver allocated memory was 274726912 and is now 276824064. 2025-12-04T11:13:50.2094503Z 2025-12-04T11:13:50.2094630Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.2095156Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_1_in_features_1024_out_features_1024_cuda_bfloat16 2025-12-04T11:13:50.2095160Z 2025-12-04T11:13:50.2095315Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.2095443Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.2095535Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.2095881Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.2096019Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.2096081Z graph_break [] 2025-12-04T11:13:50.2096203Z ----------------------------- Captured stderr call ----------------------------- 2025-12-04T11:13:50.2096895Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/constant_folding.py:256: UserWarning: Unsupported unwinding pattern: Address not in range (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/profiler/unwind/unwind.cpp:219.) 2025-12-04T11:13:50.2096965Z if out == self.unknown_value: 2025-12-04T11:13:50.2097090Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.2097181Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.2097304Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.2097647Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.2097769Z graph_break [] 2025-12-04T11:13:50.2097924Z =================================== FAILURES =================================== 2025-12-04T11:13:50.2098217Z _ TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_1_in_features_1024_out_features_1024_cuda_bfloat16 _ 2025-12-04T11:13:50.2098291Z Traceback (most recent call last): 2025-12-04T11:13:50.2098589Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.2098657Z method(*args, **kwargs) 2025-12-04T11:13:50.2098946Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.2099020Z method(*args, **kwargs) 2025-12-04T11:13:50.2099305Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T11:13:50.2099372Z with policy(): 2025-12-04T11:13:50.2099664Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T11:13:50.2099772Z raise RuntimeError(msg) 2025-12-04T11:13:50.2100593Z RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_1_in_features_1024_out_features_1024_cuda_bfloat16! Caching allocator allocated memory was 2097152 and is now reported as 3145728 on device 0. CUDA driver allocated memory was 276824064 and is now 278921216. 2025-12-04T11:13:50.2100597Z 2025-12-04T11:13:50.2100723Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.2101247Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_1_in_features_1024_out_features_1024_cuda_bfloat16 2025-12-04T11:13:50.2101252Z 2025-12-04T11:13:50.2101410Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.2101541Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.2101632Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.2101979Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.2102108Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.2102167Z graph_break [] 2025-12-04T11:13:50.2102288Z ----------------------------- Captured stderr call ----------------------------- 2025-12-04T11:13:50.2102987Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/constant_folding.py:256: UserWarning: Unsupported unwinding pattern: Address not in range (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/profiler/unwind/unwind.cpp:219.) 2025-12-04T11:13:50.2103059Z if out == self.unknown_value: 2025-12-04T11:13:50.2103187Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.2103276Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.2103403Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.2103793Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.2103852Z graph_break [] 2025-12-04T11:13:50.2103980Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.2104077Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.2104200Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.2104542Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.2104643Z graph_break [] 2025-12-04T11:13:50.2105194Z - generated xml file: /var/lib/jenkins/workspace/test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-fe4ea69fd2c99307.xml - 2025-12-04T11:13:50.2105303Z =========================== short test summary info ============================ 2025-12-04T11:13:50.2106600Z FAILED [0.5832s] inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_1_mid_dim_1_in_features_1024_out_features_1024_cuda_bfloat16 - RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_1_in_features_1024_out_features_1024_cuda_bfloat16! Caching allocator allocated memory was 2097152 and is now reported as 3145728 on device 0. CUDA driver allocated memory was 276824064 and is now 278921216. 2025-12-04T11:13:50.2106610Z 2025-12-04T11:13:50.2106733Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.2107253Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_1_in_features_1024_out_features_1024_cuda_bfloat16 2025-12-04T11:13:50.2107289Z 2025-12-04T11:13:50.2107449Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.2107553Z !!!!!!!!!!!!!!!!!!!!!!!!!! stopping after 1 failures !!!!!!!!!!!!!!!!!!!!!!!!!!! 2025-12-04T11:13:50.2107671Z ================== 1 failed, 57 deselected, 2 rerun in 12.48s ================== 2025-12-04T11:13:50.2107730Z Got exit code 1 2025-12-04T11:13:50.2107798Z Retrying single test... 2025-12-04T11:13:50.2108071Z W1204 10:45:34.235000 69273 site-packages/torch/_inductor/utils.py:1703] Not enough SMs to use max_autotune_gemm mode 2025-12-04T11:13:50.2108462Z Test results will be stored in test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-4bc55824cd7c5544.xml 2025-12-04T11:13:50.2108561Z ============================= test session starts ============================== 2025-12-04T11:13:50.2108771Z platform linux -- Python 3.10.14, pytest-7.3.2, pluggy-1.6.0 -- /opt/conda/envs/py_3.10/bin/python 2025-12-04T11:13:50.2108840Z cachedir: .pytest_cache 2025-12-04T11:13:50.2109149Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow] 2025-12-04T11:13:50.2109226Z rootdir: /var/lib/jenkins/workspace 2025-12-04T11:13:50.2109292Z configfile: pytest.ini 2025-12-04T11:13:50.2109612Z plugins: hypothesis-6.56.4, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-14.0, subtests-0.13.1, xdist-3.3.1, xdoctest-1.3.0, typeguard-4.3.0 2025-12-04T11:13:50.2109738Z collecting ... collected 58 items / 57 deselected / 1 selected 2025-12-04T11:13:50.2110308Z stepcurrent: skipping 22 already run items. Running only test/inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_1_mid_dim_1_in_features_1024_out_features_1024_cuda_bfloat16 2025-12-04T11:13:50.2110387Z Running 1 items in this shard 2025-12-04T11:13:50.2110391Z 2025-12-04T11:13:50.2111124Z inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_1_mid_dim_1_in_features_1024_out_features_1024_cuda_bfloat16 [W1204 10:45:35.502639945 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2111128Z 2025-12-04T11:13:50.2111428Z [W1204 10:45:44.782333893 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2111432Z 2025-12-04T11:13:50.2111722Z [W1204 10:45:44.782571268 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2111726Z 2025-12-04T11:13:50.2112059Z [W1204 10:45:44.788291874 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2112128Z 2025-12-04T11:13:50.2112415Z [W1204 10:45:44.788859967 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2112420Z 2025-12-04T11:13:50.2112709Z [W1204 10:45:44.789023870 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2112712Z 2025-12-04T11:13:50.2112997Z [W1204 10:45:44.794512772 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2113001Z 2025-12-04T11:13:50.2113289Z [W1204 10:45:44.795049674 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2113292Z 2025-12-04T11:13:50.2113577Z [W1204 10:45:44.795206077 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2113582Z 2025-12-04T11:13:50.2113663Z ('RERUN', {'yellow': True}) [11.3402s] [100%] 2025-12-04T11:13:50.2114443Z inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_1_mid_dim_1_in_features_1024_out_features_1024_cuda_bfloat16 [W1204 10:45:46.146625892 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2114447Z 2025-12-04T11:13:50.2114735Z [W1204 10:45:46.147163694 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2114738Z 2025-12-04T11:13:50.2115026Z [W1204 10:45:46.147302807 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2115029Z 2025-12-04T11:13:50.2115314Z [W1204 10:45:46.150257432 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2115319Z 2025-12-04T11:13:50.2115611Z [W1204 10:45:46.150827674 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2115616Z 2025-12-04T11:13:50.2115900Z [W1204 10:45:46.150964448 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2115904Z 2025-12-04T11:13:50.2116193Z [W1204 10:45:46.155481777 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2116198Z 2025-12-04T11:13:50.2116482Z [W1204 10:45:46.155947298 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2116485Z 2025-12-04T11:13:50.2116769Z [W1204 10:45:46.156082701 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2116777Z 2025-12-04T11:13:50.2116857Z ('RERUN', {'yellow': True}) [0.5893s] [100%] 2025-12-04T11:13:50.2117588Z inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_1_mid_dim_1_in_features_1024_out_features_1024_cuda_bfloat16 [W1204 10:45:46.726990275 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2117593Z 2025-12-04T11:13:50.2117886Z [W1204 10:45:46.727524537 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2117889Z 2025-12-04T11:13:50.2118174Z [W1204 10:45:46.727665580 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2118178Z 2025-12-04T11:13:50.2118465Z [W1204 10:45:46.730612895 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2118508Z 2025-12-04T11:13:50.2118859Z [W1204 10:45:46.731186048 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2118864Z 2025-12-04T11:13:50.2119153Z [W1204 10:45:46.731325231 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2119156Z 2025-12-04T11:13:50.2119441Z [W1204 10:45:46.735855821 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2119444Z 2025-12-04T11:13:50.2119730Z [W1204 10:45:46.736317741 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2119734Z 2025-12-04T11:13:50.2120021Z [W1204 10:45:46.736451574 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2120026Z 2025-12-04T11:13:50.2120087Z FAILED [0.5801s] [100%] 2025-12-04T11:13:50.2120093Z 2025-12-04T11:13:50.2120191Z ==================================== RERUNS ==================================== 2025-12-04T11:13:50.2120539Z _ TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_1_in_features_1024_out_features_1024_cuda_bfloat16 _ 2025-12-04T11:13:50.2120619Z Traceback (most recent call last): 2025-12-04T11:13:50.2120920Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.2120984Z method(*args, **kwargs) 2025-12-04T11:13:50.2121282Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.2121345Z method(*args, **kwargs) 2025-12-04T11:13:50.2121635Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T11:13:50.2121697Z with policy(): 2025-12-04T11:13:50.2121990Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T11:13:50.2122059Z raise RuntimeError(msg) 2025-12-04T11:13:50.2122868Z RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_1_in_features_1024_out_features_1024_cuda_bfloat16! Caching allocator allocated memory was 0 and is now reported as 1048576 on device 0. CUDA driver allocated memory was 230686720 and is now 274726912. 2025-12-04T11:13:50.2122872Z 2025-12-04T11:13:50.2122999Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.2123518Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_1_in_features_1024_out_features_1024_cuda_bfloat16 2025-12-04T11:13:50.2123524Z 2025-12-04T11:13:50.2123680Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.2123815Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.2123920Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.2124273Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.2124400Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.2124459Z graph_break [] 2025-12-04T11:13:50.2124586Z ----------------------------- Captured stderr call ----------------------------- 2025-12-04T11:13:50.2125277Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/constant_folding.py:256: UserWarning: Unsupported unwinding pattern: Address not in range (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/profiler/unwind/unwind.cpp:219.) 2025-12-04T11:13:50.2125394Z if out == self.unknown_value: 2025-12-04T11:13:50.2125747Z _ TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_1_in_features_1024_out_features_1024_cuda_bfloat16 _ 2025-12-04T11:13:50.2125824Z Traceback (most recent call last): 2025-12-04T11:13:50.2126119Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.2126183Z method(*args, **kwargs) 2025-12-04T11:13:50.2126470Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.2126538Z method(*args, **kwargs) 2025-12-04T11:13:50.2126821Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T11:13:50.2126885Z with policy(): 2025-12-04T11:13:50.2127172Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T11:13:50.2127238Z raise RuntimeError(msg) 2025-12-04T11:13:50.2128081Z RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_1_in_features_1024_out_features_1024_cuda_bfloat16! Caching allocator allocated memory was 1048576 and is now reported as 2097152 on device 0. CUDA driver allocated memory was 274726912 and is now 276824064. 2025-12-04T11:13:50.2128127Z 2025-12-04T11:13:50.2128254Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.2128776Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_1_in_features_1024_out_features_1024_cuda_bfloat16 2025-12-04T11:13:50.2128780Z 2025-12-04T11:13:50.2128936Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.2129061Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.2129164Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.2129511Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.2129646Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.2129705Z graph_break [] 2025-12-04T11:13:50.2129828Z ----------------------------- Captured stderr call ----------------------------- 2025-12-04T11:13:50.2130514Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/constant_folding.py:256: UserWarning: Unsupported unwinding pattern: Address not in range (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/profiler/unwind/unwind.cpp:219.) 2025-12-04T11:13:50.2130582Z if out == self.unknown_value: 2025-12-04T11:13:50.2130710Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.2130801Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.2130938Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.2131289Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.2131348Z graph_break [] 2025-12-04T11:13:50.2131432Z =================================== FAILURES =================================== 2025-12-04T11:13:50.2131726Z _ TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_1_in_features_1024_out_features_1024_cuda_bfloat16 _ 2025-12-04T11:13:50.2131799Z Traceback (most recent call last): 2025-12-04T11:13:50.2132096Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.2132160Z method(*args, **kwargs) 2025-12-04T11:13:50.2132553Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.2132626Z method(*args, **kwargs) 2025-12-04T11:13:50.2132913Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T11:13:50.2132975Z with policy(): 2025-12-04T11:13:50.2133261Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T11:13:50.2133329Z raise RuntimeError(msg) 2025-12-04T11:13:50.2134153Z RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_1_in_features_1024_out_features_1024_cuda_bfloat16! Caching allocator allocated memory was 2097152 and is now reported as 3145728 on device 0. CUDA driver allocated memory was 276824064 and is now 278921216. 2025-12-04T11:13:50.2134158Z 2025-12-04T11:13:50.2134281Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.2134805Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_1_in_features_1024_out_features_1024_cuda_bfloat16 2025-12-04T11:13:50.2134847Z 2025-12-04T11:13:50.2135005Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.2135130Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.2135226Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.2135566Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.2135692Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.2135752Z graph_break [] 2025-12-04T11:13:50.2135876Z ----------------------------- Captured stderr call ----------------------------- 2025-12-04T11:13:50.2136562Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/constant_folding.py:256: UserWarning: Unsupported unwinding pattern: Address not in range (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/profiler/unwind/unwind.cpp:219.) 2025-12-04T11:13:50.2136633Z if out == self.unknown_value: 2025-12-04T11:13:50.2136764Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.2136861Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.2136985Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.2137330Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.2137389Z graph_break [] 2025-12-04T11:13:50.2137512Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.2137606Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.2137735Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.2138084Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.2138141Z graph_break [] 2025-12-04T11:13:50.2138630Z - generated xml file: /var/lib/jenkins/workspace/test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-4bc55824cd7c5544.xml - 2025-12-04T11:13:50.2138738Z =========================== short test summary info ============================ 2025-12-04T11:13:50.2140108Z FAILED [0.5801s] inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_1_mid_dim_1_in_features_1024_out_features_1024_cuda_bfloat16 - RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_1_in_features_1024_out_features_1024_cuda_bfloat16! Caching allocator allocated memory was 2097152 and is now reported as 3145728 on device 0. CUDA driver allocated memory was 276824064 and is now 278921216. 2025-12-04T11:13:50.2140146Z 2025-12-04T11:13:50.2140282Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.2140805Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_1_in_features_1024_out_features_1024_cuda_bfloat16 2025-12-04T11:13:50.2140809Z 2025-12-04T11:13:50.2140973Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.2141080Z !!!!!!!!!!!!!!!!!!!!!!!!!! stopping after 1 failures !!!!!!!!!!!!!!!!!!!!!!!!!!! 2025-12-04T11:13:50.2141195Z ================== 1 failed, 57 deselected, 2 rerun in 12.53s ================== 2025-12-04T11:13:50.2141260Z Got exit code 1 2025-12-04T11:13:50.2141735Z FAILED CONSISTENTLY: test/inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_1_mid_dim_1_in_features_1024_out_features_1024_cuda_bfloat16 2025-12-04T11:13:50.2142017Z Test failed consistently, continuing with the rest of the tests due to continue-through-error being set 2025-12-04T11:13:50.2142282Z W1204 10:45:53.554000 69467 site-packages/torch/_inductor/utils.py:1703] Not enough SMs to use max_autotune_gemm mode 2025-12-04T11:13:50.2142667Z Test results will be stored in test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-c2fc9a8f37b6f624.xml 2025-12-04T11:13:50.2142767Z ============================= test session starts ============================== 2025-12-04T11:13:50.2142974Z platform linux -- Python 3.10.14, pytest-7.3.2, pluggy-1.6.0 -- /opt/conda/envs/py_3.10/bin/python 2025-12-04T11:13:50.2143045Z cachedir: .pytest_cache 2025-12-04T11:13:50.2143352Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow] 2025-12-04T11:13:50.2143431Z rootdir: /var/lib/jenkins/workspace 2025-12-04T11:13:50.2143501Z configfile: pytest.ini 2025-12-04T11:13:50.2143909Z plugins: hypothesis-6.56.4, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-14.0, subtests-0.13.1, xdist-3.3.1, xdoctest-1.3.0, typeguard-4.3.0 2025-12-04T11:13:50.2144040Z collecting ... collected 58 items / 23 deselected / 35 selected 2025-12-04T11:13:50.2144131Z stepcurrent: skipping 23 already run items. 2025-12-04T11:13:50.2144201Z Running 35 items in this shard 2025-12-04T11:13:50.2144205Z 2025-12-04T11:13:50.2144712Z inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_1_mid_dim_1_in_features_1024_out_features_64_cuda_bfloat16 ('RERUN', {'yellow': True}) [1.8668s] [ 2%] 2025-12-04T11:13:50.2145198Z inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_1_mid_dim_1_in_features_1024_out_features_64_cuda_bfloat16 ('RERUN', {'yellow': True}) [0.4609s] [ 2%] 2025-12-04T11:13:50.2145645Z inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_1_mid_dim_1_in_features_1024_out_features_64_cuda_bfloat16 FAILED [0.4550s] [ 2%] 2025-12-04T11:13:50.2145654Z 2025-12-04T11:13:50.2145738Z ==================================== RERUNS ==================================== 2025-12-04T11:13:50.2146034Z _ TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_1_in_features_1024_out_features_64_cuda_bfloat16 _ 2025-12-04T11:13:50.2146114Z Traceback (most recent call last): 2025-12-04T11:13:50.2146417Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.2146483Z method(*args, **kwargs) 2025-12-04T11:13:50.2146776Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.2146885Z method(*args, **kwargs) 2025-12-04T11:13:50.2147258Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T11:13:50.2147320Z with policy(): 2025-12-04T11:13:50.2147612Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T11:13:50.2147683Z raise RuntimeError(msg) 2025-12-04T11:13:50.2148484Z RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_1_in_features_1024_out_features_64_cuda_bfloat16! Caching allocator allocated memory was 0 and is now reported as 65536 on device 0. CUDA driver allocated memory was 230686720 and is now 274726912. 2025-12-04T11:13:50.2148487Z 2025-12-04T11:13:50.2148613Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.2149132Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_1_in_features_1024_out_features_64_cuda_bfloat16 2025-12-04T11:13:50.2149171Z 2025-12-04T11:13:50.2149347Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.2149479Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.2149575Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.2149930Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.2150059Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.2150119Z graph_break [] 2025-12-04T11:13:50.2150417Z _ TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_1_in_features_1024_out_features_64_cuda_bfloat16 _ 2025-12-04T11:13:50.2150494Z Traceback (most recent call last): 2025-12-04T11:13:50.2150800Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.2150866Z method(*args, **kwargs) 2025-12-04T11:13:50.2151157Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.2151223Z method(*args, **kwargs) 2025-12-04T11:13:50.2151510Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T11:13:50.2151569Z with policy(): 2025-12-04T11:13:50.2151862Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T11:13:50.2151928Z raise RuntimeError(msg) 2025-12-04T11:13:50.2152737Z RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_1_in_features_1024_out_features_64_cuda_bfloat16! Caching allocator allocated memory was 65536 and is now reported as 131072 on device 0. CUDA driver allocated memory was 274726912 and is now 276824064. 2025-12-04T11:13:50.2152744Z 2025-12-04T11:13:50.2152869Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.2153381Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_1_in_features_1024_out_features_64_cuda_bfloat16 2025-12-04T11:13:50.2153389Z 2025-12-04T11:13:50.2153543Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.2153671Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.2153767Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.2154112Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.2154429Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.2154496Z graph_break [] 2025-12-04T11:13:50.2154620Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.2154713Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.2154831Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.2155173Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.2155234Z graph_break [] 2025-12-04T11:13:50.2155318Z =================================== FAILURES =================================== 2025-12-04T11:13:50.2155605Z _ TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_1_in_features_1024_out_features_64_cuda_bfloat16 _ 2025-12-04T11:13:50.2155680Z Traceback (most recent call last): 2025-12-04T11:13:50.2155977Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.2156082Z method(*args, **kwargs) 2025-12-04T11:13:50.2156369Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.2156433Z method(*args, **kwargs) 2025-12-04T11:13:50.2156737Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T11:13:50.2156797Z with policy(): 2025-12-04T11:13:50.2157094Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T11:13:50.2157162Z raise RuntimeError(msg) 2025-12-04T11:13:50.2157972Z RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_1_in_features_1024_out_features_64_cuda_bfloat16! Caching allocator allocated memory was 131072 and is now reported as 196608 on device 0. CUDA driver allocated memory was 276824064 and is now 278921216. 2025-12-04T11:13:50.2157978Z 2025-12-04T11:13:50.2158107Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.2158617Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_1_in_features_1024_out_features_64_cuda_bfloat16 2025-12-04T11:13:50.2158620Z 2025-12-04T11:13:50.2158778Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.2158900Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.2158989Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.2159331Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.2159458Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.2159520Z graph_break [] 2025-12-04T11:13:50.2159643Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.2159733Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.2159857Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.2160197Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.2160255Z graph_break [] 2025-12-04T11:13:50.2160382Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.2160473Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.2160639Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.2161047Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.2161112Z graph_break [] 2025-12-04T11:13:50.2161601Z - generated xml file: /var/lib/jenkins/workspace/test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-c2fc9a8f37b6f624.xml - 2025-12-04T11:13:50.2161701Z =========================== short test summary info ============================ 2025-12-04T11:13:50.2162980Z FAILED [0.4550s] inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_1_mid_dim_1_in_features_1024_out_features_64_cuda_bfloat16 - RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_1_in_features_1024_out_features_64_cuda_bfloat16! Caching allocator allocated memory was 131072 and is now reported as 196608 on device 0. CUDA driver allocated memory was 276824064 and is now 278921216. 2025-12-04T11:13:50.2162988Z 2025-12-04T11:13:50.2163112Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.2163663Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_1_in_features_1024_out_features_64_cuda_bfloat16 2025-12-04T11:13:50.2163667Z 2025-12-04T11:13:50.2163821Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.2163925Z !!!!!!!!!!!!!!!!!!!!!!!!!! stopping after 1 failures !!!!!!!!!!!!!!!!!!!!!!!!!!! 2025-12-04T11:13:50.2164044Z ================== 1 failed, 23 deselected, 2 rerun in 2.81s =================== 2025-12-04T11:13:50.2164102Z Got exit code 1 2025-12-04T11:13:50.2164170Z Retrying single test... 2025-12-04T11:13:50.2164427Z W1204 10:46:03.650000 69655 site-packages/torch/_inductor/utils.py:1703] Not enough SMs to use max_autotune_gemm mode 2025-12-04T11:13:50.2164815Z Test results will be stored in test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-df0458d9683e0406.xml 2025-12-04T11:13:50.2164913Z ============================= test session starts ============================== 2025-12-04T11:13:50.2165119Z platform linux -- Python 3.10.14, pytest-7.3.2, pluggy-1.6.0 -- /opt/conda/envs/py_3.10/bin/python 2025-12-04T11:13:50.2165186Z cachedir: .pytest_cache 2025-12-04T11:13:50.2165493Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow] 2025-12-04T11:13:50.2165571Z rootdir: /var/lib/jenkins/workspace 2025-12-04T11:13:50.2165639Z configfile: pytest.ini 2025-12-04T11:13:50.2165952Z plugins: hypothesis-6.56.4, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-14.0, subtests-0.13.1, xdist-3.3.1, xdoctest-1.3.0, typeguard-4.3.0 2025-12-04T11:13:50.2166083Z collecting ... collected 58 items / 57 deselected / 1 selected 2025-12-04T11:13:50.2166654Z stepcurrent: skipping 23 already run items. Running only test/inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_1_mid_dim_1_in_features_1024_out_features_64_cuda_bfloat16 2025-12-04T11:13:50.2166725Z Running 1 items in this shard 2025-12-04T11:13:50.2166729Z 2025-12-04T11:13:50.2167457Z inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_1_mid_dim_1_in_features_1024_out_features_64_cuda_bfloat16 [W1204 10:46:04.758295350 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2167461Z 2025-12-04T11:13:50.2167757Z [W1204 10:46:13.804746493 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2167760Z 2025-12-04T11:13:50.2168053Z [W1204 10:46:13.804990749 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2168098Z 2025-12-04T11:13:50.2168450Z [W1204 10:46:13.810795737 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2168455Z 2025-12-04T11:13:50.2168744Z [W1204 10:46:13.811363050 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2168751Z 2025-12-04T11:13:50.2169037Z [W1204 10:46:13.811536354 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2169041Z 2025-12-04T11:13:50.2169327Z [W1204 10:46:13.816926673 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2169330Z 2025-12-04T11:13:50.2169625Z [W1204 10:46:13.817441994 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2169630Z 2025-12-04T11:13:50.2169918Z [W1204 10:46:13.817600208 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2169956Z 2025-12-04T11:13:50.2170046Z ('RERUN', {'yellow': True}) [10.9477s] [100%] 2025-12-04T11:13:50.2170763Z inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_1_mid_dim_1_in_features_1024_out_features_64_cuda_bfloat16 [W1204 10:46:15.001661477 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2170767Z 2025-12-04T11:13:50.2171056Z [W1204 10:46:15.002209440 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2171059Z 2025-12-04T11:13:50.2171342Z [W1204 10:46:15.002352043 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2171347Z 2025-12-04T11:13:50.2171639Z [W1204 10:46:15.005343259 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2171644Z 2025-12-04T11:13:50.2171929Z [W1204 10:46:15.005910291 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2171932Z 2025-12-04T11:13:50.2172217Z [W1204 10:46:15.006046794 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2172224Z 2025-12-04T11:13:50.2172509Z [W1204 10:46:15.010673457 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2172512Z 2025-12-04T11:13:50.2172808Z [W1204 10:46:15.011147147 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2172813Z 2025-12-04T11:13:50.2173104Z [W1204 10:46:15.011282820 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2173109Z 2025-12-04T11:13:50.2173189Z ('RERUN', {'yellow': True}) [0.4169s] [100%] 2025-12-04T11:13:50.2173908Z inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_1_mid_dim_1_in_features_1024_out_features_64_cuda_bfloat16 [W1204 10:46:15.416871891 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2173912Z 2025-12-04T11:13:50.2174201Z [W1204 10:46:15.417418143 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2174204Z 2025-12-04T11:13:50.2174488Z [W1204 10:46:15.417557806 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2174530Z 2025-12-04T11:13:50.2174918Z [W1204 10:46:15.420518891 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2174924Z 2025-12-04T11:13:50.2175214Z [W1204 10:46:15.421095744 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2175218Z 2025-12-04T11:13:50.2175504Z [W1204 10:46:15.421231747 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2175507Z 2025-12-04T11:13:50.2175790Z [W1204 10:46:15.425810588 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2175793Z 2025-12-04T11:13:50.2176079Z [W1204 10:46:15.426275989 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2176083Z 2025-12-04T11:13:50.2176371Z [W1204 10:46:15.426410282 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2176408Z 2025-12-04T11:13:50.2176473Z FAILED [0.4133s] [100%] 2025-12-04T11:13:50.2176476Z 2025-12-04T11:13:50.2176559Z ==================================== RERUNS ==================================== 2025-12-04T11:13:50.2176857Z _ TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_1_in_features_1024_out_features_64_cuda_bfloat16 _ 2025-12-04T11:13:50.2176932Z Traceback (most recent call last): 2025-12-04T11:13:50.2177234Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.2177308Z method(*args, **kwargs) 2025-12-04T11:13:50.2177595Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.2177659Z method(*args, **kwargs) 2025-12-04T11:13:50.2178115Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T11:13:50.2178179Z with policy(): 2025-12-04T11:13:50.2178473Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T11:13:50.2178537Z raise RuntimeError(msg) 2025-12-04T11:13:50.2179333Z RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_1_in_features_1024_out_features_64_cuda_bfloat16! Caching allocator allocated memory was 0 and is now reported as 65536 on device 0. CUDA driver allocated memory was 230686720 and is now 274726912. 2025-12-04T11:13:50.2179337Z 2025-12-04T11:13:50.2179471Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.2179986Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_1_in_features_1024_out_features_64_cuda_bfloat16 2025-12-04T11:13:50.2179991Z 2025-12-04T11:13:50.2180154Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.2180280Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.2180373Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.2180724Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.2180851Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.2180913Z graph_break [] 2025-12-04T11:13:50.2181037Z ----------------------------- Captured stderr call ----------------------------- 2025-12-04T11:13:50.2181845Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/constant_folding.py:256: UserWarning: Unsupported unwinding pattern: Address not in range (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/profiler/unwind/unwind.cpp:219.) 2025-12-04T11:13:50.2181973Z if out == self.unknown_value: 2025-12-04T11:13:50.2182261Z _ TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_1_in_features_1024_out_features_64_cuda_bfloat16 _ 2025-12-04T11:13:50.2182336Z Traceback (most recent call last): 2025-12-04T11:13:50.2182630Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.2182694Z method(*args, **kwargs) 2025-12-04T11:13:50.2182986Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.2183048Z method(*args, **kwargs) 2025-12-04T11:13:50.2183331Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T11:13:50.2183395Z with policy(): 2025-12-04T11:13:50.2183744Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T11:13:50.2183872Z raise RuntimeError(msg) 2025-12-04T11:13:50.2184683Z RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_1_in_features_1024_out_features_64_cuda_bfloat16! Caching allocator allocated memory was 65536 and is now reported as 131072 on device 0. CUDA driver allocated memory was 274726912 and is now 276824064. 2025-12-04T11:13:50.2184687Z 2025-12-04T11:13:50.2184814Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.2185326Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_1_in_features_1024_out_features_64_cuda_bfloat16 2025-12-04T11:13:50.2185332Z 2025-12-04T11:13:50.2185486Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.2185615Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.2185709Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.2186062Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.2186187Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.2186245Z graph_break [] 2025-12-04T11:13:50.2186374Z ----------------------------- Captured stderr call ----------------------------- 2025-12-04T11:13:50.2187057Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/constant_folding.py:256: UserWarning: Unsupported unwinding pattern: Address not in range (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/profiler/unwind/unwind.cpp:219.) 2025-12-04T11:13:50.2187129Z if out == self.unknown_value: 2025-12-04T11:13:50.2187256Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.2187349Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.2187475Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.2187818Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.2187877Z graph_break [] 2025-12-04T11:13:50.2187972Z =================================== FAILURES =================================== 2025-12-04T11:13:50.2188263Z _ TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_1_in_features_1024_out_features_64_cuda_bfloat16 _ 2025-12-04T11:13:50.2188340Z Traceback (most recent call last): 2025-12-04T11:13:50.2188638Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.2188745Z method(*args, **kwargs) 2025-12-04T11:13:50.2189100Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.2189167Z method(*args, **kwargs) 2025-12-04T11:13:50.2189460Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T11:13:50.2189524Z with policy(): 2025-12-04T11:13:50.2189815Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T11:13:50.2189884Z raise RuntimeError(msg) 2025-12-04T11:13:50.2190690Z RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_1_in_features_1024_out_features_64_cuda_bfloat16! Caching allocator allocated memory was 131072 and is now reported as 196608 on device 0. CUDA driver allocated memory was 276824064 and is now 278921216. 2025-12-04T11:13:50.2190695Z 2025-12-04T11:13:50.2190823Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.2191384Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_1_in_features_1024_out_features_64_cuda_bfloat16 2025-12-04T11:13:50.2191388Z 2025-12-04T11:13:50.2191544Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.2191672Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.2191762Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.2192106Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.2192238Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.2192297Z graph_break [] 2025-12-04T11:13:50.2192425Z ----------------------------- Captured stderr call ----------------------------- 2025-12-04T11:13:50.2193106Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/constant_folding.py:256: UserWarning: Unsupported unwinding pattern: Address not in range (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/profiler/unwind/unwind.cpp:219.) 2025-12-04T11:13:50.2193176Z if out == self.unknown_value: 2025-12-04T11:13:50.2193303Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.2193391Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.2193515Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.2193855Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.2193916Z graph_break [] 2025-12-04T11:13:50.2194046Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.2194134Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.2194256Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.2194595Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.2194655Z graph_break [] 2025-12-04T11:13:50.2195141Z - generated xml file: /var/lib/jenkins/workspace/test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-df0458d9683e0406.xml - 2025-12-04T11:13:50.2195243Z =========================== short test summary info ============================ 2025-12-04T11:13:50.2196595Z FAILED [0.4133s] inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_1_mid_dim_1_in_features_1024_out_features_64_cuda_bfloat16 - RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_1_in_features_1024_out_features_64_cuda_bfloat16! Caching allocator allocated memory was 131072 and is now reported as 196608 on device 0. CUDA driver allocated memory was 276824064 and is now 278921216. 2025-12-04T11:13:50.2196633Z 2025-12-04T11:13:50.2196763Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.2197277Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_1_in_features_1024_out_features_64_cuda_bfloat16 2025-12-04T11:13:50.2197283Z 2025-12-04T11:13:50.2197446Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.2197551Z !!!!!!!!!!!!!!!!!!!!!!!!!! stopping after 1 failures !!!!!!!!!!!!!!!!!!!!!!!!!!! 2025-12-04T11:13:50.2197668Z ================== 1 failed, 57 deselected, 2 rerun in 11.80s ================== 2025-12-04T11:13:50.2197730Z Got exit code 1 2025-12-04T11:13:50.2197794Z Retrying single test... 2025-12-04T11:13:50.2198091Z W1204 10:46:22.234000 69848 site-packages/torch/_inductor/utils.py:1703] Not enough SMs to use max_autotune_gemm mode 2025-12-04T11:13:50.2198478Z Test results will be stored in test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-1f5454d7efc5fad5.xml 2025-12-04T11:13:50.2198575Z ============================= test session starts ============================== 2025-12-04T11:13:50.2198779Z platform linux -- Python 3.10.14, pytest-7.3.2, pluggy-1.6.0 -- /opt/conda/envs/py_3.10/bin/python 2025-12-04T11:13:50.2198846Z cachedir: .pytest_cache 2025-12-04T11:13:50.2199160Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow] 2025-12-04T11:13:50.2199242Z rootdir: /var/lib/jenkins/workspace 2025-12-04T11:13:50.2199310Z configfile: pytest.ini 2025-12-04T11:13:50.2199634Z plugins: hypothesis-6.56.4, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-14.0, subtests-0.13.1, xdist-3.3.1, xdoctest-1.3.0, typeguard-4.3.0 2025-12-04T11:13:50.2199770Z collecting ... collected 58 items / 57 deselected / 1 selected 2025-12-04T11:13:50.2200345Z stepcurrent: skipping 23 already run items. Running only test/inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_1_mid_dim_1_in_features_1024_out_features_64_cuda_bfloat16 2025-12-04T11:13:50.2200417Z Running 1 items in this shard 2025-12-04T11:13:50.2200420Z 2025-12-04T11:13:50.2201150Z inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_1_mid_dim_1_in_features_1024_out_features_64_cuda_bfloat16 [W1204 10:46:23.341253909 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2201158Z 2025-12-04T11:13:50.2201457Z [W1204 10:46:32.404615808 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2201462Z 2025-12-04T11:13:50.2201750Z [W1204 10:46:32.404863594 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2201759Z 2025-12-04T11:13:50.2202045Z [W1204 10:46:32.410592440 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2202048Z 2025-12-04T11:13:50.2202333Z [W1204 10:46:32.411131962 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2202337Z 2025-12-04T11:13:50.2202626Z [W1204 10:46:32.411293906 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2202686Z 2025-12-04T11:13:50.2203038Z [W1204 10:46:32.416733326 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2203042Z 2025-12-04T11:13:50.2203335Z [W1204 10:46:32.417257007 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2203339Z 2025-12-04T11:13:50.2203625Z [W1204 10:46:32.417413791 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2203629Z 2025-12-04T11:13:50.2203713Z ('RERUN', {'yellow': True}) [10.9595s] [100%] 2025-12-04T11:13:50.2204449Z inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_1_mid_dim_1_in_features_1024_out_features_64_cuda_bfloat16 [W1204 10:46:33.598571477 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2204455Z 2025-12-04T11:13:50.2204745Z [W1204 10:46:33.599110248 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2204751Z 2025-12-04T11:13:50.2205075Z [W1204 10:46:33.599248462 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2205078Z 2025-12-04T11:13:50.2205360Z [W1204 10:46:33.602215517 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2205364Z 2025-12-04T11:13:50.2205649Z [W1204 10:46:33.602787960 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2205652Z 2025-12-04T11:13:50.2205936Z [W1204 10:46:33.602925832 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2205940Z 2025-12-04T11:13:50.2206233Z [W1204 10:46:33.607468592 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2206236Z 2025-12-04T11:13:50.2206518Z [W1204 10:46:33.607938243 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2206523Z 2025-12-04T11:13:50.2206812Z [W1204 10:46:33.608074636 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2206815Z 2025-12-04T11:13:50.2206896Z ('RERUN', {'yellow': True}) [0.4217s] [100%] 2025-12-04T11:13:50.2207617Z inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_1_mid_dim_1_in_features_1024_out_features_64_cuda_bfloat16 [W1204 10:46:34.013871567 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2207622Z 2025-12-04T11:13:50.2207911Z [W1204 10:46:34.014411709 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2207914Z 2025-12-04T11:13:50.2208202Z [W1204 10:46:34.014550242 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2208209Z 2025-12-04T11:13:50.2208493Z [W1204 10:46:34.017518518 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2208497Z 2025-12-04T11:13:50.2208778Z [W1204 10:46:34.018082051 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2208782Z 2025-12-04T11:13:50.2209069Z [W1204 10:46:34.018219614 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2209072Z 2025-12-04T11:13:50.2209395Z [W1204 10:46:34.022760564 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2209463Z 2025-12-04T11:13:50.2209755Z [W1204 10:46:34.023233574 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2209760Z 2025-12-04T11:13:50.2210046Z [W1204 10:46:34.023369137 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2210049Z 2025-12-04T11:13:50.2210115Z FAILED [0.4127s] [100%] 2025-12-04T11:13:50.2210119Z 2025-12-04T11:13:50.2210203Z ==================================== RERUNS ==================================== 2025-12-04T11:13:50.2210494Z _ TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_1_in_features_1024_out_features_64_cuda_bfloat16 _ 2025-12-04T11:13:50.2210574Z Traceback (most recent call last): 2025-12-04T11:13:50.2210885Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.2210954Z method(*args, **kwargs) 2025-12-04T11:13:50.2211246Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.2211349Z method(*args, **kwargs) 2025-12-04T11:13:50.2211639Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T11:13:50.2211699Z with policy(): 2025-12-04T11:13:50.2211990Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T11:13:50.2212060Z raise RuntimeError(msg) 2025-12-04T11:13:50.2212855Z RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_1_in_features_1024_out_features_64_cuda_bfloat16! Caching allocator allocated memory was 0 and is now reported as 65536 on device 0. CUDA driver allocated memory was 230686720 and is now 274726912. 2025-12-04T11:13:50.2212863Z 2025-12-04T11:13:50.2212994Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.2213507Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_1_in_features_1024_out_features_64_cuda_bfloat16 2025-12-04T11:13:50.2213511Z 2025-12-04T11:13:50.2213672Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.2213802Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.2213898Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.2214252Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.2214383Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.2214442Z graph_break [] 2025-12-04T11:13:50.2214574Z ----------------------------- Captured stderr call ----------------------------- 2025-12-04T11:13:50.2215266Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/constant_folding.py:256: UserWarning: Unsupported unwinding pattern: Address not in range (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/profiler/unwind/unwind.cpp:219.) 2025-12-04T11:13:50.2215341Z if out == self.unknown_value: 2025-12-04T11:13:50.2215627Z _ TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_1_in_features_1024_out_features_64_cuda_bfloat16 _ 2025-12-04T11:13:50.2215700Z Traceback (most recent call last): 2025-12-04T11:13:50.2216000Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.2216065Z method(*args, **kwargs) 2025-12-04T11:13:50.2216399Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.2216539Z method(*args, **kwargs) 2025-12-04T11:13:50.2216836Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T11:13:50.2216900Z with policy(): 2025-12-04T11:13:50.2217189Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T11:13:50.2217255Z raise RuntimeError(msg) 2025-12-04T11:13:50.2218064Z RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_1_in_features_1024_out_features_64_cuda_bfloat16! Caching allocator allocated memory was 65536 and is now reported as 131072 on device 0. CUDA driver allocated memory was 274726912 and is now 276824064. 2025-12-04T11:13:50.2218070Z 2025-12-04T11:13:50.2218195Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.2218718Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_1_in_features_1024_out_features_64_cuda_bfloat16 2025-12-04T11:13:50.2218755Z 2025-12-04T11:13:50.2218910Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.2219038Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.2219131Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.2219483Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.2219613Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.2219671Z graph_break [] 2025-12-04T11:13:50.2219800Z ----------------------------- Captured stderr call ----------------------------- 2025-12-04T11:13:50.2220499Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/constant_folding.py:256: UserWarning: Unsupported unwinding pattern: Address not in range (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/profiler/unwind/unwind.cpp:219.) 2025-12-04T11:13:50.2220570Z if out == self.unknown_value: 2025-12-04T11:13:50.2220695Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.2220783Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.2220906Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.2221251Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.2221310Z graph_break [] 2025-12-04T11:13:50.2221400Z =================================== FAILURES =================================== 2025-12-04T11:13:50.2221692Z _ TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_1_in_features_1024_out_features_64_cuda_bfloat16 _ 2025-12-04T11:13:50.2221767Z Traceback (most recent call last): 2025-12-04T11:13:50.2222072Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.2222137Z method(*args, **kwargs) 2025-12-04T11:13:50.2222435Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.2222497Z method(*args, **kwargs) 2025-12-04T11:13:50.2222784Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T11:13:50.2222847Z with policy(): 2025-12-04T11:13:50.2223139Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T11:13:50.2223251Z raise RuntimeError(msg) 2025-12-04T11:13:50.2224183Z RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_1_in_features_1024_out_features_64_cuda_bfloat16! Caching allocator allocated memory was 131072 and is now reported as 196608 on device 0. CUDA driver allocated memory was 276824064 and is now 278921216. 2025-12-04T11:13:50.2224190Z 2025-12-04T11:13:50.2224315Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.2224834Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_1_in_features_1024_out_features_64_cuda_bfloat16 2025-12-04T11:13:50.2224838Z 2025-12-04T11:13:50.2225007Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.2225138Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.2225232Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.2225578Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.2225761Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.2225818Z graph_break [] 2025-12-04T11:13:50.2225941Z ----------------------------- Captured stderr call ----------------------------- 2025-12-04T11:13:50.2226624Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/constant_folding.py:256: UserWarning: Unsupported unwinding pattern: Address not in range (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/profiler/unwind/unwind.cpp:219.) 2025-12-04T11:13:50.2226693Z if out == self.unknown_value: 2025-12-04T11:13:50.2226820Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.2226911Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.2227034Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.2227384Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.2227445Z graph_break [] 2025-12-04T11:13:50.2227571Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.2227659Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.2227780Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.2228121Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.2228178Z graph_break [] 2025-12-04T11:13:50.2228664Z - generated xml file: /var/lib/jenkins/workspace/test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-1f5454d7efc5fad5.xml - 2025-12-04T11:13:50.2228769Z =========================== short test summary info ============================ 2025-12-04T11:13:50.2230056Z FAILED [0.4127s] inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_1_mid_dim_1_in_features_1024_out_features_64_cuda_bfloat16 - RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_1_in_features_1024_out_features_64_cuda_bfloat16! Caching allocator allocated memory was 131072 and is now reported as 196608 on device 0. CUDA driver allocated memory was 276824064 and is now 278921216. 2025-12-04T11:13:50.2230063Z 2025-12-04T11:13:50.2230187Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.2230699Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_1_in_features_1024_out_features_64_cuda_bfloat16 2025-12-04T11:13:50.2230744Z 2025-12-04T11:13:50.2230987Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.2231098Z !!!!!!!!!!!!!!!!!!!!!!!!!! stopping after 1 failures !!!!!!!!!!!!!!!!!!!!!!!!!!! 2025-12-04T11:13:50.2231218Z ================== 1 failed, 57 deselected, 2 rerun in 11.82s ================== 2025-12-04T11:13:50.2231278Z Got exit code 1 2025-12-04T11:13:50.2231744Z FAILED CONSISTENTLY: test/inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_1_mid_dim_1_in_features_1024_out_features_64_cuda_bfloat16 2025-12-04T11:13:50.2231988Z Test failed consistently, continuing with the rest of the tests due to continue-through-error being set 2025-12-04T11:13:50.2232247Z W1204 10:46:40.894000 70041 site-packages/torch/_inductor/utils.py:1703] Not enough SMs to use max_autotune_gemm mode 2025-12-04T11:13:50.2232633Z Test results will be stored in test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-0e640f0818a9f631.xml 2025-12-04T11:13:50.2232734Z ============================= test session starts ============================== 2025-12-04T11:13:50.2232977Z platform linux -- Python 3.10.14, pytest-7.3.2, pluggy-1.6.0 -- /opt/conda/envs/py_3.10/bin/python 2025-12-04T11:13:50.2233048Z cachedir: .pytest_cache 2025-12-04T11:13:50.2233351Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow] 2025-12-04T11:13:50.2233427Z rootdir: /var/lib/jenkins/workspace 2025-12-04T11:13:50.2233497Z configfile: pytest.ini 2025-12-04T11:13:50.2233815Z plugins: hypothesis-6.56.4, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-14.0, subtests-0.13.1, xdist-3.3.1, xdoctest-1.3.0, typeguard-4.3.0 2025-12-04T11:13:50.2233947Z collecting ... collected 58 items / 24 deselected / 34 selected 2025-12-04T11:13:50.2234035Z stepcurrent: skipping 24 already run items. 2025-12-04T11:13:50.2234106Z Running 34 items in this shard 2025-12-04T11:13:50.2234109Z 2025-12-04T11:13:50.2234607Z inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_1_mid_dim_1_in_features_1024_out_features_65_cuda_bfloat16 ('RERUN', {'yellow': True}) [1.9818s] [ 2%] 2025-12-04T11:13:50.2235089Z inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_1_mid_dim_1_in_features_1024_out_features_65_cuda_bfloat16 ('RERUN', {'yellow': True}) [0.5515s] [ 2%] 2025-12-04T11:13:50.2235535Z inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_1_mid_dim_1_in_features_1024_out_features_65_cuda_bfloat16 FAILED [0.5391s] [ 2%] 2025-12-04T11:13:50.2235539Z 2025-12-04T11:13:50.2235619Z ==================================== RERUNS ==================================== 2025-12-04T11:13:50.2235907Z _ TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_1_in_features_1024_out_features_65_cuda_bfloat16 _ 2025-12-04T11:13:50.2235987Z Traceback (most recent call last): 2025-12-04T11:13:50.2236304Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.2236374Z method(*args, **kwargs) 2025-12-04T11:13:50.2236666Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.2236728Z method(*args, **kwargs) 2025-12-04T11:13:50.2237019Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T11:13:50.2237080Z with policy(): 2025-12-04T11:13:50.2237373Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T11:13:50.2237442Z raise RuntimeError(msg) 2025-12-04T11:13:50.2238309Z RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_1_in_features_1024_out_features_65_cuda_bfloat16! Caching allocator allocated memory was 0 and is now reported as 66560 on device 0. CUDA driver allocated memory was 230686720 and is now 274726912. 2025-12-04T11:13:50.2238348Z 2025-12-04T11:13:50.2238483Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.2238997Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_1_in_features_1024_out_features_65_cuda_bfloat16 2025-12-04T11:13:50.2239001Z 2025-12-04T11:13:50.2239160Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.2239287Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.2239380Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.2239944Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('benchmarking.InductorBenchmarker.benchmark_gpu', 2), ('pad_mm_bench', 1), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.2240118Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.2240184Z graph_break [] 2025-12-04T11:13:50.2240469Z _ TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_1_in_features_1024_out_features_65_cuda_bfloat16 _ 2025-12-04T11:13:50.2240543Z Traceback (most recent call last): 2025-12-04T11:13:50.2240841Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.2240908Z method(*args, **kwargs) 2025-12-04T11:13:50.2241196Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.2241266Z method(*args, **kwargs) 2025-12-04T11:13:50.2241555Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T11:13:50.2241617Z with policy(): 2025-12-04T11:13:50.2241910Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T11:13:50.2241979Z raise RuntimeError(msg) 2025-12-04T11:13:50.2242785Z RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_1_in_features_1024_out_features_65_cuda_bfloat16! Caching allocator allocated memory was 66560 and is now reported as 133120 on device 0. CUDA driver allocated memory was 274726912 and is now 276824064. 2025-12-04T11:13:50.2242789Z 2025-12-04T11:13:50.2242912Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.2243425Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_1_in_features_1024_out_features_65_cuda_bfloat16 2025-12-04T11:13:50.2243430Z 2025-12-04T11:13:50.2243586Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.2243713Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.2243809Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.2244351Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('benchmarking.InductorBenchmarker.benchmark_gpu', 2), ('pad_mm_bench', 1), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.2244489Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.2244550Z graph_break [] 2025-12-04T11:13:50.2244675Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.2244767Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.2244932Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.2245535Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('benchmarking.InductorBenchmarker.benchmark_gpu', 2), ('pad_mm_bench', 1), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.2245597Z graph_break [] 2025-12-04T11:13:50.2245680Z =================================== FAILURES =================================== 2025-12-04T11:13:50.2245969Z _ TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_1_in_features_1024_out_features_65_cuda_bfloat16 _ 2025-12-04T11:13:50.2246039Z Traceback (most recent call last): 2025-12-04T11:13:50.2246336Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.2246399Z method(*args, **kwargs) 2025-12-04T11:13:50.2246686Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.2246754Z method(*args, **kwargs) 2025-12-04T11:13:50.2247041Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T11:13:50.2247138Z with policy(): 2025-12-04T11:13:50.2247434Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T11:13:50.2247500Z raise RuntimeError(msg) 2025-12-04T11:13:50.2248308Z RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_1_in_features_1024_out_features_65_cuda_bfloat16! Caching allocator allocated memory was 133120 and is now reported as 199680 on device 0. CUDA driver allocated memory was 276824064 and is now 278921216. 2025-12-04T11:13:50.2248311Z 2025-12-04T11:13:50.2248434Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.2248950Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_1_in_features_1024_out_features_65_cuda_bfloat16 2025-12-04T11:13:50.2248962Z 2025-12-04T11:13:50.2249122Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.2249247Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.2249340Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.2249886Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('benchmarking.InductorBenchmarker.benchmark_gpu', 2), ('pad_mm_bench', 1), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.2250013Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.2250072Z graph_break [] 2025-12-04T11:13:50.2250196Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.2250290Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.2250410Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.2250948Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('benchmarking.InductorBenchmarker.benchmark_gpu', 2), ('pad_mm_bench', 1), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.2251009Z graph_break [] 2025-12-04T11:13:50.2251130Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.2251222Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.2251342Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.2251877Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('benchmarking.InductorBenchmarker.benchmark_gpu', 2), ('pad_mm_bench', 1), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.2252064Z graph_break [] 2025-12-04T11:13:50.2252555Z - generated xml file: /var/lib/jenkins/workspace/test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-0e640f0818a9f631.xml - 2025-12-04T11:13:50.2252661Z =========================== short test summary info ============================ 2025-12-04T11:13:50.2253938Z FAILED [0.5391s] inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_1_mid_dim_1_in_features_1024_out_features_65_cuda_bfloat16 - RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_1_in_features_1024_out_features_65_cuda_bfloat16! Caching allocator allocated memory was 133120 and is now reported as 199680 on device 0. CUDA driver allocated memory was 276824064 and is now 278921216. 2025-12-04T11:13:50.2253944Z 2025-12-04T11:13:50.2254069Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.2254585Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_1_in_features_1024_out_features_65_cuda_bfloat16 2025-12-04T11:13:50.2254624Z 2025-12-04T11:13:50.2254778Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.2254888Z !!!!!!!!!!!!!!!!!!!!!!!!!! stopping after 1 failures !!!!!!!!!!!!!!!!!!!!!!!!!!! 2025-12-04T11:13:50.2255004Z ================== 1 failed, 24 deselected, 2 rerun in 3.10s =================== 2025-12-04T11:13:50.2255065Z Got exit code 1 2025-12-04T11:13:50.2255131Z Retrying single test... 2025-12-04T11:13:50.2255394Z W1204 10:46:51.038000 70230 site-packages/torch/_inductor/utils.py:1703] Not enough SMs to use max_autotune_gemm mode 2025-12-04T11:13:50.2255785Z Test results will be stored in test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-e912b3c1622dc5a6.xml 2025-12-04T11:13:50.2255882Z ============================= test session starts ============================== 2025-12-04T11:13:50.2256097Z platform linux -- Python 3.10.14, pytest-7.3.2, pluggy-1.6.0 -- /opt/conda/envs/py_3.10/bin/python 2025-12-04T11:13:50.2256169Z cachedir: .pytest_cache 2025-12-04T11:13:50.2256473Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow] 2025-12-04T11:13:50.2256554Z rootdir: /var/lib/jenkins/workspace 2025-12-04T11:13:50.2256620Z configfile: pytest.ini 2025-12-04T11:13:50.2256932Z plugins: hypothesis-6.56.4, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-14.0, subtests-0.13.1, xdist-3.3.1, xdoctest-1.3.0, typeguard-4.3.0 2025-12-04T11:13:50.2257065Z collecting ... collected 58 items / 57 deselected / 1 selected 2025-12-04T11:13:50.2257636Z stepcurrent: skipping 24 already run items. Running only test/inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_1_mid_dim_1_in_features_1024_out_features_65_cuda_bfloat16 2025-12-04T11:13:50.2257712Z Running 1 items in this shard 2025-12-04T11:13:50.2257717Z 2025-12-04T11:13:50.2258440Z inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_1_mid_dim_1_in_features_1024_out_features_65_cuda_bfloat16 [W1204 10:46:52.677394989 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2258445Z 2025-12-04T11:13:50.2258744Z [W1204 10:47:01.571474284 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2258750Z 2025-12-04T11:13:50.2259036Z [W1204 10:47:01.571720009 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2259081Z 2025-12-04T11:13:50.2259434Z [W1204 10:47:01.577580099 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2259438Z 2025-12-04T11:13:50.2259733Z [W1204 10:47:01.578128841 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2259737Z 2025-12-04T11:13:50.2260024Z [W1204 10:47:01.578290934 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2260027Z 2025-12-04T11:13:50.2260316Z [W1204 10:47:01.583613972 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2260320Z 2025-12-04T11:13:50.2260604Z [W1204 10:47:01.584154724 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2260608Z 2025-12-04T11:13:50.2260899Z [W1204 10:47:01.584324547 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2260902Z 2025-12-04T11:13:50.2260983Z ('RERUN', {'yellow': True}) [10.8718s] [100%] 2025-12-04T11:13:50.2261830Z inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_1_mid_dim_1_in_features_1024_out_features_65_cuda_bfloat16 [W1204 10:47:02.390134978 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2261834Z 2025-12-04T11:13:50.2262120Z [W1204 10:47:02.390655379 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2262124Z 2025-12-04T11:13:50.2262409Z [W1204 10:47:02.390791372 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2262418Z 2025-12-04T11:13:50.2262707Z [W1204 10:47:02.393732877 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2262710Z 2025-12-04T11:13:50.2263000Z [W1204 10:47:02.394195887 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2263005Z 2025-12-04T11:13:50.2263297Z [W1204 10:47:02.394333950 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2263300Z 2025-12-04T11:13:50.2263627Z [W1204 10:47:02.398994422 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2263631Z 2025-12-04T11:13:50.2263917Z [W1204 10:47:02.399465173 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2263921Z 2025-12-04T11:13:50.2264208Z [W1204 10:47:02.399600976 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2264214Z 2025-12-04T11:13:50.2264295Z ('RERUN', {'yellow': True}) [0.5036s] [100%] 2025-12-04T11:13:50.2265013Z inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_1_mid_dim_1_in_features_1024_out_features_65_cuda_bfloat16 [W1204 10:47:02.890928274 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2265017Z 2025-12-04T11:13:50.2265308Z [W1204 10:47:02.891450585 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2265311Z 2025-12-04T11:13:50.2265593Z [W1204 10:47:02.891588288 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2265596Z 2025-12-04T11:13:50.2265922Z [W1204 10:47:02.894519663 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2265989Z 2025-12-04T11:13:50.2266280Z [W1204 10:47:02.894978114 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2266285Z 2025-12-04T11:13:50.2266568Z [W1204 10:47:02.895112937 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2266571Z 2025-12-04T11:13:50.2266858Z [W1204 10:47:02.899641877 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2266862Z 2025-12-04T11:13:50.2267160Z [W1204 10:47:02.900166548 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2267164Z 2025-12-04T11:13:50.2267459Z [W1204 10:47:02.900306031 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2267464Z 2025-12-04T11:13:50.2267530Z FAILED [0.4999s] [100%] 2025-12-04T11:13:50.2267533Z 2025-12-04T11:13:50.2267656Z ==================================== RERUNS ==================================== 2025-12-04T11:13:50.2267946Z _ TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_1_in_features_1024_out_features_65_cuda_bfloat16 _ 2025-12-04T11:13:50.2268021Z Traceback (most recent call last): 2025-12-04T11:13:50.2268331Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.2268396Z method(*args, **kwargs) 2025-12-04T11:13:50.2268686Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.2268751Z method(*args, **kwargs) 2025-12-04T11:13:50.2269039Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T11:13:50.2269103Z with policy(): 2025-12-04T11:13:50.2269397Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T11:13:50.2269468Z raise RuntimeError(msg) 2025-12-04T11:13:50.2270268Z RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_1_in_features_1024_out_features_65_cuda_bfloat16! Caching allocator allocated memory was 0 and is now reported as 66560 on device 0. CUDA driver allocated memory was 230686720 and is now 274726912. 2025-12-04T11:13:50.2270272Z 2025-12-04T11:13:50.2270396Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.2270917Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_1_in_features_1024_out_features_65_cuda_bfloat16 2025-12-04T11:13:50.2270922Z 2025-12-04T11:13:50.2271080Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.2271209Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.2271322Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.2271870Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('benchmarking.InductorBenchmarker.benchmark_gpu', 2), ('pad_mm_bench', 1), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.2271999Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.2272060Z graph_break [] 2025-12-04T11:13:50.2272184Z ----------------------------- Captured stderr call ----------------------------- 2025-12-04T11:13:50.2272944Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/constant_folding.py:256: UserWarning: Unsupported unwinding pattern: Address not in range (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/profiler/unwind/unwind.cpp:219.) 2025-12-04T11:13:50.2273047Z if out == self.unknown_value: 2025-12-04T11:13:50.2273341Z _ TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_1_in_features_1024_out_features_65_cuda_bfloat16 _ 2025-12-04T11:13:50.2273415Z Traceback (most recent call last): 2025-12-04T11:13:50.2273707Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.2273773Z method(*args, **kwargs) 2025-12-04T11:13:50.2274058Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.2274123Z method(*args, **kwargs) 2025-12-04T11:13:50.2274411Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T11:13:50.2274473Z with policy(): 2025-12-04T11:13:50.2274768Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T11:13:50.2274871Z raise RuntimeError(msg) 2025-12-04T11:13:50.2275676Z RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_1_in_features_1024_out_features_65_cuda_bfloat16! Caching allocator allocated memory was 66560 and is now reported as 133120 on device 0. CUDA driver allocated memory was 274726912 and is now 276824064. 2025-12-04T11:13:50.2275686Z 2025-12-04T11:13:50.2275810Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.2276329Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_1_in_features_1024_out_features_65_cuda_bfloat16 2025-12-04T11:13:50.2276334Z 2025-12-04T11:13:50.2276493Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.2276620Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.2276721Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.2277260Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('benchmarking.InductorBenchmarker.benchmark_gpu', 2), ('pad_mm_bench', 1), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.2277387Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.2277449Z graph_break [] 2025-12-04T11:13:50.2277583Z ----------------------------- Captured stderr call ----------------------------- 2025-12-04T11:13:50.2278454Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/constant_folding.py:256: UserWarning: Unsupported unwinding pattern: Address not in range (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/profiler/unwind/unwind.cpp:219.) 2025-12-04T11:13:50.2278528Z if out == self.unknown_value: 2025-12-04T11:13:50.2278654Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.2278752Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.2278874Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.2279418Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('benchmarking.InductorBenchmarker.benchmark_gpu', 2), ('pad_mm_bench', 1), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.2279480Z graph_break [] 2025-12-04T11:13:50.2279563Z =================================== FAILURES =================================== 2025-12-04T11:13:50.2279854Z _ TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_1_in_features_1024_out_features_65_cuda_bfloat16 _ 2025-12-04T11:13:50.2279996Z Traceback (most recent call last): 2025-12-04T11:13:50.2280411Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.2280480Z method(*args, **kwargs) 2025-12-04T11:13:50.2280782Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.2280848Z method(*args, **kwargs) 2025-12-04T11:13:50.2281138Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T11:13:50.2281198Z with policy(): 2025-12-04T11:13:50.2281505Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T11:13:50.2281573Z raise RuntimeError(msg) 2025-12-04T11:13:50.2282388Z RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_1_in_features_1024_out_features_65_cuda_bfloat16! Caching allocator allocated memory was 133120 and is now reported as 199680 on device 0. CUDA driver allocated memory was 276824064 and is now 278921216. 2025-12-04T11:13:50.2282447Z 2025-12-04T11:13:50.2282574Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.2283088Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_1_in_features_1024_out_features_65_cuda_bfloat16 2025-12-04T11:13:50.2283092Z 2025-12-04T11:13:50.2283252Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.2283377Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.2283470Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.2284017Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('benchmarking.InductorBenchmarker.benchmark_gpu', 2), ('pad_mm_bench', 1), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.2284143Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.2284206Z graph_break [] 2025-12-04T11:13:50.2284328Z ----------------------------- Captured stderr call ----------------------------- 2025-12-04T11:13:50.2285016Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/constant_folding.py:256: UserWarning: Unsupported unwinding pattern: Address not in range (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/profiler/unwind/unwind.cpp:219.) 2025-12-04T11:13:50.2285086Z if out == self.unknown_value: 2025-12-04T11:13:50.2285212Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.2285313Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.2285438Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.2285977Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('benchmarking.InductorBenchmarker.benchmark_gpu', 2), ('pad_mm_bench', 1), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.2286040Z graph_break [] 2025-12-04T11:13:50.2286164Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.2286259Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.2286382Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.2286913Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('benchmarking.InductorBenchmarker.benchmark_gpu', 2), ('pad_mm_bench', 1), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.2286973Z graph_break [] 2025-12-04T11:13:50.2287459Z - generated xml file: /var/lib/jenkins/workspace/test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-e912b3c1622dc5a6.xml - 2025-12-04T11:13:50.2287672Z =========================== short test summary info ============================ 2025-12-04T11:13:50.2288958Z FAILED [0.4999s] inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_1_mid_dim_1_in_features_1024_out_features_65_cuda_bfloat16 - RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_1_in_features_1024_out_features_65_cuda_bfloat16! Caching allocator allocated memory was 133120 and is now reported as 199680 on device 0. CUDA driver allocated memory was 276824064 and is now 278921216. 2025-12-04T11:13:50.2288962Z 2025-12-04T11:13:50.2289088Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.2289610Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_1_in_features_1024_out_features_65_cuda_bfloat16 2025-12-04T11:13:50.2289614Z 2025-12-04T11:13:50.2289771Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.2289913Z !!!!!!!!!!!!!!!!!!!!!!!!!! stopping after 1 failures !!!!!!!!!!!!!!!!!!!!!!!!!!! 2025-12-04T11:13:50.2290028Z ================== 1 failed, 57 deselected, 2 rerun in 11.90s ================== 2025-12-04T11:13:50.2290100Z Got exit code 1 2025-12-04T11:13:50.2290167Z Retrying single test... 2025-12-04T11:13:50.2290432Z W1204 10:47:09.711000 70424 site-packages/torch/_inductor/utils.py:1703] Not enough SMs to use max_autotune_gemm mode 2025-12-04T11:13:50.2290818Z Test results will be stored in test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-9bf082709afd4107.xml 2025-12-04T11:13:50.2290911Z ============================= test session starts ============================== 2025-12-04T11:13:50.2291121Z platform linux -- Python 3.10.14, pytest-7.3.2, pluggy-1.6.0 -- /opt/conda/envs/py_3.10/bin/python 2025-12-04T11:13:50.2291192Z cachedir: .pytest_cache 2025-12-04T11:13:50.2291497Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow] 2025-12-04T11:13:50.2291580Z rootdir: /var/lib/jenkins/workspace 2025-12-04T11:13:50.2291645Z configfile: pytest.ini 2025-12-04T11:13:50.2291957Z plugins: hypothesis-6.56.4, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-14.0, subtests-0.13.1, xdist-3.3.1, xdoctest-1.3.0, typeguard-4.3.0 2025-12-04T11:13:50.2292087Z collecting ... collected 58 items / 57 deselected / 1 selected 2025-12-04T11:13:50.2292653Z stepcurrent: skipping 24 already run items. Running only test/inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_1_mid_dim_1_in_features_1024_out_features_65_cuda_bfloat16 2025-12-04T11:13:50.2292729Z Running 1 items in this shard 2025-12-04T11:13:50.2292734Z 2025-12-04T11:13:50.2293460Z inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_1_mid_dim_1_in_features_1024_out_features_65_cuda_bfloat16 [W1204 10:47:11.366288138 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2293465Z 2025-12-04T11:13:50.2293767Z [W1204 10:47:20.420259555 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2293770Z 2025-12-04T11:13:50.2294057Z [W1204 10:47:20.420498450 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2294060Z 2025-12-04T11:13:50.2294346Z [W1204 10:47:20.426244407 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2294352Z 2025-12-04T11:13:50.2294682Z [W1204 10:47:20.426790019 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2294685Z 2025-12-04T11:13:50.2295038Z [W1204 10:47:20.426958263 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2295044Z 2025-12-04T11:13:50.2295333Z [W1204 10:47:20.432275161 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2295337Z 2025-12-04T11:13:50.2295632Z [W1204 10:47:20.432818953 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2295635Z 2025-12-04T11:13:50.2295926Z [W1204 10:47:20.432977196 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2295930Z 2025-12-04T11:13:50.2296011Z ('RERUN', {'yellow': True}) [11.0534s] [100%] 2025-12-04T11:13:50.2296737Z inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_1_mid_dim_1_in_features_1024_out_features_65_cuda_bfloat16 [W1204 10:47:21.247178588 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2296776Z 2025-12-04T11:13:50.2297063Z [W1204 10:47:21.247695749 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2297067Z 2025-12-04T11:13:50.2297355Z [W1204 10:47:21.247835962 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2297359Z 2025-12-04T11:13:50.2297643Z [W1204 10:47:21.250780868 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2297646Z 2025-12-04T11:13:50.2297931Z [W1204 10:47:21.251246028 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2297939Z 2025-12-04T11:13:50.2298226Z [W1204 10:47:21.251384591 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2298231Z 2025-12-04T11:13:50.2298513Z [W1204 10:47:21.255950222 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2298516Z 2025-12-04T11:13:50.2298805Z [W1204 10:47:21.256428013 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2298808Z 2025-12-04T11:13:50.2299093Z [W1204 10:47:21.256564496 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2299097Z 2025-12-04T11:13:50.2299178Z ('RERUN', {'yellow': True}) [0.5048s] [100%] 2025-12-04T11:13:50.2299901Z inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_1_mid_dim_1_in_features_1024_out_features_65_cuda_bfloat16 [W1204 10:47:21.748955731 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2299906Z 2025-12-04T11:13:50.2300194Z [W1204 10:47:21.749484812 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2300197Z 2025-12-04T11:13:50.2300482Z [W1204 10:47:21.749626445 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2300486Z 2025-12-04T11:13:50.2300781Z [W1204 10:47:21.752541390 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2300789Z 2025-12-04T11:13:50.2301075Z [W1204 10:47:21.753014571 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2301117Z 2025-12-04T11:13:50.2301467Z [W1204 10:47:21.753153963 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2301473Z 2025-12-04T11:13:50.2301762Z [W1204 10:47:21.757758295 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2301766Z 2025-12-04T11:13:50.2302049Z [W1204 10:47:21.758234806 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2302053Z 2025-12-04T11:13:50.2302343Z [W1204 10:47:21.758377319 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2302346Z 2025-12-04T11:13:50.2302409Z FAILED [0.5030s] [100%] 2025-12-04T11:13:50.2302412Z 2025-12-04T11:13:50.2302500Z ==================================== RERUNS ==================================== 2025-12-04T11:13:50.2302792Z _ TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_1_in_features_1024_out_features_65_cuda_bfloat16 _ 2025-12-04T11:13:50.2302902Z Traceback (most recent call last): 2025-12-04T11:13:50.2303209Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.2303273Z method(*args, **kwargs) 2025-12-04T11:13:50.2303608Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.2303675Z method(*args, **kwargs) 2025-12-04T11:13:50.2303963Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T11:13:50.2304028Z with policy(): 2025-12-04T11:13:50.2304319Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T11:13:50.2304387Z raise RuntimeError(msg) 2025-12-04T11:13:50.2305189Z RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_1_in_features_1024_out_features_65_cuda_bfloat16! Caching allocator allocated memory was 0 and is now reported as 66560 on device 0. CUDA driver allocated memory was 230686720 and is now 274726912. 2025-12-04T11:13:50.2305194Z 2025-12-04T11:13:50.2305322Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.2305844Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_1_in_features_1024_out_features_65_cuda_bfloat16 2025-12-04T11:13:50.2305848Z 2025-12-04T11:13:50.2306006Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.2306139Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.2306243Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.2306793Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('benchmarking.InductorBenchmarker.benchmark_gpu', 2), ('pad_mm_bench', 1), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.2306927Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.2306985Z graph_break [] 2025-12-04T11:13:50.2307109Z ----------------------------- Captured stderr call ----------------------------- 2025-12-04T11:13:50.2307797Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/constant_folding.py:256: UserWarning: Unsupported unwinding pattern: Address not in range (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/profiler/unwind/unwind.cpp:219.) 2025-12-04T11:13:50.2307867Z if out == self.unknown_value: 2025-12-04T11:13:50.2308237Z _ TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_1_in_features_1024_out_features_65_cuda_bfloat16 _ 2025-12-04T11:13:50.2308467Z Traceback (most recent call last): 2025-12-04T11:13:50.2308948Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.2309029Z method(*args, **kwargs) 2025-12-04T11:13:50.2309436Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.2309529Z method(*args, **kwargs) 2025-12-04T11:13:50.2309839Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T11:13:50.2309902Z with policy(): 2025-12-04T11:13:50.2310293Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T11:13:50.2310397Z raise RuntimeError(msg) 2025-12-04T11:13:50.2311227Z RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_1_in_features_1024_out_features_65_cuda_bfloat16! Caching allocator allocated memory was 66560 and is now reported as 133120 on device 0. CUDA driver allocated memory was 274726912 and is now 276824064. 2025-12-04T11:13:50.2311292Z 2025-12-04T11:13:50.2311423Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.2311942Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_1_in_features_1024_out_features_65_cuda_bfloat16 2025-12-04T11:13:50.2311946Z 2025-12-04T11:13:50.2312113Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.2312242Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.2312340Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.2312891Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('benchmarking.InductorBenchmarker.benchmark_gpu', 2), ('pad_mm_bench', 1), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.2313019Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.2313083Z graph_break [] 2025-12-04T11:13:50.2313208Z ----------------------------- Captured stderr call ----------------------------- 2025-12-04T11:13:50.2313915Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/constant_folding.py:256: UserWarning: Unsupported unwinding pattern: Address not in range (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/profiler/unwind/unwind.cpp:219.) 2025-12-04T11:13:50.2317194Z if out == self.unknown_value: 2025-12-04T11:13:50.2317367Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.2317489Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.2317638Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.2318212Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('benchmarking.InductorBenchmarker.benchmark_gpu', 2), ('pad_mm_bench', 1), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.2318280Z graph_break [] 2025-12-04T11:13:50.2318376Z =================================== FAILURES =================================== 2025-12-04T11:13:50.2318679Z _ TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_1_in_features_1024_out_features_65_cuda_bfloat16 _ 2025-12-04T11:13:50.2318761Z Traceback (most recent call last): 2025-12-04T11:13:50.2319085Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.2319155Z method(*args, **kwargs) 2025-12-04T11:13:50.2319545Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.2319689Z method(*args, **kwargs) 2025-12-04T11:13:50.2319996Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T11:13:50.2320057Z with policy(): 2025-12-04T11:13:50.2320357Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T11:13:50.2320426Z raise RuntimeError(msg) 2025-12-04T11:13:50.2321247Z RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_1_in_features_1024_out_features_65_cuda_bfloat16! Caching allocator allocated memory was 133120 and is now reported as 199680 on device 0. CUDA driver allocated memory was 276824064 and is now 278921216. 2025-12-04T11:13:50.2321254Z 2025-12-04T11:13:50.2321391Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.2321914Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_1_in_features_1024_out_features_65_cuda_bfloat16 2025-12-04T11:13:50.2321954Z 2025-12-04T11:13:50.2322118Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.2322259Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.2322354Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.2322906Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('benchmarking.InductorBenchmarker.benchmark_gpu', 2), ('pad_mm_bench', 1), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.2323035Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.2323097Z graph_break [] 2025-12-04T11:13:50.2323239Z ----------------------------- Captured stderr call ----------------------------- 2025-12-04T11:13:50.2323936Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/constant_folding.py:256: UserWarning: Unsupported unwinding pattern: Address not in range (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/profiler/unwind/unwind.cpp:219.) 2025-12-04T11:13:50.2324012Z if out == self.unknown_value: 2025-12-04T11:13:50.2324134Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.2324226Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.2324353Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.2324890Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('benchmarking.InductorBenchmarker.benchmark_gpu', 2), ('pad_mm_bench', 1), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.2324953Z graph_break [] 2025-12-04T11:13:50.2325085Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.2325178Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.2325313Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.2325848Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('benchmarking.InductorBenchmarker.benchmark_gpu', 2), ('pad_mm_bench', 1), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.2325908Z graph_break [] 2025-12-04T11:13:50.2326402Z - generated xml file: /var/lib/jenkins/workspace/test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-9bf082709afd4107.xml - 2025-12-04T11:13:50.2326506Z =========================== short test summary info ============================ 2025-12-04T11:13:50.2327918Z FAILED [0.5030s] inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_1_mid_dim_1_in_features_1024_out_features_65_cuda_bfloat16 - RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_1_in_features_1024_out_features_65_cuda_bfloat16! Caching allocator allocated memory was 133120 and is now reported as 199680 on device 0. CUDA driver allocated memory was 276824064 and is now 278921216. 2025-12-04T11:13:50.2327925Z 2025-12-04T11:13:50.2328056Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.2328676Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_1_in_features_1024_out_features_65_cuda_bfloat16 2025-12-04T11:13:50.2328680Z 2025-12-04T11:13:50.2328850Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.2328961Z !!!!!!!!!!!!!!!!!!!!!!!!!! stopping after 1 failures !!!!!!!!!!!!!!!!!!!!!!!!!!! 2025-12-04T11:13:50.2329079Z ================== 1 failed, 57 deselected, 2 rerun in 12.09s ================== 2025-12-04T11:13:50.2329173Z Got exit code 1 2025-12-04T11:13:50.2329655Z FAILED CONSISTENTLY: test/inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_1_mid_dim_1_in_features_1024_out_features_65_cuda_bfloat16 2025-12-04T11:13:50.2329897Z Test failed consistently, continuing with the rest of the tests due to continue-through-error being set 2025-12-04T11:13:50.2330159Z W1204 10:47:28.574000 70618 site-packages/torch/_inductor/utils.py:1703] Not enough SMs to use max_autotune_gemm mode 2025-12-04T11:13:50.2330543Z Test results will be stored in test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-417661a6a446d83a.xml 2025-12-04T11:13:50.2330641Z ============================= test session starts ============================== 2025-12-04T11:13:50.2330854Z platform linux -- Python 3.10.14, pytest-7.3.2, pluggy-1.6.0 -- /opt/conda/envs/py_3.10/bin/python 2025-12-04T11:13:50.2330923Z cachedir: .pytest_cache 2025-12-04T11:13:50.2331229Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow] 2025-12-04T11:13:50.2331310Z rootdir: /var/lib/jenkins/workspace 2025-12-04T11:13:50.2331376Z configfile: pytest.ini 2025-12-04T11:13:50.2331693Z plugins: hypothesis-6.56.4, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-14.0, subtests-0.13.1, xdist-3.3.1, xdoctest-1.3.0, typeguard-4.3.0 2025-12-04T11:13:50.2331827Z collecting ... collected 58 items / 25 deselected / 33 selected 2025-12-04T11:13:50.2331914Z stepcurrent: skipping 25 already run items. 2025-12-04T11:13:50.2331988Z Running 33 items in this shard 2025-12-04T11:13:50.2331992Z 2025-12-04T11:13:50.2332491Z inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_1_mid_dim_1_in_features_128_out_features_1024_cuda_bfloat16 ('RERUN', {'yellow': True}) [1.9117s] [ 3%] 2025-12-04T11:13:50.2332989Z inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_1_mid_dim_1_in_features_128_out_features_1024_cuda_bfloat16 ('RERUN', {'yellow': True}) [0.4960s] [ 3%] 2025-12-04T11:13:50.2333437Z inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_1_mid_dim_1_in_features_128_out_features_1024_cuda_bfloat16 FAILED [0.4837s] [ 3%] 2025-12-04T11:13:50.2333440Z 2025-12-04T11:13:50.2333524Z ==================================== RERUNS ==================================== 2025-12-04T11:13:50.2333823Z _ TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_1_in_features_128_out_features_1024_cuda_bfloat16 _ 2025-12-04T11:13:50.2333898Z Traceback (most recent call last): 2025-12-04T11:13:50.2334210Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.2334329Z method(*args, **kwargs) 2025-12-04T11:13:50.2334688Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.2334757Z method(*args, **kwargs) 2025-12-04T11:13:50.2335042Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T11:13:50.2335102Z with policy(): 2025-12-04T11:13:50.2335394Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T11:13:50.2335459Z raise RuntimeError(msg) 2025-12-04T11:13:50.2336263Z RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_1_in_features_128_out_features_1024_cuda_bfloat16! Caching allocator allocated memory was 0 and is now reported as 131072 on device 0. CUDA driver allocated memory was 230686720 and is now 274726912. 2025-12-04T11:13:50.2336268Z 2025-12-04T11:13:50.2336395Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.2336915Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_1_in_features_128_out_features_1024_cuda_bfloat16 2025-12-04T11:13:50.2336973Z 2025-12-04T11:13:50.2337129Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.2337257Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.2337357Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.2337706Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.2337839Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.2337900Z graph_break [] 2025-12-04T11:13:50.2338198Z _ TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_1_in_features_128_out_features_1024_cuda_bfloat16 _ 2025-12-04T11:13:50.2338277Z Traceback (most recent call last): 2025-12-04T11:13:50.2338570Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.2338633Z method(*args, **kwargs) 2025-12-04T11:13:50.2338917Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.2338979Z method(*args, **kwargs) 2025-12-04T11:13:50.2339276Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T11:13:50.2339340Z with policy(): 2025-12-04T11:13:50.2339627Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T11:13:50.2339701Z raise RuntimeError(msg) 2025-12-04T11:13:50.2340516Z RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_1_in_features_128_out_features_1024_cuda_bfloat16! Caching allocator allocated memory was 131072 and is now reported as 262144 on device 0. CUDA driver allocated memory was 274726912 and is now 276824064. 2025-12-04T11:13:50.2340522Z 2025-12-04T11:13:50.2340648Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.2341163Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_1_in_features_128_out_features_1024_cuda_bfloat16 2025-12-04T11:13:50.2341167Z 2025-12-04T11:13:50.2341323Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.2341455Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.2341602Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.2342028Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.2342155Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.2342213Z graph_break [] 2025-12-04T11:13:50.2342342Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.2342429Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.2342550Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.2342892Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.2342948Z graph_break [] 2025-12-04T11:13:50.2343036Z =================================== FAILURES =================================== 2025-12-04T11:13:50.2343324Z _ TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_1_in_features_128_out_features_1024_cuda_bfloat16 _ 2025-12-04T11:13:50.2343432Z Traceback (most recent call last): 2025-12-04T11:13:50.2343823Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.2343887Z method(*args, **kwargs) 2025-12-04T11:13:50.2344175Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.2344237Z method(*args, **kwargs) 2025-12-04T11:13:50.2344521Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T11:13:50.2344583Z with policy(): 2025-12-04T11:13:50.2344871Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T11:13:50.2344938Z raise RuntimeError(msg) 2025-12-04T11:13:50.2345753Z RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_1_in_features_128_out_features_1024_cuda_bfloat16! Caching allocator allocated memory was 262144 and is now reported as 393216 on device 0. CUDA driver allocated memory was 276824064 and is now 278921216. 2025-12-04T11:13:50.2345759Z 2025-12-04T11:13:50.2345882Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.2346397Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_1_in_features_128_out_features_1024_cuda_bfloat16 2025-12-04T11:13:50.2346400Z 2025-12-04T11:13:50.2346551Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.2346680Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.2346767Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.2347110Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.2347238Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.2347297Z graph_break [] 2025-12-04T11:13:50.2347420Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.2347509Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.2347628Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.2347969Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.2348074Z graph_break [] 2025-12-04T11:13:50.2348193Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.2348348Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.2348468Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.2348814Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.2348881Z graph_break [] 2025-12-04T11:13:50.2349367Z - generated xml file: /var/lib/jenkins/workspace/test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-417661a6a446d83a.xml - 2025-12-04T11:13:50.2349469Z =========================== short test summary info ============================ 2025-12-04T11:13:50.2350768Z FAILED [0.4837s] inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_1_mid_dim_1_in_features_128_out_features_1024_cuda_bfloat16 - RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_1_in_features_128_out_features_1024_cuda_bfloat16! Caching allocator allocated memory was 262144 and is now reported as 393216 on device 0. CUDA driver allocated memory was 276824064 and is now 278921216. 2025-12-04T11:13:50.2350810Z 2025-12-04T11:13:50.2350938Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.2351451Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_1_in_features_128_out_features_1024_cuda_bfloat16 2025-12-04T11:13:50.2351455Z 2025-12-04T11:13:50.2351608Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.2351711Z !!!!!!!!!!!!!!!!!!!!!!!!!! stopping after 1 failures !!!!!!!!!!!!!!!!!!!!!!!!!!! 2025-12-04T11:13:50.2351826Z ================== 1 failed, 25 deselected, 2 rerun in 2.92s =================== 2025-12-04T11:13:50.2351888Z Got exit code 1 2025-12-04T11:13:50.2351955Z Retrying single test... 2025-12-04T11:13:50.2352214Z W1204 10:47:38.638000 70806 site-packages/torch/_inductor/utils.py:1703] Not enough SMs to use max_autotune_gemm mode 2025-12-04T11:13:50.2352600Z Test results will be stored in test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-2d84d0ab3f5cbf16.xml 2025-12-04T11:13:50.2352695Z ============================= test session starts ============================== 2025-12-04T11:13:50.2352903Z platform linux -- Python 3.10.14, pytest-7.3.2, pluggy-1.6.0 -- /opt/conda/envs/py_3.10/bin/python 2025-12-04T11:13:50.2352971Z cachedir: .pytest_cache 2025-12-04T11:13:50.2353275Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow] 2025-12-04T11:13:50.2353357Z rootdir: /var/lib/jenkins/workspace 2025-12-04T11:13:50.2353425Z configfile: pytest.ini 2025-12-04T11:13:50.2353743Z plugins: hypothesis-6.56.4, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-14.0, subtests-0.13.1, xdist-3.3.1, xdoctest-1.3.0, typeguard-4.3.0 2025-12-04T11:13:50.2353873Z collecting ... collected 58 items / 57 deselected / 1 selected 2025-12-04T11:13:50.2354444Z stepcurrent: skipping 25 already run items. Running only test/inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_1_mid_dim_1_in_features_128_out_features_1024_cuda_bfloat16 2025-12-04T11:13:50.2354517Z Running 1 items in this shard 2025-12-04T11:13:50.2354521Z 2025-12-04T11:13:50.2355249Z inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_1_mid_dim_1_in_features_128_out_features_1024_cuda_bfloat16 [W1204 10:47:39.778248618 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2355253Z 2025-12-04T11:13:50.2355594Z [W1204 10:47:48.787248789 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2355663Z 2025-12-04T11:13:50.2355956Z [W1204 10:47:48.787489855 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2355961Z 2025-12-04T11:13:50.2356249Z [W1204 10:47:48.793238031 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2356253Z 2025-12-04T11:13:50.2356537Z [W1204 10:47:48.793795173 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2356541Z 2025-12-04T11:13:50.2356825Z [W1204 10:47:48.793963497 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2356831Z 2025-12-04T11:13:50.2357114Z [W1204 10:47:48.799319545 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2357119Z 2025-12-04T11:13:50.2357405Z [W1204 10:47:48.799846596 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2357441Z 2025-12-04T11:13:50.2357728Z [W1204 10:47:48.800021050 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2357731Z 2025-12-04T11:13:50.2357820Z ('RERUN', {'yellow': True}) [10.9489s] [100%] 2025-12-04T11:13:50.2358544Z inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_1_mid_dim_1_in_features_128_out_features_1024_cuda_bfloat16 [W1204 10:47:50.031344971 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2358548Z 2025-12-04T11:13:50.2358834Z [W1204 10:47:50.031885443 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2358839Z 2025-12-04T11:13:50.2359129Z [W1204 10:47:50.032033176 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2359133Z 2025-12-04T11:13:50.2359420Z [W1204 10:47:50.035049543 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2359426Z 2025-12-04T11:13:50.2359715Z [W1204 10:47:50.035623576 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2359718Z 2025-12-04T11:13:50.2360003Z [W1204 10:47:50.035761829 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2360006Z 2025-12-04T11:13:50.2360288Z [W1204 10:47:50.040496184 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2360302Z 2025-12-04T11:13:50.2360596Z [W1204 10:47:50.040981074 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2360601Z 2025-12-04T11:13:50.2360885Z [W1204 10:47:50.041120348 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2360889Z 2025-12-04T11:13:50.2360971Z ('RERUN', {'yellow': True}) [0.4663s] [100%] 2025-12-04T11:13:50.2361694Z inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_1_mid_dim_1_in_features_128_out_features_1024_cuda_bfloat16 [W1204 10:47:50.489998421 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2361698Z 2025-12-04T11:13:50.2361988Z [W1204 10:47:50.490563813 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2362032Z 2025-12-04T11:13:50.2362399Z [W1204 10:47:50.490703206 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2362405Z 2025-12-04T11:13:50.2362692Z [W1204 10:47:50.493763054 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2362695Z 2025-12-04T11:13:50.2362979Z [W1204 10:47:50.494330656 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2362982Z 2025-12-04T11:13:50.2363269Z [W1204 10:47:50.494468949 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2363272Z 2025-12-04T11:13:50.2363556Z [W1204 10:47:50.499126471 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2363561Z 2025-12-04T11:13:50.2363847Z [W1204 10:47:50.499595482 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2363884Z 2025-12-04T11:13:50.2364173Z [W1204 10:47:50.499732345 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2364176Z 2025-12-04T11:13:50.2364237Z FAILED [0.4537s] [100%] 2025-12-04T11:13:50.2364240Z 2025-12-04T11:13:50.2364332Z ==================================== RERUNS ==================================== 2025-12-04T11:13:50.2364626Z _ TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_1_in_features_128_out_features_1024_cuda_bfloat16 _ 2025-12-04T11:13:50.2364701Z Traceback (most recent call last): 2025-12-04T11:13:50.2365004Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.2365071Z method(*args, **kwargs) 2025-12-04T11:13:50.2365365Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.2365428Z method(*args, **kwargs) 2025-12-04T11:13:50.2365710Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T11:13:50.2365771Z with policy(): 2025-12-04T11:13:50.2366061Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T11:13:50.2366128Z raise RuntimeError(msg) 2025-12-04T11:13:50.2366934Z RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_1_in_features_128_out_features_1024_cuda_bfloat16! Caching allocator allocated memory was 0 and is now reported as 131072 on device 0. CUDA driver allocated memory was 230686720 and is now 274726912. 2025-12-04T11:13:50.2366939Z 2025-12-04T11:13:50.2367069Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.2367590Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_1_in_features_128_out_features_1024_cuda_bfloat16 2025-12-04T11:13:50.2367596Z 2025-12-04T11:13:50.2367753Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.2367884Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.2367977Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.2368327Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.2368457Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.2368570Z graph_break [] 2025-12-04T11:13:50.2368769Z ----------------------------- Captured stderr call ----------------------------- 2025-12-04T11:13:50.2369470Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/constant_folding.py:256: UserWarning: Unsupported unwinding pattern: Address not in range (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/profiler/unwind/unwind.cpp:219.) 2025-12-04T11:13:50.2369543Z if out == self.unknown_value: 2025-12-04T11:13:50.2369835Z _ TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_1_in_features_128_out_features_1024_cuda_bfloat16 _ 2025-12-04T11:13:50.2369909Z Traceback (most recent call last): 2025-12-04T11:13:50.2370208Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.2370272Z method(*args, **kwargs) 2025-12-04T11:13:50.2370565Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.2370641Z method(*args, **kwargs) 2025-12-04T11:13:50.2370932Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T11:13:50.2371296Z with policy(): 2025-12-04T11:13:50.2371591Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T11:13:50.2371656Z raise RuntimeError(msg) 2025-12-04T11:13:50.2372471Z RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_1_in_features_128_out_features_1024_cuda_bfloat16! Caching allocator allocated memory was 131072 and is now reported as 262144 on device 0. CUDA driver allocated memory was 274726912 and is now 276824064. 2025-12-04T11:13:50.2372475Z 2025-12-04T11:13:50.2372600Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.2373121Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_1_in_features_128_out_features_1024_cuda_bfloat16 2025-12-04T11:13:50.2373129Z 2025-12-04T11:13:50.2373287Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.2373412Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.2373510Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.2373855Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.2373981Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.2374045Z graph_break [] 2025-12-04T11:13:50.2374168Z ----------------------------- Captured stderr call ----------------------------- 2025-12-04T11:13:50.2374856Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/constant_folding.py:256: UserWarning: Unsupported unwinding pattern: Address not in range (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/profiler/unwind/unwind.cpp:219.) 2025-12-04T11:13:50.2374927Z if out == self.unknown_value: 2025-12-04T11:13:50.2375050Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.2375142Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.2375262Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.2375604Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.2375662Z graph_break [] 2025-12-04T11:13:50.2375744Z =================================== FAILURES =================================== 2025-12-04T11:13:50.2376082Z _ TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_1_in_features_128_out_features_1024_cuda_bfloat16 _ 2025-12-04T11:13:50.2376309Z Traceback (most recent call last): 2025-12-04T11:13:50.2376609Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.2376674Z method(*args, **kwargs) 2025-12-04T11:13:50.2376964Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.2377028Z method(*args, **kwargs) 2025-12-04T11:13:50.2377313Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T11:13:50.2377371Z with policy(): 2025-12-04T11:13:50.2377663Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T11:13:50.2377728Z raise RuntimeError(msg) 2025-12-04T11:13:50.2379000Z RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_1_in_features_128_out_features_1024_cuda_bfloat16! Caching allocator allocated memory was 262144 and is now reported as 393216 on device 0. CUDA driver allocated memory was 276824064 and is now 278921216. 2025-12-04T11:13:50.2379092Z 2025-12-04T11:13:50.2379233Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.2379753Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_1_in_features_128_out_features_1024_cuda_bfloat16 2025-12-04T11:13:50.2379759Z 2025-12-04T11:13:50.2379919Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.2380052Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.2380148Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.2380497Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.2380627Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.2380689Z graph_break [] 2025-12-04T11:13:50.2380812Z ----------------------------- Captured stderr call ----------------------------- 2025-12-04T11:13:50.2381502Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/constant_folding.py:256: UserWarning: Unsupported unwinding pattern: Address not in range (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/profiler/unwind/unwind.cpp:219.) 2025-12-04T11:13:50.2381572Z if out == self.unknown_value: 2025-12-04T11:13:50.2381695Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.2381798Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.2381927Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.2382274Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.2382333Z graph_break [] 2025-12-04T11:13:50.2382454Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.2382545Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.2382667Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.2383004Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.2383065Z graph_break [] 2025-12-04T11:13:50.2383597Z - generated xml file: /var/lib/jenkins/workspace/test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-2d84d0ab3f5cbf16.xml - 2025-12-04T11:13:50.2383771Z =========================== short test summary info ============================ 2025-12-04T11:13:50.2385156Z FAILED [0.4537s] inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_1_mid_dim_1_in_features_128_out_features_1024_cuda_bfloat16 - RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_1_in_features_128_out_features_1024_cuda_bfloat16! Caching allocator allocated memory was 262144 and is now reported as 393216 on device 0. CUDA driver allocated memory was 276824064 and is now 278921216. 2025-12-04T11:13:50.2385163Z 2025-12-04T11:13:50.2385293Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.2385811Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_1_in_features_128_out_features_1024_cuda_bfloat16 2025-12-04T11:13:50.2385817Z 2025-12-04T11:13:50.2385974Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.2386095Z !!!!!!!!!!!!!!!!!!!!!!!!!! stopping after 1 failures !!!!!!!!!!!!!!!!!!!!!!!!!!! 2025-12-04T11:13:50.2386248Z ================== 1 failed, 57 deselected, 2 rerun in 11.89s ================== 2025-12-04T11:13:50.2386309Z Got exit code 1 2025-12-04T11:13:50.2386374Z Retrying single test... 2025-12-04T11:13:50.2386639Z W1204 10:47:57.409000 70999 site-packages/torch/_inductor/utils.py:1703] Not enough SMs to use max_autotune_gemm mode 2025-12-04T11:13:50.2387029Z Test results will be stored in test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-3c429c2af5dedece.xml 2025-12-04T11:13:50.2387124Z ============================= test session starts ============================== 2025-12-04T11:13:50.2387335Z platform linux -- Python 3.10.14, pytest-7.3.2, pluggy-1.6.0 -- /opt/conda/envs/py_3.10/bin/python 2025-12-04T11:13:50.2387406Z cachedir: .pytest_cache 2025-12-04T11:13:50.2387712Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow] 2025-12-04T11:13:50.2387797Z rootdir: /var/lib/jenkins/workspace 2025-12-04T11:13:50.2387861Z configfile: pytest.ini 2025-12-04T11:13:50.2388173Z plugins: hypothesis-6.56.4, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-14.0, subtests-0.13.1, xdist-3.3.1, xdoctest-1.3.0, typeguard-4.3.0 2025-12-04T11:13:50.2388304Z collecting ... collected 58 items / 57 deselected / 1 selected 2025-12-04T11:13:50.2388868Z stepcurrent: skipping 25 already run items. Running only test/inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_1_mid_dim_1_in_features_128_out_features_1024_cuda_bfloat16 2025-12-04T11:13:50.2388944Z Running 1 items in this shard 2025-12-04T11:13:50.2388948Z 2025-12-04T11:13:50.2389679Z inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_1_mid_dim_1_in_features_128_out_features_1024_cuda_bfloat16 [W1204 10:47:58.536611992 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2389686Z 2025-12-04T11:13:50.2389987Z [W1204 10:48:07.739462009 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2389991Z 2025-12-04T11:13:50.2390281Z [W1204 10:48:07.739767856 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2390284Z 2025-12-04T11:13:50.2390568Z [W1204 10:48:07.745461992 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2390575Z 2025-12-04T11:13:50.2390858Z [W1204 10:48:07.746002024 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2390901Z 2025-12-04T11:13:50.2391272Z [W1204 10:48:07.746173838 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2391278Z 2025-12-04T11:13:50.2391579Z [W1204 10:48:07.751430214 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2391582Z 2025-12-04T11:13:50.2391865Z [W1204 10:48:07.751951115 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2391868Z 2025-12-04T11:13:50.2392157Z [W1204 10:48:07.752113589 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2392161Z 2025-12-04T11:13:50.2392242Z ('RERUN', {'yellow': True}) [11.1258s] [100%] 2025-12-04T11:13:50.2392966Z inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_1_mid_dim_1_in_features_128_out_features_1024_cuda_bfloat16 [W1204 10:48:09.974543339 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2393006Z 2025-12-04T11:13:50.2393301Z [W1204 10:48:09.975100391 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2393304Z 2025-12-04T11:13:50.2393594Z [W1204 10:48:09.975246934 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2393596Z 2025-12-04T11:13:50.2393879Z [W1204 10:48:09.978243311 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2393882Z 2025-12-04T11:13:50.2394164Z [W1204 10:48:09.978823933 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2394168Z 2025-12-04T11:13:50.2394459Z [W1204 10:48:09.978964087 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2394464Z 2025-12-04T11:13:50.2394748Z [W1204 10:48:09.983605289 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2394751Z 2025-12-04T11:13:50.2395035Z [W1204 10:48:09.984080670 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2395038Z 2025-12-04T11:13:50.2395322Z [W1204 10:48:09.984215203 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2395325Z 2025-12-04T11:13:50.2395406Z ('RERUN', {'yellow': True}) [0.4616s] [100%] 2025-12-04T11:13:50.2396126Z inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_1_mid_dim_1_in_features_128_out_features_1024_cuda_bfloat16 [W1204 10:48:09.434587717 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2396133Z 2025-12-04T11:13:50.2396432Z [W1204 10:48:09.435141139 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2396435Z 2025-12-04T11:13:50.2396719Z [W1204 10:48:09.435285563 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2396723Z 2025-12-04T11:13:50.2397011Z [W1204 10:48:09.438271209 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2397018Z 2025-12-04T11:13:50.2397302Z [W1204 10:48:09.438848021 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2397344Z 2025-12-04T11:13:50.2397695Z [W1204 10:48:09.438987545 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2397698Z 2025-12-04T11:13:50.2397987Z [W1204 10:48:09.443576936 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2397990Z 2025-12-04T11:13:50.2398274Z [W1204 10:48:09.444054407 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2398277Z 2025-12-04T11:13:50.2398566Z [W1204 10:48:09.444190460 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2398569Z 2025-12-04T11:13:50.2398630Z FAILED [0.4572s] [100%] 2025-12-04T11:13:50.2398633Z 2025-12-04T11:13:50.2398720Z ==================================== RERUNS ==================================== 2025-12-04T11:13:50.2399011Z _ TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_1_in_features_128_out_features_1024_cuda_bfloat16 _ 2025-12-04T11:13:50.2399089Z Traceback (most recent call last): 2025-12-04T11:13:50.2399440Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.2399506Z method(*args, **kwargs) 2025-12-04T11:13:50.2399796Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.2399860Z method(*args, **kwargs) 2025-12-04T11:13:50.2400143Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T11:13:50.2400204Z with policy(): 2025-12-04T11:13:50.2400492Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T11:13:50.2400560Z raise RuntimeError(msg) 2025-12-04T11:13:50.2401359Z RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_1_in_features_128_out_features_1024_cuda_bfloat16! Caching allocator allocated memory was 0 and is now reported as 131072 on device 0. CUDA driver allocated memory was 230686720 and is now 274726912. 2025-12-04T11:13:50.2401365Z 2025-12-04T11:13:50.2401494Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.2402025Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_1_in_features_128_out_features_1024_cuda_bfloat16 2025-12-04T11:13:50.2402030Z 2025-12-04T11:13:50.2402190Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.2402322Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.2402418Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.2402766Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.2402897Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.2402957Z graph_break [] 2025-12-04T11:13:50.2403084Z ----------------------------- Captured stderr call ----------------------------- 2025-12-04T11:13:50.2403774Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/constant_folding.py:256: UserWarning: Unsupported unwinding pattern: Address not in range (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/profiler/unwind/unwind.cpp:219.) 2025-12-04T11:13:50.2403845Z if out == self.unknown_value: 2025-12-04T11:13:50.2404139Z _ TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_1_in_features_128_out_features_1024_cuda_bfloat16 _ 2025-12-04T11:13:50.2404256Z Traceback (most recent call last): 2025-12-04T11:13:50.2404619Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.2404688Z method(*args, **kwargs) 2025-12-04T11:13:50.2404981Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.2405044Z method(*args, **kwargs) 2025-12-04T11:13:50.2405329Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T11:13:50.2405389Z with policy(): 2025-12-04T11:13:50.2405681Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T11:13:50.2405748Z raise RuntimeError(msg) 2025-12-04T11:13:50.2406558Z RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_1_in_features_128_out_features_1024_cuda_bfloat16! Caching allocator allocated memory was 131072 and is now reported as 262144 on device 0. CUDA driver allocated memory was 274726912 and is now 276824064. 2025-12-04T11:13:50.2406601Z 2025-12-04T11:13:50.2406729Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.2407243Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_1_in_features_128_out_features_1024_cuda_bfloat16 2025-12-04T11:13:50.2407246Z 2025-12-04T11:13:50.2407408Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.2407536Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.2407629Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.2407972Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.2408107Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.2408166Z graph_break [] 2025-12-04T11:13:50.2408293Z ----------------------------- Captured stderr call ----------------------------- 2025-12-04T11:13:50.2408975Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/constant_folding.py:256: UserWarning: Unsupported unwinding pattern: Address not in range (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/profiler/unwind/unwind.cpp:219.) 2025-12-04T11:13:50.2409045Z if out == self.unknown_value: 2025-12-04T11:13:50.2409170Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.2409258Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.2409380Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.2409724Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.2409786Z graph_break [] 2025-12-04T11:13:50.2409874Z =================================== FAILURES =================================== 2025-12-04T11:13:50.2410162Z _ TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_1_in_features_128_out_features_1024_cuda_bfloat16 _ 2025-12-04T11:13:50.2410233Z Traceback (most recent call last): 2025-12-04T11:13:50.2410530Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.2410592Z method(*args, **kwargs) 2025-12-04T11:13:50.2410885Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.2410950Z method(*args, **kwargs) 2025-12-04T11:13:50.2411243Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T11:13:50.2411359Z with policy(): 2025-12-04T11:13:50.2411720Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T11:13:50.2411788Z raise RuntimeError(msg) 2025-12-04T11:13:50.2412602Z RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_1_in_features_128_out_features_1024_cuda_bfloat16! Caching allocator allocated memory was 262144 and is now reported as 393216 on device 0. CUDA driver allocated memory was 276824064 and is now 278921216. 2025-12-04T11:13:50.2412606Z 2025-12-04T11:13:50.2412729Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.2413243Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_1_in_features_128_out_features_1024_cuda_bfloat16 2025-12-04T11:13:50.2413248Z 2025-12-04T11:13:50.2413401Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.2413529Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.2413653Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.2414006Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.2414137Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.2414194Z graph_break [] 2025-12-04T11:13:50.2414317Z ----------------------------- Captured stderr call ----------------------------- 2025-12-04T11:13:50.2415001Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/constant_folding.py:256: UserWarning: Unsupported unwinding pattern: Address not in range (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/profiler/unwind/unwind.cpp:219.) 2025-12-04T11:13:50.2415070Z if out == self.unknown_value: 2025-12-04T11:13:50.2415197Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.2415287Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.2415409Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.2415748Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.2415805Z graph_break [] 2025-12-04T11:13:50.2415930Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.2416016Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.2416134Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.2416477Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.2416537Z graph_break [] 2025-12-04T11:13:50.2417038Z - generated xml file: /var/lib/jenkins/workspace/test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-3c429c2af5dedece.xml - 2025-12-04T11:13:50.2417140Z =========================== short test summary info ============================ 2025-12-04T11:13:50.2418440Z FAILED [0.4572s] inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_1_mid_dim_1_in_features_128_out_features_1024_cuda_bfloat16 - RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_1_in_features_128_out_features_1024_cuda_bfloat16! Caching allocator allocated memory was 262144 and is now reported as 393216 on device 0. CUDA driver allocated memory was 276824064 and is now 278921216. 2025-12-04T11:13:50.2418505Z 2025-12-04T11:13:50.2418629Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.2419217Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_1_in_features_128_out_features_1024_cuda_bfloat16 2025-12-04T11:13:50.2419223Z 2025-12-04T11:13:50.2419378Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.2419481Z !!!!!!!!!!!!!!!!!!!!!!!!!! stopping after 1 failures !!!!!!!!!!!!!!!!!!!!!!!!!!! 2025-12-04T11:13:50.2419597Z ================== 1 failed, 57 deselected, 2 rerun in 12.07s ================== 2025-12-04T11:13:50.2419654Z Got exit code 1 2025-12-04T11:13:50.2420126Z FAILED CONSISTENTLY: test/inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_1_mid_dim_1_in_features_128_out_features_1024_cuda_bfloat16 2025-12-04T11:13:50.2420370Z Test failed consistently, continuing with the rest of the tests due to continue-through-error being set 2025-12-04T11:13:50.2420635Z W1204 10:48:16.253000 71192 site-packages/torch/_inductor/utils.py:1703] Not enough SMs to use max_autotune_gemm mode 2025-12-04T11:13:50.2421061Z Test results will be stored in test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-b9ae48951e1b6488.xml 2025-12-04T11:13:50.2421156Z ============================= test session starts ============================== 2025-12-04T11:13:50.2421362Z platform linux -- Python 3.10.14, pytest-7.3.2, pluggy-1.6.0 -- /opt/conda/envs/py_3.10/bin/python 2025-12-04T11:13:50.2421431Z cachedir: .pytest_cache 2025-12-04T11:13:50.2421734Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow] 2025-12-04T11:13:50.2421809Z rootdir: /var/lib/jenkins/workspace 2025-12-04T11:13:50.2421877Z configfile: pytest.ini 2025-12-04T11:13:50.2422199Z plugins: hypothesis-6.56.4, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-14.0, subtests-0.13.1, xdist-3.3.1, xdoctest-1.3.0, typeguard-4.3.0 2025-12-04T11:13:50.2422335Z collecting ... collected 58 items / 26 deselected / 32 selected 2025-12-04T11:13:50.2422424Z stepcurrent: skipping 26 already run items. 2025-12-04T11:13:50.2422494Z Running 32 items in this shard 2025-12-04T11:13:50.2422498Z 2025-12-04T11:13:50.2423001Z inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_1_mid_dim_1_in_features_128_out_features_64_cuda_bfloat16 ('RERUN', {'yellow': True}) [1.8812s] [ 3%] 2025-12-04T11:13:50.2423480Z inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_1_mid_dim_1_in_features_128_out_features_64_cuda_bfloat16 ('RERUN', {'yellow': True}) [0.4568s] [ 3%] 2025-12-04T11:13:50.2423962Z inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_1_mid_dim_1_in_features_128_out_features_64_cuda_bfloat16 FAILED [0.4599s] [ 3%] 2025-12-04T11:13:50.2423968Z 2025-12-04T11:13:50.2424051Z ==================================== RERUNS ==================================== 2025-12-04T11:13:50.2424340Z _ TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_1_in_features_128_out_features_64_cuda_bfloat16 _ 2025-12-04T11:13:50.2424430Z Traceback (most recent call last): 2025-12-04T11:13:50.2424743Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.2424823Z method(*args, **kwargs) 2025-12-04T11:13:50.2425116Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.2425179Z method(*args, **kwargs) 2025-12-04T11:13:50.2425469Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T11:13:50.2425528Z with policy(): 2025-12-04T11:13:50.2425867Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T11:13:50.2426007Z raise RuntimeError(msg) 2025-12-04T11:13:50.2426806Z RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_1_in_features_128_out_features_64_cuda_bfloat16! Caching allocator allocated memory was 0 and is now reported as 8192 on device 0. CUDA driver allocated memory was 230686720 and is now 274726912. 2025-12-04T11:13:50.2426811Z 2025-12-04T11:13:50.2426940Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.2427451Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_1_in_features_128_out_features_64_cuda_bfloat16 2025-12-04T11:13:50.2427455Z 2025-12-04T11:13:50.2427614Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.2427745Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.2427841Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.2428228Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.2428357Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.2428419Z graph_break [] 2025-12-04T11:13:50.2428707Z _ TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_1_in_features_128_out_features_64_cuda_bfloat16 _ 2025-12-04T11:13:50.2428783Z Traceback (most recent call last): 2025-12-04T11:13:50.2429083Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.2429145Z method(*args, **kwargs) 2025-12-04T11:13:50.2429441Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.2429507Z method(*args, **kwargs) 2025-12-04T11:13:50.2429792Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T11:13:50.2429856Z with policy(): 2025-12-04T11:13:50.2430145Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T11:13:50.2430221Z raise RuntimeError(msg) 2025-12-04T11:13:50.2431026Z RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_1_in_features_128_out_features_64_cuda_bfloat16! Caching allocator allocated memory was 8192 and is now reported as 16384 on device 0. CUDA driver allocated memory was 274726912 and is now 276824064. 2025-12-04T11:13:50.2431030Z 2025-12-04T11:13:50.2431152Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.2431669Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_1_in_features_128_out_features_64_cuda_bfloat16 2025-12-04T11:13:50.2431675Z 2025-12-04T11:13:50.2431828Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.2431953Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.2432048Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.2432391Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.2432517Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.2432575Z graph_break [] 2025-12-04T11:13:50.2432697Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.2432830Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.2433015Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.2433364Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.2433423Z graph_break [] 2025-12-04T11:13:50.2433506Z =================================== FAILURES =================================== 2025-12-04T11:13:50.2433794Z _ TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_1_in_features_128_out_features_64_cuda_bfloat16 _ 2025-12-04T11:13:50.2433865Z Traceback (most recent call last): 2025-12-04T11:13:50.2434156Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.2434219Z method(*args, **kwargs) 2025-12-04T11:13:50.2434511Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.2434587Z method(*args, **kwargs) 2025-12-04T11:13:50.2434874Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T11:13:50.2434970Z with policy(): 2025-12-04T11:13:50.2435264Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T11:13:50.2435329Z raise RuntimeError(msg) 2025-12-04T11:13:50.2436133Z RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_1_in_features_128_out_features_64_cuda_bfloat16! Caching allocator allocated memory was 16384 and is now reported as 24576 on device 0. CUDA driver allocated memory was 276824064 and is now 278921216. 2025-12-04T11:13:50.2436137Z 2025-12-04T11:13:50.2436261Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.2436773Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_1_in_features_128_out_features_64_cuda_bfloat16 2025-12-04T11:13:50.2436779Z 2025-12-04T11:13:50.2436935Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.2437058Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.2437149Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.2437489Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.2437610Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.2437670Z graph_break [] 2025-12-04T11:13:50.2437793Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.2437881Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.2438018Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.2438357Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.2438417Z graph_break [] 2025-12-04T11:13:50.2438537Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.2438624Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.2438746Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.2439081Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.2439140Z graph_break [] 2025-12-04T11:13:50.2439733Z - generated xml file: /var/lib/jenkins/workspace/test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-b9ae48951e1b6488.xml - 2025-12-04T11:13:50.2439834Z =========================== short test summary info ============================ 2025-12-04T11:13:50.2441111Z FAILED [0.4599s] inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_1_mid_dim_1_in_features_128_out_features_64_cuda_bfloat16 - RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_1_in_features_128_out_features_64_cuda_bfloat16! Caching allocator allocated memory was 16384 and is now reported as 24576 on device 0. CUDA driver allocated memory was 276824064 and is now 278921216. 2025-12-04T11:13:50.2441115Z 2025-12-04T11:13:50.2441235Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.2441764Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_1_in_features_128_out_features_64_cuda_bfloat16 2025-12-04T11:13:50.2441768Z 2025-12-04T11:13:50.2441921Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.2442082Z !!!!!!!!!!!!!!!!!!!!!!!!!! stopping after 1 failures !!!!!!!!!!!!!!!!!!!!!!!!!!! 2025-12-04T11:13:50.2442197Z ================== 1 failed, 26 deselected, 2 rerun in 2.82s =================== 2025-12-04T11:13:50.2442256Z Got exit code 1 2025-12-04T11:13:50.2442325Z Retrying single test... 2025-12-04T11:13:50.2442586Z W1204 10:48:26.279000 71373 site-packages/torch/_inductor/utils.py:1703] Not enough SMs to use max_autotune_gemm mode 2025-12-04T11:13:50.2442968Z Test results will be stored in test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-a9948af371c98fbd.xml 2025-12-04T11:13:50.2443067Z ============================= test session starts ============================== 2025-12-04T11:13:50.2443279Z platform linux -- Python 3.10.14, pytest-7.3.2, pluggy-1.6.0 -- /opt/conda/envs/py_3.10/bin/python 2025-12-04T11:13:50.2443351Z cachedir: .pytest_cache 2025-12-04T11:13:50.2443655Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow] 2025-12-04T11:13:50.2443730Z rootdir: /var/lib/jenkins/workspace 2025-12-04T11:13:50.2443798Z configfile: pytest.ini 2025-12-04T11:13:50.2444111Z plugins: hypothesis-6.56.4, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-14.0, subtests-0.13.1, xdist-3.3.1, xdoctest-1.3.0, typeguard-4.3.0 2025-12-04T11:13:50.2444242Z collecting ... collected 58 items / 57 deselected / 1 selected 2025-12-04T11:13:50.2444807Z stepcurrent: skipping 26 already run items. Running only test/inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_1_mid_dim_1_in_features_128_out_features_64_cuda_bfloat16 2025-12-04T11:13:50.2444879Z Running 1 items in this shard 2025-12-04T11:13:50.2444882Z 2025-12-04T11:13:50.2445610Z inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_1_mid_dim_1_in_features_128_out_features_64_cuda_bfloat16 [W1204 10:48:27.371899220 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2445616Z 2025-12-04T11:13:50.2445912Z [W1204 10:48:36.543663977 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2445915Z 2025-12-04T11:13:50.2446208Z [W1204 10:48:36.543926613 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2446212Z 2025-12-04T11:13:50.2446496Z [W1204 10:48:36.549655309 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2446541Z 2025-12-04T11:13:50.2446892Z [W1204 10:48:36.550242322 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2446896Z 2025-12-04T11:13:50.2447184Z [W1204 10:48:36.550412636 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2447189Z 2025-12-04T11:13:50.2447487Z [W1204 10:48:36.555805835 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2447491Z 2025-12-04T11:13:50.2447779Z [W1204 10:48:36.556328837 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2447783Z 2025-12-04T11:13:50.2448067Z [W1204 10:48:36.556493020 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2448073Z 2025-12-04T11:13:50.2448154Z ('RERUN', {'yellow': True}) [11.0589s] [100%] 2025-12-04T11:13:50.2448870Z inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_1_mid_dim_1_in_features_128_out_features_64_cuda_bfloat16 [W1204 10:48:37.741362083 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2448911Z 2025-12-04T11:13:50.2449203Z [W1204 10:48:37.741954086 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2449206Z 2025-12-04T11:13:50.2449491Z [W1204 10:48:37.742094709 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2449494Z 2025-12-04T11:13:50.2449783Z [W1204 10:48:37.745070815 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2449786Z 2025-12-04T11:13:50.2450073Z [W1204 10:48:37.745648288 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2450078Z 2025-12-04T11:13:50.2450366Z [W1204 10:48:37.745786180 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2450371Z 2025-12-04T11:13:50.2450653Z [W1204 10:48:37.750410133 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2450657Z 2025-12-04T11:13:50.2450944Z [W1204 10:48:37.750889124 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2450948Z 2025-12-04T11:13:50.2451232Z [W1204 10:48:37.751023896 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2451235Z 2025-12-04T11:13:50.2451312Z ('RERUN', {'yellow': True}) [0.4255s] [100%] 2025-12-04T11:13:50.2452038Z inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_1_mid_dim_1_in_features_128_out_features_64_cuda_bfloat16 [W1204 10:48:38.162685493 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2452043Z 2025-12-04T11:13:50.2452333Z [W1204 10:48:38.163259696 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2452336Z 2025-12-04T11:13:50.2452623Z [W1204 10:48:38.163405819 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2452626Z 2025-12-04T11:13:50.2452910Z [W1204 10:48:38.166343044 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2452913Z 2025-12-04T11:13:50.2453200Z [W1204 10:48:38.166911866 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2453241Z 2025-12-04T11:13:50.2453590Z [W1204 10:48:38.167047809 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2453595Z 2025-12-04T11:13:50.2453884Z [W1204 10:48:38.171642821 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2453887Z 2025-12-04T11:13:50.2454170Z [W1204 10:48:38.172117352 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2454173Z 2025-12-04T11:13:50.2454457Z [W1204 10:48:38.172251405 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2454463Z 2025-12-04T11:13:50.2454524Z FAILED [0.4174s] [100%] 2025-12-04T11:13:50.2454529Z 2025-12-04T11:13:50.2454613Z ==================================== RERUNS ==================================== 2025-12-04T11:13:50.2454903Z _ TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_1_in_features_128_out_features_64_cuda_bfloat16 _ 2025-12-04T11:13:50.2455011Z Traceback (most recent call last): 2025-12-04T11:13:50.2455327Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.2455393Z method(*args, **kwargs) 2025-12-04T11:13:50.2455682Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.2455747Z method(*args, **kwargs) 2025-12-04T11:13:50.2456033Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T11:13:50.2456092Z with policy(): 2025-12-04T11:13:50.2456387Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T11:13:50.2456454Z raise RuntimeError(msg) 2025-12-04T11:13:50.2457248Z RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_1_in_features_128_out_features_64_cuda_bfloat16! Caching allocator allocated memory was 0 and is now reported as 8192 on device 0. CUDA driver allocated memory was 230686720 and is now 274726912. 2025-12-04T11:13:50.2457254Z 2025-12-04T11:13:50.2457378Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.2457890Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_1_in_features_128_out_features_64_cuda_bfloat16 2025-12-04T11:13:50.2457896Z 2025-12-04T11:13:50.2458052Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.2458181Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.2458282Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.2458629Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.2458758Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.2458819Z graph_break [] 2025-12-04T11:13:50.2458943Z ----------------------------- Captured stderr call ----------------------------- 2025-12-04T11:13:50.2459644Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/constant_folding.py:256: UserWarning: Unsupported unwinding pattern: Address not in range (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/profiler/unwind/unwind.cpp:219.) 2025-12-04T11:13:50.2459714Z if out == self.unknown_value: 2025-12-04T11:13:50.2459998Z _ TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_1_in_features_128_out_features_64_cuda_bfloat16 _ 2025-12-04T11:13:50.2460117Z Traceback (most recent call last): 2025-12-04T11:13:50.2460480Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.2460547Z method(*args, **kwargs) 2025-12-04T11:13:50.2460837Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.2460898Z method(*args, **kwargs) 2025-12-04T11:13:50.2461200Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T11:13:50.2461259Z with policy(): 2025-12-04T11:13:50.2461550Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T11:13:50.2461617Z raise RuntimeError(msg) 2025-12-04T11:13:50.2462420Z RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_1_in_features_128_out_features_64_cuda_bfloat16! Caching allocator allocated memory was 8192 and is now reported as 16384 on device 0. CUDA driver allocated memory was 274726912 and is now 276824064. 2025-12-04T11:13:50.2462460Z 2025-12-04T11:13:50.2462591Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.2463104Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_1_in_features_128_out_features_64_cuda_bfloat16 2025-12-04T11:13:50.2463108Z 2025-12-04T11:13:50.2463273Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.2463399Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.2463491Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.2463907Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.2464033Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.2464092Z graph_break [] 2025-12-04T11:13:50.2464216Z ----------------------------- Captured stderr call ----------------------------- 2025-12-04T11:13:50.2464901Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/constant_folding.py:256: UserWarning: Unsupported unwinding pattern: Address not in range (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/profiler/unwind/unwind.cpp:219.) 2025-12-04T11:13:50.2464970Z if out == self.unknown_value: 2025-12-04T11:13:50.2465092Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.2465182Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.2465308Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.2465653Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.2465713Z graph_break [] 2025-12-04T11:13:50.2465796Z =================================== FAILURES =================================== 2025-12-04T11:13:50.2466079Z _ TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_1_in_features_128_out_features_64_cuda_bfloat16 _ 2025-12-04T11:13:50.2466152Z Traceback (most recent call last): 2025-12-04T11:13:50.2466444Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.2466507Z method(*args, **kwargs) 2025-12-04T11:13:50.2466809Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.2466872Z method(*args, **kwargs) 2025-12-04T11:13:50.2467291Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T11:13:50.2467352Z with policy(): 2025-12-04T11:13:50.2467647Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T11:13:50.2467717Z raise RuntimeError(msg) 2025-12-04T11:13:50.2468516Z RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_1_in_features_128_out_features_64_cuda_bfloat16! Caching allocator allocated memory was 16384 and is now reported as 24576 on device 0. CUDA driver allocated memory was 276824064 and is now 278921216. 2025-12-04T11:13:50.2468520Z 2025-12-04T11:13:50.2468647Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.2469160Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_1_in_features_128_out_features_64_cuda_bfloat16 2025-12-04T11:13:50.2469165Z 2025-12-04T11:13:50.2469321Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.2469487Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.2469577Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.2469922Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.2470044Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.2470101Z graph_break [] 2025-12-04T11:13:50.2470224Z ----------------------------- Captured stderr call ----------------------------- 2025-12-04T11:13:50.2470904Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/constant_folding.py:256: UserWarning: Unsupported unwinding pattern: Address not in range (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/profiler/unwind/unwind.cpp:219.) 2025-12-04T11:13:50.2470979Z if out == self.unknown_value: 2025-12-04T11:13:50.2471100Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.2471190Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.2471313Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.2471655Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.2471711Z graph_break [] 2025-12-04T11:13:50.2471846Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.2471934Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.2472055Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.2472396Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.2472454Z graph_break [] 2025-12-04T11:13:50.2472948Z - generated xml file: /var/lib/jenkins/workspace/test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-a9948af371c98fbd.xml - 2025-12-04T11:13:50.2473049Z =========================== short test summary info ============================ 2025-12-04T11:13:50.2474319Z FAILED [0.4174s] inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_1_mid_dim_1_in_features_128_out_features_64_cuda_bfloat16 - RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_1_in_features_128_out_features_64_cuda_bfloat16! Caching allocator allocated memory was 16384 and is now reported as 24576 on device 0. CUDA driver allocated memory was 276824064 and is now 278921216. 2025-12-04T11:13:50.2474365Z 2025-12-04T11:13:50.2474554Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.2475071Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_1_in_features_128_out_features_64_cuda_bfloat16 2025-12-04T11:13:50.2475075Z 2025-12-04T11:13:50.2475227Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.2475330Z !!!!!!!!!!!!!!!!!!!!!!!!!! stopping after 1 failures !!!!!!!!!!!!!!!!!!!!!!!!!!! 2025-12-04T11:13:50.2475447Z ================== 1 failed, 57 deselected, 2 rerun in 11.93s ================== 2025-12-04T11:13:50.2475506Z Got exit code 1 2025-12-04T11:13:50.2475573Z Retrying single test... 2025-12-04T11:13:50.2475834Z W1204 10:48:45.025000 71559 site-packages/torch/_inductor/utils.py:1703] Not enough SMs to use max_autotune_gemm mode 2025-12-04T11:13:50.2476221Z Test results will be stored in test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-f058047ef4d14b46.xml 2025-12-04T11:13:50.2476316Z ============================= test session starts ============================== 2025-12-04T11:13:50.2476559Z platform linux -- Python 3.10.14, pytest-7.3.2, pluggy-1.6.0 -- /opt/conda/envs/py_3.10/bin/python 2025-12-04T11:13:50.2476626Z cachedir: .pytest_cache 2025-12-04T11:13:50.2476932Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow] 2025-12-04T11:13:50.2477009Z rootdir: /var/lib/jenkins/workspace 2025-12-04T11:13:50.2477076Z configfile: pytest.ini 2025-12-04T11:13:50.2477389Z plugins: hypothesis-6.56.4, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-14.0, subtests-0.13.1, xdist-3.3.1, xdoctest-1.3.0, typeguard-4.3.0 2025-12-04T11:13:50.2477514Z collecting ... collected 58 items / 57 deselected / 1 selected 2025-12-04T11:13:50.2478221Z stepcurrent: skipping 26 already run items. Running only test/inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_1_mid_dim_1_in_features_128_out_features_64_cuda_bfloat16 2025-12-04T11:13:50.2478294Z Running 1 items in this shard 2025-12-04T11:13:50.2478297Z 2025-12-04T11:13:50.2479025Z inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_1_mid_dim_1_in_features_128_out_features_64_cuda_bfloat16 [W1204 10:48:46.125846143 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2479029Z 2025-12-04T11:13:50.2479324Z [W1204 10:48:54.858597107 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2479327Z 2025-12-04T11:13:50.2479618Z [W1204 10:48:54.858827712 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2479623Z 2025-12-04T11:13:50.2479910Z [W1204 10:48:54.864909237 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2479915Z 2025-12-04T11:13:50.2480201Z [W1204 10:48:54.865453039 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2480206Z 2025-12-04T11:13:50.2480490Z [W1204 10:48:54.865609152 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2480493Z 2025-12-04T11:13:50.2480775Z [W1204 10:48:54.870880339 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2480779Z 2025-12-04T11:13:50.2481067Z [W1204 10:48:54.871395760 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2481142Z 2025-12-04T11:13:50.2481529Z [W1204 10:48:54.871557844 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2481535Z 2025-12-04T11:13:50.2481621Z ('RERUN', {'yellow': True}) [10.6289s] [100%] 2025-12-04T11:13:50.2482339Z inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_1_mid_dim_1_in_features_128_out_features_64_cuda_bfloat16 [W1204 10:48:56.056413049 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2482343Z 2025-12-04T11:13:50.2482633Z [W1204 10:48:56.057003432 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2482637Z 2025-12-04T11:13:50.2482923Z [W1204 10:48:56.057142395 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2482928Z 2025-12-04T11:13:50.2483219Z [W1204 10:48:56.060044729 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2483270Z 2025-12-04T11:13:50.2483555Z [W1204 10:48:56.060627162 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2483558Z 2025-12-04T11:13:50.2483843Z [W1204 10:48:56.060765185 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2483849Z 2025-12-04T11:13:50.2484136Z [W1204 10:48:56.065265395 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2484140Z 2025-12-04T11:13:50.2484423Z [W1204 10:48:56.065727505 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2484428Z 2025-12-04T11:13:50.2484720Z [W1204 10:48:56.065862158 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2484723Z 2025-12-04T11:13:50.2484802Z ('RERUN', {'yellow': True}) [0.4205s] [100%] 2025-12-04T11:13:50.2485521Z inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_1_mid_dim_1_in_features_128_out_features_64_cuda_bfloat16 [W1204 10:48:56.474126461 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2485525Z 2025-12-04T11:13:50.2485812Z [W1204 10:48:56.474690873 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2485815Z 2025-12-04T11:13:50.2486102Z [W1204 10:48:56.474834046 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2486107Z 2025-12-04T11:13:50.2486394Z [W1204 10:48:56.477771591 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2486397Z 2025-12-04T11:13:50.2486685Z [W1204 10:48:56.478337744 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2486690Z 2025-12-04T11:13:50.2486976Z [W1204 10:48:56.478474827 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2486979Z 2025-12-04T11:13:50.2487265Z [W1204 10:48:56.483059758 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2487268Z 2025-12-04T11:13:50.2487557Z [W1204 10:48:56.483531758 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2487599Z 2025-12-04T11:13:50.2488024Z [W1204 10:48:56.483665811 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2488028Z 2025-12-04T11:13:50.2488093Z FAILED [0.4167s] [100%] 2025-12-04T11:13:50.2488097Z 2025-12-04T11:13:50.2488181Z ==================================== RERUNS ==================================== 2025-12-04T11:13:50.2488469Z _ TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_1_in_features_128_out_features_64_cuda_bfloat16 _ 2025-12-04T11:13:50.2488553Z Traceback (most recent call last): 2025-12-04T11:13:50.2488856Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.2488925Z method(*args, **kwargs) 2025-12-04T11:13:50.2489217Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.2489281Z method(*args, **kwargs) 2025-12-04T11:13:50.2489571Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T11:13:50.2489629Z with policy(): 2025-12-04T11:13:50.2489961Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T11:13:50.2490027Z raise RuntimeError(msg) 2025-12-04T11:13:50.2490813Z RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_1_in_features_128_out_features_64_cuda_bfloat16! Caching allocator allocated memory was 0 and is now reported as 8192 on device 0. CUDA driver allocated memory was 230686720 and is now 274726912. 2025-12-04T11:13:50.2490818Z 2025-12-04T11:13:50.2490946Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.2491461Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_1_in_features_128_out_features_64_cuda_bfloat16 2025-12-04T11:13:50.2491466Z 2025-12-04T11:13:50.2491627Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.2491757Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.2491849Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.2492200Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.2492327Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.2492388Z graph_break [] 2025-12-04T11:13:50.2492510Z ----------------------------- Captured stderr call ----------------------------- 2025-12-04T11:13:50.2493199Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/constant_folding.py:256: UserWarning: Unsupported unwinding pattern: Address not in range (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/profiler/unwind/unwind.cpp:219.) 2025-12-04T11:13:50.2493288Z if out == self.unknown_value: 2025-12-04T11:13:50.2493577Z _ TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_1_in_features_128_out_features_64_cuda_bfloat16 _ 2025-12-04T11:13:50.2493652Z Traceback (most recent call last): 2025-12-04T11:13:50.2493943Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.2494006Z method(*args, **kwargs) 2025-12-04T11:13:50.2494294Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.2494355Z method(*args, **kwargs) 2025-12-04T11:13:50.2494640Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T11:13:50.2494747Z with policy(): 2025-12-04T11:13:50.2495122Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T11:13:50.2495193Z raise RuntimeError(msg) 2025-12-04T11:13:50.2495997Z RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_1_in_features_128_out_features_64_cuda_bfloat16! Caching allocator allocated memory was 8192 and is now reported as 16384 on device 0. CUDA driver allocated memory was 274726912 and is now 276824064. 2025-12-04T11:13:50.2496002Z 2025-12-04T11:13:50.2496128Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.2496649Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_1_in_features_128_out_features_64_cuda_bfloat16 2025-12-04T11:13:50.2496653Z 2025-12-04T11:13:50.2496808Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.2496938Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.2497034Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.2497421Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.2497545Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.2497603Z graph_break [] 2025-12-04T11:13:50.2497730Z ----------------------------- Captured stderr call ----------------------------- 2025-12-04T11:13:50.2498416Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/constant_folding.py:256: UserWarning: Unsupported unwinding pattern: Address not in range (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/profiler/unwind/unwind.cpp:219.) 2025-12-04T11:13:50.2498496Z if out == self.unknown_value: 2025-12-04T11:13:50.2498618Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.2498709Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.2498836Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.2499178Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.2499236Z graph_break [] 2025-12-04T11:13:50.2499322Z =================================== FAILURES =================================== 2025-12-04T11:13:50.2499606Z _ TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_1_in_features_128_out_features_64_cuda_bfloat16 _ 2025-12-04T11:13:50.2499681Z Traceback (most recent call last): 2025-12-04T11:13:50.2499972Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.2500036Z method(*args, **kwargs) 2025-12-04T11:13:50.2500328Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.2500391Z method(*args, **kwargs) 2025-12-04T11:13:50.2500675Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T11:13:50.2500734Z with policy(): 2025-12-04T11:13:50.2501024Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T11:13:50.2501092Z raise RuntimeError(msg) 2025-12-04T11:13:50.2501891Z RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_1_in_features_128_out_features_64_cuda_bfloat16! Caching allocator allocated memory was 16384 and is now reported as 24576 on device 0. CUDA driver allocated memory was 276824064 and is now 278921216. 2025-12-04T11:13:50.2501936Z 2025-12-04T11:13:50.2502130Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.2502656Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_1_in_features_128_out_features_64_cuda_bfloat16 2025-12-04T11:13:50.2502662Z 2025-12-04T11:13:50.2502818Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.2502944Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.2503031Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.2503378Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.2503503Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.2503601Z graph_break [] 2025-12-04T11:13:50.2503726Z ----------------------------- Captured stderr call ----------------------------- 2025-12-04T11:13:50.2504411Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/constant_folding.py:256: UserWarning: Unsupported unwinding pattern: Address not in range (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/profiler/unwind/unwind.cpp:219.) 2025-12-04T11:13:50.2504520Z if out == self.unknown_value: 2025-12-04T11:13:50.2504645Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.2504731Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.2504860Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.2505206Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.2505267Z graph_break [] 2025-12-04T11:13:50.2505391Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.2505483Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.2505603Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.2505945Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.2506006Z graph_break [] 2025-12-04T11:13:50.2506491Z - generated xml file: /var/lib/jenkins/workspace/test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-f058047ef4d14b46.xml - 2025-12-04T11:13:50.2506591Z =========================== short test summary info ============================ 2025-12-04T11:13:50.2507868Z FAILED [0.4167s] inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_1_mid_dim_1_in_features_128_out_features_64_cuda_bfloat16 - RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_1_in_features_128_out_features_64_cuda_bfloat16! Caching allocator allocated memory was 16384 and is now reported as 24576 on device 0. CUDA driver allocated memory was 276824064 and is now 278921216. 2025-12-04T11:13:50.2507875Z 2025-12-04T11:13:50.2507999Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.2508507Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_1_in_features_128_out_features_64_cuda_bfloat16 2025-12-04T11:13:50.2508513Z 2025-12-04T11:13:50.2508669Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.2508771Z !!!!!!!!!!!!!!!!!!!!!!!!!! stopping after 1 failures !!!!!!!!!!!!!!!!!!!!!!!!!!! 2025-12-04T11:13:50.2508887Z ================== 1 failed, 57 deselected, 2 rerun in 11.49s ================== 2025-12-04T11:13:50.2508988Z Got exit code 1 2025-12-04T11:13:50.2509530Z FAILED CONSISTENTLY: test/inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_1_mid_dim_1_in_features_128_out_features_64_cuda_bfloat16 2025-12-04T11:13:50.2509779Z Test failed consistently, continuing with the rest of the tests due to continue-through-error being set 2025-12-04T11:13:50.2510039Z W1204 10:49:03.342000 71745 site-packages/torch/_inductor/utils.py:1703] Not enough SMs to use max_autotune_gemm mode 2025-12-04T11:13:50.2510428Z Test results will be stored in test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-5d247f52fbae6573.xml 2025-12-04T11:13:50.2510523Z ============================= test session starts ============================== 2025-12-04T11:13:50.2510729Z platform linux -- Python 3.10.14, pytest-7.3.2, pluggy-1.6.0 -- /opt/conda/envs/py_3.10/bin/python 2025-12-04T11:13:50.2510802Z cachedir: .pytest_cache 2025-12-04T11:13:50.2511107Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow] 2025-12-04T11:13:50.2511219Z rootdir: /var/lib/jenkins/workspace 2025-12-04T11:13:50.2511285Z configfile: pytest.ini 2025-12-04T11:13:50.2511600Z plugins: hypothesis-6.56.4, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-14.0, subtests-0.13.1, xdist-3.3.1, xdoctest-1.3.0, typeguard-4.3.0 2025-12-04T11:13:50.2511739Z collecting ... collected 58 items / 27 deselected / 31 selected 2025-12-04T11:13:50.2511830Z stepcurrent: skipping 27 already run items. 2025-12-04T11:13:50.2511900Z Running 31 items in this shard 2025-12-04T11:13:50.2511904Z 2025-12-04T11:13:50.2512400Z inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_1_mid_dim_1_in_features_128_out_features_65_cuda_bfloat16 ('RERUN', {'yellow': True}) [1.9838s] [ 3%] 2025-12-04T11:13:50.2512888Z inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_1_mid_dim_1_in_features_128_out_features_65_cuda_bfloat16 ('RERUN', {'yellow': True}) [0.5579s] [ 3%] 2025-12-04T11:13:50.2513332Z inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_1_mid_dim_1_in_features_128_out_features_65_cuda_bfloat16 FAILED [0.5510s] [ 3%] 2025-12-04T11:13:50.2513338Z 2025-12-04T11:13:50.2513418Z ==================================== RERUNS ==================================== 2025-12-04T11:13:50.2513707Z _ TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_1_in_features_128_out_features_65_cuda_bfloat16 _ 2025-12-04T11:13:50.2513779Z Traceback (most recent call last): 2025-12-04T11:13:50.2514082Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.2514152Z method(*args, **kwargs) 2025-12-04T11:13:50.2514442Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.2514509Z method(*args, **kwargs) 2025-12-04T11:13:50.2514799Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T11:13:50.2514858Z with policy(): 2025-12-04T11:13:50.2515149Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T11:13:50.2515215Z raise RuntimeError(msg) 2025-12-04T11:13:50.2516005Z RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_1_in_features_128_out_features_65_cuda_bfloat16! Caching allocator allocated memory was 0 and is now reported as 8704 on device 0. CUDA driver allocated memory was 230686720 and is now 274726912. 2025-12-04T11:13:50.2516012Z 2025-12-04T11:13:50.2516136Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.2516757Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_1_in_features_128_out_features_65_cuda_bfloat16 2025-12-04T11:13:50.2516764Z 2025-12-04T11:13:50.2516922Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.2517048Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.2517142Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.2517699Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('benchmarking.InductorBenchmarker.benchmark_gpu', 2), ('pad_mm_bench', 1), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.2517828Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.2517890Z graph_break [] 2025-12-04T11:13:50.2518180Z _ TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_1_in_features_128_out_features_65_cuda_bfloat16 _ 2025-12-04T11:13:50.2518255Z Traceback (most recent call last): 2025-12-04T11:13:50.2518549Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.2518650Z method(*args, **kwargs) 2025-12-04T11:13:50.2518940Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.2519002Z method(*args, **kwargs) 2025-12-04T11:13:50.2519292Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T11:13:50.2519357Z with policy(): 2025-12-04T11:13:50.2519645Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T11:13:50.2519711Z raise RuntimeError(msg) 2025-12-04T11:13:50.2520520Z RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_1_in_features_128_out_features_65_cuda_bfloat16! Caching allocator allocated memory was 8704 and is now reported as 17408 on device 0. CUDA driver allocated memory was 274726912 and is now 276824064. 2025-12-04T11:13:50.2520525Z 2025-12-04T11:13:50.2520646Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.2521163Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_1_in_features_128_out_features_65_cuda_bfloat16 2025-12-04T11:13:50.2521167Z 2025-12-04T11:13:50.2521319Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.2521448Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.2521539Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.2522095Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('benchmarking.InductorBenchmarker.benchmark_gpu', 2), ('pad_mm_bench', 1), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.2522227Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.2522284Z graph_break [] 2025-12-04T11:13:50.2522408Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.2522497Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.2522616Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.2523157Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('benchmarking.InductorBenchmarker.benchmark_gpu', 2), ('pad_mm_bench', 1), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.2523277Z graph_break [] 2025-12-04T11:13:50.2523361Z =================================== FAILURES =================================== 2025-12-04T11:13:50.2523710Z _ TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_1_in_features_128_out_features_65_cuda_bfloat16 _ 2025-12-04T11:13:50.2523784Z Traceback (most recent call last): 2025-12-04T11:13:50.2524082Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.2524144Z method(*args, **kwargs) 2025-12-04T11:13:50.2524434Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.2524501Z method(*args, **kwargs) 2025-12-04T11:13:50.2524792Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T11:13:50.2524854Z with policy(): 2025-12-04T11:13:50.2525142Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T11:13:50.2525210Z raise RuntimeError(msg) 2025-12-04T11:13:50.2526015Z RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_1_in_features_128_out_features_65_cuda_bfloat16! Caching allocator allocated memory was 17408 and is now reported as 26112 on device 0. CUDA driver allocated memory was 276824064 and is now 278921216. 2025-12-04T11:13:50.2526056Z 2025-12-04T11:13:50.2526178Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.2526696Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_1_in_features_128_out_features_65_cuda_bfloat16 2025-12-04T11:13:50.2526700Z 2025-12-04T11:13:50.2526852Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.2526976Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.2527070Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.2527610Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('benchmarking.InductorBenchmarker.benchmark_gpu', 2), ('pad_mm_bench', 1), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.2527739Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.2527796Z graph_break [] 2025-12-04T11:13:50.2527916Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.2528007Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.2528136Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.2528674Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('benchmarking.InductorBenchmarker.benchmark_gpu', 2), ('pad_mm_bench', 1), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.2528736Z graph_break [] 2025-12-04T11:13:50.2528858Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.2528949Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.2529068Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.2529599Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('benchmarking.InductorBenchmarker.benchmark_gpu', 2), ('pad_mm_bench', 1), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.2529659Z graph_break [] 2025-12-04T11:13:50.2530142Z - generated xml file: /var/lib/jenkins/workspace/test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-5d247f52fbae6573.xml - 2025-12-04T11:13:50.2530287Z =========================== short test summary info ============================ 2025-12-04T11:13:50.2531620Z FAILED [0.5510s] inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_1_mid_dim_1_in_features_128_out_features_65_cuda_bfloat16 - RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_1_in_features_128_out_features_65_cuda_bfloat16! Caching allocator allocated memory was 17408 and is now reported as 26112 on device 0. CUDA driver allocated memory was 276824064 and is now 278921216. 2025-12-04T11:13:50.2531626Z 2025-12-04T11:13:50.2531753Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.2532267Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_1_in_features_128_out_features_65_cuda_bfloat16 2025-12-04T11:13:50.2532272Z 2025-12-04T11:13:50.2532429Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.2532539Z !!!!!!!!!!!!!!!!!!!!!!!!!! stopping after 1 failures !!!!!!!!!!!!!!!!!!!!!!!!!!! 2025-12-04T11:13:50.2532688Z ================== 1 failed, 27 deselected, 2 rerun in 3.12s =================== 2025-12-04T11:13:50.2532749Z Got exit code 1 2025-12-04T11:13:50.2532814Z Retrying single test... 2025-12-04T11:13:50.2533074Z W1204 10:49:13.423000 71927 site-packages/torch/_inductor/utils.py:1703] Not enough SMs to use max_autotune_gemm mode 2025-12-04T11:13:50.2533465Z Test results will be stored in test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-b685fc5b32b50b90.xml 2025-12-04T11:13:50.2533558Z ============================= test session starts ============================== 2025-12-04T11:13:50.2533765Z platform linux -- Python 3.10.14, pytest-7.3.2, pluggy-1.6.0 -- /opt/conda/envs/py_3.10/bin/python 2025-12-04T11:13:50.2533833Z cachedir: .pytest_cache 2025-12-04T11:13:50.2534137Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow] 2025-12-04T11:13:50.2534217Z rootdir: /var/lib/jenkins/workspace 2025-12-04T11:13:50.2534284Z configfile: pytest.ini 2025-12-04T11:13:50.2534596Z plugins: hypothesis-6.56.4, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-14.0, subtests-0.13.1, xdist-3.3.1, xdoctest-1.3.0, typeguard-4.3.0 2025-12-04T11:13:50.2534725Z collecting ... collected 58 items / 57 deselected / 1 selected 2025-12-04T11:13:50.2535289Z stepcurrent: skipping 27 already run items. Running only test/inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_1_mid_dim_1_in_features_128_out_features_65_cuda_bfloat16 2025-12-04T11:13:50.2535363Z Running 1 items in this shard 2025-12-04T11:13:50.2535367Z 2025-12-04T11:13:50.2536091Z inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_1_mid_dim_1_in_features_128_out_features_65_cuda_bfloat16 [W1204 10:49:15.081806598 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2536097Z 2025-12-04T11:13:50.2536394Z [W1204 10:49:24.289734947 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2536398Z 2025-12-04T11:13:50.2536688Z [W1204 10:49:24.289984232 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2536692Z 2025-12-04T11:13:50.2536981Z [W1204 10:49:24.295649047 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2536984Z 2025-12-04T11:13:50.2537268Z [W1204 10:49:24.296247000 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2537321Z 2025-12-04T11:13:50.2537674Z [W1204 10:49:24.296419364 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2537682Z 2025-12-04T11:13:50.2537967Z [W1204 10:49:24.301734921 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2537973Z 2025-12-04T11:13:50.2538262Z [W1204 10:49:24.302246392 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2538265Z 2025-12-04T11:13:50.2538553Z [W1204 10:49:24.302404906 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2538556Z 2025-12-04T11:13:50.2538636Z ('RERUN', {'yellow': True}) [11.2071s] [100%] 2025-12-04T11:13:50.2539364Z inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_1_mid_dim_1_in_features_128_out_features_65_cuda_bfloat16 [W1204 10:49:25.113998849 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2539369Z 2025-12-04T11:13:50.2539692Z [W1204 10:49:25.114550251 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2539696Z 2025-12-04T11:13:50.2539984Z [W1204 10:49:25.114690264 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2539987Z 2025-12-04T11:13:50.2540271Z [W1204 10:49:25.117638709 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2540274Z 2025-12-04T11:13:50.2540558Z [W1204 10:49:25.118100240 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2540565Z 2025-12-04T11:13:50.2540856Z [W1204 10:49:25.118238303 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2540864Z 2025-12-04T11:13:50.2541150Z [W1204 10:49:25.122876656 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2541155Z 2025-12-04T11:13:50.2541443Z [W1204 10:49:25.123343926 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2541447Z 2025-12-04T11:13:50.2541729Z [W1204 10:49:25.123476449 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2541732Z 2025-12-04T11:13:50.2541827Z ('RERUN', {'yellow': True}) [0.5032s] [100%] 2025-12-04T11:13:50.2542546Z inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_1_mid_dim_1_in_features_128_out_features_65_cuda_bfloat16 [W1204 10:49:25.611415750 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2542553Z 2025-12-04T11:13:50.2542844Z [W1204 10:49:25.611961332 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2542849Z 2025-12-04T11:13:50.2543135Z [W1204 10:49:25.612099275 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2543138Z 2025-12-04T11:13:50.2543426Z [W1204 10:49:25.614996799 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2543430Z 2025-12-04T11:13:50.2543763Z [W1204 10:49:25.615446689 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2543766Z 2025-12-04T11:13:50.2544103Z [W1204 10:49:25.615582362 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2544110Z 2025-12-04T11:13:50.2544460Z [W1204 10:49:25.619988520 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2544466Z 2025-12-04T11:13:50.2544753Z [W1204 10:49:25.620504811 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2544756Z 2025-12-04T11:13:50.2545057Z [W1204 10:49:25.620652324 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2545061Z 2025-12-04T11:13:50.2545122Z FAILED [0.5006s] [100%] 2025-12-04T11:13:50.2545126Z 2025-12-04T11:13:50.2545210Z ==================================== RERUNS ==================================== 2025-12-04T11:13:50.2545495Z _ TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_1_in_features_128_out_features_65_cuda_bfloat16 _ 2025-12-04T11:13:50.2545570Z Traceback (most recent call last): 2025-12-04T11:13:50.2545878Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.2545995Z method(*args, **kwargs) 2025-12-04T11:13:50.2546290Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.2546355Z method(*args, **kwargs) 2025-12-04T11:13:50.2546650Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T11:13:50.2546709Z with policy(): 2025-12-04T11:13:50.2547013Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T11:13:50.2547081Z raise RuntimeError(msg) 2025-12-04T11:13:50.2547880Z RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_1_in_features_128_out_features_65_cuda_bfloat16! Caching allocator allocated memory was 0 and is now reported as 8704 on device 0. CUDA driver allocated memory was 230686720 and is now 274726912. 2025-12-04T11:13:50.2547886Z 2025-12-04T11:13:50.2548020Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.2548538Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_1_in_features_128_out_features_65_cuda_bfloat16 2025-12-04T11:13:50.2548542Z 2025-12-04T11:13:50.2548699Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.2548835Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.2548931Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.2549481Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('benchmarking.InductorBenchmarker.benchmark_gpu', 2), ('pad_mm_bench', 1), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.2549613Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.2549673Z graph_break [] 2025-12-04T11:13:50.2549802Z ----------------------------- Captured stderr call ----------------------------- 2025-12-04T11:13:50.2550489Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/constant_folding.py:256: UserWarning: Unsupported unwinding pattern: Address not in range (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/profiler/unwind/unwind.cpp:219.) 2025-12-04T11:13:50.2550560Z if out == self.unknown_value: 2025-12-04T11:13:50.2550850Z _ TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_1_in_features_128_out_features_65_cuda_bfloat16 _ 2025-12-04T11:13:50.2550979Z Traceback (most recent call last): 2025-12-04T11:13:50.2551339Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.2551406Z method(*args, **kwargs) 2025-12-04T11:13:50.2551699Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.2551767Z method(*args, **kwargs) 2025-12-04T11:13:50.2552054Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T11:13:50.2552112Z with policy(): 2025-12-04T11:13:50.2552408Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T11:13:50.2552475Z raise RuntimeError(msg) 2025-12-04T11:13:50.2553281Z RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_1_in_features_128_out_features_65_cuda_bfloat16! Caching allocator allocated memory was 8704 and is now reported as 17408 on device 0. CUDA driver allocated memory was 274726912 and is now 276824064. 2025-12-04T11:13:50.2553287Z 2025-12-04T11:13:50.2553449Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.2553960Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_1_in_features_128_out_features_65_cuda_bfloat16 2025-12-04T11:13:50.2553969Z 2025-12-04T11:13:50.2554123Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.2554250Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.2554346Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.2554892Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('benchmarking.InductorBenchmarker.benchmark_gpu', 2), ('pad_mm_bench', 1), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.2555024Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.2555087Z graph_break [] 2025-12-04T11:13:50.2555210Z ----------------------------- Captured stderr call ----------------------------- 2025-12-04T11:13:50.2555896Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/constant_folding.py:256: UserWarning: Unsupported unwinding pattern: Address not in range (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/profiler/unwind/unwind.cpp:219.) 2025-12-04T11:13:50.2555965Z if out == self.unknown_value: 2025-12-04T11:13:50.2556086Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.2556179Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.2556299Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.2556844Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('benchmarking.InductorBenchmarker.benchmark_gpu', 2), ('pad_mm_bench', 1), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.2556908Z graph_break [] 2025-12-04T11:13:50.2556991Z =================================== FAILURES =================================== 2025-12-04T11:13:50.2557279Z _ TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_1_in_features_128_out_features_65_cuda_bfloat16 _ 2025-12-04T11:13:50.2557353Z Traceback (most recent call last): 2025-12-04T11:13:50.2557663Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.2557729Z method(*args, **kwargs) 2025-12-04T11:13:50.2558020Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.2558128Z method(*args, **kwargs) 2025-12-04T11:13:50.2558479Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T11:13:50.2558539Z with policy(): 2025-12-04T11:13:50.2558833Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T11:13:50.2558901Z raise RuntimeError(msg) 2025-12-04T11:13:50.2559718Z RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_1_in_features_128_out_features_65_cuda_bfloat16! Caching allocator allocated memory was 17408 and is now reported as 26112 on device 0. CUDA driver allocated memory was 276824064 and is now 278921216. 2025-12-04T11:13:50.2559724Z 2025-12-04T11:13:50.2559852Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.2560362Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_1_in_features_128_out_features_65_cuda_bfloat16 2025-12-04T11:13:50.2560372Z 2025-12-04T11:13:50.2560527Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.2560689Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.2560783Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.2561324Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('benchmarking.InductorBenchmarker.benchmark_gpu', 2), ('pad_mm_bench', 1), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.2561449Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.2561507Z graph_break [] 2025-12-04T11:13:50.2561630Z ----------------------------- Captured stderr call ----------------------------- 2025-12-04T11:13:50.2562314Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/constant_folding.py:256: UserWarning: Unsupported unwinding pattern: Address not in range (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/profiler/unwind/unwind.cpp:219.) 2025-12-04T11:13:50.2562387Z if out == self.unknown_value: 2025-12-04T11:13:50.2562509Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.2562601Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.2562724Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.2563263Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('benchmarking.InductorBenchmarker.benchmark_gpu', 2), ('pad_mm_bench', 1), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.2563322Z graph_break [] 2025-12-04T11:13:50.2563442Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.2563532Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.2563655Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.2564190Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('benchmarking.InductorBenchmarker.benchmark_gpu', 2), ('pad_mm_bench', 1), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.2564250Z graph_break [] 2025-12-04T11:13:50.2564733Z - generated xml file: /var/lib/jenkins/workspace/test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-b685fc5b32b50b90.xml - 2025-12-04T11:13:50.2564835Z =========================== short test summary info ============================ 2025-12-04T11:13:50.2566178Z FAILED [0.5006s] inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_1_mid_dim_1_in_features_128_out_features_65_cuda_bfloat16 - RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_1_in_features_128_out_features_65_cuda_bfloat16! Caching allocator allocated memory was 17408 and is now reported as 26112 on device 0. CUDA driver allocated memory was 276824064 and is now 278921216. 2025-12-04T11:13:50.2566216Z 2025-12-04T11:13:50.2566346Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.2566857Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_1_in_features_128_out_features_65_cuda_bfloat16 2025-12-04T11:13:50.2566861Z 2025-12-04T11:13:50.2567026Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.2567134Z !!!!!!!!!!!!!!!!!!!!!!!!!! stopping after 1 failures !!!!!!!!!!!!!!!!!!!!!!!!!!! 2025-12-04T11:13:50.2567250Z ================== 1 failed, 57 deselected, 2 rerun in 12.24s ================== 2025-12-04T11:13:50.2567314Z Got exit code 1 2025-12-04T11:13:50.2567409Z Retrying single test... 2025-12-04T11:13:50.2567876Z W1204 10:49:32.479000 72114 site-packages/torch/_inductor/utils.py:1703] Not enough SMs to use max_autotune_gemm mode 2025-12-04T11:13:50.2568417Z Test results will be stored in test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-bc2410b75aa40646.xml 2025-12-04T11:13:50.2568587Z ============================= test session starts ============================== 2025-12-04T11:13:50.2568844Z platform linux -- Python 3.10.14, pytest-7.3.2, pluggy-1.6.0 -- /opt/conda/envs/py_3.10/bin/python 2025-12-04T11:13:50.2568912Z cachedir: .pytest_cache 2025-12-04T11:13:50.2569221Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow] 2025-12-04T11:13:50.2569330Z rootdir: /var/lib/jenkins/workspace 2025-12-04T11:13:50.2569454Z configfile: pytest.ini 2025-12-04T11:13:50.2569840Z plugins: hypothesis-6.56.4, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-14.0, subtests-0.13.1, xdist-3.3.1, xdoctest-1.3.0, typeguard-4.3.0 2025-12-04T11:13:50.2569975Z collecting ... collected 58 items / 57 deselected / 1 selected 2025-12-04T11:13:50.2570547Z stepcurrent: skipping 27 already run items. Running only test/inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_1_mid_dim_1_in_features_128_out_features_65_cuda_bfloat16 2025-12-04T11:13:50.2570620Z Running 1 items in this shard 2025-12-04T11:13:50.2570625Z 2025-12-04T11:13:50.2571357Z inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_1_mid_dim_1_in_features_128_out_features_65_cuda_bfloat16 [W1204 10:49:34.141034041 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2571361Z 2025-12-04T11:13:50.2571662Z [W1204 10:49:43.190516438 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2571668Z 2025-12-04T11:13:50.2571959Z [W1204 10:49:43.190761464 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2571964Z 2025-12-04T11:13:50.2572253Z [W1204 10:49:43.196630333 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2572257Z 2025-12-04T11:13:50.2572542Z [W1204 10:49:43.197200906 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2572545Z 2025-12-04T11:13:50.2572830Z [W1204 10:49:43.197376690 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2572837Z 2025-12-04T11:13:50.2573121Z [W1204 10:49:43.202813980 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2573181Z 2025-12-04T11:13:50.2573561Z [W1204 10:49:43.203340132 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2573567Z 2025-12-04T11:13:50.2573858Z [W1204 10:49:43.203505955 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2573862Z 2025-12-04T11:13:50.2573944Z ('RERUN', {'yellow': True}) [11.0574s] [100%] 2025-12-04T11:13:50.2574659Z inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_1_mid_dim_1_in_features_128_out_features_65_cuda_bfloat16 [W1204 10:49:44.021640134 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2574664Z 2025-12-04T11:13:50.2574949Z [W1204 10:49:44.022199016 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2574954Z 2025-12-04T11:13:50.2575244Z [W1204 10:49:44.022339659 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2575283Z 2025-12-04T11:13:50.2575569Z [W1204 10:49:44.025279324 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2575572Z 2025-12-04T11:13:50.2575860Z [W1204 10:49:44.025738414 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2575864Z 2025-12-04T11:13:50.2576149Z [W1204 10:49:44.025876347 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2576152Z 2025-12-04T11:13:50.2576436Z [W1204 10:49:44.030475239 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2576443Z 2025-12-04T11:13:50.2576732Z [W1204 10:49:44.030939529 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2576737Z 2025-12-04T11:13:50.2577018Z [W1204 10:49:44.031073002 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2577022Z 2025-12-04T11:13:50.2577106Z ('RERUN', {'yellow': True}) [0.5054s] [100%] 2025-12-04T11:13:50.2577947Z inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_1_mid_dim_1_in_features_128_out_features_65_cuda_bfloat16 [W1204 10:49:44.523867211 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2577951Z 2025-12-04T11:13:50.2578243Z [W1204 10:49:44.524427923 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2578248Z 2025-12-04T11:13:50.2578536Z [W1204 10:49:44.524567526 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2578541Z 2025-12-04T11:13:50.2578829Z [W1204 10:49:44.527488491 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2578833Z 2025-12-04T11:13:50.2579119Z [W1204 10:49:44.527945971 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2579122Z 2025-12-04T11:13:50.2579409Z [W1204 10:49:44.528082674 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2579412Z 2025-12-04T11:13:50.2579696Z [W1204 10:49:44.532701776 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2579769Z 2025-12-04T11:13:50.2580153Z [W1204 10:49:44.533163986 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2580159Z 2025-12-04T11:13:50.2580453Z [W1204 10:49:44.533298289 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2580457Z 2025-12-04T11:13:50.2580519Z FAILED [0.5004s] [100%] 2025-12-04T11:13:50.2580522Z 2025-12-04T11:13:50.2580610Z ==================================== RERUNS ==================================== 2025-12-04T11:13:50.2580898Z _ TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_1_in_features_128_out_features_65_cuda_bfloat16 _ 2025-12-04T11:13:50.2580973Z Traceback (most recent call last): 2025-12-04T11:13:50.2581281Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.2581350Z method(*args, **kwargs) 2025-12-04T11:13:50.2581651Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.2581774Z method(*args, **kwargs) 2025-12-04T11:13:50.2582062Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T11:13:50.2582127Z with policy(): 2025-12-04T11:13:50.2582418Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T11:13:50.2582487Z raise RuntimeError(msg) 2025-12-04T11:13:50.2583280Z RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_1_in_features_128_out_features_65_cuda_bfloat16! Caching allocator allocated memory was 0 and is now reported as 8704 on device 0. CUDA driver allocated memory was 230686720 and is now 274726912. 2025-12-04T11:13:50.2583286Z 2025-12-04T11:13:50.2583414Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.2584000Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_1_in_features_128_out_features_65_cuda_bfloat16 2025-12-04T11:13:50.2584006Z 2025-12-04T11:13:50.2584166Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.2584295Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.2584390Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.2584934Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('benchmarking.InductorBenchmarker.benchmark_gpu', 2), ('pad_mm_bench', 1), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.2585064Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.2585126Z graph_break [] 2025-12-04T11:13:50.2585255Z ----------------------------- Captured stderr call ----------------------------- 2025-12-04T11:13:50.2585947Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/constant_folding.py:256: UserWarning: Unsupported unwinding pattern: Address not in range (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/profiler/unwind/unwind.cpp:219.) 2025-12-04T11:13:50.2586022Z if out == self.unknown_value: 2025-12-04T11:13:50.2586313Z _ TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_1_in_features_128_out_features_65_cuda_bfloat16 _ 2025-12-04T11:13:50.2586387Z Traceback (most recent call last): 2025-12-04T11:13:50.2586690Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.2586753Z method(*args, **kwargs) 2025-12-04T11:13:50.2587090Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.2587222Z method(*args, **kwargs) 2025-12-04T11:13:50.2587511Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T11:13:50.2587576Z with policy(): 2025-12-04T11:13:50.2587891Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T11:13:50.2587961Z raise RuntimeError(msg) 2025-12-04T11:13:50.2588763Z RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_1_in_features_128_out_features_65_cuda_bfloat16! Caching allocator allocated memory was 8704 and is now reported as 17408 on device 0. CUDA driver allocated memory was 274726912 and is now 276824064. 2025-12-04T11:13:50.2588767Z 2025-12-04T11:13:50.2588893Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.2589421Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_1_in_features_128_out_features_65_cuda_bfloat16 2025-12-04T11:13:50.2589461Z 2025-12-04T11:13:50.2589621Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.2589747Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.2589844Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.2590390Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('benchmarking.InductorBenchmarker.benchmark_gpu', 2), ('pad_mm_bench', 1), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.2590524Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.2590584Z graph_break [] 2025-12-04T11:13:50.2590710Z ----------------------------- Captured stderr call ----------------------------- 2025-12-04T11:13:50.2591401Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/constant_folding.py:256: UserWarning: Unsupported unwinding pattern: Address not in range (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/profiler/unwind/unwind.cpp:219.) 2025-12-04T11:13:50.2591473Z if out == self.unknown_value: 2025-12-04T11:13:50.2591596Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.2591691Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.2591814Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.2592352Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('benchmarking.InductorBenchmarker.benchmark_gpu', 2), ('pad_mm_bench', 1), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.2592412Z graph_break [] 2025-12-04T11:13:50.2592506Z =================================== FAILURES =================================== 2025-12-04T11:13:50.2592804Z _ TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_1_in_features_128_out_features_65_cuda_bfloat16 _ 2025-12-04T11:13:50.2592877Z Traceback (most recent call last): 2025-12-04T11:13:50.2593177Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.2593240Z method(*args, **kwargs) 2025-12-04T11:13:50.2593528Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.2593595Z method(*args, **kwargs) 2025-12-04T11:13:50.2593884Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T11:13:50.2593944Z with policy(): 2025-12-04T11:13:50.2594282Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T11:13:50.2594415Z raise RuntimeError(msg) 2025-12-04T11:13:50.2595232Z RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_1_in_features_128_out_features_65_cuda_bfloat16! Caching allocator allocated memory was 17408 and is now reported as 26112 on device 0. CUDA driver allocated memory was 276824064 and is now 278921216. 2025-12-04T11:13:50.2595240Z 2025-12-04T11:13:50.2595365Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.2595876Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_1_in_features_128_out_features_65_cuda_bfloat16 2025-12-04T11:13:50.2595880Z 2025-12-04T11:13:50.2596036Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.2596161Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.2596257Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.2596798Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('benchmarking.InductorBenchmarker.benchmark_gpu', 2), ('pad_mm_bench', 1), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.2596964Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.2597022Z graph_break [] 2025-12-04T11:13:50.2597145Z ----------------------------- Captured stderr call ----------------------------- 2025-12-04T11:13:50.2597835Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/constant_folding.py:256: UserWarning: Unsupported unwinding pattern: Address not in range (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/profiler/unwind/unwind.cpp:219.) 2025-12-04T11:13:50.2597906Z if out == self.unknown_value: 2025-12-04T11:13:50.2598032Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.2598125Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.2598257Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.2598799Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('benchmarking.InductorBenchmarker.benchmark_gpu', 2), ('pad_mm_bench', 1), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.2598857Z graph_break [] 2025-12-04T11:13:50.2598978Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.2599070Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.2599194Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.2599746Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('benchmarking.InductorBenchmarker.benchmark_gpu', 2), ('pad_mm_bench', 1), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.2599807Z graph_break [] 2025-12-04T11:13:50.2600288Z - generated xml file: /var/lib/jenkins/workspace/test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-bc2410b75aa40646.xml - 2025-12-04T11:13:50.2600392Z =========================== short test summary info ============================ 2025-12-04T11:13:50.2601670Z FAILED [0.5004s] inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_1_mid_dim_1_in_features_128_out_features_65_cuda_bfloat16 - RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_1_in_features_128_out_features_65_cuda_bfloat16! Caching allocator allocated memory was 17408 and is now reported as 26112 on device 0. CUDA driver allocated memory was 276824064 and is now 278921216. 2025-12-04T11:13:50.2601797Z 2025-12-04T11:13:50.2602017Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.2602533Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_1_in_features_128_out_features_65_cuda_bfloat16 2025-12-04T11:13:50.2602538Z 2025-12-04T11:13:50.2602699Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.2602802Z !!!!!!!!!!!!!!!!!!!!!!!!!! stopping after 1 failures !!!!!!!!!!!!!!!!!!!!!!!!!!! 2025-12-04T11:13:50.2602918Z ================== 1 failed, 57 deselected, 2 rerun in 12.09s ================== 2025-12-04T11:13:50.2602982Z Got exit code 1 2025-12-04T11:13:50.2603464Z FAILED CONSISTENTLY: test/inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_1_mid_dim_1_in_features_128_out_features_65_cuda_bfloat16 2025-12-04T11:13:50.2603716Z Test failed consistently, continuing with the rest of the tests due to continue-through-error being set 2025-12-04T11:13:50.2603984Z W1204 10:49:51.361000 72301 site-packages/torch/_inductor/utils.py:1703] Not enough SMs to use max_autotune_gemm mode 2025-12-04T11:13:50.2604407Z Test results will be stored in test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-71cfec7d40715d91.xml 2025-12-04T11:13:50.2604504Z ============================= test session starts ============================== 2025-12-04T11:13:50.2604710Z platform linux -- Python 3.10.14, pytest-7.3.2, pluggy-1.6.0 -- /opt/conda/envs/py_3.10/bin/python 2025-12-04T11:13:50.2604779Z cachedir: .pytest_cache 2025-12-04T11:13:50.2605082Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow] 2025-12-04T11:13:50.2605157Z rootdir: /var/lib/jenkins/workspace 2025-12-04T11:13:50.2605230Z configfile: pytest.ini 2025-12-04T11:13:50.2605547Z plugins: hypothesis-6.56.4, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-14.0, subtests-0.13.1, xdist-3.3.1, xdoctest-1.3.0, typeguard-4.3.0 2025-12-04T11:13:50.2605678Z collecting ... collected 58 items / 28 deselected / 30 selected 2025-12-04T11:13:50.2605781Z stepcurrent: skipping 28 already run items. 2025-12-04T11:13:50.2605855Z Running 30 items in this shard 2025-12-04T11:13:50.2605858Z 2025-12-04T11:13:50.2606358Z inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_1_mid_dim_1_in_features_144_out_features_1024_cuda_bfloat16 ('RERUN', {'yellow': True}) [1.9014s] [ 3%] 2025-12-04T11:13:50.2606848Z inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_1_mid_dim_1_in_features_144_out_features_1024_cuda_bfloat16 ('RERUN', {'yellow': True}) [0.4962s] [ 3%] 2025-12-04T11:13:50.2607292Z inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_1_mid_dim_1_in_features_144_out_features_1024_cuda_bfloat16 FAILED [0.4975s] [ 3%] 2025-12-04T11:13:50.2607298Z 2025-12-04T11:13:50.2607382Z ==================================== RERUNS ==================================== 2025-12-04T11:13:50.2607681Z _ TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_1_in_features_144_out_features_1024_cuda_bfloat16 _ 2025-12-04T11:13:50.2607760Z Traceback (most recent call last): 2025-12-04T11:13:50.2608064Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.2608129Z method(*args, **kwargs) 2025-12-04T11:13:50.2608425Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.2608489Z method(*args, **kwargs) 2025-12-04T11:13:50.2608782Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T11:13:50.2608892Z with policy(): 2025-12-04T11:13:50.2609254Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T11:13:50.2609325Z raise RuntimeError(msg) 2025-12-04T11:13:50.2610128Z RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_1_in_features_144_out_features_1024_cuda_bfloat16! Caching allocator allocated memory was 0 and is now reported as 147456 on device 0. CUDA driver allocated memory was 230686720 and is now 274726912. 2025-12-04T11:13:50.2610132Z 2025-12-04T11:13:50.2610260Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.2610779Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_1_in_features_144_out_features_1024_cuda_bfloat16 2025-12-04T11:13:50.2610784Z 2025-12-04T11:13:50.2610944Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.2611072Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.2611199Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.2611554Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.2611684Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.2611745Z graph_break [] 2025-12-04T11:13:50.2612035Z _ TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_1_in_features_144_out_features_1024_cuda_bfloat16 _ 2025-12-04T11:13:50.2612108Z Traceback (most recent call last): 2025-12-04T11:13:50.2612406Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.2612472Z method(*args, **kwargs) 2025-12-04T11:13:50.2612762Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.2612833Z method(*args, **kwargs) 2025-12-04T11:13:50.2613124Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T11:13:50.2613185Z with policy(): 2025-12-04T11:13:50.2613488Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T11:13:50.2613553Z raise RuntimeError(msg) 2025-12-04T11:13:50.2614369Z RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_1_in_features_144_out_features_1024_cuda_bfloat16! Caching allocator allocated memory was 147456 and is now reported as 294912 on device 0. CUDA driver allocated memory was 274726912 and is now 276824064. 2025-12-04T11:13:50.2614375Z 2025-12-04T11:13:50.2614498Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.2615029Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_1_in_features_144_out_features_1024_cuda_bfloat16 2025-12-04T11:13:50.2615037Z 2025-12-04T11:13:50.2615196Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.2615321Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.2615416Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.2615764Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.2615889Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.2615990Z graph_break [] 2025-12-04T11:13:50.2616120Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.2616276Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.2616396Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.2616738Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.2616804Z graph_break [] 2025-12-04T11:13:50.2616889Z =================================== FAILURES =================================== 2025-12-04T11:13:50.2617179Z _ TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_1_in_features_144_out_features_1024_cuda_bfloat16 _ 2025-12-04T11:13:50.2617250Z Traceback (most recent call last): 2025-12-04T11:13:50.2617544Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.2617613Z method(*args, **kwargs) 2025-12-04T11:13:50.2617905Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.2618005Z method(*args, **kwargs) 2025-12-04T11:13:50.2618297Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T11:13:50.2618356Z with policy(): 2025-12-04T11:13:50.2618649Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T11:13:50.2618714Z raise RuntimeError(msg) 2025-12-04T11:13:50.2619535Z RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_1_in_features_144_out_features_1024_cuda_bfloat16! Caching allocator allocated memory was 294912 and is now reported as 442368 on device 0. CUDA driver allocated memory was 276824064 and is now 278921216. 2025-12-04T11:13:50.2619545Z 2025-12-04T11:13:50.2619669Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.2620186Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_1_in_features_144_out_features_1024_cuda_bfloat16 2025-12-04T11:13:50.2620191Z 2025-12-04T11:13:50.2620349Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.2620475Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.2620568Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.2620910Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.2621033Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.2621095Z graph_break [] 2025-12-04T11:13:50.2621219Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.2621309Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.2621436Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.2621777Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.2621838Z graph_break [] 2025-12-04T11:13:50.2621961Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.2622048Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.2622172Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.2622510Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.2622611Z graph_break [] 2025-12-04T11:13:50.2623174Z - generated xml file: /var/lib/jenkins/workspace/test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-71cfec7d40715d91.xml - 2025-12-04T11:13:50.2623279Z =========================== short test summary info ============================ 2025-12-04T11:13:50.2624613Z FAILED [0.4975s] inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_1_mid_dim_1_in_features_144_out_features_1024_cuda_bfloat16 - RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_1_in_features_144_out_features_1024_cuda_bfloat16! Caching allocator allocated memory was 294912 and is now reported as 442368 on device 0. CUDA driver allocated memory was 276824064 and is now 278921216. 2025-12-04T11:13:50.2624618Z 2025-12-04T11:13:50.2624738Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.2625259Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_1_in_features_144_out_features_1024_cuda_bfloat16 2025-12-04T11:13:50.2625299Z 2025-12-04T11:13:50.2625456Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.2625564Z !!!!!!!!!!!!!!!!!!!!!!!!!! stopping after 1 failures !!!!!!!!!!!!!!!!!!!!!!!!!!! 2025-12-04T11:13:50.2625678Z ================== 1 failed, 28 deselected, 2 rerun in 2.92s =================== 2025-12-04T11:13:50.2625737Z Got exit code 1 2025-12-04T11:13:50.2625806Z Retrying single test... 2025-12-04T11:13:50.2626068Z W1204 10:50:01.389000 72490 site-packages/torch/_inductor/utils.py:1703] Not enough SMs to use max_autotune_gemm mode 2025-12-04T11:13:50.2626451Z Test results will be stored in test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-c9dbc1d234ce26dc.xml 2025-12-04T11:13:50.2626552Z ============================= test session starts ============================== 2025-12-04T11:13:50.2626762Z platform linux -- Python 3.10.14, pytest-7.3.2, pluggy-1.6.0 -- /opt/conda/envs/py_3.10/bin/python 2025-12-04T11:13:50.2626834Z cachedir: .pytest_cache 2025-12-04T11:13:50.2627140Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow] 2025-12-04T11:13:50.2627215Z rootdir: /var/lib/jenkins/workspace 2025-12-04T11:13:50.2627286Z configfile: pytest.ini 2025-12-04T11:13:50.2627602Z plugins: hypothesis-6.56.4, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-14.0, subtests-0.13.1, xdist-3.3.1, xdoctest-1.3.0, typeguard-4.3.0 2025-12-04T11:13:50.2627732Z collecting ... collected 58 items / 57 deselected / 1 selected 2025-12-04T11:13:50.2628303Z stepcurrent: skipping 28 already run items. Running only test/inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_1_mid_dim_1_in_features_144_out_features_1024_cuda_bfloat16 2025-12-04T11:13:50.2628376Z Running 1 items in this shard 2025-12-04T11:13:50.2628382Z 2025-12-04T11:13:50.2629117Z inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_1_mid_dim_1_in_features_144_out_features_1024_cuda_bfloat16 [W1204 10:50:02.529717115 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2629122Z 2025-12-04T11:13:50.2629428Z [W1204 10:50:11.765878474 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2629432Z 2025-12-04T11:13:50.2629723Z [W1204 10:50:11.766119529 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2629727Z 2025-12-04T11:13:50.2630012Z [W1204 10:50:11.771858306 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2630075Z 2025-12-04T11:13:50.2630447Z [W1204 10:50:11.772411408 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2630453Z 2025-12-04T11:13:50.2630739Z [W1204 10:50:11.772570752 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2630742Z 2025-12-04T11:13:50.2631028Z [W1204 10:50:11.777924950 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2631036Z 2025-12-04T11:13:50.2631321Z [W1204 10:50:11.778445002 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2631324Z 2025-12-04T11:13:50.2631607Z [W1204 10:50:11.778602365 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2631612Z 2025-12-04T11:13:50.2631697Z ('RERUN', {'yellow': True}) [11.1714s] [100%] 2025-12-04T11:13:50.2632424Z inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_1_mid_dim_1_in_features_144_out_features_1024_cuda_bfloat16 [W1204 10:50:13.003689704 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2632463Z 2025-12-04T11:13:50.2632754Z [W1204 10:50:13.004224356 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2632757Z 2025-12-04T11:13:50.2633043Z [W1204 10:50:13.004363409 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2633046Z 2025-12-04T11:13:50.2633332Z [W1204 10:50:13.007370726 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2633336Z 2025-12-04T11:13:50.2633624Z [W1204 10:50:13.007943318 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2633629Z 2025-12-04T11:13:50.2633917Z [W1204 10:50:13.008081671 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2633921Z 2025-12-04T11:13:50.2634205Z [W1204 10:50:13.012714744 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2634208Z 2025-12-04T11:13:50.2634491Z [W1204 10:50:13.013193584 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2634494Z 2025-12-04T11:13:50.2634782Z [W1204 10:50:13.013330267 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2634787Z 2025-12-04T11:13:50.2634867Z ('RERUN', {'yellow': True}) [0.4630s] [100%] 2025-12-04T11:13:50.2635596Z inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_1_mid_dim_1_in_features_144_out_features_1024_cuda_bfloat16 [W1204 10:50:13.461821037 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2635601Z 2025-12-04T11:13:50.2635890Z [W1204 10:50:13.462358229 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2635893Z 2025-12-04T11:13:50.2636180Z [W1204 10:50:13.462494972 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2636183Z 2025-12-04T11:13:50.2636466Z [W1204 10:50:13.465492468 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2636506Z 2025-12-04T11:13:50.2636871Z [W1204 10:50:13.466065330 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2636878Z 2025-12-04T11:13:50.2637170Z [W1204 10:50:13.466204213 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2637173Z 2025-12-04T11:13:50.2637457Z [W1204 10:50:13.470818905 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2637463Z 2025-12-04T11:13:50.2637749Z [W1204 10:50:13.471293256 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2637752Z 2025-12-04T11:13:50.2638035Z [W1204 10:50:13.471430248 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2638040Z 2025-12-04T11:13:50.2638106Z FAILED [0.4560s] [100%] 2025-12-04T11:13:50.2638109Z 2025-12-04T11:13:50.2638194Z ==================================== RERUNS ==================================== 2025-12-04T11:13:50.2638526Z _ TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_1_in_features_144_out_features_1024_cuda_bfloat16 _ 2025-12-04T11:13:50.2638599Z Traceback (most recent call last): 2025-12-04T11:13:50.2638900Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.2638969Z method(*args, **kwargs) 2025-12-04T11:13:50.2639257Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.2639327Z method(*args, **kwargs) 2025-12-04T11:13:50.2639616Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T11:13:50.2639676Z with policy(): 2025-12-04T11:13:50.2639978Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T11:13:50.2640044Z raise RuntimeError(msg) 2025-12-04T11:13:50.2640847Z RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_1_in_features_144_out_features_1024_cuda_bfloat16! Caching allocator allocated memory was 0 and is now reported as 147456 on device 0. CUDA driver allocated memory was 230686720 and is now 274726912. 2025-12-04T11:13:50.2640855Z 2025-12-04T11:13:50.2640990Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.2641509Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_1_in_features_144_out_features_1024_cuda_bfloat16 2025-12-04T11:13:50.2641514Z 2025-12-04T11:13:50.2641678Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.2641809Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.2641906Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.2642253Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.2642377Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.2642438Z graph_break [] 2025-12-04T11:13:50.2642565Z ----------------------------- Captured stderr call ----------------------------- 2025-12-04T11:13:50.2643254Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/constant_folding.py:256: UserWarning: Unsupported unwinding pattern: Address not in range (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/profiler/unwind/unwind.cpp:219.) 2025-12-04T11:13:50.2643372Z if out == self.unknown_value: 2025-12-04T11:13:50.2643735Z _ TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_1_in_features_144_out_features_1024_cuda_bfloat16 _ 2025-12-04T11:13:50.2643817Z Traceback (most recent call last): 2025-12-04T11:13:50.2644112Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.2644175Z method(*args, **kwargs) 2025-12-04T11:13:50.2644469Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.2644530Z method(*args, **kwargs) 2025-12-04T11:13:50.2644822Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T11:13:50.2644880Z with policy(): 2025-12-04T11:13:50.2645173Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T11:13:50.2645245Z raise RuntimeError(msg) 2025-12-04T11:13:50.2646069Z RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_1_in_features_144_out_features_1024_cuda_bfloat16! Caching allocator allocated memory was 147456 and is now reported as 294912 on device 0. CUDA driver allocated memory was 274726912 and is now 276824064. 2025-12-04T11:13:50.2646110Z 2025-12-04T11:13:50.2646243Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.2646760Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_1_in_features_144_out_features_1024_cuda_bfloat16 2025-12-04T11:13:50.2646763Z 2025-12-04T11:13:50.2646921Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.2647051Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.2647152Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.2647503Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.2647630Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.2647688Z graph_break [] 2025-12-04T11:13:50.2647816Z ----------------------------- Captured stderr call ----------------------------- 2025-12-04T11:13:50.2648503Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/constant_folding.py:256: UserWarning: Unsupported unwinding pattern: Address not in range (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/profiler/unwind/unwind.cpp:219.) 2025-12-04T11:13:50.2648576Z if out == self.unknown_value: 2025-12-04T11:13:50.2648698Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.2648791Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.2648921Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.2649273Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.2649336Z graph_break [] 2025-12-04T11:13:50.2649423Z =================================== FAILURES =================================== 2025-12-04T11:13:50.2649715Z _ TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_1_in_features_144_out_features_1024_cuda_bfloat16 _ 2025-12-04T11:13:50.2649792Z Traceback (most recent call last): 2025-12-04T11:13:50.2650099Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.2650163Z method(*args, **kwargs) 2025-12-04T11:13:50.2650457Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.2650564Z method(*args, **kwargs) 2025-12-04T11:13:50.2650918Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T11:13:50.2650980Z with policy(): 2025-12-04T11:13:50.2651272Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T11:13:50.2651342Z raise RuntimeError(msg) 2025-12-04T11:13:50.2652155Z RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_1_in_features_144_out_features_1024_cuda_bfloat16! Caching allocator allocated memory was 294912 and is now reported as 442368 on device 0. CUDA driver allocated memory was 276824064 and is now 278921216. 2025-12-04T11:13:50.2652159Z 2025-12-04T11:13:50.2652286Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.2652805Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_1_in_features_144_out_features_1024_cuda_bfloat16 2025-12-04T11:13:50.2652861Z 2025-12-04T11:13:50.2653015Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.2653144Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.2653235Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.2653579Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.2653701Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.2653760Z graph_break [] 2025-12-04T11:13:50.2653890Z ----------------------------- Captured stderr call ----------------------------- 2025-12-04T11:13:50.2654581Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/constant_folding.py:256: UserWarning: Unsupported unwinding pattern: Address not in range (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/profiler/unwind/unwind.cpp:219.) 2025-12-04T11:13:50.2654656Z if out == self.unknown_value: 2025-12-04T11:13:50.2654779Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.2654870Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.2654995Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.2655335Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.2655392Z graph_break [] 2025-12-04T11:13:50.2655519Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.2655608Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.2655733Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.2656074Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.2656135Z graph_break [] 2025-12-04T11:13:50.2656639Z - generated xml file: /var/lib/jenkins/workspace/test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-c9dbc1d234ce26dc.xml - 2025-12-04T11:13:50.2656738Z =========================== short test summary info ============================ 2025-12-04T11:13:50.2658128Z FAILED [0.4560s] inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_1_mid_dim_1_in_features_144_out_features_1024_cuda_bfloat16 - RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_1_in_features_144_out_features_1024_cuda_bfloat16! Caching allocator allocated memory was 294912 and is now reported as 442368 on device 0. CUDA driver allocated memory was 276824064 and is now 278921216. 2025-12-04T11:13:50.2658175Z 2025-12-04T11:13:50.2658304Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.2658823Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_1_in_features_144_out_features_1024_cuda_bfloat16 2025-12-04T11:13:50.2658826Z 2025-12-04T11:13:50.2658979Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.2659082Z !!!!!!!!!!!!!!!!!!!!!!!!!! stopping after 1 failures !!!!!!!!!!!!!!!!!!!!!!!!!!! 2025-12-04T11:13:50.2659202Z ================== 1 failed, 57 deselected, 2 rerun in 12.11s ================== 2025-12-04T11:13:50.2659269Z Got exit code 1 2025-12-04T11:13:50.2659338Z Retrying single test... 2025-12-04T11:13:50.2659601Z W1204 10:50:20.323000 72683 site-packages/torch/_inductor/utils.py:1703] Not enough SMs to use max_autotune_gemm mode 2025-12-04T11:13:50.2659982Z Test results will be stored in test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-8a076bf12e47858e.xml 2025-12-04T11:13:50.2660125Z ============================= test session starts ============================== 2025-12-04T11:13:50.2660333Z platform linux -- Python 3.10.14, pytest-7.3.2, pluggy-1.6.0 -- /opt/conda/envs/py_3.10/bin/python 2025-12-04T11:13:50.2660398Z cachedir: .pytest_cache 2025-12-04T11:13:50.2660708Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow] 2025-12-04T11:13:50.2660786Z rootdir: /var/lib/jenkins/workspace 2025-12-04T11:13:50.2660858Z configfile: pytest.ini 2025-12-04T11:13:50.2661176Z plugins: hypothesis-6.56.4, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-14.0, subtests-0.13.1, xdist-3.3.1, xdoctest-1.3.0, typeguard-4.3.0 2025-12-04T11:13:50.2661309Z collecting ... collected 58 items / 57 deselected / 1 selected 2025-12-04T11:13:50.2661885Z stepcurrent: skipping 28 already run items. Running only test/inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_1_mid_dim_1_in_features_144_out_features_1024_cuda_bfloat16 2025-12-04T11:13:50.2661957Z Running 1 items in this shard 2025-12-04T11:13:50.2661961Z 2025-12-04T11:13:50.2662692Z inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_1_mid_dim_1_in_features_144_out_features_1024_cuda_bfloat16 [W1204 10:50:21.442879244 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2662696Z 2025-12-04T11:13:50.2662989Z [W1204 10:50:30.511125619 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2662994Z 2025-12-04T11:13:50.2663288Z [W1204 10:50:30.511403586 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2663292Z 2025-12-04T11:13:50.2663618Z [W1204 10:50:30.517171663 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2663623Z 2025-12-04T11:13:50.2663908Z [W1204 10:50:30.517708905 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2663911Z 2025-12-04T11:13:50.2664199Z [W1204 10:50:30.517872309 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2664202Z 2025-12-04T11:13:50.2664500Z [W1204 10:50:30.523245488 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2664547Z 2025-12-04T11:13:50.2664901Z [W1204 10:50:30.523767099 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2664904Z 2025-12-04T11:13:50.2665192Z [W1204 10:50:30.523922163 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2665197Z 2025-12-04T11:13:50.2665282Z ('RERUN', {'yellow': True}) [10.9874s] [100%] 2025-12-04T11:13:50.2666008Z inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_1_mid_dim_1_in_features_144_out_features_1024_cuda_bfloat16 [W1204 10:50:31.750056799 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2666012Z 2025-12-04T11:13:50.2666304Z [W1204 10:50:31.750606861 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2666308Z 2025-12-04T11:13:50.2666594Z [W1204 10:50:31.750750095 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2666597Z 2025-12-04T11:13:50.2666917Z [W1204 10:50:31.753759051 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2666924Z 2025-12-04T11:13:50.2667209Z [W1204 10:50:31.754337194 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2667212Z 2025-12-04T11:13:50.2667495Z [W1204 10:50:31.754474837 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2667499Z 2025-12-04T11:13:50.2667789Z [W1204 10:50:31.759084669 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2667792Z 2025-12-04T11:13:50.2668081Z [W1204 10:50:31.759562420 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2668087Z 2025-12-04T11:13:50.2668378Z [W1204 10:50:31.759696453 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2668383Z 2025-12-04T11:13:50.2668462Z ('RERUN', {'yellow': True}) [0.4570s] [100%] 2025-12-04T11:13:50.2669183Z inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_1_mid_dim_1_in_features_144_out_features_1024_cuda_bfloat16 [W1204 10:50:32.201628909 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2669186Z 2025-12-04T11:13:50.2669474Z [W1204 10:50:32.202169051 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2669479Z 2025-12-04T11:13:50.2669773Z [W1204 10:50:32.202312814 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2669776Z 2025-12-04T11:13:50.2670061Z [W1204 10:50:32.205298770 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2670066Z 2025-12-04T11:13:50.2670350Z [W1204 10:50:32.205868383 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2670356Z 2025-12-04T11:13:50.2670639Z [W1204 10:50:32.206004876 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2670642Z 2025-12-04T11:13:50.2670925Z [W1204 10:50:32.210635698 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2670928Z 2025-12-04T11:13:50.2671258Z [W1204 10:50:32.211111338 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2671328Z 2025-12-04T11:13:50.2671615Z [W1204 10:50:32.211246081 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2671620Z 2025-12-04T11:13:50.2671688Z FAILED [0.4503s] [100%] 2025-12-04T11:13:50.2671691Z 2025-12-04T11:13:50.2671785Z ==================================== RERUNS ==================================== 2025-12-04T11:13:50.2672079Z _ TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_1_in_features_144_out_features_1024_cuda_bfloat16 _ 2025-12-04T11:13:50.2672159Z Traceback (most recent call last): 2025-12-04T11:13:50.2672462Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.2672531Z method(*args, **kwargs) 2025-12-04T11:13:50.2672822Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.2672887Z method(*args, **kwargs) 2025-12-04T11:13:50.2673180Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T11:13:50.2673279Z with policy(): 2025-12-04T11:13:50.2673573Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T11:13:50.2673647Z raise RuntimeError(msg) 2025-12-04T11:13:50.2674452Z RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_1_in_features_144_out_features_1024_cuda_bfloat16! Caching allocator allocated memory was 0 and is now reported as 147456 on device 0. CUDA driver allocated memory was 230686720 and is now 274726912. 2025-12-04T11:13:50.2674456Z 2025-12-04T11:13:50.2674586Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.2675109Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_1_in_features_144_out_features_1024_cuda_bfloat16 2025-12-04T11:13:50.2675114Z 2025-12-04T11:13:50.2675281Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.2675414Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.2679263Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.2679673Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.2679820Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.2679883Z graph_break [] 2025-12-04T11:13:50.2680023Z ----------------------------- Captured stderr call ----------------------------- 2025-12-04T11:13:50.2680744Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/constant_folding.py:256: UserWarning: Unsupported unwinding pattern: Address not in range (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/profiler/unwind/unwind.cpp:219.) 2025-12-04T11:13:50.2680821Z if out == self.unknown_value: 2025-12-04T11:13:50.2681131Z _ TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_1_in_features_144_out_features_1024_cuda_bfloat16 _ 2025-12-04T11:13:50.2681208Z Traceback (most recent call last): 2025-12-04T11:13:50.2681512Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.2681583Z method(*args, **kwargs) 2025-12-04T11:13:50.2681869Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.2681938Z method(*args, **kwargs) 2025-12-04T11:13:50.2682347Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T11:13:50.2682512Z with policy(): 2025-12-04T11:13:50.2682821Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T11:13:50.2682891Z raise RuntimeError(msg) 2025-12-04T11:13:50.2683717Z RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_1_in_features_144_out_features_1024_cuda_bfloat16! Caching allocator allocated memory was 147456 and is now reported as 294912 on device 0. CUDA driver allocated memory was 274726912 and is now 276824064. 2025-12-04T11:13:50.2683726Z 2025-12-04T11:13:50.2683862Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.2684390Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_1_in_features_144_out_features_1024_cuda_bfloat16 2025-12-04T11:13:50.2684395Z 2025-12-04T11:13:50.2684568Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.2684755Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.2684855Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.2685206Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.2685335Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.2685398Z graph_break [] 2025-12-04T11:13:50.2685523Z ----------------------------- Captured stderr call ----------------------------- 2025-12-04T11:13:50.2686226Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/constant_folding.py:256: UserWarning: Unsupported unwinding pattern: Address not in range (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/profiler/unwind/unwind.cpp:219.) 2025-12-04T11:13:50.2686310Z if out == self.unknown_value: 2025-12-04T11:13:50.2686439Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.2686537Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.2686662Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.2687005Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.2687068Z graph_break [] 2025-12-04T11:13:50.2687152Z =================================== FAILURES =================================== 2025-12-04T11:13:50.2687451Z _ TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_1_in_features_144_out_features_1024_cuda_bfloat16 _ 2025-12-04T11:13:50.2687529Z Traceback (most recent call last): 2025-12-04T11:13:50.2687836Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.2687906Z method(*args, **kwargs) 2025-12-04T11:13:50.2688196Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.2688261Z method(*args, **kwargs) 2025-12-04T11:13:50.2688546Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T11:13:50.2688607Z with policy(): 2025-12-04T11:13:50.2688906Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T11:13:50.2688970Z raise RuntimeError(msg) 2025-12-04T11:13:50.2689877Z RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_1_in_features_144_out_features_1024_cuda_bfloat16! Caching allocator allocated memory was 294912 and is now reported as 442368 on device 0. CUDA driver allocated memory was 276824064 and is now 278921216. 2025-12-04T11:13:50.2689920Z 2025-12-04T11:13:50.2690053Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.2690579Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_1_in_features_144_out_features_1024_cuda_bfloat16 2025-12-04T11:13:50.2690583Z 2025-12-04T11:13:50.2690746Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.2690873Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.2690972Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.2691321Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.2691450Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.2691512Z graph_break [] 2025-12-04T11:13:50.2691673Z ----------------------------- Captured stderr call ----------------------------- 2025-12-04T11:13:50.2692364Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/constant_folding.py:256: UserWarning: Unsupported unwinding pattern: Address not in range (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/profiler/unwind/unwind.cpp:219.) 2025-12-04T11:13:50.2692437Z if out == self.unknown_value: 2025-12-04T11:13:50.2692560Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.2692656Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.2692779Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.2693121Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.2693187Z graph_break [] 2025-12-04T11:13:50.2693309Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.2693400Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.2693531Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.2693872Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.2693932Z graph_break [] 2025-12-04T11:13:50.2694423Z - generated xml file: /var/lib/jenkins/workspace/test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-8a076bf12e47858e.xml - 2025-12-04T11:13:50.2694523Z =========================== short test summary info ============================ 2025-12-04T11:13:50.2695825Z FAILED [0.4503s] inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_1_mid_dim_1_in_features_144_out_features_1024_cuda_bfloat16 - RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_1_in_features_144_out_features_1024_cuda_bfloat16! Caching allocator allocated memory was 294912 and is now reported as 442368 on device 0. CUDA driver allocated memory was 276824064 and is now 278921216. 2025-12-04T11:13:50.2695831Z 2025-12-04T11:13:50.2695958Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.2696489Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_1_in_features_144_out_features_1024_cuda_bfloat16 2025-12-04T11:13:50.2696492Z 2025-12-04T11:13:50.2696651Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.2696803Z !!!!!!!!!!!!!!!!!!!!!!!!!! stopping after 1 failures !!!!!!!!!!!!!!!!!!!!!!!!!!! 2025-12-04T11:13:50.2696985Z ================== 1 failed, 57 deselected, 2 rerun in 11.92s ================== 2025-12-04T11:13:50.2697047Z Got exit code 1 2025-12-04T11:13:50.2697521Z FAILED CONSISTENTLY: test/inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_1_mid_dim_1_in_features_144_out_features_1024_cuda_bfloat16 2025-12-04T11:13:50.2697761Z Test failed consistently, continuing with the rest of the tests due to continue-through-error being set 2025-12-04T11:13:50.2698027Z W1204 10:50:39.069000 72876 site-packages/torch/_inductor/utils.py:1703] Not enough SMs to use max_autotune_gemm mode 2025-12-04T11:13:50.2698414Z Test results will be stored in test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-bb0bac5f3b803919.xml 2025-12-04T11:13:50.2698509Z ============================= test session starts ============================== 2025-12-04T11:13:50.2698728Z platform linux -- Python 3.10.14, pytest-7.3.2, pluggy-1.6.0 -- /opt/conda/envs/py_3.10/bin/python 2025-12-04T11:13:50.2698796Z cachedir: .pytest_cache 2025-12-04T11:13:50.2699145Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow] 2025-12-04T11:13:50.2699221Z rootdir: /var/lib/jenkins/workspace 2025-12-04T11:13:50.2699285Z configfile: pytest.ini 2025-12-04T11:13:50.2699607Z plugins: hypothesis-6.56.4, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-14.0, subtests-0.13.1, xdist-3.3.1, xdoctest-1.3.0, typeguard-4.3.0 2025-12-04T11:13:50.2699747Z collecting ... collected 58 items / 29 deselected / 29 selected 2025-12-04T11:13:50.2699837Z stepcurrent: skipping 29 already run items. 2025-12-04T11:13:50.2699907Z Running 29 items in this shard 2025-12-04T11:13:50.2699911Z 2025-12-04T11:13:50.2700403Z inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_1_mid_dim_1_in_features_144_out_features_64_cuda_bfloat16 ('RERUN', {'yellow': True}) [1.8705s] [ 3%] 2025-12-04T11:13:50.2700892Z inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_1_mid_dim_1_in_features_144_out_features_64_cuda_bfloat16 ('RERUN', {'yellow': True}) [0.4590s] [ 3%] 2025-12-04T11:13:50.2701336Z inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_1_mid_dim_1_in_features_144_out_features_64_cuda_bfloat16 FAILED [0.4617s] [ 3%] 2025-12-04T11:13:50.2701339Z 2025-12-04T11:13:50.2701425Z ==================================== RERUNS ==================================== 2025-12-04T11:13:50.2701714Z _ TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_1_in_features_144_out_features_64_cuda_bfloat16 _ 2025-12-04T11:13:50.2701790Z Traceback (most recent call last): 2025-12-04T11:13:50.2702100Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.2702166Z method(*args, **kwargs) 2025-12-04T11:13:50.2702463Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.2702527Z method(*args, **kwargs) 2025-12-04T11:13:50.2702814Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T11:13:50.2702876Z with policy(): 2025-12-04T11:13:50.2703168Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T11:13:50.2703233Z raise RuntimeError(msg) 2025-12-04T11:13:50.2704084Z RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_1_in_features_144_out_features_64_cuda_bfloat16! Caching allocator allocated memory was 0 and is now reported as 9216 on device 0. CUDA driver allocated memory was 230686720 and is now 274726912. 2025-12-04T11:13:50.2704133Z 2025-12-04T11:13:50.2704342Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.2704864Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_1_in_features_144_out_features_64_cuda_bfloat16 2025-12-04T11:13:50.2704868Z 2025-12-04T11:13:50.2705027Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.2705156Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.2705260Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.2705611Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.2705747Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.2705806Z graph_break [] 2025-12-04T11:13:50.2706095Z _ TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_1_in_features_144_out_features_64_cuda_bfloat16 _ 2025-12-04T11:13:50.2706208Z Traceback (most recent call last): 2025-12-04T11:13:50.2706502Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.2706572Z method(*args, **kwargs) 2025-12-04T11:13:50.2706859Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.2706921Z method(*args, **kwargs) 2025-12-04T11:13:50.2707210Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T11:13:50.2707269Z with policy(): 2025-12-04T11:13:50.2707563Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T11:13:50.2707629Z raise RuntimeError(msg) 2025-12-04T11:13:50.2708425Z RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_1_in_features_144_out_features_64_cuda_bfloat16! Caching allocator allocated memory was 9216 and is now reported as 18432 on device 0. CUDA driver allocated memory was 274726912 and is now 276824064. 2025-12-04T11:13:50.2708431Z 2025-12-04T11:13:50.2708560Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.2709072Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_1_in_features_144_out_features_64_cuda_bfloat16 2025-12-04T11:13:50.2709076Z 2025-12-04T11:13:50.2709235Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.2709363Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.2709455Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.2709804Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.2709933Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.2709994Z graph_break [] 2025-12-04T11:13:50.2710117Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.2710202Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.2710323Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.2710662Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.2710719Z graph_break [] 2025-12-04T11:13:50.2710845Z =================================== FAILURES =================================== 2025-12-04T11:13:50.2711286Z _ TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_1_in_features_144_out_features_64_cuda_bfloat16 _ 2025-12-04T11:13:50.2711368Z Traceback (most recent call last): 2025-12-04T11:13:50.2711662Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.2711724Z method(*args, **kwargs) 2025-12-04T11:13:50.2712013Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.2712075Z method(*args, **kwargs) 2025-12-04T11:13:50.2712363Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T11:13:50.2712421Z with policy(): 2025-12-04T11:13:50.2712709Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T11:13:50.2712780Z raise RuntimeError(msg) 2025-12-04T11:13:50.2713582Z RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_1_in_features_144_out_features_64_cuda_bfloat16! Caching allocator allocated memory was 18432 and is now reported as 27648 on device 0. CUDA driver allocated memory was 276824064 and is now 278921216. 2025-12-04T11:13:50.2713624Z 2025-12-04T11:13:50.2713749Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.2714257Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_1_in_features_144_out_features_64_cuda_bfloat16 2025-12-04T11:13:50.2714261Z 2025-12-04T11:13:50.2714413Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.2714553Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.2714641Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.2714990Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.2715121Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.2715180Z graph_break [] 2025-12-04T11:13:50.2715306Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.2715398Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.2715517Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.2715867Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.2715926Z graph_break [] 2025-12-04T11:13:50.2716050Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.2716142Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.2716264Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.2716609Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.2716667Z graph_break [] 2025-12-04T11:13:50.2717164Z - generated xml file: /var/lib/jenkins/workspace/test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-bb0bac5f3b803919.xml - 2025-12-04T11:13:50.2717271Z =========================== short test summary info ============================ 2025-12-04T11:13:50.2718623Z FAILED [0.4617s] inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_1_mid_dim_1_in_features_144_out_features_64_cuda_bfloat16 - RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_1_in_features_144_out_features_64_cuda_bfloat16! Caching allocator allocated memory was 18432 and is now reported as 27648 on device 0. CUDA driver allocated memory was 276824064 and is now 278921216. 2025-12-04T11:13:50.2718666Z 2025-12-04T11:13:50.2718789Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.2719310Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_1_in_features_144_out_features_64_cuda_bfloat16 2025-12-04T11:13:50.2719313Z 2025-12-04T11:13:50.2719471Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.2719574Z !!!!!!!!!!!!!!!!!!!!!!!!!! stopping after 1 failures !!!!!!!!!!!!!!!!!!!!!!!!!!! 2025-12-04T11:13:50.2719694Z ================== 1 failed, 29 deselected, 2 rerun in 2.82s =================== 2025-12-04T11:13:50.2719754Z Got exit code 1 2025-12-04T11:13:50.2719818Z Retrying single test... 2025-12-04T11:13:50.2720092Z W1204 10:50:49.091000 73057 site-packages/torch/_inductor/utils.py:1703] Not enough SMs to use max_autotune_gemm mode 2025-12-04T11:13:50.2720507Z Test results will be stored in test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-0645969e8b49cd4b.xml 2025-12-04T11:13:50.2720611Z ============================= test session starts ============================== 2025-12-04T11:13:50.2720824Z platform linux -- Python 3.10.14, pytest-7.3.2, pluggy-1.6.0 -- /opt/conda/envs/py_3.10/bin/python 2025-12-04T11:13:50.2720889Z cachedir: .pytest_cache 2025-12-04T11:13:50.2721198Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow] 2025-12-04T11:13:50.2721274Z rootdir: /var/lib/jenkins/workspace 2025-12-04T11:13:50.2721339Z configfile: pytest.ini 2025-12-04T11:13:50.2721657Z plugins: hypothesis-6.56.4, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-14.0, subtests-0.13.1, xdist-3.3.1, xdoctest-1.3.0, typeguard-4.3.0 2025-12-04T11:13:50.2721789Z collecting ... collected 58 items / 57 deselected / 1 selected 2025-12-04T11:13:50.2722350Z stepcurrent: skipping 29 already run items. Running only test/inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_1_mid_dim_1_in_features_144_out_features_64_cuda_bfloat16 2025-12-04T11:13:50.2722425Z Running 1 items in this shard 2025-12-04T11:13:50.2722429Z 2025-12-04T11:13:50.2723153Z inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_1_mid_dim_1_in_features_144_out_features_64_cuda_bfloat16 [W1204 10:50:50.184728903 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2723158Z 2025-12-04T11:13:50.2723460Z [W1204 10:50:59.347351410 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2723467Z 2025-12-04T11:13:50.2723755Z [W1204 10:50:59.347656777 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2723761Z 2025-12-04T11:13:50.2724047Z [W1204 10:50:59.353442305 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2724051Z 2025-12-04T11:13:50.2724336Z [W1204 10:50:59.354000667 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2724339Z 2025-12-04T11:13:50.2724626Z [W1204 10:50:59.354175441 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2724629Z 2025-12-04T11:13:50.2724912Z [W1204 10:50:59.359548390 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2724956Z 2025-12-04T11:13:50.2725311Z [W1204 10:50:59.360097892 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2725316Z 2025-12-04T11:13:50.2725602Z [W1204 10:50:59.360265126 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2725605Z 2025-12-04T11:13:50.2725695Z ('RERUN', {'yellow': True}) [11.0535s] [100%] 2025-12-04T11:13:50.2726417Z inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_1_mid_dim_1_in_features_144_out_features_64_cuda_bfloat16 [W1204 10:51:00.548822458 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2726421Z 2025-12-04T11:13:50.2726707Z [W1204 10:51:00.549424792 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2726712Z 2025-12-04T11:13:50.2726999Z [W1204 10:51:00.549570265 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2727039Z 2025-12-04T11:13:50.2727329Z [W1204 10:51:00.552586251 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2727333Z 2025-12-04T11:13:50.2727619Z [W1204 10:51:00.553179604 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2727622Z 2025-12-04T11:13:50.2727906Z [W1204 10:51:00.553322197 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2727910Z 2025-12-04T11:13:50.2728194Z [W1204 10:51:00.557913478 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2728198Z 2025-12-04T11:13:50.2728484Z [W1204 10:51:00.558388179 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2728489Z 2025-12-04T11:13:50.2728775Z [W1204 10:51:00.558525962 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2728777Z 2025-12-04T11:13:50.2728855Z ('RERUN', {'yellow': True}) [0.4226s] [100%] 2025-12-04T11:13:50.2729575Z inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_1_mid_dim_1_in_features_144_out_features_64_cuda_bfloat16 [W1204 10:51:01.967147717 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2729579Z 2025-12-04T11:13:50.2729869Z [W1204 10:51:01.967739561 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2729874Z 2025-12-04T11:13:50.2730158Z [W1204 10:51:01.967883384 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2730162Z 2025-12-04T11:13:50.2730449Z [W1204 10:51:01.970875990 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2730452Z 2025-12-04T11:13:50.2730737Z [W1204 10:51:01.971460003 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2730740Z 2025-12-04T11:13:50.2731025Z [W1204 10:51:01.971598586 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2731029Z 2025-12-04T11:13:50.2731322Z [W1204 10:51:01.976171467 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2731365Z 2025-12-04T11:13:50.2731737Z [W1204 10:51:01.976655898 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2731743Z 2025-12-04T11:13:50.2732029Z [W1204 10:51:01.976792371 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2732032Z 2025-12-04T11:13:50.2732093Z FAILED [0.4149s] [100%] 2025-12-04T11:13:50.2732097Z 2025-12-04T11:13:50.2732183Z ==================================== RERUNS ==================================== 2025-12-04T11:13:50.2732468Z _ TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_1_in_features_144_out_features_64_cuda_bfloat16 _ 2025-12-04T11:13:50.2732542Z Traceback (most recent call last): 2025-12-04T11:13:50.2732847Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.2732914Z method(*args, **kwargs) 2025-12-04T11:13:50.2733209Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.2733307Z method(*args, **kwargs) 2025-12-04T11:13:50.2733608Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T11:13:50.2733667Z with policy(): 2025-12-04T11:13:50.2733959Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T11:13:50.2734024Z raise RuntimeError(msg) 2025-12-04T11:13:50.2734813Z RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_1_in_features_144_out_features_64_cuda_bfloat16! Caching allocator allocated memory was 0 and is now reported as 9216 on device 0. CUDA driver allocated memory was 230686720 and is now 274726912. 2025-12-04T11:13:50.2734819Z 2025-12-04T11:13:50.2734953Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.2735465Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_1_in_features_144_out_features_64_cuda_bfloat16 2025-12-04T11:13:50.2735471Z 2025-12-04T11:13:50.2735627Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.2735758Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.2735853Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.2736203Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.2736330Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.2736389Z graph_break [] 2025-12-04T11:13:50.2736514Z ----------------------------- Captured stderr call ----------------------------- 2025-12-04T11:13:50.2737205Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/constant_folding.py:256: UserWarning: Unsupported unwinding pattern: Address not in range (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/profiler/unwind/unwind.cpp:219.) 2025-12-04T11:13:50.2737280Z if out == self.unknown_value: 2025-12-04T11:13:50.2737565Z _ TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_1_in_features_144_out_features_64_cuda_bfloat16 _ 2025-12-04T11:13:50.2737640Z Traceback (most recent call last): 2025-12-04T11:13:50.2737935Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.2737997Z method(*args, **kwargs) 2025-12-04T11:13:50.2738281Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.2738389Z method(*args, **kwargs) 2025-12-04T11:13:50.2738749Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T11:13:50.2738817Z with policy(): 2025-12-04T11:13:50.2739109Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T11:13:50.2739172Z raise RuntimeError(msg) 2025-12-04T11:13:50.2739983Z RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_1_in_features_144_out_features_64_cuda_bfloat16! Caching allocator allocated memory was 9216 and is now reported as 18432 on device 0. CUDA driver allocated memory was 274726912 and is now 276824064. 2025-12-04T11:13:50.2739986Z 2025-12-04T11:13:50.2740111Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.2740636Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_1_in_features_144_out_features_64_cuda_bfloat16 2025-12-04T11:13:50.2740934Z 2025-12-04T11:13:50.2741098Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.2741226Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.2741322Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.2741667Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.2741796Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.2741857Z graph_break [] 2025-12-04T11:13:50.2741978Z ----------------------------- Captured stderr call ----------------------------- 2025-12-04T11:13:50.2742668Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/constant_folding.py:256: UserWarning: Unsupported unwinding pattern: Address not in range (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/profiler/unwind/unwind.cpp:219.) 2025-12-04T11:13:50.2742738Z if out == self.unknown_value: 2025-12-04T11:13:50.2742862Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.2742952Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.2743077Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.2743420Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.2743477Z graph_break [] 2025-12-04T11:13:50.2743650Z =================================== FAILURES =================================== 2025-12-04T11:13:50.2743939Z _ TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_1_in_features_144_out_features_64_cuda_bfloat16 _ 2025-12-04T11:13:50.2744014Z Traceback (most recent call last): 2025-12-04T11:13:50.2744317Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.2744382Z method(*args, **kwargs) 2025-12-04T11:13:50.2744668Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.2744734Z method(*args, **kwargs) 2025-12-04T11:13:50.2745018Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T11:13:50.2745079Z with policy(): 2025-12-04T11:13:50.2745369Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T11:13:50.2745432Z raise RuntimeError(msg) 2025-12-04T11:13:50.2746366Z RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_1_in_features_144_out_features_64_cuda_bfloat16! Caching allocator allocated memory was 18432 and is now reported as 27648 on device 0. CUDA driver allocated memory was 276824064 and is now 278921216. 2025-12-04T11:13:50.2746373Z 2025-12-04T11:13:50.2746504Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.2747017Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_1_in_features_144_out_features_64_cuda_bfloat16 2025-12-04T11:13:50.2747021Z 2025-12-04T11:13:50.2747177Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.2747301Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.2747397Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.2747742Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.2747866Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.2747962Z graph_break [] 2025-12-04T11:13:50.2748088Z ----------------------------- Captured stderr call ----------------------------- 2025-12-04T11:13:50.2748774Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/constant_folding.py:256: UserWarning: Unsupported unwinding pattern: Address not in range (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/profiler/unwind/unwind.cpp:219.) 2025-12-04T11:13:50.2748843Z if out == self.unknown_value: 2025-12-04T11:13:50.2748966Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.2749054Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.2749178Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.2749524Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.2749583Z graph_break [] 2025-12-04T11:13:50.2749705Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.2749795Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.2749915Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.2750255Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.2750311Z graph_break [] 2025-12-04T11:13:50.2750791Z - generated xml file: /var/lib/jenkins/workspace/test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-0645969e8b49cd4b.xml - 2025-12-04T11:13:50.2750904Z =========================== short test summary info ============================ 2025-12-04T11:13:50.2752185Z FAILED [0.4149s] inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_1_mid_dim_1_in_features_144_out_features_64_cuda_bfloat16 - RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_1_in_features_144_out_features_64_cuda_bfloat16! Caching allocator allocated memory was 18432 and is now reported as 27648 on device 0. CUDA driver allocated memory was 276824064 and is now 278921216. 2025-12-04T11:13:50.2752191Z 2025-12-04T11:13:50.2752318Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.2752830Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_1_in_features_144_out_features_64_cuda_bfloat16 2025-12-04T11:13:50.2752874Z 2025-12-04T11:13:50.2753032Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.2753200Z !!!!!!!!!!!!!!!!!!!!!!!!!! stopping after 1 failures !!!!!!!!!!!!!!!!!!!!!!!!!!! 2025-12-04T11:13:50.2753318Z ================== 1 failed, 57 deselected, 2 rerun in 11.92s ================== 2025-12-04T11:13:50.2753380Z Got exit code 1 2025-12-04T11:13:50.2753453Z Retrying single test... 2025-12-04T11:13:50.2753715Z W1204 10:51:07.778000 73243 site-packages/torch/_inductor/utils.py:1703] Not enough SMs to use max_autotune_gemm mode 2025-12-04T11:13:50.2754101Z Test results will be stored in test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-20accf8a0f4e3024.xml 2025-12-04T11:13:50.2754194Z ============================= test session starts ============================== 2025-12-04T11:13:50.2754402Z platform linux -- Python 3.10.14, pytest-7.3.2, pluggy-1.6.0 -- /opt/conda/envs/py_3.10/bin/python 2025-12-04T11:13:50.2754468Z cachedir: .pytest_cache 2025-12-04T11:13:50.2754775Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow] 2025-12-04T11:13:50.2754854Z rootdir: /var/lib/jenkins/workspace 2025-12-04T11:13:50.2754958Z configfile: pytest.ini 2025-12-04T11:13:50.2755275Z plugins: hypothesis-6.56.4, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-14.0, subtests-0.13.1, xdist-3.3.1, xdoctest-1.3.0, typeguard-4.3.0 2025-12-04T11:13:50.2755403Z collecting ... collected 58 items / 57 deselected / 1 selected 2025-12-04T11:13:50.2755966Z stepcurrent: skipping 29 already run items. Running only test/inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_1_mid_dim_1_in_features_144_out_features_64_cuda_bfloat16 2025-12-04T11:13:50.2756041Z Running 1 items in this shard 2025-12-04T11:13:50.2756045Z 2025-12-04T11:13:50.2756768Z inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_1_mid_dim_1_in_features_144_out_features_64_cuda_bfloat16 [W1204 10:51:08.865281856 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2756775Z 2025-12-04T11:13:50.2757083Z [W1204 10:51:18.003948642 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2757089Z 2025-12-04T11:13:50.2757381Z [W1204 10:51:18.004186617 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2757384Z 2025-12-04T11:13:50.2757674Z [W1204 10:51:18.009916353 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2757678Z 2025-12-04T11:13:50.2757962Z [W1204 10:51:18.010515416 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2757967Z 2025-12-04T11:13:50.2758252Z [W1204 10:51:18.010692220 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2758260Z 2025-12-04T11:13:50.2758546Z [W1204 10:51:18.016120250 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2758549Z 2025-12-04T11:13:50.2758833Z [W1204 10:51:18.016651202 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2758836Z 2025-12-04T11:13:50.2759122Z [W1204 10:51:18.016807505 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2759125Z 2025-12-04T11:13:50.2759204Z ('RERUN', {'yellow': True}) [11.0210s] [100%] 2025-12-04T11:13:50.2760011Z inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_1_mid_dim_1_in_features_144_out_features_64_cuda_bfloat16 [W1204 10:51:19.199647706 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2760048Z 2025-12-04T11:13:50.2760336Z [W1204 10:51:19.200254599 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2760340Z 2025-12-04T11:13:50.2760627Z [W1204 10:51:19.200397102 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2760630Z 2025-12-04T11:13:50.2760914Z [W1204 10:51:19.203391539 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2760917Z 2025-12-04T11:13:50.2761203Z [W1204 10:51:19.203969492 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2761207Z 2025-12-04T11:13:50.2761496Z [W1204 10:51:19.204106785 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2761499Z 2025-12-04T11:13:50.2761783Z [W1204 10:51:19.208747027 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2761826Z 2025-12-04T11:13:50.2762112Z [W1204 10:51:19.209224708 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2762116Z 2025-12-04T11:13:50.2762398Z [W1204 10:51:19.209360451 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2762401Z 2025-12-04T11:13:50.2762482Z ('RERUN', {'yellow': True}) [0.4193s] [100%] 2025-12-04T11:13:50.2763195Z inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_1_mid_dim_1_in_features_144_out_features_64_cuda_bfloat16 [W1204 10:51:19.614173894 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2763200Z 2025-12-04T11:13:50.2763488Z [W1204 10:51:19.614751767 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2763493Z 2025-12-04T11:13:50.2763778Z [W1204 10:51:19.614887180 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2763782Z 2025-12-04T11:13:50.2764065Z [W1204 10:51:19.617881076 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2764069Z 2025-12-04T11:13:50.2764352Z [W1204 10:51:19.618449429 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2764355Z 2025-12-04T11:13:50.2764646Z [W1204 10:51:19.618584052 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2764652Z 2025-12-04T11:13:50.2764938Z [W1204 10:51:19.623252805 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2764943Z 2025-12-04T11:13:50.2765228Z [W1204 10:51:19.623737346 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2765231Z 2025-12-04T11:13:50.2765516Z [W1204 10:51:19.623871149 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2765520Z 2025-12-04T11:13:50.2765582Z FAILED [0.4141s] [100%] 2025-12-04T11:13:50.2765585Z 2025-12-04T11:13:50.2765672Z ==================================== RERUNS ==================================== 2025-12-04T11:13:50.2765957Z _ TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_1_in_features_144_out_features_64_cuda_bfloat16 _ 2025-12-04T11:13:50.2766071Z Traceback (most recent call last): 2025-12-04T11:13:50.2766443Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.2766510Z method(*args, **kwargs) 2025-12-04T11:13:50.2766801Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.2766863Z method(*args, **kwargs) 2025-12-04T11:13:50.2767148Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T11:13:50.2767211Z with policy(): 2025-12-04T11:13:50.2767508Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T11:13:50.2767573Z raise RuntimeError(msg) 2025-12-04T11:13:50.2768364Z RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_1_in_features_144_out_features_64_cuda_bfloat16! Caching allocator allocated memory was 0 and is now reported as 9216 on device 0. CUDA driver allocated memory was 230686720 and is now 274726912. 2025-12-04T11:13:50.2768407Z 2025-12-04T11:13:50.2768536Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.2769049Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_1_in_features_144_out_features_64_cuda_bfloat16 2025-12-04T11:13:50.2769053Z 2025-12-04T11:13:50.2769225Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.2769360Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.2769458Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.2769808Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.2769933Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.2769998Z graph_break [] 2025-12-04T11:13:50.2770120Z ----------------------------- Captured stderr call ----------------------------- 2025-12-04T11:13:50.2770808Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/constant_folding.py:256: UserWarning: Unsupported unwinding pattern: Address not in range (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/profiler/unwind/unwind.cpp:219.) 2025-12-04T11:13:50.2770878Z if out == self.unknown_value: 2025-12-04T11:13:50.2771162Z _ TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_1_in_features_144_out_features_64_cuda_bfloat16 _ 2025-12-04T11:13:50.2771237Z Traceback (most recent call last): 2025-12-04T11:13:50.2771531Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.2771595Z method(*args, **kwargs) 2025-12-04T11:13:50.2771886Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.2771948Z method(*args, **kwargs) 2025-12-04T11:13:50.2772233Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T11:13:50.2772293Z with policy(): 2025-12-04T11:13:50.2772584Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T11:13:50.2772652Z raise RuntimeError(msg) 2025-12-04T11:13:50.2773457Z RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_1_in_features_144_out_features_64_cuda_bfloat16! Caching allocator allocated memory was 9216 and is now reported as 18432 on device 0. CUDA driver allocated memory was 274726912 and is now 276824064. 2025-12-04T11:13:50.2773567Z 2025-12-04T11:13:50.2773701Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.2774215Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_1_in_features_144_out_features_64_cuda_bfloat16 2025-12-04T11:13:50.2774219Z 2025-12-04T11:13:50.2774374Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.2774509Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.2774601Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.2774944Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.2775071Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.2775130Z graph_break [] 2025-12-04T11:13:50.2775257Z ----------------------------- Captured stderr call ----------------------------- 2025-12-04T11:13:50.2775979Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/constant_folding.py:256: UserWarning: Unsupported unwinding pattern: Address not in range (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/profiler/unwind/unwind.cpp:219.) 2025-12-04T11:13:50.2776050Z if out == self.unknown_value: 2025-12-04T11:13:50.2776171Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.2776259Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.2776384Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.2776724Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.2776784Z graph_break [] 2025-12-04T11:13:50.2776871Z =================================== FAILURES =================================== 2025-12-04T11:13:50.2777155Z _ TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_1_in_features_144_out_features_64_cuda_bfloat16 _ 2025-12-04T11:13:50.2777231Z Traceback (most recent call last): 2025-12-04T11:13:50.2777534Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.2777597Z method(*args, **kwargs) 2025-12-04T11:13:50.2778140Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.2778246Z method(*args, **kwargs) 2025-12-04T11:13:50.2778731Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T11:13:50.2778804Z with policy(): 2025-12-04T11:13:50.2779114Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T11:13:50.2779184Z raise RuntimeError(msg) 2025-12-04T11:13:50.2779991Z RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_1_in_features_144_out_features_64_cuda_bfloat16! Caching allocator allocated memory was 18432 and is now reported as 27648 on device 0. CUDA driver allocated memory was 276824064 and is now 278921216. 2025-12-04T11:13:50.2779996Z 2025-12-04T11:13:50.2780124Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.2780632Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_1_in_features_144_out_features_64_cuda_bfloat16 2025-12-04T11:13:50.2780636Z 2025-12-04T11:13:50.2780897Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.2781122Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.2781216Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.2781566Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.2781692Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.2781750Z graph_break [] 2025-12-04T11:13:50.2781875Z ----------------------------- Captured stderr call ----------------------------- 2025-12-04T11:13:50.2782565Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/constant_folding.py:256: UserWarning: Unsupported unwinding pattern: Address not in range (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/profiler/unwind/unwind.cpp:219.) 2025-12-04T11:13:50.2782645Z if out == self.unknown_value: 2025-12-04T11:13:50.2782770Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.2782864Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.2783066Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.2783406Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.2783464Z graph_break [] 2025-12-04T11:13:50.2783671Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.2783762Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.2783887Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.2784223Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.2784283Z graph_break [] 2025-12-04T11:13:50.2784778Z - generated xml file: /var/lib/jenkins/workspace/test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-20accf8a0f4e3024.xml - 2025-12-04T11:13:50.2784878Z =========================== short test summary info ============================ 2025-12-04T11:13:50.2786170Z FAILED [0.4141s] inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_1_mid_dim_1_in_features_144_out_features_64_cuda_bfloat16 - RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_1_in_features_144_out_features_64_cuda_bfloat16! Caching allocator allocated memory was 18432 and is now reported as 27648 on device 0. CUDA driver allocated memory was 276824064 and is now 278921216. 2025-12-04T11:13:50.2786175Z 2025-12-04T11:13:50.2786300Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.2786828Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_1_in_features_144_out_features_64_cuda_bfloat16 2025-12-04T11:13:50.2786834Z 2025-12-04T11:13:50.2786997Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.2787104Z !!!!!!!!!!!!!!!!!!!!!!!!!! stopping after 1 failures !!!!!!!!!!!!!!!!!!!!!!!!!!! 2025-12-04T11:13:50.2787224Z ================== 1 failed, 57 deselected, 2 rerun in 11.88s ================== 2025-12-04T11:13:50.2787284Z Got exit code 1 2025-12-04T11:13:50.2787758Z FAILED CONSISTENTLY: test/inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_1_mid_dim_1_in_features_144_out_features_64_cuda_bfloat16 2025-12-04T11:13:50.2788003Z Test failed consistently, continuing with the rest of the tests due to continue-through-error being set 2025-12-04T11:13:50.2788322Z W1204 10:51:26.510000 73429 site-packages/torch/_inductor/utils.py:1703] Not enough SMs to use max_autotune_gemm mode 2025-12-04T11:13:50.2788778Z Test results will be stored in test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-55dc1e1f737da98a.xml 2025-12-04T11:13:50.2788877Z ============================= test session starts ============================== 2025-12-04T11:13:50.2789083Z platform linux -- Python 3.10.14, pytest-7.3.2, pluggy-1.6.0 -- /opt/conda/envs/py_3.10/bin/python 2025-12-04T11:13:50.2789153Z cachedir: .pytest_cache 2025-12-04T11:13:50.2789456Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow] 2025-12-04T11:13:50.2789544Z rootdir: /var/lib/jenkins/workspace 2025-12-04T11:13:50.2789608Z configfile: pytest.ini 2025-12-04T11:13:50.2789920Z plugins: hypothesis-6.56.4, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-14.0, subtests-0.13.1, xdist-3.3.1, xdoctest-1.3.0, typeguard-4.3.0 2025-12-04T11:13:50.2790052Z collecting ... collected 58 items / 30 deselected / 28 selected 2025-12-04T11:13:50.2790141Z stepcurrent: skipping 30 already run items. 2025-12-04T11:13:50.2790211Z Running 28 items in this shard 2025-12-04T11:13:50.2790255Z 2025-12-04T11:13:50.2790749Z inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_1_mid_dim_1_in_features_144_out_features_65_cuda_bfloat16 ('RERUN', {'yellow': True}) [1.9589s] [ 3%] 2025-12-04T11:13:50.2791228Z inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_1_mid_dim_1_in_features_144_out_features_65_cuda_bfloat16 ('RERUN', {'yellow': True}) [0.5409s] [ 3%] 2025-12-04T11:13:50.2791668Z inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_1_mid_dim_1_in_features_144_out_features_65_cuda_bfloat16 FAILED [0.5296s] [ 3%] 2025-12-04T11:13:50.2791672Z 2025-12-04T11:13:50.2791764Z ==================================== RERUNS ==================================== 2025-12-04T11:13:50.2792058Z _ TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_1_in_features_144_out_features_65_cuda_bfloat16 _ 2025-12-04T11:13:50.2792132Z Traceback (most recent call last): 2025-12-04T11:13:50.2792438Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.2792507Z method(*args, **kwargs) 2025-12-04T11:13:50.2792801Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.2792866Z method(*args, **kwargs) 2025-12-04T11:13:50.2793151Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T11:13:50.2793214Z with policy(): 2025-12-04T11:13:50.2793506Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T11:13:50.2793572Z raise RuntimeError(msg) 2025-12-04T11:13:50.2794367Z RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_1_in_features_144_out_features_65_cuda_bfloat16! Caching allocator allocated memory was 0 and is now reported as 9728 on device 0. CUDA driver allocated memory was 230686720 and is now 274726912. 2025-12-04T11:13:50.2794376Z 2025-12-04T11:13:50.2794501Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.2795014Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_1_in_features_144_out_features_65_cuda_bfloat16 2025-12-04T11:13:50.2795018Z 2025-12-04T11:13:50.2795177Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.2795309Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.2795450Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.2796066Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('benchmarking.InductorBenchmarker.benchmark_gpu', 2), ('pad_mm_bench', 1), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.2796199Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.2796273Z graph_break [] 2025-12-04T11:13:50.2796565Z _ TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_1_in_features_144_out_features_65_cuda_bfloat16 _ 2025-12-04T11:13:50.2796642Z Traceback (most recent call last): 2025-12-04T11:13:50.2796942Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.2797007Z method(*args, **kwargs) 2025-12-04T11:13:50.2797295Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.2797361Z method(*args, **kwargs) 2025-12-04T11:13:50.2797646Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T11:13:50.2797747Z with policy(): 2025-12-04T11:13:50.2798038Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T11:13:50.2798106Z raise RuntimeError(msg) 2025-12-04T11:13:50.2798911Z RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_1_in_features_144_out_features_65_cuda_bfloat16! Caching allocator allocated memory was 9728 and is now reported as 19456 on device 0. CUDA driver allocated memory was 274726912 and is now 276824064. 2025-12-04T11:13:50.2798915Z 2025-12-04T11:13:50.2799038Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.2799565Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_1_in_features_144_out_features_65_cuda_bfloat16 2025-12-04T11:13:50.2799570Z 2025-12-04T11:13:50.2799725Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.2799857Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.2799948Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.2800490Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('benchmarking.InductorBenchmarker.benchmark_gpu', 2), ('pad_mm_bench', 1), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.2800623Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.2800682Z graph_break [] 2025-12-04T11:13:50.2800808Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.2800898Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.2801017Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.2801560Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('benchmarking.InductorBenchmarker.benchmark_gpu', 2), ('pad_mm_bench', 1), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.2801619Z graph_break [] 2025-12-04T11:13:50.2801702Z =================================== FAILURES =================================== 2025-12-04T11:13:50.2801990Z _ TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_1_in_features_144_out_features_65_cuda_bfloat16 _ 2025-12-04T11:13:50.2802062Z Traceback (most recent call last): 2025-12-04T11:13:50.2802368Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.2802475Z method(*args, **kwargs) 2025-12-04T11:13:50.2802826Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.2802894Z method(*args, **kwargs) 2025-12-04T11:13:50.2803178Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T11:13:50.2803238Z with policy(): 2025-12-04T11:13:50.2803529Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T11:13:50.2803596Z raise RuntimeError(msg) 2025-12-04T11:13:50.2804403Z RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_1_in_features_144_out_features_65_cuda_bfloat16! Caching allocator allocated memory was 19456 and is now reported as 29184 on device 0. CUDA driver allocated memory was 276824064 and is now 278921216. 2025-12-04T11:13:50.2804409Z 2025-12-04T11:13:50.2804535Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.2805054Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_1_in_features_144_out_features_65_cuda_bfloat16 2025-12-04T11:13:50.2805093Z 2025-12-04T11:13:50.2805248Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.2805374Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.2805463Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.2806000Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('benchmarking.InductorBenchmarker.benchmark_gpu', 2), ('pad_mm_bench', 1), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.2806130Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.2806187Z graph_break [] 2025-12-04T11:13:50.2806311Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.2806405Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.2806524Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.2807061Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('benchmarking.InductorBenchmarker.benchmark_gpu', 2), ('pad_mm_bench', 1), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.2807118Z graph_break [] 2025-12-04T11:13:50.2807239Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.2807329Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.2807447Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.2807983Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('benchmarking.InductorBenchmarker.benchmark_gpu', 2), ('pad_mm_bench', 1), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.2808047Z graph_break [] 2025-12-04T11:13:50.2808540Z - generated xml file: /var/lib/jenkins/workspace/test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-55dc1e1f737da98a.xml - 2025-12-04T11:13:50.2808644Z =========================== short test summary info ============================ 2025-12-04T11:13:50.2809992Z FAILED [0.5296s] inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_1_mid_dim_1_in_features_144_out_features_65_cuda_bfloat16 - RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_1_in_features_144_out_features_65_cuda_bfloat16! Caching allocator allocated memory was 19456 and is now reported as 29184 on device 0. CUDA driver allocated memory was 276824064 and is now 278921216. 2025-12-04T11:13:50.2810047Z 2025-12-04T11:13:50.2810175Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.2810686Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_1_in_features_144_out_features_65_cuda_bfloat16 2025-12-04T11:13:50.2810690Z 2025-12-04T11:13:50.2810845Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.2810947Z !!!!!!!!!!!!!!!!!!!!!!!!!! stopping after 1 failures !!!!!!!!!!!!!!!!!!!!!!!!!!! 2025-12-04T11:13:50.2811060Z ================== 1 failed, 30 deselected, 2 rerun in 3.05s =================== 2025-12-04T11:13:50.2811121Z Got exit code 1 2025-12-04T11:13:50.2811197Z Retrying single test... 2025-12-04T11:13:50.2811460Z W1204 10:51:36.514000 73611 site-packages/torch/_inductor/utils.py:1703] Not enough SMs to use max_autotune_gemm mode 2025-12-04T11:13:50.2811851Z Test results will be stored in test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-458d9d61d3eb81db.xml 2025-12-04T11:13:50.2811984Z ============================= test session starts ============================== 2025-12-04T11:13:50.2812190Z platform linux -- Python 3.10.14, pytest-7.3.2, pluggy-1.6.0 -- /opt/conda/envs/py_3.10/bin/python 2025-12-04T11:13:50.2812260Z cachedir: .pytest_cache 2025-12-04T11:13:50.2812566Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow] 2025-12-04T11:13:50.2812645Z rootdir: /var/lib/jenkins/workspace 2025-12-04T11:13:50.2812709Z configfile: pytest.ini 2025-12-04T11:13:50.2813025Z plugins: hypothesis-6.56.4, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-14.0, subtests-0.13.1, xdist-3.3.1, xdoctest-1.3.0, typeguard-4.3.0 2025-12-04T11:13:50.2813154Z collecting ... collected 58 items / 57 deselected / 1 selected 2025-12-04T11:13:50.2813718Z stepcurrent: skipping 30 already run items. Running only test/inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_1_mid_dim_1_in_features_144_out_features_65_cuda_bfloat16 2025-12-04T11:13:50.2813792Z Running 1 items in this shard 2025-12-04T11:13:50.2813796Z 2025-12-04T11:13:50.2814519Z inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_1_mid_dim_1_in_features_144_out_features_65_cuda_bfloat16 [W1204 10:51:38.152854774 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2814523Z 2025-12-04T11:13:50.2814826Z [W1204 10:51:47.032466963 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2814829Z 2025-12-04T11:13:50.2815119Z [W1204 10:51:47.032730689 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2815125Z 2025-12-04T11:13:50.2815416Z [W1204 10:51:47.038637379 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2815422Z 2025-12-04T11:13:50.2815708Z [W1204 10:51:47.039208001 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2815711Z 2025-12-04T11:13:50.2815994Z [W1204 10:51:47.039387905 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2816000Z 2025-12-04T11:13:50.2816286Z [W1204 10:51:47.044816805 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2816290Z 2025-12-04T11:13:50.2816572Z [W1204 10:51:47.045344477 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2816615Z 2025-12-04T11:13:50.2816967Z [W1204 10:51:47.045504851 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2816972Z 2025-12-04T11:13:50.2817054Z ('RERUN', {'yellow': True}) [10.8607s] [100%] 2025-12-04T11:13:50.2817774Z inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_1_mid_dim_1_in_features_144_out_features_65_cuda_bfloat16 [W1204 10:51:47.854990235 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2817778Z 2025-12-04T11:13:50.2818064Z [W1204 10:51:47.855553128 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2818067Z 2025-12-04T11:13:50.2818353Z [W1204 10:51:47.855694351 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2818359Z 2025-12-04T11:13:50.2818645Z [W1204 10:51:47.858626406 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2818682Z 2025-12-04T11:13:50.2818969Z [W1204 10:51:47.859091726 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2818972Z 2025-12-04T11:13:50.2819258Z [W1204 10:51:47.859228389 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2819261Z 2025-12-04T11:13:50.2819546Z [W1204 10:51:47.863877882 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2819561Z 2025-12-04T11:13:50.2819853Z [W1204 10:51:47.864347792 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2819857Z 2025-12-04T11:13:50.2820143Z [W1204 10:51:47.864481425 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2820148Z 2025-12-04T11:13:50.2820230Z ('RERUN', {'yellow': True}) [0.5071s] [100%] 2025-12-04T11:13:50.2820943Z inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_1_mid_dim_1_in_features_144_out_features_65_cuda_bfloat16 [W1204 10:51:48.358904466 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2820947Z 2025-12-04T11:13:50.2821238Z [W1204 10:51:48.359461288 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2821242Z 2025-12-04T11:13:50.2821525Z [W1204 10:51:48.359602941 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2821530Z 2025-12-04T11:13:50.2821821Z [W1204 10:51:48.362555746 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2821826Z 2025-12-04T11:13:50.2822108Z [W1204 10:51:48.363045367 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2822112Z 2025-12-04T11:13:50.2822396Z [W1204 10:51:48.363183731 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2822399Z 2025-12-04T11:13:50.2822684Z [W1204 10:51:48.367702631 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2822687Z 2025-12-04T11:13:50.2822972Z [W1204 10:51:48.368166711 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2823015Z 2025-12-04T11:13:50.2823450Z [W1204 10:51:48.368301124 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2823455Z 2025-12-04T11:13:50.2823572Z FAILED [0.5024s] [100%] 2025-12-04T11:13:50.2823575Z 2025-12-04T11:13:50.2823663Z ==================================== RERUNS ==================================== 2025-12-04T11:13:50.2823951Z _ TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_1_in_features_144_out_features_65_cuda_bfloat16 _ 2025-12-04T11:13:50.2824025Z Traceback (most recent call last): 2025-12-04T11:13:50.2824340Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.2824406Z method(*args, **kwargs) 2025-12-04T11:13:50.2824699Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.2824763Z method(*args, **kwargs) 2025-12-04T11:13:50.2825051Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T11:13:50.2825155Z with policy(): 2025-12-04T11:13:50.2825445Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T11:13:50.2825510Z raise RuntimeError(msg) 2025-12-04T11:13:50.2826303Z RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_1_in_features_144_out_features_65_cuda_bfloat16! Caching allocator allocated memory was 0 and is now reported as 9728 on device 0. CUDA driver allocated memory was 230686720 and is now 274726912. 2025-12-04T11:13:50.2826307Z 2025-12-04T11:13:50.2826431Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.2826955Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_1_in_features_144_out_features_65_cuda_bfloat16 2025-12-04T11:13:50.2826958Z 2025-12-04T11:13:50.2827115Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.2827243Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.2827336Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.2827879Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('benchmarking.InductorBenchmarker.benchmark_gpu', 2), ('pad_mm_bench', 1), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.2828008Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.2828068Z graph_break [] 2025-12-04T11:13:50.2828194Z ----------------------------- Captured stderr call ----------------------------- 2025-12-04T11:13:50.2828889Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/constant_folding.py:256: UserWarning: Unsupported unwinding pattern: Address not in range (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/profiler/unwind/unwind.cpp:219.) 2025-12-04T11:13:50.2828963Z if out == self.unknown_value: 2025-12-04T11:13:50.2829269Z _ TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_1_in_features_144_out_features_65_cuda_bfloat16 _ 2025-12-04T11:13:50.2829344Z Traceback (most recent call last): 2025-12-04T11:13:50.2829638Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.2829702Z method(*args, **kwargs) 2025-12-04T11:13:50.2829987Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.2830051Z method(*args, **kwargs) 2025-12-04T11:13:50.2830383Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T11:13:50.2830509Z with policy(): 2025-12-04T11:13:50.2830802Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T11:13:50.2830869Z raise RuntimeError(msg) 2025-12-04T11:13:50.2831671Z RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_1_in_features_144_out_features_65_cuda_bfloat16! Caching allocator allocated memory was 9728 and is now reported as 19456 on device 0. CUDA driver allocated memory was 274726912 and is now 276824064. 2025-12-04T11:13:50.2831675Z 2025-12-04T11:13:50.2831797Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.2832322Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_1_in_features_144_out_features_65_cuda_bfloat16 2025-12-04T11:13:50.2832327Z 2025-12-04T11:13:50.2832485Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.2832644Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.2832738Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.2833281Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('benchmarking.InductorBenchmarker.benchmark_gpu', 2), ('pad_mm_bench', 1), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.2833410Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.2833466Z graph_break [] 2025-12-04T11:13:50.2833588Z ----------------------------- Captured stderr call ----------------------------- 2025-12-04T11:13:50.2834278Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/constant_folding.py:256: UserWarning: Unsupported unwinding pattern: Address not in range (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/profiler/unwind/unwind.cpp:219.) 2025-12-04T11:13:50.2834350Z if out == self.unknown_value: 2025-12-04T11:13:50.2834471Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.2834564Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.2834685Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.2835223Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('benchmarking.InductorBenchmarker.benchmark_gpu', 2), ('pad_mm_bench', 1), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.2835280Z graph_break [] 2025-12-04T11:13:50.2835361Z =================================== FAILURES =================================== 2025-12-04T11:13:50.2835649Z _ TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_1_in_features_144_out_features_65_cuda_bfloat16 _ 2025-12-04T11:13:50.2835724Z Traceback (most recent call last): 2025-12-04T11:13:50.2836031Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.2836095Z method(*args, **kwargs) 2025-12-04T11:13:50.2836382Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.2836449Z method(*args, **kwargs) 2025-12-04T11:13:50.2836737Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T11:13:50.2836795Z with policy(): 2025-12-04T11:13:50.2837089Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T11:13:50.2837153Z raise RuntimeError(msg) 2025-12-04T11:13:50.2838082Z RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_1_in_features_144_out_features_65_cuda_bfloat16! Caching allocator allocated memory was 19456 and is now reported as 29184 on device 0. CUDA driver allocated memory was 276824064 and is now 278921216. 2025-12-04T11:13:50.2838089Z 2025-12-04T11:13:50.2838214Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.2838727Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_1_in_features_144_out_features_65_cuda_bfloat16 2025-12-04T11:13:50.2838731Z 2025-12-04T11:13:50.2838886Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.2839012Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.2839105Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.2839653Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('benchmarking.InductorBenchmarker.benchmark_gpu', 2), ('pad_mm_bench', 1), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.2839813Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.2839871Z graph_break [] 2025-12-04T11:13:50.2839993Z ----------------------------- Captured stderr call ----------------------------- 2025-12-04T11:13:50.2840677Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/constant_folding.py:256: UserWarning: Unsupported unwinding pattern: Address not in range (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/profiler/unwind/unwind.cpp:219.) 2025-12-04T11:13:50.2840745Z if out == self.unknown_value: 2025-12-04T11:13:50.2840869Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.2840957Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.2841077Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.2841622Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('benchmarking.InductorBenchmarker.benchmark_gpu', 2), ('pad_mm_bench', 1), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.2841684Z graph_break [] 2025-12-04T11:13:50.2841806Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.2841897Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.2842014Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.2842544Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('benchmarking.InductorBenchmarker.benchmark_gpu', 2), ('pad_mm_bench', 1), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.2842603Z graph_break [] 2025-12-04T11:13:50.2843088Z - generated xml file: /var/lib/jenkins/workspace/test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-458d9d61d3eb81db.xml - 2025-12-04T11:13:50.2843190Z =========================== short test summary info ============================ 2025-12-04T11:13:50.2844465Z FAILED [0.5024s] inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_1_mid_dim_1_in_features_144_out_features_65_cuda_bfloat16 - RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_1_in_features_144_out_features_65_cuda_bfloat16! Caching allocator allocated memory was 19456 and is now reported as 29184 on device 0. CUDA driver allocated memory was 276824064 and is now 278921216. 2025-12-04T11:13:50.2844470Z 2025-12-04T11:13:50.2844593Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.2845210Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_1_in_features_144_out_features_65_cuda_bfloat16 2025-12-04T11:13:50.2845216Z 2025-12-04T11:13:50.2845375Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.2845477Z !!!!!!!!!!!!!!!!!!!!!!!!!! stopping after 1 failures !!!!!!!!!!!!!!!!!!!!!!!!!!! 2025-12-04T11:13:50.2845603Z ================== 1 failed, 57 deselected, 2 rerun in 11.90s ================== 2025-12-04T11:13:50.2845665Z Got exit code 1 2025-12-04T11:13:50.2845728Z Retrying single test... 2025-12-04T11:13:50.2845992Z W1204 10:51:55.221000 73798 site-packages/torch/_inductor/utils.py:1703] Not enough SMs to use max_autotune_gemm mode 2025-12-04T11:13:50.2846376Z Test results will be stored in test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-0cf2b0bead25849b.xml 2025-12-04T11:13:50.2846470Z ============================= test session starts ============================== 2025-12-04T11:13:50.2846680Z platform linux -- Python 3.10.14, pytest-7.3.2, pluggy-1.6.0 -- /opt/conda/envs/py_3.10/bin/python 2025-12-04T11:13:50.2846784Z cachedir: .pytest_cache 2025-12-04T11:13:50.2847087Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow] 2025-12-04T11:13:50.2847164Z rootdir: /var/lib/jenkins/workspace 2025-12-04T11:13:50.2847228Z configfile: pytest.ini 2025-12-04T11:13:50.2847543Z plugins: hypothesis-6.56.4, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-14.0, subtests-0.13.1, xdist-3.3.1, xdoctest-1.3.0, typeguard-4.3.0 2025-12-04T11:13:50.2847669Z collecting ... collected 58 items / 57 deselected / 1 selected 2025-12-04T11:13:50.2848231Z stepcurrent: skipping 30 already run items. Running only test/inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_1_mid_dim_1_in_features_144_out_features_65_cuda_bfloat16 2025-12-04T11:13:50.2848302Z Running 1 items in this shard 2025-12-04T11:13:50.2848305Z 2025-12-04T11:13:50.2849032Z inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_1_mid_dim_1_in_features_144_out_features_65_cuda_bfloat16 [W1204 10:51:56.859273022 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2849038Z 2025-12-04T11:13:50.2849335Z [W1204 10:52:06.953363792 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2849338Z 2025-12-04T11:13:50.2849632Z [W1204 10:52:06.953671259 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2849636Z 2025-12-04T11:13:50.2849923Z [W1204 10:52:06.959480116 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2849929Z 2025-12-04T11:13:50.2850227Z [W1204 10:52:06.960100240 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2850232Z 2025-12-04T11:13:50.2850524Z [W1204 10:52:06.960283504 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2850528Z 2025-12-04T11:13:50.2850813Z [W1204 10:52:06.965626012 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2850817Z 2025-12-04T11:13:50.2851102Z [W1204 10:52:06.966147714 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2851108Z 2025-12-04T11:13:50.2851392Z [W1204 10:52:06.966307087 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2851434Z 2025-12-04T11:13:50.2851517Z ('RERUN', {'yellow': True}) [11.0782s] [100%] 2025-12-04T11:13:50.2852297Z inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_1_mid_dim_1_in_features_144_out_features_65_cuda_bfloat16 [W1204 10:52:06.778919332 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2852303Z 2025-12-04T11:13:50.2852590Z [W1204 10:52:06.779477205 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2852593Z 2025-12-04T11:13:50.2852880Z [W1204 10:52:06.779618218 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2852883Z 2025-12-04T11:13:50.2853163Z [W1204 10:52:06.782521782 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2853168Z 2025-12-04T11:13:50.2853456Z [W1204 10:52:06.782990672 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2853497Z 2025-12-04T11:13:50.2853782Z [W1204 10:52:06.783131485 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2853785Z 2025-12-04T11:13:50.2854075Z [W1204 10:52:06.787697085 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2854079Z 2025-12-04T11:13:50.2854363Z [W1204 10:52:06.788169146 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2854366Z 2025-12-04T11:13:50.2854649Z [W1204 10:52:06.788307159 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2854653Z 2025-12-04T11:13:50.2854734Z ('RERUN', {'yellow': True}) [0.5049s] [100%] 2025-12-04T11:13:50.2855448Z inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_1_mid_dim_1_in_features_144_out_features_65_cuda_bfloat16 [W1204 10:52:07.280986118 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2855454Z 2025-12-04T11:13:50.2855747Z [W1204 10:52:07.281514560 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2855750Z 2025-12-04T11:13:50.2856034Z [W1204 10:52:07.281653343 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2856037Z 2025-12-04T11:13:50.2856322Z [W1204 10:52:07.284537167 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2856327Z 2025-12-04T11:13:50.2856612Z [W1204 10:52:07.285000517 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2856615Z 2025-12-04T11:13:50.2856902Z [W1204 10:52:07.285137310 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2856906Z 2025-12-04T11:13:50.2857187Z [W1204 10:52:07.289694491 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2857190Z 2025-12-04T11:13:50.2857475Z [W1204 10:52:07.290226473 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2857481Z 2025-12-04T11:13:50.2857765Z [W1204 10:52:07.290369096 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2857808Z 2025-12-04T11:13:50.2857873Z FAILED [0.5028s] [100%] 2025-12-04T11:13:50.2857877Z 2025-12-04T11:13:50.2858030Z ==================================== RERUNS ==================================== 2025-12-04T11:13:50.2858320Z _ TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_1_in_features_144_out_features_65_cuda_bfloat16 _ 2025-12-04T11:13:50.2858411Z Traceback (most recent call last): 2025-12-04T11:13:50.2858722Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.2858787Z method(*args, **kwargs) 2025-12-04T11:13:50.2859080Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.2859141Z method(*args, **kwargs) 2025-12-04T11:13:50.2859427Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T11:13:50.2859488Z with policy(): 2025-12-04T11:13:50.2859780Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T11:13:50.2859849Z raise RuntimeError(msg) 2025-12-04T11:13:50.2860673Z RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_1_in_features_144_out_features_65_cuda_bfloat16! Caching allocator allocated memory was 0 and is now reported as 9728 on device 0. CUDA driver allocated memory was 230686720 and is now 274726912. 2025-12-04T11:13:50.2860677Z 2025-12-04T11:13:50.2860802Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.2861316Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_1_in_features_144_out_features_65_cuda_bfloat16 2025-12-04T11:13:50.2861320Z 2025-12-04T11:13:50.2861476Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.2861611Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.2861707Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.2862254Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('benchmarking.InductorBenchmarker.benchmark_gpu', 2), ('pad_mm_bench', 1), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.2862383Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.2862451Z graph_break [] 2025-12-04T11:13:50.2862581Z ----------------------------- Captured stderr call ----------------------------- 2025-12-04T11:13:50.2863268Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/constant_folding.py:256: UserWarning: Unsupported unwinding pattern: Address not in range (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/profiler/unwind/unwind.cpp:219.) 2025-12-04T11:13:50.2863340Z if out == self.unknown_value: 2025-12-04T11:13:50.2863664Z _ TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_1_in_features_144_out_features_65_cuda_bfloat16 _ 2025-12-04T11:13:50.2863740Z Traceback (most recent call last): 2025-12-04T11:13:50.2864034Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.2864096Z method(*args, **kwargs) 2025-12-04T11:13:50.2864379Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.2864445Z method(*args, **kwargs) 2025-12-04T11:13:50.2864728Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T11:13:50.2864787Z with policy(): 2025-12-04T11:13:50.2865087Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T11:13:50.2865220Z raise RuntimeError(msg) 2025-12-04T11:13:50.2866090Z RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_1_in_features_144_out_features_65_cuda_bfloat16! Caching allocator allocated memory was 9728 and is now reported as 19456 on device 0. CUDA driver allocated memory was 274726912 and is now 276824064. 2025-12-04T11:13:50.2866096Z 2025-12-04T11:13:50.2866220Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.2866734Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_1_in_features_144_out_features_65_cuda_bfloat16 2025-12-04T11:13:50.2866737Z 2025-12-04T11:13:50.2866890Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.2867015Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.2867111Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.2867653Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('benchmarking.InductorBenchmarker.benchmark_gpu', 2), ('pad_mm_bench', 1), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.2867819Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.2867877Z graph_break [] 2025-12-04T11:13:50.2867998Z ----------------------------- Captured stderr call ----------------------------- 2025-12-04T11:13:50.2868695Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/constant_folding.py:256: UserWarning: Unsupported unwinding pattern: Address not in range (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/profiler/unwind/unwind.cpp:219.) 2025-12-04T11:13:50.2868766Z if out == self.unknown_value: 2025-12-04T11:13:50.2868895Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.2868985Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.2869105Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.2869651Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('benchmarking.InductorBenchmarker.benchmark_gpu', 2), ('pad_mm_bench', 1), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.2869709Z graph_break [] 2025-12-04T11:13:50.2869790Z =================================== FAILURES =================================== 2025-12-04T11:13:50.2870081Z _ TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_1_in_features_144_out_features_65_cuda_bfloat16 _ 2025-12-04T11:13:50.2870153Z Traceback (most recent call last): 2025-12-04T11:13:50.2870450Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.2870515Z method(*args, **kwargs) 2025-12-04T11:13:50.2870806Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.2870874Z method(*args, **kwargs) 2025-12-04T11:13:50.2871158Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T11:13:50.2871220Z with policy(): 2025-12-04T11:13:50.2871507Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T11:13:50.2871571Z raise RuntimeError(msg) 2025-12-04T11:13:50.2872376Z RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_1_in_features_144_out_features_65_cuda_bfloat16! Caching allocator allocated memory was 19456 and is now reported as 29184 on device 0. CUDA driver allocated memory was 276824064 and is now 278921216. 2025-12-04T11:13:50.2872422Z 2025-12-04T11:13:50.2872609Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.2873125Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_1_in_features_144_out_features_65_cuda_bfloat16 2025-12-04T11:13:50.2873131Z 2025-12-04T11:13:50.2873286Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.2873406Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.2873498Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.2874033Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('benchmarking.InductorBenchmarker.benchmark_gpu', 2), ('pad_mm_bench', 1), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.2874160Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.2874216Z graph_break [] 2025-12-04T11:13:50.2874338Z ----------------------------- Captured stderr call ----------------------------- 2025-12-04T11:13:50.2875070Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/constant_folding.py:256: UserWarning: Unsupported unwinding pattern: Address not in range (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/profiler/unwind/unwind.cpp:219.) 2025-12-04T11:13:50.2875138Z if out == self.unknown_value: 2025-12-04T11:13:50.2875263Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.2875353Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.2875475Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.2876012Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('benchmarking.InductorBenchmarker.benchmark_gpu', 2), ('pad_mm_bench', 1), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.2876074Z graph_break [] 2025-12-04T11:13:50.2876199Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.2876288Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.2876407Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.2876941Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('benchmarking.InductorBenchmarker.benchmark_gpu', 2), ('pad_mm_bench', 1), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.2876997Z graph_break [] 2025-12-04T11:13:50.2877484Z - generated xml file: /var/lib/jenkins/workspace/test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-0cf2b0bead25849b.xml - 2025-12-04T11:13:50.2877588Z =========================== short test summary info ============================ 2025-12-04T11:13:50.2879070Z FAILED [0.5028s] inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_1_mid_dim_1_in_features_144_out_features_65_cuda_bfloat16 - RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_1_in_features_144_out_features_65_cuda_bfloat16! Caching allocator allocated memory was 19456 and is now reported as 29184 on device 0. CUDA driver allocated memory was 276824064 and is now 278921216. 2025-12-04T11:13:50.2879086Z 2025-12-04T11:13:50.2879213Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.2879722Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_1_in_features_144_out_features_65_cuda_bfloat16 2025-12-04T11:13:50.2879797Z 2025-12-04T11:13:50.2879965Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.2880158Z !!!!!!!!!!!!!!!!!!!!!!!!!! stopping after 1 failures !!!!!!!!!!!!!!!!!!!!!!!!!!! 2025-12-04T11:13:50.2880276Z ================== 1 failed, 57 deselected, 2 rerun in 12.11s ================== 2025-12-04T11:13:50.2880341Z Got exit code 1 2025-12-04T11:13:50.2880812Z FAILED CONSISTENTLY: test/inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_1_mid_dim_1_in_features_144_out_features_65_cuda_bfloat16 2025-12-04T11:13:50.2881056Z Test failed consistently, continuing with the rest of the tests due to continue-through-error being set 2025-12-04T11:13:50.2881317Z W1204 10:52:14.163000 73985 site-packages/torch/_inductor/utils.py:1703] Not enough SMs to use max_autotune_gemm mode 2025-12-04T11:13:50.2881699Z Test results will be stored in test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-e47d4d68d26f0ad1.xml 2025-12-04T11:13:50.2881842Z ============================= test session starts ============================== 2025-12-04T11:13:50.2882204Z platform linux -- Python 3.10.14, pytest-7.3.2, pluggy-1.6.0 -- /opt/conda/envs/py_3.10/bin/python 2025-12-04T11:13:50.2882429Z cachedir: .pytest_cache 2025-12-04T11:13:50.2882751Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow] 2025-12-04T11:13:50.2882873Z rootdir: /var/lib/jenkins/workspace 2025-12-04T11:13:50.2882997Z configfile: pytest.ini 2025-12-04T11:13:50.2883349Z plugins: hypothesis-6.56.4, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-14.0, subtests-0.13.1, xdist-3.3.1, xdoctest-1.3.0, typeguard-4.3.0 2025-12-04T11:13:50.2883482Z collecting ... collected 58 items / 31 deselected / 27 selected 2025-12-04T11:13:50.2883569Z stepcurrent: skipping 31 already run items. 2025-12-04T11:13:50.2883642Z Running 27 items in this shard 2025-12-04T11:13:50.2883647Z 2025-12-04T11:13:50.2884307Z inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_1_mid_dim_8_in_features_1024_out_features_1024_cuda_bfloat16 ('RERUN', {'yellow': True}) [2.0321s] [ 3%] 2025-12-04T11:13:50.2884814Z inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_1_mid_dim_8_in_features_1024_out_features_1024_cuda_bfloat16 ('RERUN', {'yellow': True}) [0.6160s] [ 3%] 2025-12-04T11:13:50.2885269Z inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_1_mid_dim_8_in_features_1024_out_features_1024_cuda_bfloat16 FAILED [0.6208s] [ 3%] 2025-12-04T11:13:50.2885274Z 2025-12-04T11:13:50.2885359Z ==================================== RERUNS ==================================== 2025-12-04T11:13:50.2885657Z _ TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_8_in_features_1024_out_features_1024_cuda_bfloat16 _ 2025-12-04T11:13:50.2885737Z Traceback (most recent call last): 2025-12-04T11:13:50.2886051Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.2886122Z method(*args, **kwargs) 2025-12-04T11:13:50.2886412Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.2886476Z method(*args, **kwargs) 2025-12-04T11:13:50.2886767Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T11:13:50.2886827Z with policy(): 2025-12-04T11:13:50.2887117Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T11:13:50.2887186Z raise RuntimeError(msg) 2025-12-04T11:13:50.2888095Z RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_8_in_features_1024_out_features_1024_cuda_bfloat16! Caching allocator allocated memory was 0 and is now reported as 1048576 on device 0. CUDA driver allocated memory was 230686720 and is now 274726912. 2025-12-04T11:13:50.2888133Z 2025-12-04T11:13:50.2888271Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.2888793Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_8_in_features_1024_out_features_1024_cuda_bfloat16 2025-12-04T11:13:50.2888797Z 2025-12-04T11:13:50.2888961Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.2889091Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.2889195Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.2889550Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.2889678Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.2889740Z graph_break [] 2025-12-04T11:13:50.2890031Z _ TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_8_in_features_1024_out_features_1024_cuda_bfloat16 _ 2025-12-04T11:13:50.2890163Z Traceback (most recent call last): 2025-12-04T11:13:50.2890465Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.2890527Z method(*args, **kwargs) 2025-12-04T11:13:50.2890815Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.2890882Z method(*args, **kwargs) 2025-12-04T11:13:50.2891168Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T11:13:50.2891230Z with policy(): 2025-12-04T11:13:50.2891520Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T11:13:50.2891586Z raise RuntimeError(msg) 2025-12-04T11:13:50.2892412Z RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_8_in_features_1024_out_features_1024_cuda_bfloat16! Caching allocator allocated memory was 1048576 and is now reported as 2097152 on device 0. CUDA driver allocated memory was 274726912 and is now 276824064. 2025-12-04T11:13:50.2892416Z 2025-12-04T11:13:50.2892542Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.2893074Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_8_in_features_1024_out_features_1024_cuda_bfloat16 2025-12-04T11:13:50.2893078Z 2025-12-04T11:13:50.2893234Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.2893370Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.2893465Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.2893812Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.2893943Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.2894001Z graph_break [] 2025-12-04T11:13:50.2894124Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.2894215Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.2894335Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.2894674Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.2894778Z graph_break [] 2025-12-04T11:13:50.2894926Z =================================== FAILURES =================================== 2025-12-04T11:13:50.2895221Z _ TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_8_in_features_1024_out_features_1024_cuda_bfloat16 _ 2025-12-04T11:13:50.2895295Z Traceback (most recent call last): 2025-12-04T11:13:50.2895590Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.2895657Z method(*args, **kwargs) 2025-12-04T11:13:50.2895944Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.2896008Z method(*args, **kwargs) 2025-12-04T11:13:50.2896292Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T11:13:50.2896353Z with policy(): 2025-12-04T11:13:50.2896646Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T11:13:50.2896711Z raise RuntimeError(msg) 2025-12-04T11:13:50.2897570Z RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_8_in_features_1024_out_features_1024_cuda_bfloat16! Caching allocator allocated memory was 2097152 and is now reported as 3145728 on device 0. CUDA driver allocated memory was 276824064 and is now 278921216. 2025-12-04T11:13:50.2897580Z 2025-12-04T11:13:50.2897702Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.2898218Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_8_in_features_1024_out_features_1024_cuda_bfloat16 2025-12-04T11:13:50.2898222Z 2025-12-04T11:13:50.2898386Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.2898511Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.2898602Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.2898947Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.2899071Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.2899132Z graph_break [] 2025-12-04T11:13:50.2899251Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.2899347Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.2899468Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.2899807Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.2899867Z graph_break [] 2025-12-04T11:13:50.2899991Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.2900080Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.2900202Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.2900538Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.2900600Z graph_break [] 2025-12-04T11:13:50.2901090Z - generated xml file: /var/lib/jenkins/workspace/test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-e47d4d68d26f0ad1.xml - 2025-12-04T11:13:50.2901190Z =========================== short test summary info ============================ 2025-12-04T11:13:50.2902584Z FAILED [0.6208s] inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_1_mid_dim_8_in_features_1024_out_features_1024_cuda_bfloat16 - RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_8_in_features_1024_out_features_1024_cuda_bfloat16! Caching allocator allocated memory was 2097152 and is now reported as 3145728 on device 0. CUDA driver allocated memory was 276824064 and is now 278921216. 2025-12-04T11:13:50.2902627Z 2025-12-04T11:13:50.2902753Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.2903277Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_8_in_features_1024_out_features_1024_cuda_bfloat16 2025-12-04T11:13:50.2903281Z 2025-12-04T11:13:50.2903436Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.2903623Z !!!!!!!!!!!!!!!!!!!!!!!!!! stopping after 1 failures !!!!!!!!!!!!!!!!!!!!!!!!!!! 2025-12-04T11:13:50.2903746Z ================== 1 failed, 31 deselected, 2 rerun in 3.29s =================== 2025-12-04T11:13:50.2903806Z Got exit code 1 2025-12-04T11:13:50.2903916Z Retrying single test... 2025-12-04T11:13:50.2904180Z W1204 10:52:24.395000 74174 site-packages/torch/_inductor/utils.py:1703] Not enough SMs to use max_autotune_gemm mode 2025-12-04T11:13:50.2904564Z Test results will be stored in test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-044dfd62cda981dc.xml 2025-12-04T11:13:50.2904662Z ============================= test session starts ============================== 2025-12-04T11:13:50.2904877Z platform linux -- Python 3.10.14, pytest-7.3.2, pluggy-1.6.0 -- /opt/conda/envs/py_3.10/bin/python 2025-12-04T11:13:50.2904947Z cachedir: .pytest_cache 2025-12-04T11:13:50.2905249Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow] 2025-12-04T11:13:50.2905328Z rootdir: /var/lib/jenkins/workspace 2025-12-04T11:13:50.2905395Z configfile: pytest.ini 2025-12-04T11:13:50.2905710Z plugins: hypothesis-6.56.4, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-14.0, subtests-0.13.1, xdist-3.3.1, xdoctest-1.3.0, typeguard-4.3.0 2025-12-04T11:13:50.2905843Z collecting ... collected 58 items / 57 deselected / 1 selected 2025-12-04T11:13:50.2906420Z stepcurrent: skipping 31 already run items. Running only test/inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_1_mid_dim_8_in_features_1024_out_features_1024_cuda_bfloat16 2025-12-04T11:13:50.2906491Z Running 1 items in this shard 2025-12-04T11:13:50.2906494Z 2025-12-04T11:13:50.2907230Z inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_1_mid_dim_8_in_features_1024_out_features_1024_cuda_bfloat16 [W1204 10:52:25.670922244 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2907235Z 2025-12-04T11:13:50.2907535Z [W1204 10:52:35.017608699 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2907540Z 2025-12-04T11:13:50.2907832Z [W1204 10:52:35.017846684 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2907835Z 2025-12-04T11:13:50.2908120Z [W1204 10:52:35.023519400 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2908124Z 2025-12-04T11:13:50.2908411Z [W1204 10:52:35.024070112 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2908414Z 2025-12-04T11:13:50.2908698Z [W1204 10:52:35.024238256 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2908748Z 2025-12-04T11:13:50.2909102Z [W1204 10:52:35.029718307 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2909108Z 2025-12-04T11:13:50.2909401Z [W1204 10:52:35.030305540 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2909404Z 2025-12-04T11:13:50.2909689Z [W1204 10:52:35.030477154 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2909695Z 2025-12-04T11:13:50.2909776Z ('RERUN', {'yellow': True}) [11.4156s] [100%] 2025-12-04T11:13:50.2910504Z inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_1_mid_dim_8_in_features_1024_out_features_1024_cuda_bfloat16 [W1204 10:52:36.379835774 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2910510Z 2025-12-04T11:13:50.2910797Z [W1204 10:52:36.380404286 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2910836Z 2025-12-04T11:13:50.2911121Z [W1204 10:52:36.380545579 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2911125Z 2025-12-04T11:13:50.2911415Z [W1204 10:52:36.383610707 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2911418Z 2025-12-04T11:13:50.2911704Z [W1204 10:52:36.384176879 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2911707Z 2025-12-04T11:13:50.2911994Z [W1204 10:52:36.384313692 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2911999Z 2025-12-04T11:13:50.2912286Z [W1204 10:52:36.388950454 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2912291Z 2025-12-04T11:13:50.2912576Z [W1204 10:52:36.389417244 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2912583Z 2025-12-04T11:13:50.2912866Z [W1204 10:52:36.389551377 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2912869Z 2025-12-04T11:13:50.2912949Z ('RERUN', {'yellow': True}) [0.5861s] [100%] 2025-12-04T11:13:50.2913673Z inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_1_mid_dim_8_in_features_1024_out_features_1024_cuda_bfloat16 [W1204 10:52:37.962795747 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2913678Z 2025-12-04T11:13:50.2913965Z [W1204 10:52:37.963336149 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2913970Z 2025-12-04T11:13:50.2914258Z [W1204 10:52:37.963477133 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2914262Z 2025-12-04T11:13:50.2914545Z [W1204 10:52:37.966483959 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2914548Z 2025-12-04T11:13:50.2914841Z [W1204 10:52:37.967048731 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2914844Z 2025-12-04T11:13:50.2915131Z [W1204 10:52:37.967186504 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2915173Z 2025-12-04T11:13:50.2915548Z [W1204 10:52:37.971857357 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2915551Z 2025-12-04T11:13:50.2915839Z [W1204 10:52:37.972337677 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2915842Z 2025-12-04T11:13:50.2916125Z [W1204 10:52:37.972471971 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2916132Z 2025-12-04T11:13:50.2916193Z FAILED [0.5866s] [100%] 2025-12-04T11:13:50.2916196Z 2025-12-04T11:13:50.2916280Z ==================================== RERUNS ==================================== 2025-12-04T11:13:50.2916580Z _ TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_8_in_features_1024_out_features_1024_cuda_bfloat16 _ 2025-12-04T11:13:50.2916654Z Traceback (most recent call last): 2025-12-04T11:13:50.2916963Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.2917031Z method(*args, **kwargs) 2025-12-04T11:13:50.2917358Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.2917422Z method(*args, **kwargs) 2025-12-04T11:13:50.2917706Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T11:13:50.2917767Z with policy(): 2025-12-04T11:13:50.2918059Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T11:13:50.2918123Z raise RuntimeError(msg) 2025-12-04T11:13:50.2918944Z RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_8_in_features_1024_out_features_1024_cuda_bfloat16! Caching allocator allocated memory was 0 and is now reported as 1048576 on device 0. CUDA driver allocated memory was 230686720 and is now 274726912. 2025-12-04T11:13:50.2918950Z 2025-12-04T11:13:50.2919078Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.2919605Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_8_in_features_1024_out_features_1024_cuda_bfloat16 2025-12-04T11:13:50.2919608Z 2025-12-04T11:13:50.2919767Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.2919894Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.2919989Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.2920341Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.2920466Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.2920531Z graph_break [] 2025-12-04T11:13:50.2920653Z ----------------------------- Captured stderr call ----------------------------- 2025-12-04T11:13:50.2921348Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/constant_folding.py:256: UserWarning: Unsupported unwinding pattern: Address not in range (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/profiler/unwind/unwind.cpp:219.) 2025-12-04T11:13:50.2921420Z if out == self.unknown_value: 2025-12-04T11:13:50.2921709Z _ TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_8_in_features_1024_out_features_1024_cuda_bfloat16 _ 2025-12-04T11:13:50.2921793Z Traceback (most recent call last): 2025-12-04T11:13:50.2922091Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.2922196Z method(*args, **kwargs) 2025-12-04T11:13:50.2922551Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.2922614Z method(*args, **kwargs) 2025-12-04T11:13:50.2922906Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T11:13:50.2922965Z with policy(): 2025-12-04T11:13:50.2923256Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T11:13:50.2923324Z raise RuntimeError(msg) 2025-12-04T11:13:50.2924144Z RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_8_in_features_1024_out_features_1024_cuda_bfloat16! Caching allocator allocated memory was 1048576 and is now reported as 2097152 on device 0. CUDA driver allocated memory was 274726912 and is now 276824064. 2025-12-04T11:13:50.2924150Z 2025-12-04T11:13:50.2924276Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.2924796Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_8_in_features_1024_out_features_1024_cuda_bfloat16 2025-12-04T11:13:50.2924834Z 2025-12-04T11:13:50.2924992Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.2925118Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.2925209Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.2925560Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.2925692Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.2925754Z graph_break [] 2025-12-04T11:13:50.2925882Z ----------------------------- Captured stderr call ----------------------------- 2025-12-04T11:13:50.2926565Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/constant_folding.py:256: UserWarning: Unsupported unwinding pattern: Address not in range (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/profiler/unwind/unwind.cpp:219.) 2025-12-04T11:13:50.2926641Z if out == self.unknown_value: 2025-12-04T11:13:50.2926765Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.2926854Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.2926982Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.2927331Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.2927391Z graph_break [] 2025-12-04T11:13:50.2927475Z =================================== FAILURES =================================== 2025-12-04T11:13:50.2927767Z _ TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_8_in_features_1024_out_features_1024_cuda_bfloat16 _ 2025-12-04T11:13:50.2927845Z Traceback (most recent call last): 2025-12-04T11:13:50.2928141Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.2928203Z method(*args, **kwargs) 2025-12-04T11:13:50.2928495Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.2928556Z method(*args, **kwargs) 2025-12-04T11:13:50.2928846Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T11:13:50.2928907Z with policy(): 2025-12-04T11:13:50.2929196Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T11:13:50.2929320Z raise RuntimeError(msg) 2025-12-04T11:13:50.2930207Z RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_8_in_features_1024_out_features_1024_cuda_bfloat16! Caching allocator allocated memory was 2097152 and is now reported as 3145728 on device 0. CUDA driver allocated memory was 276824064 and is now 278921216. 2025-12-04T11:13:50.2930213Z 2025-12-04T11:13:50.2930340Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.2930857Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_8_in_features_1024_out_features_1024_cuda_bfloat16 2025-12-04T11:13:50.2930861Z 2025-12-04T11:13:50.2931013Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.2931141Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.2931234Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.2931577Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.2931734Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.2931791Z graph_break [] 2025-12-04T11:13:50.2931919Z ----------------------------- Captured stderr call ----------------------------- 2025-12-04T11:13:50.2932602Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/constant_folding.py:256: UserWarning: Unsupported unwinding pattern: Address not in range (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/profiler/unwind/unwind.cpp:219.) 2025-12-04T11:13:50.2932673Z if out == self.unknown_value: 2025-12-04T11:13:50.2932793Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.2932883Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.2933009Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.2933351Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.2933409Z graph_break [] 2025-12-04T11:13:50.2933538Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.2933624Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.2933746Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.2934083Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.2934140Z graph_break [] 2025-12-04T11:13:50.2934630Z - generated xml file: /var/lib/jenkins/workspace/test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-044dfd62cda981dc.xml - 2025-12-04T11:13:50.2934729Z =========================== short test summary info ============================ 2025-12-04T11:13:50.2936033Z FAILED [0.5866s] inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_1_mid_dim_8_in_features_1024_out_features_1024_cuda_bfloat16 - RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_8_in_features_1024_out_features_1024_cuda_bfloat16! Caching allocator allocated memory was 2097152 and is now reported as 3145728 on device 0. CUDA driver allocated memory was 276824064 and is now 278921216. 2025-12-04T11:13:50.2936037Z 2025-12-04T11:13:50.2936159Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.2936834Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_8_in_features_1024_out_features_1024_cuda_bfloat16 2025-12-04T11:13:50.2936873Z 2025-12-04T11:13:50.2937029Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.2937131Z !!!!!!!!!!!!!!!!!!!!!!!!!! stopping after 1 failures !!!!!!!!!!!!!!!!!!!!!!!!!!! 2025-12-04T11:13:50.2937247Z ================== 1 failed, 57 deselected, 2 rerun in 12.61s ================== 2025-12-04T11:13:50.2937305Z Got exit code 1 2025-12-04T11:13:50.2937377Z Retrying single test... 2025-12-04T11:13:50.2937641Z W1204 10:52:43.835000 74368 site-packages/torch/_inductor/utils.py:1703] Not enough SMs to use max_autotune_gemm mode 2025-12-04T11:13:50.2938021Z Test results will be stored in test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-f4c669524343c1b1.xml 2025-12-04T11:13:50.2938118Z ============================= test session starts ============================== 2025-12-04T11:13:50.2938327Z platform linux -- Python 3.10.14, pytest-7.3.2, pluggy-1.6.0 -- /opt/conda/envs/py_3.10/bin/python 2025-12-04T11:13:50.2938395Z cachedir: .pytest_cache 2025-12-04T11:13:50.2938743Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow] 2025-12-04T11:13:50.2938818Z rootdir: /var/lib/jenkins/workspace 2025-12-04T11:13:50.2938885Z configfile: pytest.ini 2025-12-04T11:13:50.2939197Z plugins: hypothesis-6.56.4, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-14.0, subtests-0.13.1, xdist-3.3.1, xdoctest-1.3.0, typeguard-4.3.0 2025-12-04T11:13:50.2939324Z collecting ... collected 58 items / 57 deselected / 1 selected 2025-12-04T11:13:50.2939894Z stepcurrent: skipping 31 already run items. Running only test/inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_1_mid_dim_8_in_features_1024_out_features_1024_cuda_bfloat16 2025-12-04T11:13:50.2939965Z Running 1 items in this shard 2025-12-04T11:13:50.2939969Z 2025-12-04T11:13:50.2940706Z inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_1_mid_dim_8_in_features_1024_out_features_1024_cuda_bfloat16 [W1204 10:52:45.108979521 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2940712Z 2025-12-04T11:13:50.2941010Z [W1204 10:52:54.145300130 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2941013Z 2025-12-04T11:13:50.2941302Z [W1204 10:52:54.145536955 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2941306Z 2025-12-04T11:13:50.2941588Z [W1204 10:52:54.151279582 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2941593Z 2025-12-04T11:13:50.2941879Z [W1204 10:52:54.151828704 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2941885Z 2025-12-04T11:13:50.2942172Z [W1204 10:52:54.152010758 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2942175Z 2025-12-04T11:13:50.2942460Z [W1204 10:52:54.157325765 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2942464Z 2025-12-04T11:13:50.2942752Z [W1204 10:52:54.157876997 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2942755Z 2025-12-04T11:13:50.2943038Z [W1204 10:52:54.158050231 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2943080Z 2025-12-04T11:13:50.2943172Z ('RERUN', {'yellow': True}) [11.1088s] [100%] 2025-12-04T11:13:50.2944035Z inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_1_mid_dim_8_in_features_1024_out_features_1024_cuda_bfloat16 [W1204 10:52:55.525903663 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2944042Z 2025-12-04T11:13:50.2944336Z [W1204 10:52:55.526444215 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2944339Z 2025-12-04T11:13:50.2944625Z [W1204 10:52:55.526582998 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2944628Z 2025-12-04T11:13:50.2944915Z [W1204 10:52:55.529508563 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2944920Z 2025-12-04T11:13:50.2945205Z [W1204 10:52:55.530087086 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2945208Z 2025-12-04T11:13:50.2945492Z [W1204 10:52:55.530227969 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2945533Z 2025-12-04T11:13:50.2945819Z [W1204 10:52:55.534758939 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2945822Z 2025-12-04T11:13:50.2946106Z [W1204 10:52:55.535221959 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2946110Z 2025-12-04T11:13:50.2946395Z [W1204 10:52:55.535355232 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2946398Z 2025-12-04T11:13:50.2946478Z ('RERUN', {'yellow': True}) [0.6054s] [100%] 2025-12-04T11:13:50.2947205Z inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_1_mid_dim_8_in_features_1024_out_features_1024_cuda_bfloat16 [W1204 10:52:56.126999335 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2947210Z 2025-12-04T11:13:50.2947492Z [W1204 10:52:56.127556668 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2947496Z 2025-12-04T11:13:50.2947782Z [W1204 10:52:56.127700171 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2947786Z 2025-12-04T11:13:50.2948069Z [W1204 10:52:56.130665516 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2948072Z 2025-12-04T11:13:50.2948368Z [W1204 10:52:56.131236259 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2948373Z 2025-12-04T11:13:50.2948659Z [W1204 10:52:56.131373202 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2948664Z 2025-12-04T11:13:50.2948948Z [W1204 10:52:56.135935653 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2948959Z 2025-12-04T11:13:50.2949241Z [W1204 10:52:56.136405623 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2949244Z 2025-12-04T11:13:50.2949532Z [W1204 10:52:56.136540557 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2949535Z 2025-12-04T11:13:50.2949602Z FAILED [0.6051s] [100%] 2025-12-04T11:13:50.2949644Z 2025-12-04T11:13:50.2949728Z ==================================== RERUNS ==================================== 2025-12-04T11:13:50.2950089Z _ TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_8_in_features_1024_out_features_1024_cuda_bfloat16 _ 2025-12-04T11:13:50.2950165Z Traceback (most recent call last): 2025-12-04T11:13:50.2950465Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.2950534Z method(*args, **kwargs) 2025-12-04T11:13:50.2950836Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.2950899Z method(*args, **kwargs) 2025-12-04T11:13:50.2951189Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T11:13:50.2951250Z with policy(): 2025-12-04T11:13:50.2951546Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T11:13:50.2951618Z raise RuntimeError(msg) 2025-12-04T11:13:50.2952428Z RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_8_in_features_1024_out_features_1024_cuda_bfloat16! Caching allocator allocated memory was 0 and is now reported as 1048576 on device 0. CUDA driver allocated memory was 230686720 and is now 274726912. 2025-12-04T11:13:50.2952473Z 2025-12-04T11:13:50.2952598Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.2953120Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_8_in_features_1024_out_features_1024_cuda_bfloat16 2025-12-04T11:13:50.2953124Z 2025-12-04T11:13:50.2953283Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.2953412Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.2953509Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.2953858Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.2953987Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.2954048Z graph_break [] 2025-12-04T11:13:50.2954172Z ----------------------------- Captured stderr call ----------------------------- 2025-12-04T11:13:50.2954859Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/constant_folding.py:256: UserWarning: Unsupported unwinding pattern: Address not in range (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/profiler/unwind/unwind.cpp:219.) 2025-12-04T11:13:50.2954931Z if out == self.unknown_value: 2025-12-04T11:13:50.2955221Z _ TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_8_in_features_1024_out_features_1024_cuda_bfloat16 _ 2025-12-04T11:13:50.2955299Z Traceback (most recent call last): 2025-12-04T11:13:50.2955602Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.2955668Z method(*args, **kwargs) 2025-12-04T11:13:50.2955960Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.2956023Z method(*args, **kwargs) 2025-12-04T11:13:50.2956311Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T11:13:50.2956370Z with policy(): 2025-12-04T11:13:50.2956659Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T11:13:50.2956729Z raise RuntimeError(msg) 2025-12-04T11:13:50.2957661Z RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_8_in_features_1024_out_features_1024_cuda_bfloat16! Caching allocator allocated memory was 1048576 and is now reported as 2097152 on device 0. CUDA driver allocated memory was 274726912 and is now 276824064. 2025-12-04T11:13:50.2957667Z 2025-12-04T11:13:50.2957805Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.2958326Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_8_in_features_1024_out_features_1024_cuda_bfloat16 2025-12-04T11:13:50.2958330Z 2025-12-04T11:13:50.2958487Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.2958615Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.2958705Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.2959057Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.2959217Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.2959276Z graph_break [] 2025-12-04T11:13:50.2959410Z ----------------------------- Captured stderr call ----------------------------- 2025-12-04T11:13:50.2960094Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/constant_folding.py:256: UserWarning: Unsupported unwinding pattern: Address not in range (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/profiler/unwind/unwind.cpp:219.) 2025-12-04T11:13:50.2960165Z if out == self.unknown_value: 2025-12-04T11:13:50.2960287Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.2960374Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.2960501Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.2960844Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.2960903Z graph_break [] 2025-12-04T11:13:50.2960989Z =================================== FAILURES =================================== 2025-12-04T11:13:50.2961278Z _ TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_8_in_features_1024_out_features_1024_cuda_bfloat16 _ 2025-12-04T11:13:50.2961352Z Traceback (most recent call last): 2025-12-04T11:13:50.2961646Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.2961708Z method(*args, **kwargs) 2025-12-04T11:13:50.2961998Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.2962063Z method(*args, **kwargs) 2025-12-04T11:13:50.2962348Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T11:13:50.2962412Z with policy(): 2025-12-04T11:13:50.2962698Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T11:13:50.2962766Z raise RuntimeError(msg) 2025-12-04T11:13:50.2963588Z RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_8_in_features_1024_out_features_1024_cuda_bfloat16! Caching allocator allocated memory was 2097152 and is now reported as 3145728 on device 0. CUDA driver allocated memory was 276824064 and is now 278921216. 2025-12-04T11:13:50.2963593Z 2025-12-04T11:13:50.2963719Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.2964311Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_8_in_features_1024_out_features_1024_cuda_bfloat16 2025-12-04T11:13:50.2964347Z 2025-12-04T11:13:50.2964506Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.2964634Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.2964724Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.2965072Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.2965194Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.2965261Z graph_break [] 2025-12-04T11:13:50.2965389Z ----------------------------- Captured stderr call ----------------------------- 2025-12-04T11:13:50.2966074Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/constant_folding.py:256: UserWarning: Unsupported unwinding pattern: Address not in range (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/profiler/unwind/unwind.cpp:219.) 2025-12-04T11:13:50.2966186Z if out == self.unknown_value: 2025-12-04T11:13:50.2966314Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.2966402Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.2966527Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.2966869Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.2966928Z graph_break [] 2025-12-04T11:13:50.2967053Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.2967139Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.2967264Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.2967603Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.2967663Z graph_break [] 2025-12-04T11:13:50.2968149Z - generated xml file: /var/lib/jenkins/workspace/test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-f4c669524343c1b1.xml - 2025-12-04T11:13:50.2968250Z =========================== short test summary info ============================ 2025-12-04T11:13:50.2969561Z FAILED [0.6051s] inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_1_mid_dim_8_in_features_1024_out_features_1024_cuda_bfloat16 - RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_8_in_features_1024_out_features_1024_cuda_bfloat16! Caching allocator allocated memory was 2097152 and is now reported as 3145728 on device 0. CUDA driver allocated memory was 276824064 and is now 278921216. 2025-12-04T11:13:50.2969567Z 2025-12-04T11:13:50.2969689Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.2970223Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_8_in_features_1024_out_features_1024_cuda_bfloat16 2025-12-04T11:13:50.2970227Z 2025-12-04T11:13:50.2970380Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.2970484Z !!!!!!!!!!!!!!!!!!!!!!!!!! stopping after 1 failures !!!!!!!!!!!!!!!!!!!!!!!!!!! 2025-12-04T11:13:50.2970603Z ================== 1 failed, 57 deselected, 2 rerun in 12.34s ================== 2025-12-04T11:13:50.2970663Z Got exit code 1 2025-12-04T11:13:50.2971133Z FAILED CONSISTENTLY: test/inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_1_mid_dim_8_in_features_1024_out_features_1024_cuda_bfloat16 2025-12-04T11:13:50.2971504Z Test failed consistently, continuing with the rest of the tests due to continue-through-error being set 2025-12-04T11:13:50.2971768Z W1204 10:53:02.964000 74562 site-packages/torch/_inductor/utils.py:1703] Not enough SMs to use max_autotune_gemm mode 2025-12-04T11:13:50.2972149Z Test results will be stored in test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-9f9264fbba4173a1.xml 2025-12-04T11:13:50.2972245Z ============================= test session starts ============================== 2025-12-04T11:13:50.2972451Z platform linux -- Python 3.10.14, pytest-7.3.2, pluggy-1.6.0 -- /opt/conda/envs/py_3.10/bin/python 2025-12-04T11:13:50.2972520Z cachedir: .pytest_cache 2025-12-04T11:13:50.2972825Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow] 2025-12-04T11:13:50.2972904Z rootdir: /var/lib/jenkins/workspace 2025-12-04T11:13:50.2972968Z configfile: pytest.ini 2025-12-04T11:13:50.2973281Z plugins: hypothesis-6.56.4, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-14.0, subtests-0.13.1, xdist-3.3.1, xdoctest-1.3.0, typeguard-4.3.0 2025-12-04T11:13:50.2973457Z collecting ... collected 58 items / 32 deselected / 26 selected 2025-12-04T11:13:50.2973544Z stepcurrent: skipping 32 already run items. 2025-12-04T11:13:50.2973613Z Running 26 items in this shard 2025-12-04T11:13:50.2973620Z 2025-12-04T11:13:50.2974117Z inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_1_mid_dim_8_in_features_1024_out_features_64_cuda_bfloat16 ('RERUN', {'yellow': True}) [1.8581s] [ 3%] 2025-12-04T11:13:50.2974601Z inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_1_mid_dim_8_in_features_1024_out_features_64_cuda_bfloat16 ('RERUN', {'yellow': True}) [0.4484s] [ 3%] 2025-12-04T11:13:50.2975049Z inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_1_mid_dim_8_in_features_1024_out_features_64_cuda_bfloat16 FAILED [0.4576s] [ 3%] 2025-12-04T11:13:50.2975053Z 2025-12-04T11:13:50.2975134Z ==================================== RERUNS ==================================== 2025-12-04T11:13:50.2975428Z _ TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_8_in_features_1024_out_features_64_cuda_bfloat16 _ 2025-12-04T11:13:50.2975501Z Traceback (most recent call last): 2025-12-04T11:13:50.2975804Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.2975873Z method(*args, **kwargs) 2025-12-04T11:13:50.2976162Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.2976227Z method(*args, **kwargs) 2025-12-04T11:13:50.2976516Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T11:13:50.2976580Z with policy(): 2025-12-04T11:13:50.2976874Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T11:13:50.2976943Z raise RuntimeError(msg) 2025-12-04T11:13:50.2977744Z RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_8_in_features_1024_out_features_64_cuda_bfloat16! Caching allocator allocated memory was 0 and is now reported as 65536 on device 0. CUDA driver allocated memory was 230686720 and is now 274726912. 2025-12-04T11:13:50.2977751Z 2025-12-04T11:13:50.2978057Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.2978579Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_8_in_features_1024_out_features_64_cuda_bfloat16 2025-12-04T11:13:50.2978655Z 2025-12-04T11:13:50.2978913Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.2979050Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.2979150Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.2979497Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.2979626Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.2979689Z graph_break [] 2025-12-04T11:13:50.2979982Z _ TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_8_in_features_1024_out_features_64_cuda_bfloat16 _ 2025-12-04T11:13:50.2980056Z Traceback (most recent call last): 2025-12-04T11:13:50.2980356Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.2980423Z method(*args, **kwargs) 2025-12-04T11:13:50.2980717Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.2980834Z method(*args, **kwargs) 2025-12-04T11:13:50.2981121Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T11:13:50.2981185Z with policy(): 2025-12-04T11:13:50.2981502Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T11:13:50.2981572Z raise RuntimeError(msg) 2025-12-04T11:13:50.2982391Z RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_8_in_features_1024_out_features_64_cuda_bfloat16! Caching allocator allocated memory was 65536 and is now reported as 131072 on device 0. CUDA driver allocated memory was 274726912 and is now 276824064. 2025-12-04T11:13:50.2982397Z 2025-12-04T11:13:50.2982531Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.2983068Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_8_in_features_1024_out_features_64_cuda_bfloat16 2025-12-04T11:13:50.2983072Z 2025-12-04T11:13:50.2983232Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.2983368Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.2983462Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.2983873Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.2984008Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.2984067Z graph_break [] 2025-12-04T11:13:50.2984200Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.2984291Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.2984410Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.2984755Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.2984812Z graph_break [] 2025-12-04T11:13:50.2984898Z =================================== FAILURES =================================== 2025-12-04T11:13:50.2985195Z _ TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_8_in_features_1024_out_features_64_cuda_bfloat16 _ 2025-12-04T11:13:50.2985268Z Traceback (most recent call last): 2025-12-04T11:13:50.2985574Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.2985693Z method(*args, **kwargs) 2025-12-04T11:13:50.2986051Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.2986124Z method(*args, **kwargs) 2025-12-04T11:13:50.2986411Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T11:13:50.2986469Z with policy(): 2025-12-04T11:13:50.2986766Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T11:13:50.2986829Z raise RuntimeError(msg) 2025-12-04T11:13:50.2987646Z RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_8_in_features_1024_out_features_64_cuda_bfloat16! Caching allocator allocated memory was 131072 and is now reported as 196608 on device 0. CUDA driver allocated memory was 276824064 and is now 278921216. 2025-12-04T11:13:50.2987652Z 2025-12-04T11:13:50.2987780Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.2988334Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_8_in_features_1024_out_features_64_cuda_bfloat16 2025-12-04T11:13:50.2988338Z 2025-12-04T11:13:50.2988496Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.2988621Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.2988716Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.2989057Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.2989186Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.2989242Z graph_break [] 2025-12-04T11:13:50.2989368Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.2989468Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.2989589Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.2989927Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.2989989Z graph_break [] 2025-12-04T11:13:50.2990113Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.2990206Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.2990327Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.2990663Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.2990730Z graph_break [] 2025-12-04T11:13:50.2991219Z - generated xml file: /var/lib/jenkins/workspace/test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-9f9264fbba4173a1.xml - 2025-12-04T11:13:50.2991325Z =========================== short test summary info ============================ 2025-12-04T11:13:50.2992619Z FAILED [0.4576s] inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_1_mid_dim_8_in_features_1024_out_features_64_cuda_bfloat16 - RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_8_in_features_1024_out_features_64_cuda_bfloat16! Caching allocator allocated memory was 131072 and is now reported as 196608 on device 0. CUDA driver allocated memory was 276824064 and is now 278921216. 2025-12-04T11:13:50.2992686Z 2025-12-04T11:13:50.2992818Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.2993400Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_8_in_features_1024_out_features_64_cuda_bfloat16 2025-12-04T11:13:50.2993406Z 2025-12-04T11:13:50.2993560Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.2993666Z !!!!!!!!!!!!!!!!!!!!!!!!!! stopping after 1 failures !!!!!!!!!!!!!!!!!!!!!!!!!!! 2025-12-04T11:13:50.2993781Z ================== 1 failed, 32 deselected, 2 rerun in 2.79s =================== 2025-12-04T11:13:50.2993845Z Got exit code 1 2025-12-04T11:13:50.2993909Z Retrying single test... 2025-12-04T11:13:50.2994173Z W1204 10:53:12.964000 74750 site-packages/torch/_inductor/utils.py:1703] Not enough SMs to use max_autotune_gemm mode 2025-12-04T11:13:50.2994566Z Test results will be stored in test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-c85f34f0c90d933d.xml 2025-12-04T11:13:50.2994666Z ============================= test session starts ============================== 2025-12-04T11:13:50.2994929Z platform linux -- Python 3.10.14, pytest-7.3.2, pluggy-1.6.0 -- /opt/conda/envs/py_3.10/bin/python 2025-12-04T11:13:50.2994999Z cachedir: .pytest_cache 2025-12-04T11:13:50.2995303Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow] 2025-12-04T11:13:50.2995382Z rootdir: /var/lib/jenkins/workspace 2025-12-04T11:13:50.2995451Z configfile: pytest.ini 2025-12-04T11:13:50.2995767Z plugins: hypothesis-6.56.4, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-14.0, subtests-0.13.1, xdist-3.3.1, xdoctest-1.3.0, typeguard-4.3.0 2025-12-04T11:13:50.2995899Z collecting ... collected 58 items / 57 deselected / 1 selected 2025-12-04T11:13:50.2996466Z stepcurrent: skipping 32 already run items. Running only test/inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_1_mid_dim_8_in_features_1024_out_features_64_cuda_bfloat16 2025-12-04T11:13:50.2996543Z Running 1 items in this shard 2025-12-04T11:13:50.2996548Z 2025-12-04T11:13:50.2997275Z inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_1_mid_dim_8_in_features_1024_out_features_64_cuda_bfloat16 [W1204 10:53:14.037587203 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2997279Z 2025-12-04T11:13:50.2997575Z [W1204 10:53:23.383097005 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2997581Z 2025-12-04T11:13:50.2997869Z [W1204 10:53:23.383343330 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2997874Z 2025-12-04T11:13:50.2998160Z [W1204 10:53:23.389146929 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2998164Z 2025-12-04T11:13:50.2998453Z [W1204 10:53:23.389719061 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2998458Z 2025-12-04T11:13:50.2998739Z [W1204 10:53:23.389877565 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2998742Z 2025-12-04T11:13:50.2999033Z [W1204 10:53:23.395333545 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2999037Z 2025-12-04T11:13:50.2999322Z [W1204 10:53:23.395868596 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2999325Z 2025-12-04T11:13:50.2999655Z [W1204 10:53:23.396033280 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.2999735Z 2025-12-04T11:13:50.2999818Z ('RERUN', {'yellow': True}) [11.2164s] [100%] 2025-12-04T11:13:50.3000535Z inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_1_mid_dim_8_in_features_1024_out_features_64_cuda_bfloat16 [W1204 10:53:24.576971178 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3000541Z 2025-12-04T11:13:50.3000828Z [W1204 10:53:24.577513989 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3000831Z 2025-12-04T11:13:50.3001116Z [W1204 10:53:24.577653952 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3001122Z 2025-12-04T11:13:50.3001411Z [W1204 10:53:24.580676831 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3001416Z 2025-12-04T11:13:50.3001700Z [W1204 10:53:24.581253203 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3001738Z 2025-12-04T11:13:50.3002027Z [W1204 10:53:24.581389716 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3002031Z 2025-12-04T11:13:50.3002313Z [W1204 10:53:24.585900234 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3002316Z 2025-12-04T11:13:50.3002602Z [W1204 10:53:24.586367064 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3002605Z 2025-12-04T11:13:50.3002888Z [W1204 10:53:24.586504617 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3002892Z 2025-12-04T11:13:50.3002977Z ('RERUN', {'yellow': True}) [0.4159s] [100%] 2025-12-04T11:13:50.3003694Z inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_1_mid_dim_8_in_features_1024_out_features_64_cuda_bfloat16 [W1204 10:53:25.991177735 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3003698Z 2025-12-04T11:13:50.3003983Z [W1204 10:53:25.991722476 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3003991Z 2025-12-04T11:13:50.3004276Z [W1204 10:53:25.991860179 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3004279Z 2025-12-04T11:13:50.3004565Z [W1204 10:53:25.994807237 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3004569Z 2025-12-04T11:13:50.3004867Z [W1204 10:53:25.995371158 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3004872Z 2025-12-04T11:13:50.3005158Z [W1204 10:53:25.995507751 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3005161Z 2025-12-04T11:13:50.3005448Z [W1204 10:53:25.000046122 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3005451Z 2025-12-04T11:13:50.3005738Z [W1204 10:53:25.000522831 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3005741Z 2025-12-04T11:13:50.3006027Z [W1204 10:53:25.000667364 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3006070Z 2025-12-04T11:13:50.3006195Z FAILED [0.4097s] [100%] 2025-12-04T11:13:50.3006199Z 2025-12-04T11:13:50.3006282Z ==================================== RERUNS ==================================== 2025-12-04T11:13:50.3006577Z _ TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_8_in_features_1024_out_features_64_cuda_bfloat16 _ 2025-12-04T11:13:50.3006650Z Traceback (most recent call last): 2025-12-04T11:13:50.3006963Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.3007033Z method(*args, **kwargs) 2025-12-04T11:13:50.3007323Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.3007390Z method(*args, **kwargs) 2025-12-04T11:13:50.3007675Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T11:13:50.3007738Z with policy(): 2025-12-04T11:13:50.3008033Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T11:13:50.3008136Z raise RuntimeError(msg) 2025-12-04T11:13:50.3008937Z RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_8_in_features_1024_out_features_64_cuda_bfloat16! Caching allocator allocated memory was 0 and is now reported as 65536 on device 0. CUDA driver allocated memory was 230686720 and is now 274726912. 2025-12-04T11:13:50.3008941Z 2025-12-04T11:13:50.3009069Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.3009586Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_8_in_features_1024_out_features_64_cuda_bfloat16 2025-12-04T11:13:50.3009592Z 2025-12-04T11:13:50.3009750Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.3009877Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.3009976Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.3010322Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.3010456Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.3010513Z graph_break [] 2025-12-04T11:13:50.3010638Z ----------------------------- Captured stderr call ----------------------------- 2025-12-04T11:13:50.3011341Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/constant_folding.py:256: UserWarning: Unsupported unwinding pattern: Address not in range (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/profiler/unwind/unwind.cpp:219.) 2025-12-04T11:13:50.3011413Z if out == self.unknown_value: 2025-12-04T11:13:50.3011713Z _ TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_8_in_features_1024_out_features_64_cuda_bfloat16 _ 2025-12-04T11:13:50.3011789Z Traceback (most recent call last): 2025-12-04T11:13:50.3012082Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.3012148Z method(*args, **kwargs) 2025-12-04T11:13:50.3012439Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.3012502Z method(*args, **kwargs) 2025-12-04T11:13:50.3012793Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T11:13:50.3012850Z with policy(): 2025-12-04T11:13:50.3013142Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T11:13:50.3013319Z raise RuntimeError(msg) 2025-12-04T11:13:50.3014129Z RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_8_in_features_1024_out_features_64_cuda_bfloat16! Caching allocator allocated memory was 65536 and is now reported as 131072 on device 0. CUDA driver allocated memory was 274726912 and is now 276824064. 2025-12-04T11:13:50.3014139Z 2025-12-04T11:13:50.3014264Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.3014778Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_8_in_features_1024_out_features_64_cuda_bfloat16 2025-12-04T11:13:50.3014782Z 2025-12-04T11:13:50.3014942Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.3015069Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.3015164Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.3015511Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.3015672Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.3015731Z graph_break [] 2025-12-04T11:13:50.3015868Z ----------------------------- Captured stderr call ----------------------------- 2025-12-04T11:13:50.3016557Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/constant_folding.py:256: UserWarning: Unsupported unwinding pattern: Address not in range (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/profiler/unwind/unwind.cpp:219.) 2025-12-04T11:13:50.3016631Z if out == self.unknown_value: 2025-12-04T11:13:50.3016754Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.3016847Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.3016969Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.3017312Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.3017375Z graph_break [] 2025-12-04T11:13:50.3017457Z =================================== FAILURES =================================== 2025-12-04T11:13:50.3017744Z _ TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_8_in_features_1024_out_features_64_cuda_bfloat16 _ 2025-12-04T11:13:50.3017819Z Traceback (most recent call last): 2025-12-04T11:13:50.3018112Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.3018176Z method(*args, **kwargs) 2025-12-04T11:13:50.3018467Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.3018533Z method(*args, **kwargs) 2025-12-04T11:13:50.3018829Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T11:13:50.3018888Z with policy(): 2025-12-04T11:13:50.3019184Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T11:13:50.3019249Z raise RuntimeError(msg) 2025-12-04T11:13:50.3020057Z RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_8_in_features_1024_out_features_64_cuda_bfloat16! Caching allocator allocated memory was 131072 and is now reported as 196608 on device 0. CUDA driver allocated memory was 276824064 and is now 278921216. 2025-12-04T11:13:50.3020109Z 2025-12-04T11:13:50.3020241Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.3020838Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_8_in_features_1024_out_features_64_cuda_bfloat16 2025-12-04T11:13:50.3020844Z 2025-12-04T11:13:50.3021002Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.3021124Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.3021214Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.3021558Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.3021679Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.3021739Z graph_break [] 2025-12-04T11:13:50.3021863Z ----------------------------- Captured stderr call ----------------------------- 2025-12-04T11:13:50.3022547Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/constant_folding.py:256: UserWarning: Unsupported unwinding pattern: Address not in range (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/profiler/unwind/unwind.cpp:219.) 2025-12-04T11:13:50.3022663Z if out == self.unknown_value: 2025-12-04T11:13:50.3022785Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.3022879Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.3022999Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.3023341Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.3023411Z graph_break [] 2025-12-04T11:13:50.3023582Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.3023675Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.3023802Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.3024141Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.3024203Z graph_break [] 2025-12-04T11:13:50.3024687Z - generated xml file: /var/lib/jenkins/workspace/test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-c85f34f0c90d933d.xml - 2025-12-04T11:13:50.3024784Z =========================== short test summary info ============================ 2025-12-04T11:13:50.3026070Z FAILED [0.4097s] inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_1_mid_dim_8_in_features_1024_out_features_64_cuda_bfloat16 - RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_8_in_features_1024_out_features_64_cuda_bfloat16! Caching allocator allocated memory was 131072 and is now reported as 196608 on device 0. CUDA driver allocated memory was 276824064 and is now 278921216. 2025-12-04T11:13:50.3026076Z 2025-12-04T11:13:50.3026198Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.3026714Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_8_in_features_1024_out_features_64_cuda_bfloat16 2025-12-04T11:13:50.3026717Z 2025-12-04T11:13:50.3026870Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.3026978Z !!!!!!!!!!!!!!!!!!!!!!!!!! stopping after 1 failures !!!!!!!!!!!!!!!!!!!!!!!!!!! 2025-12-04T11:13:50.3027091Z ================== 1 failed, 57 deselected, 2 rerun in 12.07s ================== 2025-12-04T11:13:50.3027197Z Got exit code 1 2025-12-04T11:13:50.3027266Z Retrying single test... 2025-12-04T11:13:50.3027597Z W1204 10:53:31.815000 74943 site-packages/torch/_inductor/utils.py:1703] Not enough SMs to use max_autotune_gemm mode 2025-12-04T11:13:50.3027991Z Test results will be stored in test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-4b2e7fd1ed886dfe.xml 2025-12-04T11:13:50.3028085Z ============================= test session starts ============================== 2025-12-04T11:13:50.3028296Z platform linux -- Python 3.10.14, pytest-7.3.2, pluggy-1.6.0 -- /opt/conda/envs/py_3.10/bin/python 2025-12-04T11:13:50.3028363Z cachedir: .pytest_cache 2025-12-04T11:13:50.3028669Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow] 2025-12-04T11:13:50.3028745Z rootdir: /var/lib/jenkins/workspace 2025-12-04T11:13:50.3028814Z configfile: pytest.ini 2025-12-04T11:13:50.3029127Z plugins: hypothesis-6.56.4, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-14.0, subtests-0.13.1, xdist-3.3.1, xdoctest-1.3.0, typeguard-4.3.0 2025-12-04T11:13:50.3029260Z collecting ... collected 58 items / 57 deselected / 1 selected 2025-12-04T11:13:50.3029874Z stepcurrent: skipping 32 already run items. Running only test/inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_1_mid_dim_8_in_features_1024_out_features_64_cuda_bfloat16 2025-12-04T11:13:50.3029945Z Running 1 items in this shard 2025-12-04T11:13:50.3029948Z 2025-12-04T11:13:50.3030674Z inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_1_mid_dim_8_in_features_1024_out_features_64_cuda_bfloat16 [W1204 10:53:32.911417213 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3030678Z 2025-12-04T11:13:50.3030976Z [W1204 10:53:42.074726319 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3030982Z 2025-12-04T11:13:50.3031274Z [W1204 10:53:42.074967684 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3031279Z 2025-12-04T11:13:50.3031566Z [W1204 10:53:42.081413116 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3031569Z 2025-12-04T11:13:50.3031863Z [W1204 10:53:42.081977499 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3031866Z 2025-12-04T11:13:50.3032149Z [W1204 10:53:42.082149953 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3032152Z 2025-12-04T11:13:50.3032445Z [W1204 10:53:42.087530152 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3032450Z 2025-12-04T11:13:50.3032753Z [W1204 10:53:42.088050013 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3032758Z 2025-12-04T11:13:50.3033049Z [W1204 10:53:42.088211267 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3033059Z 2025-12-04T11:13:50.3033140Z ('RERUN', {'yellow': True}) [11.0555s] [100%] 2025-12-04T11:13:50.3033857Z inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_1_mid_dim_8_in_features_1024_out_features_64_cuda_bfloat16 [W1204 10:53:43.273964055 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3033861Z 2025-12-04T11:13:50.3034151Z [W1204 10:53:43.274527637 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3034196Z 2025-12-04T11:13:50.3034551Z [W1204 10:53:43.274669460 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3034557Z 2025-12-04T11:13:50.3034848Z [W1204 10:53:43.277689327 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3034851Z 2025-12-04T11:13:50.3035142Z [W1204 10:53:43.278268450 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3035145Z 2025-12-04T11:13:50.3035434Z [W1204 10:53:43.278407893 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3035437Z 2025-12-04T11:13:50.3035724Z [W1204 10:53:43.283101907 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3035729Z 2025-12-04T11:13:50.3036018Z [W1204 10:53:43.283585057 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3036055Z 2025-12-04T11:13:50.3036344Z [W1204 10:53:43.283722130 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3036347Z 2025-12-04T11:13:50.3039747Z ('RERUN', {'yellow': True}) [0.4356s] [100%] 2025-12-04T11:13:50.3040556Z inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_1_mid_dim_8_in_features_1024_out_features_64_cuda_bfloat16 [W1204 10:53:43.722601130 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3040562Z 2025-12-04T11:13:50.3040875Z [W1204 10:53:43.723151692 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3040886Z 2025-12-04T11:13:50.3041190Z [W1204 10:53:43.723291925 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3041195Z 2025-12-04T11:13:50.3041483Z [W1204 10:53:43.726341992 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3041487Z 2025-12-04T11:13:50.3041776Z [W1204 10:53:43.726923075 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3041779Z 2025-12-04T11:13:50.3042066Z [W1204 10:53:43.727060038 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3042070Z 2025-12-04T11:13:50.3042350Z [W1204 10:53:43.731756371 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3042355Z 2025-12-04T11:13:50.3042645Z [W1204 10:53:43.732236362 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3042649Z 2025-12-04T11:13:50.3042941Z [W1204 10:53:43.732368115 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3042945Z 2025-12-04T11:13:50.3043009Z FAILED [0.4313s] [100%] 2025-12-04T11:13:50.3043013Z 2025-12-04T11:13:50.3043106Z ==================================== RERUNS ==================================== 2025-12-04T11:13:50.3043403Z _ TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_8_in_features_1024_out_features_64_cuda_bfloat16 _ 2025-12-04T11:13:50.3043482Z Traceback (most recent call last): 2025-12-04T11:13:50.3043800Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.3043945Z method(*args, **kwargs) 2025-12-04T11:13:50.3044313Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.3044380Z method(*args, **kwargs) 2025-12-04T11:13:50.3044670Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T11:13:50.3044735Z with policy(): 2025-12-04T11:13:50.3045037Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T11:13:50.3045106Z raise RuntimeError(msg) 2025-12-04T11:13:50.3045906Z RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_8_in_features_1024_out_features_64_cuda_bfloat16! Caching allocator allocated memory was 0 and is now reported as 65536 on device 0. CUDA driver allocated memory was 230686720 and is now 274726912. 2025-12-04T11:13:50.3045912Z 2025-12-04T11:13:50.3046042Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.3046568Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_8_in_features_1024_out_features_64_cuda_bfloat16 2025-12-04T11:13:50.3046693Z 2025-12-04T11:13:50.3046858Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.3046995Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.3047101Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.3047456Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.3047586Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.3047646Z graph_break [] 2025-12-04T11:13:50.3047780Z ----------------------------- Captured stderr call ----------------------------- 2025-12-04T11:13:50.3048481Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/constant_folding.py:256: UserWarning: Unsupported unwinding pattern: Address not in range (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/profiler/unwind/unwind.cpp:219.) 2025-12-04T11:13:50.3048555Z if out == self.unknown_value: 2025-12-04T11:13:50.3048856Z _ TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_8_in_features_1024_out_features_64_cuda_bfloat16 _ 2025-12-04T11:13:50.3048931Z Traceback (most recent call last): 2025-12-04T11:13:50.3049237Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.3049302Z method(*args, **kwargs) 2025-12-04T11:13:50.3049596Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.3049662Z method(*args, **kwargs) 2025-12-04T11:13:50.3049958Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T11:13:50.3050024Z with policy(): 2025-12-04T11:13:50.3050315Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T11:13:50.3050381Z raise RuntimeError(msg) 2025-12-04T11:13:50.3051196Z RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_8_in_features_1024_out_features_64_cuda_bfloat16! Caching allocator allocated memory was 65536 and is now reported as 131072 on device 0. CUDA driver allocated memory was 274726912 and is now 276824064. 2025-12-04T11:13:50.3051200Z 2025-12-04T11:13:50.3051328Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.3051935Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_8_in_features_1024_out_features_64_cuda_bfloat16 2025-12-04T11:13:50.3051971Z 2025-12-04T11:13:50.3052129Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.3052258Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.3052355Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.3052699Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.3052830Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.3052889Z graph_break [] 2025-12-04T11:13:50.3053012Z ----------------------------- Captured stderr call ----------------------------- 2025-12-04T11:13:50.3053700Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/constant_folding.py:256: UserWarning: Unsupported unwinding pattern: Address not in range (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/profiler/unwind/unwind.cpp:219.) 2025-12-04T11:13:50.3053773Z if out == self.unknown_value: 2025-12-04T11:13:50.3053934Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.3054028Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.3054150Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.3054498Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.3054555Z graph_break [] 2025-12-04T11:13:50.3054643Z =================================== FAILURES =================================== 2025-12-04T11:13:50.3054934Z _ TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_8_in_features_1024_out_features_64_cuda_bfloat16 _ 2025-12-04T11:13:50.3055009Z Traceback (most recent call last): 2025-12-04T11:13:50.3055309Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.3055374Z method(*args, **kwargs) 2025-12-04T11:13:50.3055668Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.3055733Z method(*args, **kwargs) 2025-12-04T11:13:50.3056021Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T11:13:50.3056086Z with policy(): 2025-12-04T11:13:50.3056380Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T11:13:50.3056446Z raise RuntimeError(msg) 2025-12-04T11:13:50.3057269Z RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_8_in_features_1024_out_features_64_cuda_bfloat16! Caching allocator allocated memory was 131072 and is now reported as 196608 on device 0. CUDA driver allocated memory was 276824064 and is now 278921216. 2025-12-04T11:13:50.3057280Z 2025-12-04T11:13:50.3057408Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.3057924Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_8_in_features_1024_out_features_64_cuda_bfloat16 2025-12-04T11:13:50.3057928Z 2025-12-04T11:13:50.3058083Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.3058204Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.3058294Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.3058635Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.3058871Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.3058933Z graph_break [] 2025-12-04T11:13:50.3059053Z ----------------------------- Captured stderr call ----------------------------- 2025-12-04T11:13:50.3059735Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/constant_folding.py:256: UserWarning: Unsupported unwinding pattern: Address not in range (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/profiler/unwind/unwind.cpp:219.) 2025-12-04T11:13:50.3059803Z if out == self.unknown_value: 2025-12-04T11:13:50.3059923Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.3060016Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.3060137Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.3060483Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.3060540Z graph_break [] 2025-12-04T11:13:50.3060700Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.3060790Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.3060909Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.3061252Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.3061310Z graph_break [] 2025-12-04T11:13:50.3061801Z - generated xml file: /var/lib/jenkins/workspace/test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-4b2e7fd1ed886dfe.xml - 2025-12-04T11:13:50.3061903Z =========================== short test summary info ============================ 2025-12-04T11:13:50.3063192Z FAILED [0.4313s] inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_1_mid_dim_8_in_features_1024_out_features_64_cuda_bfloat16 - RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_8_in_features_1024_out_features_64_cuda_bfloat16! Caching allocator allocated memory was 131072 and is now reported as 196608 on device 0. CUDA driver allocated memory was 276824064 and is now 278921216. 2025-12-04T11:13:50.3063198Z 2025-12-04T11:13:50.3063325Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.3063917Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_8_in_features_1024_out_features_64_cuda_bfloat16 2025-12-04T11:13:50.3063922Z 2025-12-04T11:13:50.3064081Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.3064185Z !!!!!!!!!!!!!!!!!!!!!!!!!! stopping after 1 failures !!!!!!!!!!!!!!!!!!!!!!!!!!! 2025-12-04T11:13:50.3064299Z ================== 1 failed, 57 deselected, 2 rerun in 11.95s ================== 2025-12-04T11:13:50.3064364Z Got exit code 1 2025-12-04T11:13:50.3064835Z FAILED CONSISTENTLY: test/inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_1_mid_dim_8_in_features_1024_out_features_64_cuda_bfloat16 2025-12-04T11:13:50.3065077Z Test failed consistently, continuing with the rest of the tests due to continue-through-error being set 2025-12-04T11:13:50.3065340Z W1204 10:53:50.559000 75136 site-packages/torch/_inductor/utils.py:1703] Not enough SMs to use max_autotune_gemm mode 2025-12-04T11:13:50.3065721Z Test results will be stored in test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-f54f533501201f35.xml 2025-12-04T11:13:50.3065869Z ============================= test session starts ============================== 2025-12-04T11:13:50.3066142Z platform linux -- Python 3.10.14, pytest-7.3.2, pluggy-1.6.0 -- /opt/conda/envs/py_3.10/bin/python 2025-12-04T11:13:50.3066217Z cachedir: .pytest_cache 2025-12-04T11:13:50.3066521Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow] 2025-12-04T11:13:50.3066597Z rootdir: /var/lib/jenkins/workspace 2025-12-04T11:13:50.3066662Z configfile: pytest.ini 2025-12-04T11:13:50.3066976Z plugins: hypothesis-6.56.4, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-14.0, subtests-0.13.1, xdist-3.3.1, xdoctest-1.3.0, typeguard-4.3.0 2025-12-04T11:13:50.3067103Z collecting ... collected 58 items / 33 deselected / 25 selected 2025-12-04T11:13:50.3067194Z stepcurrent: skipping 33 already run items. 2025-12-04T11:13:50.3067264Z Running 25 items in this shard 2025-12-04T11:13:50.3067267Z 2025-12-04T11:13:50.3067772Z inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_1_mid_dim_8_in_features_1024_out_features_65_cuda_bfloat16 ('RERUN', {'yellow': True}) [1.9752s] [ 4%] 2025-12-04T11:13:50.3068254Z inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_1_mid_dim_8_in_features_1024_out_features_65_cuda_bfloat16 ('RERUN', {'yellow': True}) [0.5599s] [ 4%] 2025-12-04T11:13:50.3068733Z inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_1_mid_dim_8_in_features_1024_out_features_65_cuda_bfloat16 FAILED [0.5494s] [ 4%] 2025-12-04T11:13:50.3068737Z 2025-12-04T11:13:50.3068822Z ==================================== RERUNS ==================================== 2025-12-04T11:13:50.3069111Z _ TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_8_in_features_1024_out_features_65_cuda_bfloat16 _ 2025-12-04T11:13:50.3069186Z Traceback (most recent call last): 2025-12-04T11:13:50.3069498Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.3069569Z method(*args, **kwargs) 2025-12-04T11:13:50.3069863Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.3069927Z method(*args, **kwargs) 2025-12-04T11:13:50.3070211Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T11:13:50.3070273Z with policy(): 2025-12-04T11:13:50.3070559Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T11:13:50.3070629Z raise RuntimeError(msg) 2025-12-04T11:13:50.3071432Z RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_8_in_features_1024_out_features_65_cuda_bfloat16! Caching allocator allocated memory was 0 and is now reported as 66560 on device 0. CUDA driver allocated memory was 230686720 and is now 274726912. 2025-12-04T11:13:50.3071441Z 2025-12-04T11:13:50.3071570Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.3072085Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_8_in_features_1024_out_features_65_cuda_bfloat16 2025-12-04T11:13:50.3072089Z 2025-12-04T11:13:50.3072242Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.3072372Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.3072466Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.3073018Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('benchmarking.InductorBenchmarker.benchmark_gpu', 2), ('pad_mm_bench', 1), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.3073253Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.3073313Z graph_break [] 2025-12-04T11:13:50.3073606Z _ TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_8_in_features_1024_out_features_65_cuda_bfloat16 _ 2025-12-04T11:13:50.3073678Z Traceback (most recent call last): 2025-12-04T11:13:50.3073986Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.3074053Z method(*args, **kwargs) 2025-12-04T11:13:50.3074342Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.3074407Z method(*args, **kwargs) 2025-12-04T11:13:50.3074692Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T11:13:50.3074754Z with policy(): 2025-12-04T11:13:50.3075047Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T11:13:50.3075112Z raise RuntimeError(msg) 2025-12-04T11:13:50.3075966Z RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_8_in_features_1024_out_features_65_cuda_bfloat16! Caching allocator allocated memory was 66560 and is now reported as 133120 on device 0. CUDA driver allocated memory was 274726912 and is now 276824064. 2025-12-04T11:13:50.3075971Z 2025-12-04T11:13:50.3076105Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.3076625Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_8_in_features_1024_out_features_65_cuda_bfloat16 2025-12-04T11:13:50.3076633Z 2025-12-04T11:13:50.3076790Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.3076920Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.3077015Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.3077558Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('benchmarking.InductorBenchmarker.benchmark_gpu', 2), ('pad_mm_bench', 1), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.3077681Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.3077746Z graph_break [] 2025-12-04T11:13:50.3078048Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.3078141Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.3078259Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.3078802Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('benchmarking.InductorBenchmarker.benchmark_gpu', 2), ('pad_mm_bench', 1), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.3078868Z graph_break [] 2025-12-04T11:13:50.3078952Z =================================== FAILURES =================================== 2025-12-04T11:13:50.3079245Z _ TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_8_in_features_1024_out_features_65_cuda_bfloat16 _ 2025-12-04T11:13:50.3079316Z Traceback (most recent call last): 2025-12-04T11:13:50.3079607Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.3079686Z method(*args, **kwargs) 2025-12-04T11:13:50.3079974Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.3080037Z method(*args, **kwargs) 2025-12-04T11:13:50.3080533Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T11:13:50.3080597Z with policy(): 2025-12-04T11:13:50.3080898Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T11:13:50.3080963Z raise RuntimeError(msg) 2025-12-04T11:13:50.3081775Z RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_8_in_features_1024_out_features_65_cuda_bfloat16! Caching allocator allocated memory was 133120 and is now reported as 199680 on device 0. CUDA driver allocated memory was 276824064 and is now 278921216. 2025-12-04T11:13:50.3081781Z 2025-12-04T11:13:50.3081906Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.3082417Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_8_in_features_1024_out_features_65_cuda_bfloat16 2025-12-04T11:13:50.3082423Z 2025-12-04T11:13:50.3082595Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.3082778Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.3082868Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.3083408Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('benchmarking.InductorBenchmarker.benchmark_gpu', 2), ('pad_mm_bench', 1), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.3083531Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.3083593Z graph_break [] 2025-12-04T11:13:50.3083714Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.3083803Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.3083928Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.3084464Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('benchmarking.InductorBenchmarker.benchmark_gpu', 2), ('pad_mm_bench', 1), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.3084534Z graph_break [] 2025-12-04T11:13:50.3084658Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.3084747Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.3084867Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.3085395Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('benchmarking.InductorBenchmarker.benchmark_gpu', 2), ('pad_mm_bench', 1), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.3085456Z graph_break [] 2025-12-04T11:13:50.3085949Z - generated xml file: /var/lib/jenkins/workspace/test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-f54f533501201f35.xml - 2025-12-04T11:13:50.3086049Z =========================== short test summary info ============================ 2025-12-04T11:13:50.3087336Z FAILED [0.5494s] inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_1_mid_dim_8_in_features_1024_out_features_65_cuda_bfloat16 - RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_8_in_features_1024_out_features_65_cuda_bfloat16! Caching allocator allocated memory was 133120 and is now reported as 199680 on device 0. CUDA driver allocated memory was 276824064 and is now 278921216. 2025-12-04T11:13:50.3087340Z 2025-12-04T11:13:50.3087464Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.3088121Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_8_in_features_1024_out_features_65_cuda_bfloat16 2025-12-04T11:13:50.3088128Z 2025-12-04T11:13:50.3088285Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.3088389Z !!!!!!!!!!!!!!!!!!!!!!!!!! stopping after 1 failures !!!!!!!!!!!!!!!!!!!!!!!!!!! 2025-12-04T11:13:50.3088503Z ================== 1 failed, 33 deselected, 2 rerun in 3.11s =================== 2025-12-04T11:13:50.3088561Z Got exit code 1 2025-12-04T11:13:50.3088628Z Retrying single test... 2025-12-04T11:13:50.3088886Z W1204 10:54:00.606000 75325 site-packages/torch/_inductor/utils.py:1703] Not enough SMs to use max_autotune_gemm mode 2025-12-04T11:13:50.3089271Z Test results will be stored in test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-cbbd83fa84eb990f.xml 2025-12-04T11:13:50.3089377Z ============================= test session starts ============================== 2025-12-04T11:13:50.3089584Z platform linux -- Python 3.10.14, pytest-7.3.2, pluggy-1.6.0 -- /opt/conda/envs/py_3.10/bin/python 2025-12-04T11:13:50.3089696Z cachedir: .pytest_cache 2025-12-04T11:13:50.3090005Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow] 2025-12-04T11:13:50.3090081Z rootdir: /var/lib/jenkins/workspace 2025-12-04T11:13:50.3090148Z configfile: pytest.ini 2025-12-04T11:13:50.3090460Z plugins: hypothesis-6.56.4, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-14.0, subtests-0.13.1, xdist-3.3.1, xdoctest-1.3.0, typeguard-4.3.0 2025-12-04T11:13:50.3090591Z collecting ... collected 58 items / 57 deselected / 1 selected 2025-12-04T11:13:50.3091156Z stepcurrent: skipping 33 already run items. Running only test/inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_1_mid_dim_8_in_features_1024_out_features_65_cuda_bfloat16 2025-12-04T11:13:50.3091230Z Running 1 items in this shard 2025-12-04T11:13:50.3091234Z 2025-12-04T11:13:50.3091962Z inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_1_mid_dim_8_in_features_1024_out_features_65_cuda_bfloat16 [W1204 10:54:02.232232700 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3091968Z 2025-12-04T11:13:50.3092265Z [W1204 10:54:11.164525719 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3092269Z 2025-12-04T11:13:50.3092558Z [W1204 10:54:11.164778705 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3092561Z 2025-12-04T11:13:50.3092843Z [W1204 10:54:11.170721826 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3092849Z 2025-12-04T11:13:50.3093135Z [W1204 10:54:11.171289678 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3093140Z 2025-12-04T11:13:50.3093423Z [W1204 10:54:11.171464812 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3093427Z 2025-12-04T11:13:50.3093708Z [W1204 10:54:11.176774440 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3093711Z 2025-12-04T11:13:50.3093992Z [W1204 10:54:11.177289331 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3093996Z 2025-12-04T11:13:50.3094278Z [W1204 10:54:11.177450685 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3094332Z 2025-12-04T11:13:50.3094415Z ('RERUN', {'yellow': True}) [10.8997s] [100%] 2025-12-04T11:13:50.3095201Z inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_1_mid_dim_8_in_features_1024_out_features_65_cuda_bfloat16 [W1204 10:54:12.994475574 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3095207Z 2025-12-04T11:13:50.3095493Z [W1204 10:54:12.994998216 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3095497Z 2025-12-04T11:13:50.3095782Z [W1204 10:54:12.995134059 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3095786Z 2025-12-04T11:13:50.3096069Z [W1204 10:54:12.998085444 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3096074Z 2025-12-04T11:13:50.3096359Z [W1204 10:54:12.998542564 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3096395Z 2025-12-04T11:13:50.3096677Z [W1204 10:54:12.998680737 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3096680Z 2025-12-04T11:13:50.3096966Z [W1204 10:54:12.003307939 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3096969Z 2025-12-04T11:13:50.3097258Z [W1204 10:54:12.003774589 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3097261Z 2025-12-04T11:13:50.3097542Z [W1204 10:54:12.003907782 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3097546Z 2025-12-04T11:13:50.3097624Z ('RERUN', {'yellow': True}) [0.5172s] [100%] 2025-12-04T11:13:50.3098342Z inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_1_mid_dim_8_in_features_1024_out_features_65_cuda_bfloat16 [W1204 10:54:12.504165179 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3098348Z 2025-12-04T11:13:50.3098633Z [W1204 10:54:12.504694281 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3098636Z 2025-12-04T11:13:50.3098920Z [W1204 10:54:12.504831244 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3098923Z 2025-12-04T11:13:50.3099204Z [W1204 10:54:12.507716177 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3099209Z 2025-12-04T11:13:50.3099502Z [W1204 10:54:12.508180527 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3099505Z 2025-12-04T11:13:50.3099787Z [W1204 10:54:12.508314940 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3099790Z 2025-12-04T11:13:50.3100075Z [W1204 10:54:12.512933802 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3100078Z 2025-12-04T11:13:50.3100359Z [W1204 10:54:12.513404053 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3100362Z 2025-12-04T11:13:50.3100644Z [W1204 10:54:12.513538706 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3100696Z 2025-12-04T11:13:50.3100760Z FAILED [0.5036s] [100%] 2025-12-04T11:13:50.3100763Z 2025-12-04T11:13:50.3100912Z ==================================== RERUNS ==================================== 2025-12-04T11:13:50.3101213Z _ TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_8_in_features_1024_out_features_65_cuda_bfloat16 _ 2025-12-04T11:13:50.3101291Z Traceback (most recent call last): 2025-12-04T11:13:50.3101607Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.3101674Z method(*args, **kwargs) 2025-12-04T11:13:50.3101964Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.3102034Z method(*args, **kwargs) 2025-12-04T11:13:50.3102327Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T11:13:50.3102387Z with policy(): 2025-12-04T11:13:50.3102682Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T11:13:50.3102753Z raise RuntimeError(msg) 2025-12-04T11:13:50.3103668Z RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_8_in_features_1024_out_features_65_cuda_bfloat16! Caching allocator allocated memory was 0 and is now reported as 66560 on device 0. CUDA driver allocated memory was 230686720 and is now 274726912. 2025-12-04T11:13:50.3103673Z 2025-12-04T11:13:50.3103804Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.3104320Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_8_in_features_1024_out_features_65_cuda_bfloat16 2025-12-04T11:13:50.3104327Z 2025-12-04T11:13:50.3104483Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.3104612Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.3104712Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.3105260Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('benchmarking.InductorBenchmarker.benchmark_gpu', 2), ('pad_mm_bench', 1), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.3105386Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.3105447Z graph_break [] 2025-12-04T11:13:50.3105571Z ----------------------------- Captured stderr call ----------------------------- 2025-12-04T11:13:50.3106264Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/constant_folding.py:256: UserWarning: Unsupported unwinding pattern: Address not in range (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/profiler/unwind/unwind.cpp:219.) 2025-12-04T11:13:50.3106335Z if out == self.unknown_value: 2025-12-04T11:13:50.3106626Z _ TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_8_in_features_1024_out_features_65_cuda_bfloat16 _ 2025-12-04T11:13:50.3106703Z Traceback (most recent call last): 2025-12-04T11:13:50.3106997Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.3107063Z method(*args, **kwargs) 2025-12-04T11:13:50.3107349Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.3107411Z method(*args, **kwargs) 2025-12-04T11:13:50.3107697Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T11:13:50.3107756Z with policy(): 2025-12-04T11:13:50.3108046Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T11:13:50.3108162Z raise RuntimeError(msg) 2025-12-04T11:13:50.3109033Z RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_8_in_features_1024_out_features_65_cuda_bfloat16! Caching allocator allocated memory was 66560 and is now reported as 133120 on device 0. CUDA driver allocated memory was 274726912 and is now 276824064. 2025-12-04T11:13:50.3109040Z 2025-12-04T11:13:50.3109169Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.3111735Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_8_in_features_1024_out_features_65_cuda_bfloat16 2025-12-04T11:13:50.3111752Z 2025-12-04T11:13:50.3111951Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.3112097Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.3112203Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.3112766Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('benchmarking.InductorBenchmarker.benchmark_gpu', 2), ('pad_mm_bench', 1), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.3112965Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.3113027Z graph_break [] 2025-12-04T11:13:50.3113163Z ----------------------------- Captured stderr call ----------------------------- 2025-12-04T11:13:50.3113888Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/constant_folding.py:256: UserWarning: Unsupported unwinding pattern: Address not in range (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/profiler/unwind/unwind.cpp:219.) 2025-12-04T11:13:50.3113965Z if out == self.unknown_value: 2025-12-04T11:13:50.3114097Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.3114195Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.3114324Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.3114874Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('benchmarking.InductorBenchmarker.benchmark_gpu', 2), ('pad_mm_bench', 1), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.3114935Z graph_break [] 2025-12-04T11:13:50.3115023Z =================================== FAILURES =================================== 2025-12-04T11:13:50.3115323Z _ TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_8_in_features_1024_out_features_65_cuda_bfloat16 _ 2025-12-04T11:13:50.3115401Z Traceback (most recent call last): 2025-12-04T11:13:50.3115714Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.3115784Z method(*args, **kwargs) 2025-12-04T11:13:50.3116078Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.3116144Z method(*args, **kwargs) 2025-12-04T11:13:50.3116453Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T11:13:50.3116521Z with policy(): 2025-12-04T11:13:50.3116820Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T11:13:50.3116887Z raise RuntimeError(msg) 2025-12-04T11:13:50.3117709Z RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_8_in_features_1024_out_features_65_cuda_bfloat16! Caching allocator allocated memory was 133120 and is now reported as 199680 on device 0. CUDA driver allocated memory was 276824064 and is now 278921216. 2025-12-04T11:13:50.3117757Z 2025-12-04T11:13:50.3117926Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.3118449Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_8_in_features_1024_out_features_65_cuda_bfloat16 2025-12-04T11:13:50.3118455Z 2025-12-04T11:13:50.3118615Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.3118750Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.3118845Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.3119458Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('benchmarking.InductorBenchmarker.benchmark_gpu', 2), ('pad_mm_bench', 1), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.3119601Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.3119665Z graph_break [] 2025-12-04T11:13:50.3119795Z ----------------------------- Captured stderr call ----------------------------- 2025-12-04T11:13:50.3120522Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/constant_folding.py:256: UserWarning: Unsupported unwinding pattern: Address not in range (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/profiler/unwind/unwind.cpp:219.) 2025-12-04T11:13:50.3120594Z if out == self.unknown_value: 2025-12-04T11:13:50.3120721Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.3120818Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.3120944Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.3121487Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('benchmarking.InductorBenchmarker.benchmark_gpu', 2), ('pad_mm_bench', 1), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.3121550Z graph_break [] 2025-12-04T11:13:50.3121679Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.3121772Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.3121894Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.3122431Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('benchmarking.InductorBenchmarker.benchmark_gpu', 2), ('pad_mm_bench', 1), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.3122493Z graph_break [] 2025-12-04T11:13:50.3122989Z - generated xml file: /var/lib/jenkins/workspace/test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-cbbd83fa84eb990f.xml - 2025-12-04T11:13:50.3123092Z =========================== short test summary info ============================ 2025-12-04T11:13:50.3124390Z FAILED [0.5036s] inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_1_mid_dim_8_in_features_1024_out_features_65_cuda_bfloat16 - RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_8_in_features_1024_out_features_65_cuda_bfloat16! Caching allocator allocated memory was 133120 and is now reported as 199680 on device 0. CUDA driver allocated memory was 276824064 and is now 278921216. 2025-12-04T11:13:50.3124399Z 2025-12-04T11:13:50.3124529Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.3125055Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_8_in_features_1024_out_features_65_cuda_bfloat16 2025-12-04T11:13:50.3125100Z 2025-12-04T11:13:50.3125266Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.3125410Z !!!!!!!!!!!!!!!!!!!!!!!!!! stopping after 1 failures !!!!!!!!!!!!!!!!!!!!!!!!!!! 2025-12-04T11:13:50.3125531Z ================== 1 failed, 57 deselected, 2 rerun in 11.95s ================== 2025-12-04T11:13:50.3125593Z Got exit code 1 2025-12-04T11:13:50.3125659Z Retrying single test... 2025-12-04T11:13:50.3125929Z W1204 10:54:19.343000 75519 site-packages/torch/_inductor/utils.py:1703] Not enough SMs to use max_autotune_gemm mode 2025-12-04T11:13:50.3126311Z Test results will be stored in test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-67e17348ec4a01d8.xml 2025-12-04T11:13:50.3126461Z ============================= test session starts ============================== 2025-12-04T11:13:50.3126675Z platform linux -- Python 3.10.14, pytest-7.3.2, pluggy-1.6.0 -- /opt/conda/envs/py_3.10/bin/python 2025-12-04T11:13:50.3126744Z cachedir: .pytest_cache 2025-12-04T11:13:50.3127057Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow] 2025-12-04T11:13:50.3127134Z rootdir: /var/lib/jenkins/workspace 2025-12-04T11:13:50.3127237Z configfile: pytest.ini 2025-12-04T11:13:50.3127552Z plugins: hypothesis-6.56.4, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-14.0, subtests-0.13.1, xdist-3.3.1, xdoctest-1.3.0, typeguard-4.3.0 2025-12-04T11:13:50.3127685Z collecting ... collected 58 items / 57 deselected / 1 selected 2025-12-04T11:13:50.3128262Z stepcurrent: skipping 33 already run items. Running only test/inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_1_mid_dim_8_in_features_1024_out_features_65_cuda_bfloat16 2025-12-04T11:13:50.3128337Z Running 1 items in this shard 2025-12-04T11:13:50.3128341Z 2025-12-04T11:13:50.3129068Z inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_1_mid_dim_8_in_features_1024_out_features_65_cuda_bfloat16 [W1204 10:54:21.986319661 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3129077Z 2025-12-04T11:13:50.3129384Z [W1204 10:54:30.207302786 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3129388Z 2025-12-04T11:13:50.3129677Z [W1204 10:54:30.207554331 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3129681Z 2025-12-04T11:13:50.3129970Z [W1204 10:54:30.213513092 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3129974Z 2025-12-04T11:13:50.3130261Z [W1204 10:54:30.214087435 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3130266Z 2025-12-04T11:13:50.3130556Z [W1204 10:54:30.214267599 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3130560Z 2025-12-04T11:13:50.3130845Z [W1204 10:54:30.219680339 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3130850Z 2025-12-04T11:13:50.3131137Z [W1204 10:54:30.220220991 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3131140Z 2025-12-04T11:13:50.3131426Z [W1204 10:54:30.220388384 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3131429Z 2025-12-04T11:13:50.3131517Z ('RERUN', {'yellow': True}) [11.2063s] [100%] 2025-12-04T11:13:50.3132283Z inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_1_mid_dim_8_in_features_1024_out_features_65_cuda_bfloat16 [W1204 10:54:31.033811374 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3132337Z 2025-12-04T11:13:50.3132628Z [W1204 10:54:31.034326766 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3132631Z 2025-12-04T11:13:50.3132923Z [W1204 10:54:31.034464779 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3132926Z 2025-12-04T11:13:50.3133210Z [W1204 10:54:31.037369983 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3133213Z 2025-12-04T11:13:50.3133538Z [W1204 10:54:31.037829643 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3133545Z 2025-12-04T11:13:50.3133833Z [W1204 10:54:31.037967926 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3133838Z 2025-12-04T11:13:50.3134279Z [W1204 10:54:31.042535237 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3134348Z 2025-12-04T11:13:50.3134726Z [W1204 10:54:31.043001287 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3134730Z 2025-12-04T11:13:50.3135017Z [W1204 10:54:31.043137690 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3135020Z 2025-12-04T11:13:50.3135107Z ('RERUN', {'yellow': True}) [0.5077s] [100%] 2025-12-04T11:13:50.3135830Z inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_1_mid_dim_8_in_features_1024_out_features_65_cuda_bfloat16 [W1204 10:54:31.534640135 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3135838Z 2025-12-04T11:13:50.3136127Z [W1204 10:54:31.535154196 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3136133Z 2025-12-04T11:13:50.3136418Z [W1204 10:54:31.535292079 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3136421Z 2025-12-04T11:13:50.3136713Z [W1204 10:54:31.538210653 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3136716Z 2025-12-04T11:13:50.3137007Z [W1204 10:54:31.538670753 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3137010Z 2025-12-04T11:13:50.3137295Z [W1204 10:54:31.538808806 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3137298Z 2025-12-04T11:13:50.3137587Z [W1204 10:54:31.543295535 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3137591Z 2025-12-04T11:13:50.3137875Z [W1204 10:54:31.543760936 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3137879Z 2025-12-04T11:13:50.3138167Z [W1204 10:54:31.543896759 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3138171Z 2025-12-04T11:13:50.3138235Z FAILED [0.5027s] [100%] 2025-12-04T11:13:50.3138239Z 2025-12-04T11:13:50.3138331Z ==================================== RERUNS ==================================== 2025-12-04T11:13:50.3138624Z _ TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_8_in_features_1024_out_features_65_cuda_bfloat16 _ 2025-12-04T11:13:50.3138761Z Traceback (most recent call last): 2025-12-04T11:13:50.3139118Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.3139191Z method(*args, **kwargs) 2025-12-04T11:13:50.3139489Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.3139560Z method(*args, **kwargs) 2025-12-04T11:13:50.3139845Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T11:13:50.3139908Z with policy(): 2025-12-04T11:13:50.3140242Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T11:13:50.3140311Z raise RuntimeError(msg) 2025-12-04T11:13:50.3141114Z RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_8_in_features_1024_out_features_65_cuda_bfloat16! Caching allocator allocated memory was 0 and is now reported as 66560 on device 0. CUDA driver allocated memory was 230686720 and is now 274726912. 2025-12-04T11:13:50.3141155Z 2025-12-04T11:13:50.3141287Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.3141802Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_8_in_features_1024_out_features_65_cuda_bfloat16 2025-12-04T11:13:50.3141806Z 2025-12-04T11:13:50.3141967Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.3142103Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.3142199Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.3142747Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('benchmarking.InductorBenchmarker.benchmark_gpu', 2), ('pad_mm_bench', 1), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.3142893Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.3142953Z graph_break [] 2025-12-04T11:13:50.3143079Z ----------------------------- Captured stderr call ----------------------------- 2025-12-04T11:13:50.3143847Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/constant_folding.py:256: UserWarning: Unsupported unwinding pattern: Address not in range (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/profiler/unwind/unwind.cpp:219.) 2025-12-04T11:13:50.3143923Z if out == self.unknown_value: 2025-12-04T11:13:50.3144213Z _ TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_8_in_features_1024_out_features_65_cuda_bfloat16 _ 2025-12-04T11:13:50.3144290Z Traceback (most recent call last): 2025-12-04T11:13:50.3144584Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.3144654Z method(*args, **kwargs) 2025-12-04T11:13:50.3145014Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.3145128Z method(*args, **kwargs) 2025-12-04T11:13:50.3145565Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T11:13:50.3145633Z with policy(): 2025-12-04T11:13:50.3145930Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T11:13:50.3146001Z raise RuntimeError(msg) 2025-12-04T11:13:50.3146872Z RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_8_in_features_1024_out_features_65_cuda_bfloat16! Caching allocator allocated memory was 66560 and is now reported as 133120 on device 0. CUDA driver allocated memory was 274726912 and is now 276824064. 2025-12-04T11:13:50.3146912Z 2025-12-04T11:13:50.3147043Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.3147557Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_8_in_features_1024_out_features_65_cuda_bfloat16 2025-12-04T11:13:50.3147565Z 2025-12-04T11:13:50.3147721Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.3147849Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.3147989Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.3148535Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('benchmarking.InductorBenchmarker.benchmark_gpu', 2), ('pad_mm_bench', 1), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.3148668Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.3148764Z graph_break [] 2025-12-04T11:13:50.3148889Z ----------------------------- Captured stderr call ----------------------------- 2025-12-04T11:13:50.3149585Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/constant_folding.py:256: UserWarning: Unsupported unwinding pattern: Address not in range (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/profiler/unwind/unwind.cpp:219.) 2025-12-04T11:13:50.3149655Z if out == self.unknown_value: 2025-12-04T11:13:50.3149780Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.3149874Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.3149999Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.3150545Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('benchmarking.InductorBenchmarker.benchmark_gpu', 2), ('pad_mm_bench', 1), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.3150605Z graph_break [] 2025-12-04T11:13:50.3150689Z =================================== FAILURES =================================== 2025-12-04T11:13:50.3150979Z _ TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_8_in_features_1024_out_features_65_cuda_bfloat16 _ 2025-12-04T11:13:50.3151055Z Traceback (most recent call last): 2025-12-04T11:13:50.3151348Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.3151429Z method(*args, **kwargs) 2025-12-04T11:13:50.3151720Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.3151789Z method(*args, **kwargs) 2025-12-04T11:13:50.3152090Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T11:13:50.3152153Z with policy(): 2025-12-04T11:13:50.3152448Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T11:13:50.3152514Z raise RuntimeError(msg) 2025-12-04T11:13:50.3153331Z RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_8_in_features_1024_out_features_65_cuda_bfloat16! Caching allocator allocated memory was 133120 and is now reported as 199680 on device 0. CUDA driver allocated memory was 276824064 and is now 278921216. 2025-12-04T11:13:50.3153336Z 2025-12-04T11:13:50.3153463Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.3154022Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_8_in_features_1024_out_features_65_cuda_bfloat16 2025-12-04T11:13:50.3154063Z 2025-12-04T11:13:50.3154222Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.3154350Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.3154445Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.3154986Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('benchmarking.InductorBenchmarker.benchmark_gpu', 2), ('pad_mm_bench', 1), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.3155149Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.3155216Z graph_break [] 2025-12-04T11:13:50.3155339Z ----------------------------- Captured stderr call ----------------------------- 2025-12-04T11:13:50.3156032Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/constant_folding.py:256: UserWarning: Unsupported unwinding pattern: Address not in range (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/profiler/unwind/unwind.cpp:219.) 2025-12-04T11:13:50.3156141Z if out == self.unknown_value: 2025-12-04T11:13:50.3156265Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.3156361Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.3156483Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.3157022Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('benchmarking.InductorBenchmarker.benchmark_gpu', 2), ('pad_mm_bench', 1), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.3157081Z graph_break [] 2025-12-04T11:13:50.3157203Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.3157297Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.3157419Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.3157955Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('benchmarking.InductorBenchmarker.benchmark_gpu', 2), ('pad_mm_bench', 1), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.3158014Z graph_break [] 2025-12-04T11:13:50.3158500Z - generated xml file: /var/lib/jenkins/workspace/test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-67e17348ec4a01d8.xml - 2025-12-04T11:13:50.3158603Z =========================== short test summary info ============================ 2025-12-04T11:13:50.3159891Z FAILED [0.5027s] inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_1_mid_dim_8_in_features_1024_out_features_65_cuda_bfloat16 - RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_8_in_features_1024_out_features_65_cuda_bfloat16! Caching allocator allocated memory was 133120 and is now reported as 199680 on device 0. CUDA driver allocated memory was 276824064 and is now 278921216. 2025-12-04T11:13:50.3159899Z 2025-12-04T11:13:50.3160029Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.3160542Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_8_in_features_1024_out_features_65_cuda_bfloat16 2025-12-04T11:13:50.3160545Z 2025-12-04T11:13:50.3160705Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.3160810Z !!!!!!!!!!!!!!!!!!!!!!!!!! stopping after 1 failures !!!!!!!!!!!!!!!!!!!!!!!!!!! 2025-12-04T11:13:50.3160925Z ================== 1 failed, 57 deselected, 2 rerun in 12.24s ================== 2025-12-04T11:13:50.3161051Z Got exit code 1 2025-12-04T11:13:50.3161637Z FAILED CONSISTENTLY: test/inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_1_mid_dim_8_in_features_1024_out_features_65_cuda_bfloat16 2025-12-04T11:13:50.3161884Z Test failed consistently, continuing with the rest of the tests due to continue-through-error being set 2025-12-04T11:13:50.3162146Z W1204 10:54:38.374000 75713 site-packages/torch/_inductor/utils.py:1703] Not enough SMs to use max_autotune_gemm mode 2025-12-04T11:13:50.3162532Z Test results will be stored in test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-f72cf4cc8675ff8a.xml 2025-12-04T11:13:50.3162682Z ============================= test session starts ============================== 2025-12-04T11:13:50.3162896Z platform linux -- Python 3.10.14, pytest-7.3.2, pluggy-1.6.0 -- /opt/conda/envs/py_3.10/bin/python 2025-12-04T11:13:50.3162967Z cachedir: .pytest_cache 2025-12-04T11:13:50.3163277Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow] 2025-12-04T11:13:50.3163389Z rootdir: /var/lib/jenkins/workspace 2025-12-04T11:13:50.3163459Z configfile: pytest.ini 2025-12-04T11:13:50.3163771Z plugins: hypothesis-6.56.4, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-14.0, subtests-0.13.1, xdist-3.3.1, xdoctest-1.3.0, typeguard-4.3.0 2025-12-04T11:13:50.3163899Z collecting ... collected 58 items / 34 deselected / 24 selected 2025-12-04T11:13:50.3163987Z stepcurrent: skipping 34 already run items. 2025-12-04T11:13:50.3164058Z Running 24 items in this shard 2025-12-04T11:13:50.3164062Z 2025-12-04T11:13:50.3164728Z inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_1_mid_dim_8_in_features_128_out_features_1024_cuda_bfloat16 ('RERUN', {'yellow': True}) [1.9208s] [ 4%] 2025-12-04T11:13:50.3165368Z inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_1_mid_dim_8_in_features_128_out_features_1024_cuda_bfloat16 ('RERUN', {'yellow': True}) [0.5150s] [ 4%] 2025-12-04T11:13:50.3165827Z inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_1_mid_dim_8_in_features_128_out_features_1024_cuda_bfloat16 FAILED [0.5055s] [ 4%] 2025-12-04T11:13:50.3165842Z 2025-12-04T11:13:50.3165991Z ==================================== RERUNS ==================================== 2025-12-04T11:13:50.3166351Z _ TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_8_in_features_128_out_features_1024_cuda_bfloat16 _ 2025-12-04T11:13:50.3166432Z Traceback (most recent call last): 2025-12-04T11:13:50.3166743Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.3166811Z method(*args, **kwargs) 2025-12-04T11:13:50.3167120Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.3167189Z method(*args, **kwargs) 2025-12-04T11:13:50.3167489Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T11:13:50.3167553Z with policy(): 2025-12-04T11:13:50.3167848Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T11:13:50.3167917Z raise RuntimeError(msg) 2025-12-04T11:13:50.3168725Z RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_8_in_features_128_out_features_1024_cuda_bfloat16! Caching allocator allocated memory was 0 and is now reported as 131072 on device 0. CUDA driver allocated memory was 230686720 and is now 274726912. 2025-12-04T11:13:50.3168729Z 2025-12-04T11:13:50.3168865Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.3169478Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_8_in_features_128_out_features_1024_cuda_bfloat16 2025-12-04T11:13:50.3169485Z 2025-12-04T11:13:50.3169656Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.3169798Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.3169897Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.3170248Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.3170417Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.3170480Z graph_break [] 2025-12-04T11:13:50.3170774Z _ TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_8_in_features_128_out_features_1024_cuda_bfloat16 _ 2025-12-04T11:13:50.3170854Z Traceback (most recent call last): 2025-12-04T11:13:50.3171160Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.3171267Z method(*args, **kwargs) 2025-12-04T11:13:50.3171558Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.3171625Z method(*args, **kwargs) 2025-12-04T11:13:50.3171913Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T11:13:50.3171972Z with policy(): 2025-12-04T11:13:50.3172269Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T11:13:50.3172338Z raise RuntimeError(msg) 2025-12-04T11:13:50.3173164Z RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_8_in_features_128_out_features_1024_cuda_bfloat16! Caching allocator allocated memory was 131072 and is now reported as 262144 on device 0. CUDA driver allocated memory was 274726912 and is now 276824064. 2025-12-04T11:13:50.3173171Z 2025-12-04T11:13:50.3173298Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.3173815Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_8_in_features_128_out_features_1024_cuda_bfloat16 2025-12-04T11:13:50.3173822Z 2025-12-04T11:13:50.3173979Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.3174108Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.3174205Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.3174552Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.3174683Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.3174747Z graph_break [] 2025-12-04T11:13:50.3174873Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.3174966Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.3175089Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.3175428Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.3175493Z graph_break [] 2025-12-04T11:13:50.3175587Z =================================== FAILURES =================================== 2025-12-04T11:13:50.3175880Z _ TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_8_in_features_128_out_features_1024_cuda_bfloat16 _ 2025-12-04T11:13:50.3176000Z Traceback (most recent call last): 2025-12-04T11:13:50.3176331Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.3176400Z method(*args, **kwargs) 2025-12-04T11:13:50.3176688Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.3176754Z method(*args, **kwargs) 2025-12-04T11:13:50.3177044Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T11:13:50.3177104Z with policy(): 2025-12-04T11:13:50.3177436Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T11:13:50.3177505Z raise RuntimeError(msg) 2025-12-04T11:13:50.3178491Z RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_8_in_features_128_out_features_1024_cuda_bfloat16! Caching allocator allocated memory was 262144 and is now reported as 393216 on device 0. CUDA driver allocated memory was 276824064 and is now 278921216. 2025-12-04T11:13:50.3178574Z 2025-12-04T11:13:50.3178721Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.3179244Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_8_in_features_128_out_features_1024_cuda_bfloat16 2025-12-04T11:13:50.3179248Z 2025-12-04T11:13:50.3179408Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.3179538Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.3179632Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.3179984Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.3180109Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.3180174Z graph_break [] 2025-12-04T11:13:50.3180298Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.3180387Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.3180509Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.3180846Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.3180907Z graph_break [] 2025-12-04T11:13:50.3181035Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.3181125Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.3181250Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.3181591Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.3181650Z graph_break [] 2025-12-04T11:13:50.3182144Z - generated xml file: /var/lib/jenkins/workspace/test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-f72cf4cc8675ff8a.xml - 2025-12-04T11:13:50.3182245Z =========================== short test summary info ============================ 2025-12-04T11:13:50.3183681Z FAILED [0.5055s] inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_1_mid_dim_8_in_features_128_out_features_1024_cuda_bfloat16 - RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_8_in_features_128_out_features_1024_cuda_bfloat16! Caching allocator allocated memory was 262144 and is now reported as 393216 on device 0. CUDA driver allocated memory was 276824064 and is now 278921216. 2025-12-04T11:13:50.3183732Z 2025-12-04T11:13:50.3183868Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.3184396Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_8_in_features_128_out_features_1024_cuda_bfloat16 2025-12-04T11:13:50.3184400Z 2025-12-04T11:13:50.3184557Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.3184662Z !!!!!!!!!!!!!!!!!!!!!!!!!! stopping after 1 failures !!!!!!!!!!!!!!!!!!!!!!!!!!! 2025-12-04T11:13:50.3184837Z ================== 1 failed, 34 deselected, 2 rerun in 2.97s =================== 2025-12-04T11:13:50.3184899Z Got exit code 1 2025-12-04T11:13:50.3184967Z Retrying single test... 2025-12-04T11:13:50.3185231Z W1204 10:54:48.398000 75902 site-packages/torch/_inductor/utils.py:1703] Not enough SMs to use max_autotune_gemm mode 2025-12-04T11:13:50.3185613Z Test results will be stored in test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-bd700a0c553d5cfe.xml 2025-12-04T11:13:50.3185769Z ============================= test session starts ============================== 2025-12-04T11:13:50.3185978Z platform linux -- Python 3.10.14, pytest-7.3.2, pluggy-1.6.0 -- /opt/conda/envs/py_3.10/bin/python 2025-12-04T11:13:50.3186045Z cachedir: .pytest_cache 2025-12-04T11:13:50.3186357Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow] 2025-12-04T11:13:50.3186434Z rootdir: /var/lib/jenkins/workspace 2025-12-04T11:13:50.3186503Z configfile: pytest.ini 2025-12-04T11:13:50.3186816Z plugins: hypothesis-6.56.4, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-14.0, subtests-0.13.1, xdist-3.3.1, xdoctest-1.3.0, typeguard-4.3.0 2025-12-04T11:13:50.3186948Z collecting ... collected 58 items / 57 deselected / 1 selected 2025-12-04T11:13:50.3187541Z stepcurrent: skipping 34 already run items. Running only test/inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_1_mid_dim_8_in_features_128_out_features_1024_cuda_bfloat16 2025-12-04T11:13:50.3187616Z Running 1 items in this shard 2025-12-04T11:13:50.3187620Z 2025-12-04T11:13:50.3188359Z inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_1_mid_dim_8_in_features_128_out_features_1024_cuda_bfloat16 [W1204 10:54:49.528433493 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3188364Z 2025-12-04T11:13:50.3188662Z [W1204 10:54:58.408611455 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3188667Z 2025-12-04T11:13:50.3188959Z [W1204 10:54:58.408860071 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3188963Z 2025-12-04T11:13:50.3189252Z [W1204 10:54:58.414506646 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3189255Z 2025-12-04T11:13:50.3189540Z [W1204 10:54:58.415065948 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3189545Z 2025-12-04T11:13:50.3189829Z [W1204 10:54:58.415227642 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3189832Z 2025-12-04T11:13:50.3190122Z [W1204 10:54:58.420615681 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3190175Z 2025-12-04T11:13:50.3190468Z [W1204 10:54:58.421143862 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3190506Z 2025-12-04T11:13:50.3190792Z [W1204 10:54:58.421301076 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3190797Z 2025-12-04T11:13:50.3190881Z ('RERUN', {'yellow': True}) [10.8010s] [100%] 2025-12-04T11:13:50.3191604Z inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_1_mid_dim_8_in_features_128_out_features_1024_cuda_bfloat16 [W1204 10:54:59.638619670 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3191608Z 2025-12-04T11:13:50.3191935Z [W1204 10:54:59.639172263 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3191940Z 2025-12-04T11:13:50.3192226Z [W1204 10:54:59.639316046 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3192230Z 2025-12-04T11:13:50.3192517Z [W1204 10:54:59.642415094 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3192553Z 2025-12-04T11:13:50.3192839Z [W1204 10:54:59.643004207 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3192843Z 2025-12-04T11:13:50.3193128Z [W1204 10:54:59.643143240 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3193136Z 2025-12-04T11:13:50.3193421Z [W1204 10:54:59.647849233 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3193424Z 2025-12-04T11:13:50.3193710Z [W1204 10:54:59.648326824 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3193715Z 2025-12-04T11:13:50.3194002Z [W1204 10:54:59.648462667 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3194007Z 2025-12-04T11:13:50.3194086Z ('RERUN', {'yellow': True}) [0.4548s] [100%] 2025-12-04T11:13:50.3194814Z inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_1_mid_dim_8_in_features_128_out_features_1024_cuda_bfloat16 [W1204 10:55:00.090300335 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3194817Z 2025-12-04T11:13:50.3195104Z [W1204 10:55:00.090857657 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3195107Z 2025-12-04T11:13:50.3195394Z [W1204 10:55:00.091003330 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3195399Z 2025-12-04T11:13:50.3195684Z [W1204 10:55:00.094062298 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3195689Z 2025-12-04T11:13:50.3195971Z [W1204 10:55:00.094645091 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3195977Z 2025-12-04T11:13:50.3196261Z [W1204 10:55:00.094783014 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3196264Z 2025-12-04T11:13:50.3196548Z [W1204 10:55:00.099526808 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3196552Z 2025-12-04T11:13:50.3196840Z [W1204 10:55:00.100027599 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3196881Z 2025-12-04T11:13:50.3197204Z [W1204 10:55:00.100171142 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3197210Z 2025-12-04T11:13:50.3197274Z FAILED [0.4496s] [100%] 2025-12-04T11:13:50.3197278Z 2025-12-04T11:13:50.3197362Z ==================================== RERUNS ==================================== 2025-12-04T11:13:50.3197660Z _ TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_8_in_features_128_out_features_1024_cuda_bfloat16 _ 2025-12-04T11:13:50.3197746Z Traceback (most recent call last): 2025-12-04T11:13:50.3198091Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.3198162Z method(*args, **kwargs) 2025-12-04T11:13:50.3198451Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.3198517Z method(*args, **kwargs) 2025-12-04T11:13:50.3198811Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T11:13:50.3198908Z with policy(): 2025-12-04T11:13:50.3199211Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T11:13:50.3199278Z raise RuntimeError(msg) 2025-12-04T11:13:50.3200088Z RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_8_in_features_128_out_features_1024_cuda_bfloat16! Caching allocator allocated memory was 0 and is now reported as 131072 on device 0. CUDA driver allocated memory was 230686720 and is now 274726912. 2025-12-04T11:13:50.3200092Z 2025-12-04T11:13:50.3200223Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.3200747Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_8_in_features_128_out_features_1024_cuda_bfloat16 2025-12-04T11:13:50.3200752Z 2025-12-04T11:13:50.3200914Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.3201044Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.3201138Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.3201496Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.3201625Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.3201688Z graph_break [] 2025-12-04T11:13:50.3201814Z ----------------------------- Captured stderr call ----------------------------- 2025-12-04T11:13:50.3202507Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/constant_folding.py:256: UserWarning: Unsupported unwinding pattern: Address not in range (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/profiler/unwind/unwind.cpp:219.) 2025-12-04T11:13:50.3202584Z if out == self.unknown_value: 2025-12-04T11:13:50.3202875Z _ TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_8_in_features_128_out_features_1024_cuda_bfloat16 _ 2025-12-04T11:13:50.3202956Z Traceback (most recent call last): 2025-12-04T11:13:50.3203258Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.3203329Z method(*args, **kwargs) 2025-12-04T11:13:50.3203625Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.3203690Z method(*args, **kwargs) 2025-12-04T11:13:50.3204019Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T11:13:50.3204115Z with policy(): 2025-12-04T11:13:50.3204409Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T11:13:50.3204484Z raise RuntimeError(msg) 2025-12-04T11:13:50.3205299Z RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_8_in_features_128_out_features_1024_cuda_bfloat16! Caching allocator allocated memory was 131072 and is now reported as 262144 on device 0. CUDA driver allocated memory was 274726912 and is now 276824064. 2025-12-04T11:13:50.3205303Z 2025-12-04T11:13:50.3205468Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.3205986Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_8_in_features_128_out_features_1024_cuda_bfloat16 2025-12-04T11:13:50.3205992Z 2025-12-04T11:13:50.3206151Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.3206317Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.3206412Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.3206762Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.3206888Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.3206948Z graph_break [] 2025-12-04T11:13:50.3207077Z ----------------------------- Captured stderr call ----------------------------- 2025-12-04T11:13:50.3207766Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/constant_folding.py:256: UserWarning: Unsupported unwinding pattern: Address not in range (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/profiler/unwind/unwind.cpp:219.) 2025-12-04T11:13:50.3207840Z if out == self.unknown_value: 2025-12-04T11:13:50.3207968Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.3208061Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.3208190Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.3208541Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.3208599Z graph_break [] 2025-12-04T11:13:50.3208687Z =================================== FAILURES =================================== 2025-12-04T11:13:50.3208981Z _ TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_8_in_features_128_out_features_1024_cuda_bfloat16 _ 2025-12-04T11:13:50.3209060Z Traceback (most recent call last): 2025-12-04T11:13:50.3209356Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.3209430Z method(*args, **kwargs) 2025-12-04T11:13:50.3209723Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.3209786Z method(*args, **kwargs) 2025-12-04T11:13:50.3210071Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T11:13:50.3210135Z with policy(): 2025-12-04T11:13:50.3210427Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T11:13:50.3210496Z raise RuntimeError(msg) 2025-12-04T11:13:50.3211354Z RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_8_in_features_128_out_features_1024_cuda_bfloat16! Caching allocator allocated memory was 262144 and is now reported as 393216 on device 0. CUDA driver allocated memory was 276824064 and is now 278921216. 2025-12-04T11:13:50.3211392Z 2025-12-04T11:13:50.3211520Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.3212038Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_8_in_features_128_out_features_1024_cuda_bfloat16 2025-12-04T11:13:50.3212042Z 2025-12-04T11:13:50.3212199Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.3212329Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.3212476Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.3212819Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.3212948Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.3213006Z graph_break [] 2025-12-04T11:13:50.3213183Z ----------------------------- Captured stderr call ----------------------------- 2025-12-04T11:13:50.3213868Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/constant_folding.py:256: UserWarning: Unsupported unwinding pattern: Address not in range (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/profiler/unwind/unwind.cpp:219.) 2025-12-04T11:13:50.3213938Z if out == self.unknown_value: 2025-12-04T11:13:50.3214066Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.3214156Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.3214281Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.3214622Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.3214684Z graph_break [] 2025-12-04T11:13:50.3214810Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.3214900Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.3215021Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.3215361Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.3215421Z graph_break [] 2025-12-04T11:13:50.3215909Z - generated xml file: /var/lib/jenkins/workspace/test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-bd700a0c553d5cfe.xml - 2025-12-04T11:13:50.3216011Z =========================== short test summary info ============================ 2025-12-04T11:13:50.3217309Z FAILED [0.4496s] inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_1_mid_dim_8_in_features_128_out_features_1024_cuda_bfloat16 - RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_8_in_features_128_out_features_1024_cuda_bfloat16! Caching allocator allocated memory was 262144 and is now reported as 393216 on device 0. CUDA driver allocated memory was 276824064 and is now 278921216. 2025-12-04T11:13:50.3217315Z 2025-12-04T11:13:50.3217437Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.3217956Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_8_in_features_128_out_features_1024_cuda_bfloat16 2025-12-04T11:13:50.3217962Z 2025-12-04T11:13:50.3218119Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.3218264Z !!!!!!!!!!!!!!!!!!!!!!!!!! stopping after 1 failures !!!!!!!!!!!!!!!!!!!!!!!!!!! 2025-12-04T11:13:50.3218416Z ================== 1 failed, 57 deselected, 2 rerun in 11.73s ================== 2025-12-04T11:13:50.3218479Z Got exit code 1 2025-12-04T11:13:50.3218545Z Retrying single test... 2025-12-04T11:13:50.3218810Z W1204 10:55:06.904000 76095 site-packages/torch/_inductor/utils.py:1703] Not enough SMs to use max_autotune_gemm mode 2025-12-04T11:13:50.3219192Z Test results will be stored in test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-28b04335f093e360.xml 2025-12-04T11:13:50.3219292Z ============================= test session starts ============================== 2025-12-04T11:13:50.3219557Z platform linux -- Python 3.10.14, pytest-7.3.2, pluggy-1.6.0 -- /opt/conda/envs/py_3.10/bin/python 2025-12-04T11:13:50.3219627Z cachedir: .pytest_cache 2025-12-04T11:13:50.3219941Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow] 2025-12-04T11:13:50.3220021Z rootdir: /var/lib/jenkins/workspace 2025-12-04T11:13:50.3220089Z configfile: pytest.ini 2025-12-04T11:13:50.3220415Z plugins: hypothesis-6.56.4, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-14.0, subtests-0.13.1, xdist-3.3.1, xdoctest-1.3.0, typeguard-4.3.0 2025-12-04T11:13:50.3220578Z collecting ... collected 58 items / 57 deselected / 1 selected 2025-12-04T11:13:50.3221158Z stepcurrent: skipping 34 already run items. Running only test/inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_1_mid_dim_8_in_features_128_out_features_1024_cuda_bfloat16 2025-12-04T11:13:50.3221231Z Running 1 items in this shard 2025-12-04T11:13:50.3221235Z 2025-12-04T11:13:50.3221979Z inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_1_mid_dim_8_in_features_128_out_features_1024_cuda_bfloat16 [W1204 10:55:08.023143797 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3221987Z 2025-12-04T11:13:50.3222289Z [W1204 10:55:17.199846415 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3222294Z 2025-12-04T11:13:50.3222584Z [W1204 10:55:17.200131051 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3222598Z 2025-12-04T11:13:50.3222884Z [W1204 10:55:17.205713033 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3222888Z 2025-12-04T11:13:50.3223178Z [W1204 10:55:17.206259695 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3223182Z 2025-12-04T11:13:50.3223469Z [W1204 10:55:17.206420189 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3223474Z 2025-12-04T11:13:50.3223803Z [W1204 10:55:17.211749706 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3223808Z 2025-12-04T11:13:50.3224097Z [W1204 10:55:17.212269037 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3224100Z 2025-12-04T11:13:50.3224383Z [W1204 10:55:17.212431671 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3224387Z 2025-12-04T11:13:50.3224471Z ('RERUN', {'yellow': True}) [11.0915s] [100%] 2025-12-04T11:13:50.3225199Z inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_1_mid_dim_8_in_features_128_out_features_1024_cuda_bfloat16 [W1204 10:55:18.433621538 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3225245Z 2025-12-04T11:13:50.3225572Z [W1204 10:55:18.434167180 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3225582Z 2025-12-04T11:13:50.3225869Z [W1204 10:55:18.434306783 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3225872Z 2025-12-04T11:13:50.3226156Z [W1204 10:55:18.437282408 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3226160Z 2025-12-04T11:13:50.3226488Z [W1204 10:55:18.437858461 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3226491Z 2025-12-04T11:13:50.3226782Z [W1204 10:55:18.437997244 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3226787Z 2025-12-04T11:13:50.3227082Z [W1204 10:55:18.442607045 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3227119Z 2025-12-04T11:13:50.3227408Z [W1204 10:55:18.443083726 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3227411Z 2025-12-04T11:13:50.3227701Z [W1204 10:55:18.443219569 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3227704Z 2025-12-04T11:13:50.3227784Z ('RERUN', {'yellow': True}) [0.4550s] [100%] 2025-12-04T11:13:50.3228508Z inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_1_mid_dim_8_in_features_128_out_features_1024_cuda_bfloat16 [W1204 10:55:18.886072847 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3228514Z 2025-12-04T11:13:50.3228804Z [W1204 10:55:18.886621029 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3228809Z 2025-12-04T11:13:50.3229096Z [W1204 10:55:18.886764683 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3229103Z 2025-12-04T11:13:50.3229396Z [W1204 10:55:18.889719408 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3229399Z 2025-12-04T11:13:50.3229695Z [W1204 10:55:18.890316401 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3229698Z 2025-12-04T11:13:50.3229986Z [W1204 10:55:18.890459414 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3229990Z 2025-12-04T11:13:50.3230276Z [W1204 10:55:18.895006524 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3230281Z 2025-12-04T11:13:50.3230569Z [W1204 10:55:18.895479244 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3230572Z 2025-12-04T11:13:50.3230857Z [W1204 10:55:18.895617368 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3230861Z 2025-12-04T11:13:50.3230926Z FAILED [0.4547s] [100%] 2025-12-04T11:13:50.3230929Z 2025-12-04T11:13:50.3231015Z ==================================== RERUNS ==================================== 2025-12-04T11:13:50.3231315Z _ TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_8_in_features_128_out_features_1024_cuda_bfloat16 _ 2025-12-04T11:13:50.3231437Z Traceback (most recent call last): 2025-12-04T11:13:50.3231779Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.3231850Z method(*args, **kwargs) 2025-12-04T11:13:50.3232142Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.3232206Z method(*args, **kwargs) 2025-12-04T11:13:50.3232497Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T11:13:50.3232557Z with policy(): 2025-12-04T11:13:50.3232850Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T11:13:50.3232967Z raise RuntimeError(msg) 2025-12-04T11:13:50.3233776Z RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_8_in_features_128_out_features_1024_cuda_bfloat16! Caching allocator allocated memory was 0 and is now reported as 131072 on device 0. CUDA driver allocated memory was 230686720 and is now 274726912. 2025-12-04T11:13:50.3233814Z 2025-12-04T11:13:50.3233948Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.3234480Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_8_in_features_128_out_features_1024_cuda_bfloat16 2025-12-04T11:13:50.3234484Z 2025-12-04T11:13:50.3234650Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.3234781Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.3234878Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.3235230Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.3235361Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.3235422Z graph_break [] 2025-12-04T11:13:50.3235556Z ----------------------------- Captured stderr call ----------------------------- 2025-12-04T11:13:50.3236250Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/constant_folding.py:256: UserWarning: Unsupported unwinding pattern: Address not in range (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/profiler/unwind/unwind.cpp:219.) 2025-12-04T11:13:50.3236326Z if out == self.unknown_value: 2025-12-04T11:13:50.3236616Z _ TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_8_in_features_128_out_features_1024_cuda_bfloat16 _ 2025-12-04T11:13:50.3236690Z Traceback (most recent call last): 2025-12-04T11:13:50.3236987Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.3237055Z method(*args, **kwargs) 2025-12-04T11:13:50.3237348Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.3237412Z method(*args, **kwargs) 2025-12-04T11:13:50.3237701Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T11:13:50.3237763Z with policy(): 2025-12-04T11:13:50.3238056Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T11:13:50.3238123Z raise RuntimeError(msg) 2025-12-04T11:13:50.3238959Z RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_8_in_features_128_out_features_1024_cuda_bfloat16! Caching allocator allocated memory was 131072 and is now reported as 262144 on device 0. CUDA driver allocated memory was 274726912 and is now 276824064. 2025-12-04T11:13:50.3239004Z 2025-12-04T11:13:50.3239189Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.3239722Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_8_in_features_128_out_features_1024_cuda_bfloat16 2025-12-04T11:13:50.3239728Z 2025-12-04T11:13:50.3239887Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.3240017Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.3240110Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.3240495Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.3240628Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.3240691Z graph_break [] 2025-12-04T11:13:50.3240821Z ----------------------------- Captured stderr call ----------------------------- 2025-12-04T11:13:50.3241515Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/constant_folding.py:256: UserWarning: Unsupported unwinding pattern: Address not in range (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/profiler/unwind/unwind.cpp:219.) 2025-12-04T11:13:50.3241622Z if out == self.unknown_value: 2025-12-04T11:13:50.3241765Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.3241868Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.3242002Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.3242358Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.3242422Z graph_break [] 2025-12-04T11:13:50.3242517Z =================================== FAILURES =================================== 2025-12-04T11:13:50.3242819Z _ TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_8_in_features_128_out_features_1024_cuda_bfloat16 _ 2025-12-04T11:13:50.3242897Z Traceback (most recent call last): 2025-12-04T11:13:50.3243207Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.3243278Z method(*args, **kwargs) 2025-12-04T11:13:50.3243580Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.3243645Z method(*args, **kwargs) 2025-12-04T11:13:50.3243937Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T11:13:50.3244002Z with policy(): 2025-12-04T11:13:50.3244295Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T11:13:50.3244365Z raise RuntimeError(msg) 2025-12-04T11:13:50.3245189Z RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_8_in_features_128_out_features_1024_cuda_bfloat16! Caching allocator allocated memory was 262144 and is now reported as 393216 on device 0. CUDA driver allocated memory was 276824064 and is now 278921216. 2025-12-04T11:13:50.3245194Z 2025-12-04T11:13:50.3245327Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.3245856Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_8_in_features_128_out_features_1024_cuda_bfloat16 2025-12-04T11:13:50.3245860Z 2025-12-04T11:13:50.3246024Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.3246202Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.3246332Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.3246681Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.3246818Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.3246880Z graph_break [] 2025-12-04T11:13:50.3247007Z ----------------------------- Captured stderr call ----------------------------- 2025-12-04T11:13:50.3247731Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/constant_folding.py:256: UserWarning: Unsupported unwinding pattern: Address not in range (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/profiler/unwind/unwind.cpp:219.) 2025-12-04T11:13:50.3247804Z if out == self.unknown_value: 2025-12-04T11:13:50.3247932Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.3248026Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.3248150Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.3248550Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.3248610Z graph_break [] 2025-12-04T11:13:50.3248745Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.3248836Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.3248960Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.3249307Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.3249375Z graph_break [] 2025-12-04T11:13:50.3249862Z - generated xml file: /var/lib/jenkins/workspace/test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-28b04335f093e360.xml - 2025-12-04T11:13:50.3249968Z =========================== short test summary info ============================ 2025-12-04T11:13:50.3251267Z FAILED [0.4547s] inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_1_mid_dim_8_in_features_128_out_features_1024_cuda_bfloat16 - RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_8_in_features_128_out_features_1024_cuda_bfloat16! Caching allocator allocated memory was 262144 and is now reported as 393216 on device 0. CUDA driver allocated memory was 276824064 and is now 278921216. 2025-12-04T11:13:50.3251276Z 2025-12-04T11:13:50.3251405Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.3251941Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_8_in_features_128_out_features_1024_cuda_bfloat16 2025-12-04T11:13:50.3251944Z 2025-12-04T11:13:50.3252106Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.3252212Z !!!!!!!!!!!!!!!!!!!!!!!!!! stopping after 1 failures !!!!!!!!!!!!!!!!!!!!!!!!!!! 2025-12-04T11:13:50.3252335Z ================== 1 failed, 57 deselected, 2 rerun in 12.03s ================== 2025-12-04T11:13:50.3252396Z Got exit code 1 2025-12-04T11:13:50.3252871Z FAILED CONSISTENTLY: test/inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_1_mid_dim_8_in_features_128_out_features_1024_cuda_bfloat16 2025-12-04T11:13:50.3253121Z Test failed consistently, continuing with the rest of the tests due to continue-through-error being set 2025-12-04T11:13:50.3253387Z W1204 10:55:25.742000 76288 site-packages/torch/_inductor/utils.py:1703] Not enough SMs to use max_autotune_gemm mode 2025-12-04T11:13:50.3253858Z Test results will be stored in test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-857ce5cc87a4a050.xml 2025-12-04T11:13:50.3253966Z ============================= test session starts ============================== 2025-12-04T11:13:50.3254176Z platform linux -- Python 3.10.14, pytest-7.3.2, pluggy-1.6.0 -- /opt/conda/envs/py_3.10/bin/python 2025-12-04T11:13:50.3254248Z cachedir: .pytest_cache 2025-12-04T11:13:50.3254555Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow] 2025-12-04T11:13:50.3254633Z rootdir: /var/lib/jenkins/workspace 2025-12-04T11:13:50.3254705Z configfile: pytest.ini 2025-12-04T11:13:50.3255057Z plugins: hypothesis-6.56.4, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-14.0, subtests-0.13.1, xdist-3.3.1, xdoctest-1.3.0, typeguard-4.3.0 2025-12-04T11:13:50.3255200Z collecting ... collected 58 items / 35 deselected / 23 selected 2025-12-04T11:13:50.3255288Z stepcurrent: skipping 35 already run items. 2025-12-04T11:13:50.3255362Z Running 23 items in this shard 2025-12-04T11:13:50.3255366Z 2025-12-04T11:13:50.3255893Z inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_1_mid_dim_8_in_features_128_out_features_64_cuda_bfloat16 ('RERUN', {'yellow': True}) [1.8584s] [ 4%] 2025-12-04T11:13:50.3256377Z inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_1_mid_dim_8_in_features_128_out_features_64_cuda_bfloat16 ('RERUN', {'yellow': True}) [0.4533s] [ 4%] 2025-12-04T11:13:50.3256818Z inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_1_mid_dim_8_in_features_128_out_features_64_cuda_bfloat16 FAILED [0.4597s] [ 4%] 2025-12-04T11:13:50.3256823Z 2025-12-04T11:13:50.3256908Z ==================================== RERUNS ==================================== 2025-12-04T11:13:50.3257201Z _ TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_8_in_features_128_out_features_64_cuda_bfloat16 _ 2025-12-04T11:13:50.3257281Z Traceback (most recent call last): 2025-12-04T11:13:50.3257589Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.3257657Z method(*args, **kwargs) 2025-12-04T11:13:50.3257948Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.3258011Z method(*args, **kwargs) 2025-12-04T11:13:50.3258305Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T11:13:50.3258368Z with policy(): 2025-12-04T11:13:50.3258660Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T11:13:50.3258730Z raise RuntimeError(msg) 2025-12-04T11:13:50.3259524Z RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_8_in_features_128_out_features_64_cuda_bfloat16! Caching allocator allocated memory was 0 and is now reported as 8192 on device 0. CUDA driver allocated memory was 230686720 and is now 274726912. 2025-12-04T11:13:50.3259529Z 2025-12-04T11:13:50.3259660Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.3260174Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_8_in_features_128_out_features_64_cuda_bfloat16 2025-12-04T11:13:50.3260178Z 2025-12-04T11:13:50.3260343Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.3260476Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.3260613Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.3261001Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.3261138Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.3261203Z graph_break [] 2025-12-04T11:13:50.3261493Z _ TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_8_in_features_128_out_features_64_cuda_bfloat16 _ 2025-12-04T11:13:50.3261570Z Traceback (most recent call last): 2025-12-04T11:13:50.3261879Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.3261943Z method(*args, **kwargs) 2025-12-04T11:13:50.3262268Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.3262338Z method(*args, **kwargs) 2025-12-04T11:13:50.3262627Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T11:13:50.3262691Z with policy(): 2025-12-04T11:13:50.3263020Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T11:13:50.3263100Z raise RuntimeError(msg) 2025-12-04T11:13:50.3263942Z RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_8_in_features_128_out_features_64_cuda_bfloat16! Caching allocator allocated memory was 8192 and is now reported as 16384 on device 0. CUDA driver allocated memory was 274726912 and is now 276824064. 2025-12-04T11:13:50.3263946Z 2025-12-04T11:13:50.3264074Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.3264591Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_8_in_features_128_out_features_64_cuda_bfloat16 2025-12-04T11:13:50.3264597Z 2025-12-04T11:13:50.3264757Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.3264887Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.3264984Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.3265333Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.3265463Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.3265525Z graph_break [] 2025-12-04T11:13:50.3265652Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.3265748Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.3265873Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.3266218Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.3266284Z graph_break [] 2025-12-04T11:13:50.3266371Z =================================== FAILURES =================================== 2025-12-04T11:13:50.3266667Z _ TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_8_in_features_128_out_features_64_cuda_bfloat16 _ 2025-12-04T11:13:50.3266743Z Traceback (most recent call last): 2025-12-04T11:13:50.3267041Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.3267110Z method(*args, **kwargs) 2025-12-04T11:13:50.3267400Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.3267527Z method(*args, **kwargs) 2025-12-04T11:13:50.3267849Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T11:13:50.3267911Z with policy(): 2025-12-04T11:13:50.3268207Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T11:13:50.3268275Z raise RuntimeError(msg) 2025-12-04T11:13:50.3269076Z RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_8_in_features_128_out_features_64_cuda_bfloat16! Caching allocator allocated memory was 16384 and is now reported as 24576 on device 0. CUDA driver allocated memory was 276824064 and is now 278921216. 2025-12-04T11:13:50.3269083Z 2025-12-04T11:13:50.3269244Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.3269767Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_8_in_features_128_out_features_64_cuda_bfloat16 2025-12-04T11:13:50.3269773Z 2025-12-04T11:13:50.3269935Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.3270098Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.3270195Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.3270546Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.3270672Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.3270735Z graph_break [] 2025-12-04T11:13:50.3270865Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.3270955Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.3271086Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.3271426Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.3271489Z graph_break [] 2025-12-04T11:13:50.3271612Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.3271702Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.3271827Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.3272163Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.3272226Z graph_break [] 2025-12-04T11:13:50.3272713Z - generated xml file: /var/lib/jenkins/workspace/test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-857ce5cc87a4a050.xml - 2025-12-04T11:13:50.3272816Z =========================== short test summary info ============================ 2025-12-04T11:13:50.3274094Z FAILED [0.4597s] inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_1_mid_dim_8_in_features_128_out_features_64_cuda_bfloat16 - RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_8_in_features_128_out_features_64_cuda_bfloat16! Caching allocator allocated memory was 16384 and is now reported as 24576 on device 0. CUDA driver allocated memory was 276824064 and is now 278921216. 2025-12-04T11:13:50.3274099Z 2025-12-04T11:13:50.3274223Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.3274738Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_8_in_features_128_out_features_64_cuda_bfloat16 2025-12-04T11:13:50.3274873Z 2025-12-04T11:13:50.3275034Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.3275181Z !!!!!!!!!!!!!!!!!!!!!!!!!! stopping after 1 failures !!!!!!!!!!!!!!!!!!!!!!!!!!! 2025-12-04T11:13:50.3275302Z ================== 1 failed, 35 deselected, 2 rerun in 2.80s =================== 2025-12-04T11:13:50.3275364Z Got exit code 1 2025-12-04T11:13:50.3275431Z Retrying single test... 2025-12-04T11:13:50.3275695Z W1204 10:55:35.738000 76469 site-packages/torch/_inductor/utils.py:1703] Not enough SMs to use max_autotune_gemm mode 2025-12-04T11:13:50.3276080Z Test results will be stored in test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-4e0b90b7cea75841.xml 2025-12-04T11:13:50.3276216Z ============================= test session starts ============================== 2025-12-04T11:13:50.3276429Z platform linux -- Python 3.10.14, pytest-7.3.2, pluggy-1.6.0 -- /opt/conda/envs/py_3.10/bin/python 2025-12-04T11:13:50.3276500Z cachedir: .pytest_cache 2025-12-04T11:13:50.3276809Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow] 2025-12-04T11:13:50.3276924Z rootdir: /var/lib/jenkins/workspace 2025-12-04T11:13:50.3276992Z configfile: pytest.ini 2025-12-04T11:13:50.3277302Z plugins: hypothesis-6.56.4, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-14.0, subtests-0.13.1, xdist-3.3.1, xdoctest-1.3.0, typeguard-4.3.0 2025-12-04T11:13:50.3277432Z collecting ... collected 58 items / 57 deselected / 1 selected 2025-12-04T11:13:50.3278165Z stepcurrent: skipping 35 already run items. Running only test/inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_1_mid_dim_8_in_features_128_out_features_64_cuda_bfloat16 2025-12-04T11:13:50.3278241Z Running 1 items in this shard 2025-12-04T11:13:50.3278244Z 2025-12-04T11:13:50.3278974Z inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_1_mid_dim_8_in_features_128_out_features_64_cuda_bfloat16 [W1204 10:55:36.824505696 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3278982Z 2025-12-04T11:13:50.3279282Z [W1204 10:55:45.871863131 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3279286Z 2025-12-04T11:13:50.3279596Z [W1204 10:55:45.872097146 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3279599Z 2025-12-04T11:13:50.3279893Z [W1204 10:55:45.877669909 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3279897Z 2025-12-04T11:13:50.3280190Z [W1204 10:55:45.878225672 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3280196Z 2025-12-04T11:13:50.3280489Z [W1204 10:55:45.878394076 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3280494Z 2025-12-04T11:13:50.3280786Z [W1204 10:55:45.883800965 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3280789Z 2025-12-04T11:13:50.3281077Z [W1204 10:55:45.884334257 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3281081Z 2025-12-04T11:13:50.3281368Z [W1204 10:55:45.884485660 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3281373Z 2025-12-04T11:13:50.3281462Z ('RERUN', {'yellow': True}) [10.9286s] [100%] 2025-12-04T11:13:50.3282245Z inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_1_mid_dim_8_in_features_128_out_features_64_cuda_bfloat16 [W1204 10:55:47.072390589 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3282295Z 2025-12-04T11:13:50.3282594Z [W1204 10:55:47.072989772 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3282597Z 2025-12-04T11:13:50.3282885Z [W1204 10:55:47.073130185 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3282889Z 2025-12-04T11:13:50.3283181Z [W1204 10:55:47.076100721 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3283231Z 2025-12-04T11:13:50.3283524Z [W1204 10:55:47.076688904 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3283528Z 2025-12-04T11:13:50.3283821Z [W1204 10:55:47.076827247 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3283824Z 2025-12-04T11:13:50.3284161Z [W1204 10:55:47.081491490 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3284164Z 2025-12-04T11:13:50.3284449Z [W1204 10:55:47.081970901 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3284456Z 2025-12-04T11:13:50.3284744Z [W1204 10:55:47.082105864 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3284747Z 2025-12-04T11:13:50.3284830Z ('RERUN', {'yellow': True}) [0.4268s] [100%] 2025-12-04T11:13:50.3285549Z inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_1_mid_dim_8_in_features_128_out_features_64_cuda_bfloat16 [W1204 10:55:47.494148722 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3285554Z 2025-12-04T11:13:50.3285844Z [W1204 10:55:47.494738465 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3285847Z 2025-12-04T11:13:50.3286139Z [W1204 10:55:47.494878048 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3286143Z 2025-12-04T11:13:50.3286429Z [W1204 10:55:47.497851843 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3286433Z 2025-12-04T11:13:50.3286724Z [W1204 10:55:47.498418816 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3286729Z 2025-12-04T11:13:50.3287019Z [W1204 10:55:47.498554999 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3287022Z 2025-12-04T11:13:50.3287311Z [W1204 10:55:47.503200882 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3287316Z 2025-12-04T11:13:50.3287606Z [W1204 10:55:47.503678763 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3287609Z 2025-12-04T11:13:50.3287900Z [W1204 10:55:47.503814855 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3287910Z 2025-12-04T11:13:50.3287984Z FAILED [0.4194s] [100%] 2025-12-04T11:13:50.3287988Z 2025-12-04T11:13:50.3288075Z ==================================== RERUNS ==================================== 2025-12-04T11:13:50.3288416Z _ TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_8_in_features_128_out_features_64_cuda_bfloat16 _ 2025-12-04T11:13:50.3288529Z Traceback (most recent call last): 2025-12-04T11:13:50.3288835Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.3288910Z method(*args, **kwargs) 2025-12-04T11:13:50.3289202Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.3289270Z method(*args, **kwargs) 2025-12-04T11:13:50.3289570Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T11:13:50.3289631Z with policy(): 2025-12-04T11:13:50.3289963Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T11:13:50.3290033Z raise RuntimeError(msg) 2025-12-04T11:13:50.3290828Z RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_8_in_features_128_out_features_64_cuda_bfloat16! Caching allocator allocated memory was 0 and is now reported as 8192 on device 0. CUDA driver allocated memory was 230686720 and is now 274726912. 2025-12-04T11:13:50.3290888Z 2025-12-04T11:13:50.3291020Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.3291535Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_8_in_features_128_out_features_64_cuda_bfloat16 2025-12-04T11:13:50.3291539Z 2025-12-04T11:13:50.3291704Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.3291834Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.3291935Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.3292289Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.3292420Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.3292486Z graph_break [] 2025-12-04T11:13:50.3292610Z ----------------------------- Captured stderr call ----------------------------- 2025-12-04T11:13:50.3293302Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/constant_folding.py:256: UserWarning: Unsupported unwinding pattern: Address not in range (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/profiler/unwind/unwind.cpp:219.) 2025-12-04T11:13:50.3293376Z if out == self.unknown_value: 2025-12-04T11:13:50.3293667Z _ TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_8_in_features_128_out_features_64_cuda_bfloat16 _ 2025-12-04T11:13:50.3293750Z Traceback (most recent call last): 2025-12-04T11:13:50.3294048Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.3294114Z method(*args, **kwargs) 2025-12-04T11:13:50.3294407Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.3294473Z method(*args, **kwargs) 2025-12-04T11:13:50.3294777Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T11:13:50.3294844Z with policy(): 2025-12-04T11:13:50.3295137Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T11:13:50.3295207Z raise RuntimeError(msg) 2025-12-04T11:13:50.3296051Z RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_8_in_features_128_out_features_64_cuda_bfloat16! Caching allocator allocated memory was 8192 and is now reported as 16384 on device 0. CUDA driver allocated memory was 274726912 and is now 276824064. 2025-12-04T11:13:50.3296089Z 2025-12-04T11:13:50.3296223Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.3296741Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_8_in_features_128_out_features_64_cuda_bfloat16 2025-12-04T11:13:50.3296745Z 2025-12-04T11:13:50.3296908Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.3297034Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.3297166Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.3297518Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.3297647Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.3297708Z graph_break [] 2025-12-04T11:13:50.3297836Z ----------------------------- Captured stderr call ----------------------------- 2025-12-04T11:13:50.3298561Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/constant_folding.py:256: UserWarning: Unsupported unwinding pattern: Address not in range (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/profiler/unwind/unwind.cpp:219.) 2025-12-04T11:13:50.3298633Z if out == self.unknown_value: 2025-12-04T11:13:50.3298757Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.3298858Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.3298989Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.3299333Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.3299396Z graph_break [] 2025-12-04T11:13:50.3299491Z =================================== FAILURES =================================== 2025-12-04T11:13:50.3299787Z _ TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_8_in_features_128_out_features_64_cuda_bfloat16 _ 2025-12-04T11:13:50.3299865Z Traceback (most recent call last): 2025-12-04T11:13:50.3300160Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.3300225Z method(*args, **kwargs) 2025-12-04T11:13:50.3300517Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.3300584Z method(*args, **kwargs) 2025-12-04T11:13:50.3300876Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T11:13:50.3300937Z with policy(): 2025-12-04T11:13:50.3301231Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T11:13:50.3301302Z raise RuntimeError(msg) 2025-12-04T11:13:50.3302110Z RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_8_in_features_128_out_features_64_cuda_bfloat16! Caching allocator allocated memory was 16384 and is now reported as 24576 on device 0. CUDA driver allocated memory was 276824064 and is now 278921216. 2025-12-04T11:13:50.3302114Z 2025-12-04T11:13:50.3302248Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.3302770Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_8_in_features_128_out_features_64_cuda_bfloat16 2025-12-04T11:13:50.3302814Z 2025-12-04T11:13:50.3302972Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.3303133Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.3303229Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.3303613Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.3303743Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.3303803Z graph_break [] 2025-12-04T11:13:50.3303931Z ----------------------------- Captured stderr call ----------------------------- 2025-12-04T11:13:50.3304654Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/constant_folding.py:256: UserWarning: Unsupported unwinding pattern: Address not in range (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/profiler/unwind/unwind.cpp:219.) 2025-12-04T11:13:50.3304732Z if out == self.unknown_value: 2025-12-04T11:13:50.3304856Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.3304948Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.3305114Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.3305460Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.3305520Z graph_break [] 2025-12-04T11:13:50.3305648Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.3305736Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.3305864Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.3306207Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.3306268Z graph_break [] 2025-12-04T11:13:50.3306760Z - generated xml file: /var/lib/jenkins/workspace/test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-4e0b90b7cea75841.xml - 2025-12-04T11:13:50.3306867Z =========================== short test summary info ============================ 2025-12-04T11:13:50.3308157Z FAILED [0.4194s] inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_1_mid_dim_8_in_features_128_out_features_64_cuda_bfloat16 - RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_8_in_features_128_out_features_64_cuda_bfloat16! Caching allocator allocated memory was 16384 and is now reported as 24576 on device 0. CUDA driver allocated memory was 276824064 and is now 278921216. 2025-12-04T11:13:50.3308162Z 2025-12-04T11:13:50.3308291Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.3308809Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_8_in_features_128_out_features_64_cuda_bfloat16 2025-12-04T11:13:50.3308815Z 2025-12-04T11:13:50.3308974Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.3309079Z !!!!!!!!!!!!!!!!!!!!!!!!!! stopping after 1 failures !!!!!!!!!!!!!!!!!!!!!!!!!!! 2025-12-04T11:13:50.3309199Z ================== 1 failed, 57 deselected, 2 rerun in 11.80s ================== 2025-12-04T11:13:50.3309262Z Got exit code 1 2025-12-04T11:13:50.3309327Z Retrying single test... 2025-12-04T11:13:50.3309597Z W1204 10:55:54.328000 76655 site-packages/torch/_inductor/utils.py:1703] Not enough SMs to use max_autotune_gemm mode 2025-12-04T11:13:50.3309983Z Test results will be stored in test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-14983efcd69bf6ac.xml 2025-12-04T11:13:50.3310127Z ============================= test session starts ============================== 2025-12-04T11:13:50.3310371Z platform linux -- Python 3.10.14, pytest-7.3.2, pluggy-1.6.0 -- /opt/conda/envs/py_3.10/bin/python 2025-12-04T11:13:50.3310441Z cachedir: .pytest_cache 2025-12-04T11:13:50.3310751Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow] 2025-12-04T11:13:50.3310828Z rootdir: /var/lib/jenkins/workspace 2025-12-04T11:13:50.3310901Z configfile: pytest.ini 2025-12-04T11:13:50.3311216Z plugins: hypothesis-6.56.4, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-14.0, subtests-0.13.1, xdist-3.3.1, xdoctest-1.3.0, typeguard-4.3.0 2025-12-04T11:13:50.3311377Z collecting ... collected 58 items / 57 deselected / 1 selected 2025-12-04T11:13:50.3311951Z stepcurrent: skipping 35 already run items. Running only test/inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_1_mid_dim_8_in_features_128_out_features_64_cuda_bfloat16 2025-12-04T11:13:50.3312029Z Running 1 items in this shard 2025-12-04T11:13:50.3312032Z 2025-12-04T11:13:50.3312760Z inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_1_mid_dim_8_in_features_128_out_features_64_cuda_bfloat16 [W1204 10:55:55.407317854 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3312798Z 2025-12-04T11:13:50.3313100Z [W1204 10:56:04.628314509 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3313103Z 2025-12-04T11:13:50.3313403Z [W1204 10:56:04.628553544 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3313407Z 2025-12-04T11:13:50.3313698Z [W1204 10:56:04.634259319 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3313702Z 2025-12-04T11:13:50.3313990Z [W1204 10:56:04.634815892 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3314002Z 2025-12-04T11:13:50.3314289Z [W1204 10:56:04.634982975 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3314293Z 2025-12-04T11:13:50.3314581Z [W1204 10:56:04.640315133 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3314584Z 2025-12-04T11:13:50.3314874Z [W1204 10:56:04.640840504 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3314877Z 2025-12-04T11:13:50.3315165Z [W1204 10:56:04.640999378 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3315169Z 2025-12-04T11:13:50.3315257Z ('RERUN', {'yellow': True}) [11.0934s] [100%] 2025-12-04T11:13:50.3315972Z inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_1_mid_dim_8_in_features_128_out_features_64_cuda_bfloat16 [W1204 10:56:05.821535703 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3315978Z 2025-12-04T11:13:50.3316269Z [W1204 10:56:05.822110716 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3316272Z 2025-12-04T11:13:50.3316563Z [W1204 10:56:05.822264220 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3316566Z 2025-12-04T11:13:50.3316858Z [W1204 10:56:05.825193374 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3316900Z 2025-12-04T11:13:50.3317219Z [W1204 10:56:05.825771317 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3317224Z 2025-12-04T11:13:50.3317511Z [W1204 10:56:05.825909400 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3317515Z 2025-12-04T11:13:50.3317806Z [W1204 10:56:05.830452970 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3317809Z 2025-12-04T11:13:50.3318148Z [W1204 10:56:05.830924521 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3318151Z 2025-12-04T11:13:50.3318444Z [W1204 10:56:05.831059914 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3318449Z 2025-12-04T11:13:50.3318540Z ('RERUN', {'yellow': True}) [0.4174s] [100%] 2025-12-04T11:13:50.3319267Z inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_1_mid_dim_8_in_features_128_out_features_64_cuda_bfloat16 [W1204 10:56:06.232445824 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3319309Z 2025-12-04T11:13:50.3319597Z [W1204 10:56:06.233045337 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3319601Z 2025-12-04T11:13:50.3319894Z [W1204 10:56:06.233190070 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3319897Z 2025-12-04T11:13:50.3320185Z [W1204 10:56:06.236077624 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3320190Z 2025-12-04T11:13:50.3320481Z [W1204 10:56:06.236654066 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3320488Z 2025-12-04T11:13:50.3320775Z [W1204 10:56:06.236792239 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3320778Z 2025-12-04T11:13:50.3321064Z [W1204 10:56:06.241301569 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3321068Z 2025-12-04T11:13:50.3321364Z [W1204 10:56:06.241778729 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3321367Z 2025-12-04T11:13:50.3321653Z [W1204 10:56:06.241913362 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3321658Z 2025-12-04T11:13:50.3321727Z FAILED [0.4089s] [100%] 2025-12-04T11:13:50.3321730Z 2025-12-04T11:13:50.3321816Z ==================================== RERUNS ==================================== 2025-12-04T11:13:50.3322111Z _ TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_8_in_features_128_out_features_64_cuda_bfloat16 _ 2025-12-04T11:13:50.3322190Z Traceback (most recent call last): 2025-12-04T11:13:50.3322506Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.3322576Z method(*args, **kwargs) 2025-12-04T11:13:50.3322867Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.3322934Z method(*args, **kwargs) 2025-12-04T11:13:50.3323226Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T11:13:50.3323328Z with policy(): 2025-12-04T11:13:50.3323660Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T11:13:50.3323730Z raise RuntimeError(msg) 2025-12-04T11:13:50.3324522Z RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_8_in_features_128_out_features_64_cuda_bfloat16! Caching allocator allocated memory was 0 and is now reported as 8192 on device 0. CUDA driver allocated memory was 230686720 and is now 274726912. 2025-12-04T11:13:50.3324527Z 2025-12-04T11:13:50.3324658Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.3325202Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_8_in_features_128_out_features_64_cuda_bfloat16 2025-12-04T11:13:50.3325208Z 2025-12-04T11:13:50.3325376Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.3325506Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.3325636Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.3325986Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.3326117Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.3326180Z graph_break [] 2025-12-04T11:13:50.3326305Z ----------------------------- Captured stderr call ----------------------------- 2025-12-04T11:13:50.3326994Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/constant_folding.py:256: UserWarning: Unsupported unwinding pattern: Address not in range (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/profiler/unwind/unwind.cpp:219.) 2025-12-04T11:13:50.3327072Z if out == self.unknown_value: 2025-12-04T11:13:50.3327365Z _ TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_8_in_features_128_out_features_64_cuda_bfloat16 _ 2025-12-04T11:13:50.3327446Z Traceback (most recent call last): 2025-12-04T11:13:50.3327742Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.3327806Z method(*args, **kwargs) 2025-12-04T11:13:50.3328102Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.3328169Z method(*args, **kwargs) 2025-12-04T11:13:50.3328459Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T11:13:50.3328525Z with policy(): 2025-12-04T11:13:50.3328820Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T11:13:50.3328896Z raise RuntimeError(msg) 2025-12-04T11:13:50.3329695Z RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_8_in_features_128_out_features_64_cuda_bfloat16! Caching allocator allocated memory was 8192 and is now reported as 16384 on device 0. CUDA driver allocated memory was 274726912 and is now 276824064. 2025-12-04T11:13:50.3329701Z 2025-12-04T11:13:50.3329829Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.3330359Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_8_in_features_128_out_features_64_cuda_bfloat16 2025-12-04T11:13:50.3330364Z 2025-12-04T11:13:50.3330524Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.3330655Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.3330794Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.3331176Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.3331309Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.3331370Z graph_break [] 2025-12-04T11:13:50.3331498Z ----------------------------- Captured stderr call ----------------------------- 2025-12-04T11:13:50.3332216Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/constant_folding.py:256: UserWarning: Unsupported unwinding pattern: Address not in range (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/profiler/unwind/unwind.cpp:219.) 2025-12-04T11:13:50.3332289Z if out == self.unknown_value: 2025-12-04T11:13:50.3332416Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.3332510Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.3332644Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.3332996Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.3333091Z graph_break [] 2025-12-04T11:13:50.3333183Z =================================== FAILURES =================================== 2025-12-04T11:13:50.3333469Z _ TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_8_in_features_128_out_features_64_cuda_bfloat16 _ 2025-12-04T11:13:50.3333544Z Traceback (most recent call last): 2025-12-04T11:13:50.3333846Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.3333915Z method(*args, **kwargs) 2025-12-04T11:13:50.3334210Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.3334276Z method(*args, **kwargs) 2025-12-04T11:13:50.3334567Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T11:13:50.3334633Z with policy(): 2025-12-04T11:13:50.3334926Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T11:13:50.3334996Z raise RuntimeError(msg) 2025-12-04T11:13:50.3335801Z RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_8_in_features_128_out_features_64_cuda_bfloat16! Caching allocator allocated memory was 16384 and is now reported as 24576 on device 0. CUDA driver allocated memory was 276824064 and is now 278921216. 2025-12-04T11:13:50.3335805Z 2025-12-04T11:13:50.3335930Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.3336443Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_8_in_features_128_out_features_64_cuda_bfloat16 2025-12-04T11:13:50.3336447Z 2025-12-04T11:13:50.3336604Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.3336743Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.3336840Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.3337183Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.3337315Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.3337375Z graph_break [] 2025-12-04T11:13:50.3337503Z ----------------------------- Captured stderr call ----------------------------- 2025-12-04T11:13:50.3338265Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/constant_folding.py:256: UserWarning: Unsupported unwinding pattern: Address not in range (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/profiler/unwind/unwind.cpp:219.) 2025-12-04T11:13:50.3338337Z if out == self.unknown_value: 2025-12-04T11:13:50.3338465Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.3338555Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.3338682Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.3339024Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.3339122Z graph_break [] 2025-12-04T11:13:50.3339249Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.3339343Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.3339467Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.3339811Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.3339911Z graph_break [] 2025-12-04T11:13:50.3340400Z - generated xml file: /var/lib/jenkins/workspace/test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-14983efcd69bf6ac.xml - 2025-12-04T11:13:50.3340503Z =========================== short test summary info ============================ 2025-12-04T11:13:50.3341778Z FAILED [0.4089s] inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_1_mid_dim_8_in_features_128_out_features_64_cuda_bfloat16 - RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_8_in_features_128_out_features_64_cuda_bfloat16! Caching allocator allocated memory was 16384 and is now reported as 24576 on device 0. CUDA driver allocated memory was 276824064 and is now 278921216. 2025-12-04T11:13:50.3341788Z 2025-12-04T11:13:50.3341913Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.3342424Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_8_in_features_128_out_features_64_cuda_bfloat16 2025-12-04T11:13:50.3342432Z 2025-12-04T11:13:50.3342590Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.3342694Z !!!!!!!!!!!!!!!!!!!!!!!!!! stopping after 1 failures !!!!!!!!!!!!!!!!!!!!!!!!!!! 2025-12-04T11:13:50.3342816Z ================== 1 failed, 57 deselected, 2 rerun in 11.94s ================== 2025-12-04T11:13:50.3342877Z Got exit code 1 2025-12-04T11:13:50.3343347Z FAILED CONSISTENTLY: test/inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_1_mid_dim_8_in_features_128_out_features_64_cuda_bfloat16 2025-12-04T11:13:50.3343632Z Test failed consistently, continuing with the rest of the tests due to continue-through-error being set 2025-12-04T11:13:50.3343900Z W1204 10:56:13.077000 76841 site-packages/torch/_inductor/utils.py:1703] Not enough SMs to use max_autotune_gemm mode 2025-12-04T11:13:50.3344285Z Test results will be stored in test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-2104559418af7772.xml 2025-12-04T11:13:50.3344381Z ============================= test session starts ============================== 2025-12-04T11:13:50.3344591Z platform linux -- Python 3.10.14, pytest-7.3.2, pluggy-1.6.0 -- /opt/conda/envs/py_3.10/bin/python 2025-12-04T11:13:50.3344665Z cachedir: .pytest_cache 2025-12-04T11:13:50.3344972Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow] 2025-12-04T11:13:50.3345097Z rootdir: /var/lib/jenkins/workspace 2025-12-04T11:13:50.3345216Z configfile: pytest.ini 2025-12-04T11:13:50.3345534Z plugins: hypothesis-6.56.4, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-14.0, subtests-0.13.1, xdist-3.3.1, xdoctest-1.3.0, typeguard-4.3.0 2025-12-04T11:13:50.3345670Z collecting ... collected 58 items / 36 deselected / 22 selected 2025-12-04T11:13:50.3345758Z stepcurrent: skipping 36 already run items. 2025-12-04T11:13:50.3345831Z Running 22 items in this shard 2025-12-04T11:13:50.3345835Z 2025-12-04T11:13:50.3346334Z inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_1_mid_dim_8_in_features_128_out_features_65_cuda_bfloat16 ('RERUN', {'yellow': True}) [1.9697s] [ 4%] 2025-12-04T11:13:50.3346863Z inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_1_mid_dim_8_in_features_128_out_features_65_cuda_bfloat16 ('RERUN', {'yellow': True}) [0.5473s] [ 4%] 2025-12-04T11:13:50.3347312Z inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_1_mid_dim_8_in_features_128_out_features_65_cuda_bfloat16 FAILED [0.5361s] [ 4%] 2025-12-04T11:13:50.3347349Z 2025-12-04T11:13:50.3347434Z ==================================== RERUNS ==================================== 2025-12-04T11:13:50.3347725Z _ TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_8_in_features_128_out_features_65_cuda_bfloat16 _ 2025-12-04T11:13:50.3347803Z Traceback (most recent call last): 2025-12-04T11:13:50.3348106Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.3348176Z method(*args, **kwargs) 2025-12-04T11:13:50.3348471Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.3348535Z method(*args, **kwargs) 2025-12-04T11:13:50.3348830Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T11:13:50.3348892Z with policy(): 2025-12-04T11:13:50.3349193Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T11:13:50.3349264Z raise RuntimeError(msg) 2025-12-04T11:13:50.3350056Z RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_8_in_features_128_out_features_65_cuda_bfloat16! Caching allocator allocated memory was 0 and is now reported as 8704 on device 0. CUDA driver allocated memory was 230686720 and is now 274726912. 2025-12-04T11:13:50.3350060Z 2025-12-04T11:13:50.3350192Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.3350705Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_8_in_features_128_out_features_65_cuda_bfloat16 2025-12-04T11:13:50.3350710Z 2025-12-04T11:13:50.3350874Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.3351004Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.3351110Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.3351664Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('benchmarking.InductorBenchmarker.benchmark_gpu', 2), ('pad_mm_bench', 1), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.3351793Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.3351858Z graph_break [] 2025-12-04T11:13:50.3352146Z _ TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_8_in_features_128_out_features_65_cuda_bfloat16 _ 2025-12-04T11:13:50.3352261Z Traceback (most recent call last): 2025-12-04T11:13:50.3352597Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.3352664Z method(*args, **kwargs) 2025-12-04T11:13:50.3352962Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.3353024Z method(*args, **kwargs) 2025-12-04T11:13:50.3353311Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T11:13:50.3353375Z with policy(): 2025-12-04T11:13:50.3353670Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T11:13:50.3353771Z raise RuntimeError(msg) 2025-12-04T11:13:50.3354576Z RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_8_in_features_128_out_features_65_cuda_bfloat16! Caching allocator allocated memory was 8704 and is now reported as 17408 on device 0. CUDA driver allocated memory was 274726912 and is now 276824064. 2025-12-04T11:13:50.3354614Z 2025-12-04T11:13:50.3354742Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.3355259Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_8_in_features_128_out_features_65_cuda_bfloat16 2025-12-04T11:13:50.3355263Z 2025-12-04T11:13:50.3355418Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.3355550Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.3355649Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.3356192Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('benchmarking.InductorBenchmarker.benchmark_gpu', 2), ('pad_mm_bench', 1), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.3356324Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.3356387Z graph_break [] 2025-12-04T11:13:50.3356517Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.3356608Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.3356730Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.3357270Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('benchmarking.InductorBenchmarker.benchmark_gpu', 2), ('pad_mm_bench', 1), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.3357330Z graph_break [] 2025-12-04T11:13:50.3357421Z =================================== FAILURES =================================== 2025-12-04T11:13:50.3357717Z _ TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_8_in_features_128_out_features_65_cuda_bfloat16 _ 2025-12-04T11:13:50.3357796Z Traceback (most recent call last): 2025-12-04T11:13:50.3358095Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.3358162Z method(*args, **kwargs) 2025-12-04T11:13:50.3358452Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.3358521Z method(*args, **kwargs) 2025-12-04T11:13:50.3358810Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T11:13:50.3358872Z with policy(): 2025-12-04T11:13:50.3359169Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T11:13:50.3359276Z raise RuntimeError(msg) 2025-12-04T11:13:50.3360130Z RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_8_in_features_128_out_features_65_cuda_bfloat16! Caching allocator allocated memory was 17408 and is now reported as 26112 on device 0. CUDA driver allocated memory was 276824064 and is now 278921216. 2025-12-04T11:13:50.3360136Z 2025-12-04T11:13:50.3360270Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.3360790Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_8_in_features_128_out_features_65_cuda_bfloat16 2025-12-04T11:13:50.3360794Z 2025-12-04T11:13:50.3360987Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.3361116Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.3361214Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.3361758Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('benchmarking.InductorBenchmarker.benchmark_gpu', 2), ('pad_mm_bench', 1), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.3361923Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.3361984Z graph_break [] 2025-12-04T11:13:50.3362110Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.3362202Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.3362324Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.3362862Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('benchmarking.InductorBenchmarker.benchmark_gpu', 2), ('pad_mm_bench', 1), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.3362925Z graph_break [] 2025-12-04T11:13:50.3363050Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.3363146Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.3363271Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.3363806Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('benchmarking.InductorBenchmarker.benchmark_gpu', 2), ('pad_mm_bench', 1), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.3363869Z graph_break [] 2025-12-04T11:13:50.3364358Z - generated xml file: /var/lib/jenkins/workspace/test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-2104559418af7772.xml - 2025-12-04T11:13:50.3364466Z =========================== short test summary info ============================ 2025-12-04T11:13:50.3365737Z FAILED [0.5361s] inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_1_mid_dim_8_in_features_128_out_features_65_cuda_bfloat16 - RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_8_in_features_128_out_features_65_cuda_bfloat16! Caching allocator allocated memory was 17408 and is now reported as 26112 on device 0. CUDA driver allocated memory was 276824064 and is now 278921216. 2025-12-04T11:13:50.3365744Z 2025-12-04T11:13:50.3365874Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.3366388Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_8_in_features_128_out_features_65_cuda_bfloat16 2025-12-04T11:13:50.3366392Z 2025-12-04T11:13:50.3366548Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.3366655Z !!!!!!!!!!!!!!!!!!!!!!!!!! stopping after 1 failures !!!!!!!!!!!!!!!!!!!!!!!!!!! 2025-12-04T11:13:50.3366847Z ================== 1 failed, 36 deselected, 2 rerun in 3.08s =================== 2025-12-04T11:13:50.3366916Z Got exit code 1 2025-12-04T11:13:50.3366986Z Retrying single test... 2025-12-04T11:13:50.3367248Z W1204 10:56:23.140000 77023 site-packages/torch/_inductor/utils.py:1703] Not enough SMs to use max_autotune_gemm mode 2025-12-04T11:13:50.3367634Z Test results will be stored in test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-49eccf27aa184bc6.xml 2025-12-04T11:13:50.3367730Z ============================= test session starts ============================== 2025-12-04T11:13:50.3367941Z platform linux -- Python 3.10.14, pytest-7.3.2, pluggy-1.6.0 -- /opt/conda/envs/py_3.10/bin/python 2025-12-04T11:13:50.3368055Z cachedir: .pytest_cache 2025-12-04T11:13:50.3368363Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow] 2025-12-04T11:13:50.3368447Z rootdir: /var/lib/jenkins/workspace 2025-12-04T11:13:50.3368513Z configfile: pytest.ini 2025-12-04T11:13:50.3368830Z plugins: hypothesis-6.56.4, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-14.0, subtests-0.13.1, xdist-3.3.1, xdoctest-1.3.0, typeguard-4.3.0 2025-12-04T11:13:50.3368996Z collecting ... collected 58 items / 57 deselected / 1 selected 2025-12-04T11:13:50.3369570Z stepcurrent: skipping 36 already run items. Running only test/inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_1_mid_dim_8_in_features_128_out_features_65_cuda_bfloat16 2025-12-04T11:13:50.3369645Z Running 1 items in this shard 2025-12-04T11:13:50.3369649Z 2025-12-04T11:13:50.3370374Z inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_1_mid_dim_8_in_features_128_out_features_65_cuda_bfloat16 [W1204 10:56:24.782271024 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3370379Z 2025-12-04T11:13:50.3370682Z [W1204 10:56:34.059380531 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3370687Z 2025-12-04T11:13:50.3370980Z [W1204 10:56:34.059613066 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3370984Z 2025-12-04T11:13:50.3371272Z [W1204 10:56:34.065972396 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3371280Z 2025-12-04T11:13:50.3371569Z [W1204 10:56:34.066535159 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3371573Z 2025-12-04T11:13:50.3371861Z [W1204 10:56:34.066697763 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3371866Z 2025-12-04T11:13:50.3372159Z [W1204 10:56:34.072113292 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3372164Z 2025-12-04T11:13:50.3372454Z [W1204 10:56:34.072674184 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3372457Z 2025-12-04T11:13:50.3372746Z [W1204 10:56:34.072844538 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3372750Z 2025-12-04T11:13:50.3372833Z ('RERUN', {'yellow': True}) [11.2620s] [100%] 2025-12-04T11:13:50.3373567Z inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_1_mid_dim_8_in_features_128_out_features_65_cuda_bfloat16 [W1204 10:56:34.888222149 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3373627Z 2025-12-04T11:13:50.3373961Z [W1204 10:56:34.888791252 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3373966Z 2025-12-04T11:13:50.3374256Z [W1204 10:56:34.888928735 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3374259Z 2025-12-04T11:13:50.3374546Z [W1204 10:56:34.891914291 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3374549Z 2025-12-04T11:13:50.3374834Z [W1204 10:56:34.892377081 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3374877Z 2025-12-04T11:13:50.3375163Z [W1204 10:56:34.892515124 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3375167Z 2025-12-04T11:13:50.3375454Z [W1204 10:56:34.897107125 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3375457Z 2025-12-04T11:13:50.3375780Z [W1204 10:56:34.897570405 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3375784Z 2025-12-04T11:13:50.3376071Z [W1204 10:56:34.897704428 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3376074Z 2025-12-04T11:13:50.3376159Z ('RERUN', {'yellow': True}) [0.5104s] [100%] 2025-12-04T11:13:50.3376876Z inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_1_mid_dim_8_in_features_128_out_features_65_cuda_bfloat16 [W1204 10:56:35.394251149 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3376881Z 2025-12-04T11:13:50.3377176Z [W1204 10:56:35.394804611 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3377179Z 2025-12-04T11:13:50.3377483Z [W1204 10:56:35.394940414 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3377486Z 2025-12-04T11:13:50.3377776Z [W1204 10:56:35.397912170 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3377782Z 2025-12-04T11:13:50.3378221Z [W1204 10:56:35.398371110 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3378224Z 2025-12-04T11:13:50.3378514Z [W1204 10:56:35.398504613 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3378519Z 2025-12-04T11:13:50.3378813Z [W1204 10:56:35.403113124 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3378816Z 2025-12-04T11:13:50.3379102Z [W1204 10:56:35.403581665 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3379107Z 2025-12-04T11:13:50.3379401Z [W1204 10:56:35.403714678 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3379405Z 2025-12-04T11:13:50.3379467Z FAILED [0.5031s] [100%] 2025-12-04T11:13:50.3379471Z 2025-12-04T11:13:50.3379559Z ==================================== RERUNS ==================================== 2025-12-04T11:13:50.3379852Z _ TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_8_in_features_128_out_features_65_cuda_bfloat16 _ 2025-12-04T11:13:50.3379931Z Traceback (most recent call last): 2025-12-04T11:13:50.3380315Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.3380434Z method(*args, **kwargs) 2025-12-04T11:13:50.3380729Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.3380801Z method(*args, **kwargs) 2025-12-04T11:13:50.3381090Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T11:13:50.3381157Z with policy(): 2025-12-04T11:13:50.3381450Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T11:13:50.3381515Z raise RuntimeError(msg) 2025-12-04T11:13:50.3382359Z RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_8_in_features_128_out_features_65_cuda_bfloat16! Caching allocator allocated memory was 0 and is now reported as 8704 on device 0. CUDA driver allocated memory was 230686720 and is now 274726912. 2025-12-04T11:13:50.3382367Z 2025-12-04T11:13:50.3382497Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.3383142Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_8_in_features_128_out_features_65_cuda_bfloat16 2025-12-04T11:13:50.3383146Z 2025-12-04T11:13:50.3383306Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.3383439Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.3383603Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.3384158Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('benchmarking.InductorBenchmarker.benchmark_gpu', 2), ('pad_mm_bench', 1), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.3384297Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.3384358Z graph_break [] 2025-12-04T11:13:50.3384486Z ----------------------------- Captured stderr call ----------------------------- 2025-12-04T11:13:50.3385182Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/constant_folding.py:256: UserWarning: Unsupported unwinding pattern: Address not in range (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/profiler/unwind/unwind.cpp:219.) 2025-12-04T11:13:50.3385255Z if out == self.unknown_value: 2025-12-04T11:13:50.3385549Z _ TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_8_in_features_128_out_features_65_cuda_bfloat16 _ 2025-12-04T11:13:50.3385627Z Traceback (most recent call last): 2025-12-04T11:13:50.3385925Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.3385996Z method(*args, **kwargs) 2025-12-04T11:13:50.3386288Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.3386358Z method(*args, **kwargs) 2025-12-04T11:13:50.3386649Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T11:13:50.3386710Z with policy(): 2025-12-04T11:13:50.3387008Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T11:13:50.3387075Z raise RuntimeError(msg) 2025-12-04T11:13:50.3387888Z RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_8_in_features_128_out_features_65_cuda_bfloat16! Caching allocator allocated memory was 8704 and is now reported as 17408 on device 0. CUDA driver allocated memory was 274726912 and is now 276824064. 2025-12-04T11:13:50.3387937Z 2025-12-04T11:13:50.3388102Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.3388617Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_8_in_features_128_out_features_65_cuda_bfloat16 2025-12-04T11:13:50.3388622Z 2025-12-04T11:13:50.3388785Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.3388917Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.3389016Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.3389596Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('benchmarking.InductorBenchmarker.benchmark_gpu', 2), ('pad_mm_bench', 1), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.3389731Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.3389797Z graph_break [] 2025-12-04T11:13:50.3389932Z ----------------------------- Captured stderr call ----------------------------- 2025-12-04T11:13:50.3390659Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/constant_folding.py:256: UserWarning: Unsupported unwinding pattern: Address not in range (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/profiler/unwind/unwind.cpp:219.) 2025-12-04T11:13:50.3390732Z if out == self.unknown_value: 2025-12-04T11:13:50.3390861Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.3390970Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.3391104Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.3391650Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('benchmarking.InductorBenchmarker.benchmark_gpu', 2), ('pad_mm_bench', 1), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.3391713Z graph_break [] 2025-12-04T11:13:50.3391801Z =================================== FAILURES =================================== 2025-12-04T11:13:50.3392095Z _ TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_8_in_features_128_out_features_65_cuda_bfloat16 _ 2025-12-04T11:13:50.3392173Z Traceback (most recent call last): 2025-12-04T11:13:50.3392473Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.3392544Z method(*args, **kwargs) 2025-12-04T11:13:50.3392838Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.3392907Z method(*args, **kwargs) 2025-12-04T11:13:50.3393196Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T11:13:50.3393257Z with policy(): 2025-12-04T11:13:50.3393562Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T11:13:50.3393630Z raise RuntimeError(msg) 2025-12-04T11:13:50.3394434Z RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_8_in_features_128_out_features_65_cuda_bfloat16! Caching allocator allocated memory was 17408 and is now reported as 26112 on device 0. CUDA driver allocated memory was 276824064 and is now 278921216. 2025-12-04T11:13:50.3394438Z 2025-12-04T11:13:50.3394569Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.3395085Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_8_in_features_128_out_features_65_cuda_bfloat16 2025-12-04T11:13:50.3395134Z 2025-12-04T11:13:50.3395297Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.3395459Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.3395561Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.3396102Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('benchmarking.InductorBenchmarker.benchmark_gpu', 2), ('pad_mm_bench', 1), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.3396232Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.3396298Z graph_break [] 2025-12-04T11:13:50.3396462Z ----------------------------- Captured stderr call ----------------------------- 2025-12-04T11:13:50.3397162Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/constant_folding.py:256: UserWarning: Unsupported unwinding pattern: Address not in range (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/profiler/unwind/unwind.cpp:219.) 2025-12-04T11:13:50.3397238Z if out == self.unknown_value: 2025-12-04T11:13:50.3397363Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.3397514Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.3397636Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.3398181Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('benchmarking.InductorBenchmarker.benchmark_gpu', 2), ('pad_mm_bench', 1), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.3398243Z graph_break [] 2025-12-04T11:13:50.3398371Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.3398468Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.3398591Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.3399130Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('benchmarking.InductorBenchmarker.benchmark_gpu', 2), ('pad_mm_bench', 1), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.3399194Z graph_break [] 2025-12-04T11:13:50.3399694Z - generated xml file: /var/lib/jenkins/workspace/test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-49eccf27aa184bc6.xml - 2025-12-04T11:13:50.3399797Z =========================== short test summary info ============================ 2025-12-04T11:13:50.3401073Z FAILED [0.5031s] inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_1_mid_dim_8_in_features_128_out_features_65_cuda_bfloat16 - RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_8_in_features_128_out_features_65_cuda_bfloat16! Caching allocator allocated memory was 17408 and is now reported as 26112 on device 0. CUDA driver allocated memory was 276824064 and is now 278921216. 2025-12-04T11:13:50.3401079Z 2025-12-04T11:13:50.3401212Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.3401728Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_8_in_features_128_out_features_65_cuda_bfloat16 2025-12-04T11:13:50.3401732Z 2025-12-04T11:13:50.3401894Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.3402000Z !!!!!!!!!!!!!!!!!!!!!!!!!! stopping after 1 failures !!!!!!!!!!!!!!!!!!!!!!!!!!! 2025-12-04T11:13:50.3402118Z ================== 1 failed, 57 deselected, 2 rerun in 12.30s ================== 2025-12-04T11:13:50.3402185Z Got exit code 1 2025-12-04T11:13:50.3402251Z Retrying single test... 2025-12-04T11:13:50.3402572Z W1204 10:56:42.200000 77210 site-packages/torch/_inductor/utils.py:1703] Not enough SMs to use max_autotune_gemm mode 2025-12-04T11:13:50.3402995Z Test results will be stored in test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-71736827b1650ca5.xml 2025-12-04T11:13:50.3406616Z ============================= test session starts ============================== 2025-12-04T11:13:50.3406879Z platform linux -- Python 3.10.14, pytest-7.3.2, pluggy-1.6.0 -- /opt/conda/envs/py_3.10/bin/python 2025-12-04T11:13:50.3406952Z cachedir: .pytest_cache 2025-12-04T11:13:50.3407279Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow] 2025-12-04T11:13:50.3407361Z rootdir: /var/lib/jenkins/workspace 2025-12-04T11:13:50.3407507Z configfile: pytest.ini 2025-12-04T11:13:50.3407838Z plugins: hypothesis-6.56.4, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-14.0, subtests-0.13.1, xdist-3.3.1, xdoctest-1.3.0, typeguard-4.3.0 2025-12-04T11:13:50.3407978Z collecting ... collected 58 items / 57 deselected / 1 selected 2025-12-04T11:13:50.3408557Z stepcurrent: skipping 36 already run items. Running only test/inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_1_mid_dim_8_in_features_128_out_features_65_cuda_bfloat16 2025-12-04T11:13:50.3408670Z Running 1 items in this shard 2025-12-04T11:13:50.3408675Z 2025-12-04T11:13:50.3409417Z inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_1_mid_dim_8_in_features_128_out_features_65_cuda_bfloat16 [W1204 10:56:43.832115229 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3409423Z 2025-12-04T11:13:50.3409731Z [W1204 10:56:52.599280001 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3409737Z 2025-12-04T11:13:50.3410028Z [W1204 10:56:52.599521336 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3410033Z 2025-12-04T11:13:50.3410315Z [W1204 10:56:52.605468088 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3410321Z 2025-12-04T11:13:50.3410602Z [W1204 10:56:52.606038130 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3410606Z 2025-12-04T11:13:50.3410889Z [W1204 10:56:52.606208214 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3410892Z 2025-12-04T11:13:50.3411178Z [W1204 10:56:52.611594273 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3411181Z 2025-12-04T11:13:50.3411476Z [W1204 10:56:52.612149655 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3411480Z 2025-12-04T11:13:50.3411766Z [W1204 10:56:52.612318249 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3411771Z 2025-12-04T11:13:50.3411857Z ('RERUN', {'yellow': True}) [10.7434s] [100%] 2025-12-04T11:13:50.3412580Z inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_1_mid_dim_8_in_features_128_out_features_65_cuda_bfloat16 [W1204 10:56:53.423948795 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3412584Z 2025-12-04T11:13:50.3412884Z [W1204 10:56:53.424502707 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3412887Z 2025-12-04T11:13:50.3413223Z [W1204 10:56:53.424653791 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3413260Z 2025-12-04T11:13:50.3413547Z [W1204 10:56:53.427590745 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3413556Z 2025-12-04T11:13:50.3413850Z [W1204 10:56:53.428050155 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3413855Z 2025-12-04T11:13:50.3414142Z [W1204 10:56:53.428191128 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3414145Z 2025-12-04T11:13:50.3414469Z [W1204 10:56:53.432806350 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3414473Z 2025-12-04T11:13:50.3414761Z [W1204 10:56:53.433273101 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3414766Z 2025-12-04T11:13:50.3415069Z [W1204 10:56:53.433408933 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3415107Z 2025-12-04T11:13:50.3415191Z ('RERUN', {'yellow': True}) [0.5066s] [100%] 2025-12-04T11:13:50.3415931Z inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_1_mid_dim_8_in_features_128_out_features_65_cuda_bfloat16 [W1204 10:56:53.927621304 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3415935Z 2025-12-04T11:13:50.3416238Z [W1204 10:56:53.928174846 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3416241Z 2025-12-04T11:13:50.3416534Z [W1204 10:56:53.928315279 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3416540Z 2025-12-04T11:13:50.3416827Z [W1204 10:56:53.931316536 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3416832Z 2025-12-04T11:13:50.3417119Z [W1204 10:56:53.931783006 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3417122Z 2025-12-04T11:13:50.3417411Z [W1204 10:56:53.931920289 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3417414Z 2025-12-04T11:13:50.3417701Z [W1204 10:56:53.936404297 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3417704Z 2025-12-04T11:13:50.3417990Z [W1204 10:56:53.936879098 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3417995Z 2025-12-04T11:13:50.3418282Z [W1204 10:56:53.937014221 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3418287Z 2025-12-04T11:13:50.3418354Z FAILED [0.5056s] [100%] 2025-12-04T11:13:50.3418358Z 2025-12-04T11:13:50.3418448Z ==================================== RERUNS ==================================== 2025-12-04T11:13:50.3418746Z _ TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_8_in_features_128_out_features_65_cuda_bfloat16 _ 2025-12-04T11:13:50.3418827Z Traceback (most recent call last): 2025-12-04T11:13:50.3419152Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.3419222Z method(*args, **kwargs) 2025-12-04T11:13:50.3419513Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.3419627Z method(*args, **kwargs) 2025-12-04T11:13:50.3419950Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T11:13:50.3420015Z with policy(): 2025-12-04T11:13:50.3420309Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T11:13:50.3420378Z raise RuntimeError(msg) 2025-12-04T11:13:50.3421208Z RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_8_in_features_128_out_features_65_cuda_bfloat16! Caching allocator allocated memory was 0 and is now reported as 8704 on device 0. CUDA driver allocated memory was 230686720 and is now 274726912. 2025-12-04T11:13:50.3421213Z 2025-12-04T11:13:50.3421348Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.3421875Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_8_in_features_128_out_features_65_cuda_bfloat16 2025-12-04T11:13:50.3421911Z 2025-12-04T11:13:50.3422074Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.3422211Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.3422309Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.3422856Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('benchmarking.InductorBenchmarker.benchmark_gpu', 2), ('pad_mm_bench', 1), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.3422989Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.3423052Z graph_break [] 2025-12-04T11:13:50.3423179Z ----------------------------- Captured stderr call ----------------------------- 2025-12-04T11:13:50.3423954Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/constant_folding.py:256: UserWarning: Unsupported unwinding pattern: Address not in range (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/profiler/unwind/unwind.cpp:219.) 2025-12-04T11:13:50.3424032Z if out == self.unknown_value: 2025-12-04T11:13:50.3424326Z _ TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_8_in_features_128_out_features_65_cuda_bfloat16 _ 2025-12-04T11:13:50.3424405Z Traceback (most recent call last): 2025-12-04T11:13:50.3424704Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.3424771Z method(*args, **kwargs) 2025-12-04T11:13:50.3425066Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.3425132Z method(*args, **kwargs) 2025-12-04T11:13:50.3425421Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T11:13:50.3425485Z with policy(): 2025-12-04T11:13:50.3425779Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T11:13:50.3425847Z raise RuntimeError(msg) 2025-12-04T11:13:50.3426647Z RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_8_in_features_128_out_features_65_cuda_bfloat16! Caching allocator allocated memory was 8704 and is now reported as 17408 on device 0. CUDA driver allocated memory was 274726912 and is now 276824064. 2025-12-04T11:13:50.3426652Z 2025-12-04T11:13:50.3426785Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.3427305Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_8_in_features_128_out_features_65_cuda_bfloat16 2025-12-04T11:13:50.3427352Z 2025-12-04T11:13:50.3427548Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.3427681Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.3427775Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.3428319Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('benchmarking.InductorBenchmarker.benchmark_gpu', 2), ('pad_mm_bench', 1), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.3428447Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.3428565Z graph_break [] 2025-12-04T11:13:50.3428695Z ----------------------------- Captured stderr call ----------------------------- 2025-12-04T11:13:50.3429388Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/constant_folding.py:256: UserWarning: Unsupported unwinding pattern: Address not in range (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/profiler/unwind/unwind.cpp:219.) 2025-12-04T11:13:50.3429498Z if out == self.unknown_value: 2025-12-04T11:13:50.3429629Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.3429721Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.3429849Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.3430396Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('benchmarking.InductorBenchmarker.benchmark_gpu', 2), ('pad_mm_bench', 1), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.3430456Z graph_break [] 2025-12-04T11:13:50.3430546Z =================================== FAILURES =================================== 2025-12-04T11:13:50.3430842Z _ TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_8_in_features_128_out_features_65_cuda_bfloat16 _ 2025-12-04T11:13:50.3430927Z Traceback (most recent call last): 2025-12-04T11:13:50.3431242Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.3431308Z method(*args, **kwargs) 2025-12-04T11:13:50.3431599Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.3431663Z method(*args, **kwargs) 2025-12-04T11:13:50.3431948Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T11:13:50.3432012Z with policy(): 2025-12-04T11:13:50.3432303Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T11:13:50.3432373Z raise RuntimeError(msg) 2025-12-04T11:13:50.3433184Z RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_8_in_features_128_out_features_65_cuda_bfloat16! Caching allocator allocated memory was 17408 and is now reported as 26112 on device 0. CUDA driver allocated memory was 276824064 and is now 278921216. 2025-12-04T11:13:50.3433190Z 2025-12-04T11:13:50.3433320Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.3433840Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_8_in_features_128_out_features_65_cuda_bfloat16 2025-12-04T11:13:50.3433843Z 2025-12-04T11:13:50.3434004Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.3434133Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.3434269Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.3434844Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('benchmarking.InductorBenchmarker.benchmark_gpu', 2), ('pad_mm_bench', 1), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.3434973Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.3435032Z graph_break [] 2025-12-04T11:13:50.3435156Z ----------------------------- Captured stderr call ----------------------------- 2025-12-04T11:13:50.3435889Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/constant_folding.py:256: UserWarning: Unsupported unwinding pattern: Address not in range (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/profiler/unwind/unwind.cpp:219.) 2025-12-04T11:13:50.3435966Z if out == self.unknown_value: 2025-12-04T11:13:50.3436095Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.3436189Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.3436321Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.3436866Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('benchmarking.InductorBenchmarker.benchmark_gpu', 2), ('pad_mm_bench', 1), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.3436961Z graph_break [] 2025-12-04T11:13:50.3437092Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.3437184Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.3437308Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.3437857Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('benchmarking.InductorBenchmarker.benchmark_gpu', 2), ('pad_mm_bench', 1), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.3437918Z graph_break [] 2025-12-04T11:13:50.3438416Z - generated xml file: /var/lib/jenkins/workspace/test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-71736827b1650ca5.xml - 2025-12-04T11:13:50.3438520Z =========================== short test summary info ============================ 2025-12-04T11:13:50.3439817Z FAILED [0.5056s] inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_1_mid_dim_8_in_features_128_out_features_65_cuda_bfloat16 - RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_8_in_features_128_out_features_65_cuda_bfloat16! Caching allocator allocated memory was 17408 and is now reported as 26112 on device 0. CUDA driver allocated memory was 276824064 and is now 278921216. 2025-12-04T11:13:50.3439821Z 2025-12-04T11:13:50.3439949Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.3440470Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_8_in_features_128_out_features_65_cuda_bfloat16 2025-12-04T11:13:50.3440476Z 2025-12-04T11:13:50.3440633Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.3440741Z !!!!!!!!!!!!!!!!!!!!!!!!!! stopping after 1 failures !!!!!!!!!!!!!!!!!!!!!!!!!!! 2025-12-04T11:13:50.3440861Z ================== 1 failed, 57 deselected, 2 rerun in 11.78s ================== 2025-12-04T11:13:50.3440922Z Got exit code 1 2025-12-04T11:13:50.3441397Z FAILED CONSISTENTLY: test/inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_1_mid_dim_8_in_features_128_out_features_65_cuda_bfloat16 2025-12-04T11:13:50.3441638Z Test failed consistently, continuing with the rest of the tests due to continue-through-error being set 2025-12-04T11:13:50.3441954Z W1204 10:57:00.815000 77397 site-packages/torch/_inductor/utils.py:1703] Not enough SMs to use max_autotune_gemm mode 2025-12-04T11:13:50.3442381Z Test results will be stored in test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-b5c5c102e82be0c3.xml 2025-12-04T11:13:50.3442484Z ============================= test session starts ============================== 2025-12-04T11:13:50.3442693Z platform linux -- Python 3.10.14, pytest-7.3.2, pluggy-1.6.0 -- /opt/conda/envs/py_3.10/bin/python 2025-12-04T11:13:50.3442760Z cachedir: .pytest_cache 2025-12-04T11:13:50.3443068Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow] 2025-12-04T11:13:50.3443150Z rootdir: /var/lib/jenkins/workspace 2025-12-04T11:13:50.3443253Z configfile: pytest.ini 2025-12-04T11:13:50.3443565Z plugins: hypothesis-6.56.4, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-14.0, subtests-0.13.1, xdist-3.3.1, xdoctest-1.3.0, typeguard-4.3.0 2025-12-04T11:13:50.3443700Z collecting ... collected 58 items / 37 deselected / 21 selected 2025-12-04T11:13:50.3443790Z stepcurrent: skipping 37 already run items. 2025-12-04T11:13:50.3443862Z Running 21 items in this shard 2025-12-04T11:13:50.3443902Z 2025-12-04T11:13:50.3444405Z inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_1_mid_dim_8_in_features_144_out_features_1024_cuda_bfloat16 ('RERUN', {'yellow': True}) [1.9165s] [ 4%] 2025-12-04T11:13:50.3444898Z inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_1_mid_dim_8_in_features_144_out_features_1024_cuda_bfloat16 ('RERUN', {'yellow': True}) [0.4908s] [ 4%] 2025-12-04T11:13:50.3445340Z inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_1_mid_dim_8_in_features_144_out_features_1024_cuda_bfloat16 FAILED [0.4788s] [ 4%] 2025-12-04T11:13:50.3445344Z 2025-12-04T11:13:50.3445430Z ==================================== RERUNS ==================================== 2025-12-04T11:13:50.3445724Z _ TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_8_in_features_144_out_features_1024_cuda_bfloat16 _ 2025-12-04T11:13:50.3445803Z Traceback (most recent call last): 2025-12-04T11:13:50.3446112Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.3446181Z method(*args, **kwargs) 2025-12-04T11:13:50.3446483Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.3446551Z method(*args, **kwargs) 2025-12-04T11:13:50.3446842Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T11:13:50.3446903Z with policy(): 2025-12-04T11:13:50.3447202Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T11:13:50.3447269Z raise RuntimeError(msg) 2025-12-04T11:13:50.3448089Z RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_8_in_features_144_out_features_1024_cuda_bfloat16! Caching allocator allocated memory was 0 and is now reported as 147456 on device 0. CUDA driver allocated memory was 230686720 and is now 274726912. 2025-12-04T11:13:50.3448095Z 2025-12-04T11:13:50.3448228Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.3448755Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_8_in_features_144_out_features_1024_cuda_bfloat16 2025-12-04T11:13:50.3448762Z 2025-12-04T11:13:50.3448922Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.3449055Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.3449198Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.3449581Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.3449715Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.3449779Z graph_break [] 2025-12-04T11:13:50.3450071Z _ TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_8_in_features_144_out_features_1024_cuda_bfloat16 _ 2025-12-04T11:13:50.3450148Z Traceback (most recent call last): 2025-12-04T11:13:50.3450480Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.3450547Z method(*args, **kwargs) 2025-12-04T11:13:50.3450845Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.3450910Z method(*args, **kwargs) 2025-12-04T11:13:50.3451209Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T11:13:50.3451306Z with policy(): 2025-12-04T11:13:50.3451602Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T11:13:50.3451671Z raise RuntimeError(msg) 2025-12-04T11:13:50.3452489Z RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_8_in_features_144_out_features_1024_cuda_bfloat16! Caching allocator allocated memory was 147456 and is now reported as 294912 on device 0. CUDA driver allocated memory was 274726912 and is now 276824064. 2025-12-04T11:13:50.3452493Z 2025-12-04T11:13:50.3452621Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.3453146Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_8_in_features_144_out_features_1024_cuda_bfloat16 2025-12-04T11:13:50.3453151Z 2025-12-04T11:13:50.3453310Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.3453445Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.3453540Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.3453891Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.3454019Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.3454079Z graph_break [] 2025-12-04T11:13:50.3454216Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.3454311Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.3454433Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.3454781Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.3454843Z graph_break [] 2025-12-04T11:13:50.3454933Z =================================== FAILURES =================================== 2025-12-04T11:13:50.3455223Z _ TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_8_in_features_144_out_features_1024_cuda_bfloat16 _ 2025-12-04T11:13:50.3455298Z Traceback (most recent call last): 2025-12-04T11:13:50.3455601Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.3455665Z method(*args, **kwargs) 2025-12-04T11:13:50.3455955Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.3456076Z method(*args, **kwargs) 2025-12-04T11:13:50.3456415Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T11:13:50.3456480Z with policy(): 2025-12-04T11:13:50.3456769Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T11:13:50.3456836Z raise RuntimeError(msg) 2025-12-04T11:13:50.3457689Z RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_8_in_features_144_out_features_1024_cuda_bfloat16! Caching allocator allocated memory was 294912 and is now reported as 442368 on device 0. CUDA driver allocated memory was 276824064 and is now 278921216. 2025-12-04T11:13:50.3457693Z 2025-12-04T11:13:50.3457818Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.3458349Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_8_in_features_144_out_features_1024_cuda_bfloat16 2025-12-04T11:13:50.3458388Z 2025-12-04T11:13:50.3458546Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.3458672Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.3458765Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.3459105Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.3459238Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.3459298Z graph_break [] 2025-12-04T11:13:50.3459421Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.3459514Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.3459645Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.3459991Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.3460051Z graph_break [] 2025-12-04T11:13:50.3460175Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.3460275Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.3460402Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.3460741Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.3460803Z graph_break [] 2025-12-04T11:13:50.3461289Z - generated xml file: /var/lib/jenkins/workspace/test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-b5c5c102e82be0c3.xml - 2025-12-04T11:13:50.3461395Z =========================== short test summary info ============================ 2025-12-04T11:13:50.3462680Z FAILED [0.4788s] inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_1_mid_dim_8_in_features_144_out_features_1024_cuda_bfloat16 - RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_8_in_features_144_out_features_1024_cuda_bfloat16! Caching allocator allocated memory was 294912 and is now reported as 442368 on device 0. CUDA driver allocated memory was 276824064 and is now 278921216. 2025-12-04T11:13:50.3462685Z 2025-12-04T11:13:50.3462813Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.3463327Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_8_in_features_144_out_features_1024_cuda_bfloat16 2025-12-04T11:13:50.3463371Z 2025-12-04T11:13:50.3463605Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.3463715Z !!!!!!!!!!!!!!!!!!!!!!!!!! stopping after 1 failures !!!!!!!!!!!!!!!!!!!!!!!!!!! 2025-12-04T11:13:50.3463843Z ================== 1 failed, 37 deselected, 2 rerun in 2.91s =================== 2025-12-04T11:13:50.3463907Z Got exit code 1 2025-12-04T11:13:50.3463974Z Retrying single test... 2025-12-04T11:13:50.3464354Z W1204 10:57:10.884000 77585 site-packages/torch/_inductor/utils.py:1703] Not enough SMs to use max_autotune_gemm mode 2025-12-04T11:13:50.3465026Z Test results will be stored in test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-6b2f788f9a7d50ec.xml 2025-12-04T11:13:50.3465159Z ============================= test session starts ============================== 2025-12-04T11:13:50.3465382Z platform linux -- Python 3.10.14, pytest-7.3.2, pluggy-1.6.0 -- /opt/conda/envs/py_3.10/bin/python 2025-12-04T11:13:50.3465451Z cachedir: .pytest_cache 2025-12-04T11:13:50.3465871Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow] 2025-12-04T11:13:50.3466032Z rootdir: /var/lib/jenkins/workspace 2025-12-04T11:13:50.3466103Z configfile: pytest.ini 2025-12-04T11:13:50.3466425Z plugins: hypothesis-6.56.4, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-14.0, subtests-0.13.1, xdist-3.3.1, xdoctest-1.3.0, typeguard-4.3.0 2025-12-04T11:13:50.3466558Z collecting ... collected 58 items / 57 deselected / 1 selected 2025-12-04T11:13:50.3467138Z stepcurrent: skipping 37 already run items. Running only test/inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_1_mid_dim_8_in_features_144_out_features_1024_cuda_bfloat16 2025-12-04T11:13:50.3467211Z Running 1 items in this shard 2025-12-04T11:13:50.3467217Z 2025-12-04T11:13:50.3467962Z inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_1_mid_dim_8_in_features_144_out_features_1024_cuda_bfloat16 [W1204 10:57:12.012600890 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3467967Z 2025-12-04T11:13:50.3468272Z [W1204 10:57:21.153525947 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3468275Z 2025-12-04T11:13:50.3468565Z [W1204 10:57:21.153763262 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3468569Z 2025-12-04T11:13:50.3468856Z [W1204 10:57:21.159459208 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3468861Z 2025-12-04T11:13:50.3469146Z [W1204 10:57:21.159997290 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3469151Z 2025-12-04T11:13:50.3469439Z [W1204 10:57:21.160197284 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3469444Z 2025-12-04T11:13:50.3469741Z [W1204 10:57:21.165497862 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3469744Z 2025-12-04T11:13:50.3470028Z [W1204 10:57:21.166015653 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3470031Z 2025-12-04T11:13:50.3470324Z [W1204 10:57:21.166170827 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3470328Z 2025-12-04T11:13:50.3470412Z ('RERUN', {'yellow': True}) [11.0558s] [100%] 2025-12-04T11:13:50.3471228Z inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_1_mid_dim_8_in_features_144_out_features_1024_cuda_bfloat16 [W1204 10:57:22.385076732 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3471235Z 2025-12-04T11:13:50.3471525Z [W1204 10:57:22.385623634 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3471528Z 2025-12-04T11:13:50.3471816Z [W1204 10:57:22.385768787 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3471819Z 2025-12-04T11:13:50.3472137Z [W1204 10:57:22.388819135 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3472141Z 2025-12-04T11:13:50.3472425Z [W1204 10:57:22.389394268 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3472430Z 2025-12-04T11:13:50.3472716Z [W1204 10:57:22.389531911 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3472752Z 2025-12-04T11:13:50.3473039Z [W1204 10:57:22.394207524 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3473042Z 2025-12-04T11:13:50.3473330Z [W1204 10:57:22.394688505 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3473334Z 2025-12-04T11:13:50.3473622Z [W1204 10:57:22.394826338 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3473625Z 2025-12-04T11:13:50.3473707Z ('RERUN', {'yellow': True}) [0.4576s] [100%] 2025-12-04T11:13:50.3474432Z inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_1_mid_dim_8_in_features_144_out_features_1024_cuda_bfloat16 [W1204 10:57:22.839924543 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3474437Z 2025-12-04T11:13:50.3474726Z [W1204 10:57:22.840550688 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3474729Z 2025-12-04T11:13:50.3475015Z [W1204 10:57:22.840706521 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3475018Z 2025-12-04T11:13:50.3475306Z [W1204 10:57:22.843609335 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3475314Z 2025-12-04T11:13:50.3475597Z [W1204 10:57:22.844166508 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3475602Z 2025-12-04T11:13:50.3475889Z [W1204 10:57:22.844305300 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3475893Z 2025-12-04T11:13:50.3476183Z [W1204 10:57:22.848823560 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3476186Z 2025-12-04T11:13:50.3476473Z [W1204 10:57:22.849285731 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3476476Z 2025-12-04T11:13:50.3476764Z [W1204 10:57:22.849419084 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3476767Z 2025-12-04T11:13:50.3476830Z FAILED [0.4542s] [100%] 2025-12-04T11:13:50.3476833Z 2025-12-04T11:13:50.3476964Z ==================================== RERUNS ==================================== 2025-12-04T11:13:50.3477294Z _ TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_8_in_features_144_out_features_1024_cuda_bfloat16 _ 2025-12-04T11:13:50.3477374Z Traceback (most recent call last): 2025-12-04T11:13:50.3477685Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.3477762Z method(*args, **kwargs) 2025-12-04T11:13:50.3478249Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.3478321Z method(*args, **kwargs) 2025-12-04T11:13:50.3478691Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T11:13:50.3478756Z with policy(): 2025-12-04T11:13:50.3479048Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T11:13:50.3479117Z raise RuntimeError(msg) 2025-12-04T11:13:50.3479938Z RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_8_in_features_144_out_features_1024_cuda_bfloat16! Caching allocator allocated memory was 0 and is now reported as 147456 on device 0. CUDA driver allocated memory was 230686720 and is now 274726912. 2025-12-04T11:13:50.3479998Z 2025-12-04T11:13:50.3480138Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.3480667Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_8_in_features_144_out_features_1024_cuda_bfloat16 2025-12-04T11:13:50.3480673Z 2025-12-04T11:13:50.3480834Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.3480971Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.3481069Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.3481418Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.3481551Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.3481612Z graph_break [] 2025-12-04T11:13:50.3481742Z ----------------------------- Captured stderr call ----------------------------- 2025-12-04T11:13:50.3482452Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/constant_folding.py:256: UserWarning: Unsupported unwinding pattern: Address not in range (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/profiler/unwind/unwind.cpp:219.) 2025-12-04T11:13:50.3482524Z if out == self.unknown_value: 2025-12-04T11:13:50.3482821Z _ TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_8_in_features_144_out_features_1024_cuda_bfloat16 _ 2025-12-04T11:13:50.3482898Z Traceback (most recent call last): 2025-12-04T11:13:50.3483205Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.3483276Z method(*args, **kwargs) 2025-12-04T11:13:50.3483575Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.3483643Z method(*args, **kwargs) 2025-12-04T11:13:50.3483934Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T11:13:50.3483993Z with policy(): 2025-12-04T11:13:50.3484290Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T11:13:50.3484354Z raise RuntimeError(msg) 2025-12-04T11:13:50.3485252Z RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_8_in_features_144_out_features_1024_cuda_bfloat16! Caching allocator allocated memory was 147456 and is now reported as 294912 on device 0. CUDA driver allocated memory was 274726912 and is now 276824064. 2025-12-04T11:13:50.3485315Z 2025-12-04T11:13:50.3485448Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.3485969Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_8_in_features_144_out_features_1024_cuda_bfloat16 2025-12-04T11:13:50.3485973Z 2025-12-04T11:13:50.3486134Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.3486571Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.3486686Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.3487040Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.3487173Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.3487282Z graph_break [] 2025-12-04T11:13:50.3487413Z ----------------------------- Captured stderr call ----------------------------- 2025-12-04T11:13:50.3488108Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/constant_folding.py:256: UserWarning: Unsupported unwinding pattern: Address not in range (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/profiler/unwind/unwind.cpp:219.) 2025-12-04T11:13:50.3488182Z if out == self.unknown_value: 2025-12-04T11:13:50.3488311Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.3488409Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.3488536Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.3488886Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.3488956Z graph_break [] 2025-12-04T11:13:50.3489040Z =================================== FAILURES =================================== 2025-12-04T11:13:50.3489350Z _ TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_8_in_features_144_out_features_1024_cuda_bfloat16 _ 2025-12-04T11:13:50.3489427Z Traceback (most recent call last): 2025-12-04T11:13:50.3489727Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.3489797Z method(*args, **kwargs) 2025-12-04T11:13:50.3490089Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.3490152Z method(*args, **kwargs) 2025-12-04T11:13:50.3490444Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T11:13:50.3490506Z with policy(): 2025-12-04T11:13:50.3490807Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T11:13:50.3490875Z raise RuntimeError(msg) 2025-12-04T11:13:50.3491691Z RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_8_in_features_144_out_features_1024_cuda_bfloat16! Caching allocator allocated memory was 294912 and is now reported as 442368 on device 0. CUDA driver allocated memory was 276824064 and is now 278921216. 2025-12-04T11:13:50.3491698Z 2025-12-04T11:13:50.3491829Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.3492350Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_8_in_features_144_out_features_1024_cuda_bfloat16 2025-12-04T11:13:50.3492396Z 2025-12-04T11:13:50.3492596Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.3492729Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.3492824Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.3493168Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.3493298Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.3493359Z graph_break [] 2025-12-04T11:13:50.3493520Z ----------------------------- Captured stderr call ----------------------------- 2025-12-04T11:13:50.3494208Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/constant_folding.py:256: UserWarning: Unsupported unwinding pattern: Address not in range (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/profiler/unwind/unwind.cpp:219.) 2025-12-04T11:13:50.3494289Z if out == self.unknown_value: 2025-12-04T11:13:50.3494415Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.3494547Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.3494671Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.3495013Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.3495073Z graph_break [] 2025-12-04T11:13:50.3495199Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.3495293Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.3495414Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.3495755Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.3495816Z graph_break [] 2025-12-04T11:13:50.3496303Z - generated xml file: /var/lib/jenkins/workspace/test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-6b2f788f9a7d50ec.xml - 2025-12-04T11:13:50.3496416Z =========================== short test summary info ============================ 2025-12-04T11:13:50.3497723Z FAILED [0.4542s] inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_1_mid_dim_8_in_features_144_out_features_1024_cuda_bfloat16 - RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_8_in_features_144_out_features_1024_cuda_bfloat16! Caching allocator allocated memory was 294912 and is now reported as 442368 on device 0. CUDA driver allocated memory was 276824064 and is now 278921216. 2025-12-04T11:13:50.3497729Z 2025-12-04T11:13:50.3497856Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.3498381Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_8_in_features_144_out_features_1024_cuda_bfloat16 2025-12-04T11:13:50.3498387Z 2025-12-04T11:13:50.3498548Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.3498657Z !!!!!!!!!!!!!!!!!!!!!!!!!! stopping after 1 failures !!!!!!!!!!!!!!!!!!!!!!!!!!! 2025-12-04T11:13:50.3498772Z ================== 1 failed, 57 deselected, 2 rerun in 11.99s ================== 2025-12-04T11:13:50.3498833Z Got exit code 1 2025-12-04T11:13:50.3498905Z Retrying single test... 2025-12-04T11:13:50.3499166Z W1204 10:57:29.668000 77778 site-packages/torch/_inductor/utils.py:1703] Not enough SMs to use max_autotune_gemm mode 2025-12-04T11:13:50.3499637Z Test results will be stored in test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-61d7ccaaf8cfd370.xml 2025-12-04T11:13:50.3499737Z ============================= test session starts ============================== 2025-12-04T11:13:50.3499950Z platform linux -- Python 3.10.14, pytest-7.3.2, pluggy-1.6.0 -- /opt/conda/envs/py_3.10/bin/python 2025-12-04T11:13:50.3500018Z cachedir: .pytest_cache 2025-12-04T11:13:50.3500321Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow] 2025-12-04T11:13:50.3500398Z rootdir: /var/lib/jenkins/workspace 2025-12-04T11:13:50.3500466Z configfile: pytest.ini 2025-12-04T11:13:50.3500898Z plugins: hypothesis-6.56.4, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-14.0, subtests-0.13.1, xdist-3.3.1, xdoctest-1.3.0, typeguard-4.3.0 2025-12-04T11:13:50.3501033Z collecting ... collected 58 items / 57 deselected / 1 selected 2025-12-04T11:13:50.3501604Z stepcurrent: skipping 37 already run items. Running only test/inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_1_mid_dim_8_in_features_144_out_features_1024_cuda_bfloat16 2025-12-04T11:13:50.3501711Z Running 1 items in this shard 2025-12-04T11:13:50.3501715Z 2025-12-04T11:13:50.3502455Z inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_1_mid_dim_8_in_features_144_out_features_1024_cuda_bfloat16 [W1204 10:57:30.792285112 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3502460Z 2025-12-04T11:13:50.3502760Z [W1204 10:57:40.159389311 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3502764Z 2025-12-04T11:13:50.3503061Z [W1204 10:57:40.159641887 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3503066Z 2025-12-04T11:13:50.3503354Z [W1204 10:57:40.165926956 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3503359Z 2025-12-04T11:13:50.3503730Z [W1204 10:57:40.166542029 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3503734Z 2025-12-04T11:13:50.3504031Z [W1204 10:57:40.166714563 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3504035Z 2025-12-04T11:13:50.3504330Z [W1204 10:57:40.172178554 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3504334Z 2025-12-04T11:13:50.3504623Z [W1204 10:57:40.172713546 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3504628Z 2025-12-04T11:13:50.3504918Z [W1204 10:57:40.172875080 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3504926Z 2025-12-04T11:13:50.3505009Z ('RERUN', {'yellow': True}) [11.2931s] [100%] 2025-12-04T11:13:50.3505733Z inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_1_mid_dim_8_in_features_144_out_features_1024_cuda_bfloat16 [W1204 10:57:41.406721689 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3505736Z 2025-12-04T11:13:50.3506028Z [W1204 10:57:41.407287561 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3506031Z 2025-12-04T11:13:50.3506316Z [W1204 10:57:41.407430885 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3506363Z 2025-12-04T11:13:50.3506687Z [W1204 10:57:41.410453552 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3506693Z 2025-12-04T11:13:50.3506982Z [W1204 10:57:41.411038165 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3506986Z 2025-12-04T11:13:50.3507277Z [W1204 10:57:41.411175238 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3507281Z 2025-12-04T11:13:50.3507566Z [W1204 10:57:41.415852641 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3507603Z 2025-12-04T11:13:50.3507892Z [W1204 10:57:41.416331882 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3507897Z 2025-12-04T11:13:50.3508183Z [W1204 10:57:41.416467635 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3508186Z 2025-12-04T11:13:50.3508320Z ('RERUN', {'yellow': True}) [0.4647s] [100%] 2025-12-04T11:13:50.3509038Z inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_1_mid_dim_8_in_features_144_out_features_1024_cuda_bfloat16 [W1204 10:57:41.866741846 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3509042Z 2025-12-04T11:13:50.3509332Z [W1204 10:57:41.867285618 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3509337Z 2025-12-04T11:13:50.3509634Z [W1204 10:57:41.867428151 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3509639Z 2025-12-04T11:13:50.3509926Z [W1204 10:57:41.870548910 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3509929Z 2025-12-04T11:13:50.3510218Z [W1204 10:57:41.871128273 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3510221Z 2025-12-04T11:13:50.3510506Z [W1204 10:57:41.871265126 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3510510Z 2025-12-04T11:13:50.3510798Z [W1204 10:57:41.875903858 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3510801Z 2025-12-04T11:13:50.3511089Z [W1204 10:57:41.876375119 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3511094Z 2025-12-04T11:13:50.3511380Z [W1204 10:57:41.876509922 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3511387Z 2025-12-04T11:13:50.3511449Z FAILED [0.4594s] [100%] 2025-12-04T11:13:50.3511454Z 2025-12-04T11:13:50.3511552Z ==================================== RERUNS ==================================== 2025-12-04T11:13:50.3511852Z _ TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_8_in_features_144_out_features_1024_cuda_bfloat16 _ 2025-12-04T11:13:50.3511928Z Traceback (most recent call last): 2025-12-04T11:13:50.3512233Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.3512301Z method(*args, **kwargs) 2025-12-04T11:13:50.3512592Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.3512702Z method(*args, **kwargs) 2025-12-04T11:13:50.3513022Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T11:13:50.3513083Z with policy(): 2025-12-04T11:13:50.3513381Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T11:13:50.3513446Z raise RuntimeError(msg) 2025-12-04T11:13:50.3514256Z RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_8_in_features_144_out_features_1024_cuda_bfloat16! Caching allocator allocated memory was 0 and is now reported as 147456 on device 0. CUDA driver allocated memory was 230686720 and is now 274726912. 2025-12-04T11:13:50.3514260Z 2025-12-04T11:13:50.3514426Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.3514948Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_8_in_features_144_out_features_1024_cuda_bfloat16 2025-12-04T11:13:50.3514957Z 2025-12-04T11:13:50.3515120Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.3515289Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.3515392Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.3515739Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.3515871Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.3515948Z graph_break [] 2025-12-04T11:13:50.3516079Z ----------------------------- Captured stderr call ----------------------------- 2025-12-04T11:13:50.3516780Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/constant_folding.py:256: UserWarning: Unsupported unwinding pattern: Address not in range (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/profiler/unwind/unwind.cpp:219.) 2025-12-04T11:13:50.3516852Z if out == self.unknown_value: 2025-12-04T11:13:50.3517147Z _ TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_8_in_features_144_out_features_1024_cuda_bfloat16 _ 2025-12-04T11:13:50.3517226Z Traceback (most recent call last): 2025-12-04T11:13:50.3517523Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.3517590Z method(*args, **kwargs) 2025-12-04T11:13:50.3517880Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.3517945Z method(*args, **kwargs) 2025-12-04T11:13:50.3518235Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T11:13:50.3518298Z with policy(): 2025-12-04T11:13:50.3518590Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T11:13:50.3518661Z raise RuntimeError(msg) 2025-12-04T11:13:50.3519479Z RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_8_in_features_144_out_features_1024_cuda_bfloat16! Caching allocator allocated memory was 147456 and is now reported as 294912 on device 0. CUDA driver allocated memory was 274726912 and is now 276824064. 2025-12-04T11:13:50.3519483Z 2025-12-04T11:13:50.3519616Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.3520137Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_8_in_features_144_out_features_1024_cuda_bfloat16 2025-12-04T11:13:50.3520190Z 2025-12-04T11:13:50.3520357Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.3520518Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.3520615Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.3520967Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.3521095Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.3521154Z graph_break [] 2025-12-04T11:13:50.3521281Z ----------------------------- Captured stderr call ----------------------------- 2025-12-04T11:13:50.3522003Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/constant_folding.py:256: UserWarning: Unsupported unwinding pattern: Address not in range (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/profiler/unwind/unwind.cpp:219.) 2025-12-04T11:13:50.3522079Z if out == self.unknown_value: 2025-12-04T11:13:50.3522204Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.3522296Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.3522457Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.3522798Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.3522864Z graph_break [] 2025-12-04T11:13:50.3522953Z =================================== FAILURES =================================== 2025-12-04T11:13:50.3523248Z _ TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_8_in_features_144_out_features_1024_cuda_bfloat16 _ 2025-12-04T11:13:50.3523327Z Traceback (most recent call last): 2025-12-04T11:13:50.3523623Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.3523691Z method(*args, **kwargs) 2025-12-04T11:13:50.3523984Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.3524050Z method(*args, **kwargs) 2025-12-04T11:13:50.3524342Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T11:13:50.3524402Z with policy(): 2025-12-04T11:13:50.3524692Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T11:13:50.3524762Z raise RuntimeError(msg) 2025-12-04T11:13:50.3525582Z RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_8_in_features_144_out_features_1024_cuda_bfloat16! Caching allocator allocated memory was 294912 and is now reported as 442368 on device 0. CUDA driver allocated memory was 276824064 and is now 278921216. 2025-12-04T11:13:50.3525588Z 2025-12-04T11:13:50.3525718Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.3526232Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_8_in_features_144_out_features_1024_cuda_bfloat16 2025-12-04T11:13:50.3526237Z 2025-12-04T11:13:50.3526398Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.3526522Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.3526613Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.3526966Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.3527089Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.3527194Z graph_break [] 2025-12-04T11:13:50.3527356Z ----------------------------- Captured stderr call ----------------------------- 2025-12-04T11:13:50.3528040Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/constant_folding.py:256: UserWarning: Unsupported unwinding pattern: Address not in range (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/profiler/unwind/unwind.cpp:219.) 2025-12-04T11:13:50.3528113Z if out == self.unknown_value: 2025-12-04T11:13:50.3528236Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.3528326Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.3528461Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.3528839Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.3528906Z graph_break [] 2025-12-04T11:13:50.3529026Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.3529117Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.3529276Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.3529622Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.3529679Z graph_break [] 2025-12-04T11:13:50.3530168Z - generated xml file: /var/lib/jenkins/workspace/test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-61d7ccaaf8cfd370.xml - 2025-12-04T11:13:50.3530269Z =========================== short test summary info ============================ 2025-12-04T11:13:50.3531567Z FAILED [0.4594s] inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_1_mid_dim_8_in_features_144_out_features_1024_cuda_bfloat16 - RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_8_in_features_144_out_features_1024_cuda_bfloat16! Caching allocator allocated memory was 294912 and is now reported as 442368 on device 0. CUDA driver allocated memory was 276824064 and is now 278921216. 2025-12-04T11:13:50.3531574Z 2025-12-04T11:13:50.3531700Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.3532224Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_8_in_features_144_out_features_1024_cuda_bfloat16 2025-12-04T11:13:50.3532228Z 2025-12-04T11:13:50.3532385Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.3532490Z !!!!!!!!!!!!!!!!!!!!!!!!!! stopping after 1 failures !!!!!!!!!!!!!!!!!!!!!!!!!!! 2025-12-04T11:13:50.3532608Z ================== 1 failed, 57 deselected, 2 rerun in 12.24s ================== 2025-12-04T11:13:50.3532666Z Got exit code 1 2025-12-04T11:13:50.3533143Z FAILED CONSISTENTLY: test/inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_1_mid_dim_8_in_features_144_out_features_1024_cuda_bfloat16 2025-12-04T11:13:50.3533386Z Test failed consistently, continuing with the rest of the tests due to continue-through-error being set 2025-12-04T11:13:50.3533646Z W1204 10:57:48.745000 77971 site-packages/torch/_inductor/utils.py:1703] Not enough SMs to use max_autotune_gemm mode 2025-12-04T11:13:50.3534033Z Test results will be stored in test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-cc96100da976ce4e.xml 2025-12-04T11:13:50.3534130Z ============================= test session starts ============================== 2025-12-04T11:13:50.3534342Z platform linux -- Python 3.10.14, pytest-7.3.2, pluggy-1.6.0 -- /opt/conda/envs/py_3.10/bin/python 2025-12-04T11:13:50.3534451Z cachedir: .pytest_cache 2025-12-04T11:13:50.3534790Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow] 2025-12-04T11:13:50.3534873Z rootdir: /var/lib/jenkins/workspace 2025-12-04T11:13:50.3534937Z configfile: pytest.ini 2025-12-04T11:13:50.3535250Z plugins: hypothesis-6.56.4, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-14.0, subtests-0.13.1, xdist-3.3.1, xdoctest-1.3.0, typeguard-4.3.0 2025-12-04T11:13:50.3535384Z collecting ... collected 58 items / 38 deselected / 20 selected 2025-12-04T11:13:50.3535472Z stepcurrent: skipping 38 already run items. 2025-12-04T11:13:50.3535544Z Running 20 items in this shard 2025-12-04T11:13:50.3535547Z 2025-12-04T11:13:50.3536088Z inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_1_mid_dim_8_in_features_144_out_features_64_cuda_bfloat16 ('RERUN', {'yellow': True}) [1.8601s] [ 5%] 2025-12-04T11:13:50.3536577Z inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_1_mid_dim_8_in_features_144_out_features_64_cuda_bfloat16 ('RERUN', {'yellow': True}) [0.4592s] [ 5%] 2025-12-04T11:13:50.3537054Z inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_1_mid_dim_8_in_features_144_out_features_64_cuda_bfloat16 FAILED [0.4467s] [ 5%] 2025-12-04T11:13:50.3537058Z 2025-12-04T11:13:50.3537139Z ==================================== RERUNS ==================================== 2025-12-04T11:13:50.3537430Z _ TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_8_in_features_144_out_features_64_cuda_bfloat16 _ 2025-12-04T11:13:50.3537505Z Traceback (most recent call last): 2025-12-04T11:13:50.3537814Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.3537880Z method(*args, **kwargs) 2025-12-04T11:13:50.3538174Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.3538244Z method(*args, **kwargs) 2025-12-04T11:13:50.3538533Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T11:13:50.3538595Z with policy(): 2025-12-04T11:13:50.3538891Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T11:13:50.3538956Z raise RuntimeError(msg) 2025-12-04T11:13:50.3539755Z RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_8_in_features_144_out_features_64_cuda_bfloat16! Caching allocator allocated memory was 0 and is now reported as 9216 on device 0. CUDA driver allocated memory was 230686720 and is now 274726912. 2025-12-04T11:13:50.3539760Z 2025-12-04T11:13:50.3539888Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.3540397Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_8_in_features_144_out_features_64_cuda_bfloat16 2025-12-04T11:13:50.3540406Z 2025-12-04T11:13:50.3540564Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.3540690Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.3540786Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.3541136Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.3541264Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.3541326Z graph_break [] 2025-12-04T11:13:50.3541614Z _ TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_8_in_features_144_out_features_64_cuda_bfloat16 _ 2025-12-04T11:13:50.3541745Z Traceback (most recent call last): 2025-12-04T11:13:50.3542078Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.3542146Z method(*args, **kwargs) 2025-12-04T11:13:50.3542440Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.3542507Z method(*args, **kwargs) 2025-12-04T11:13:50.3542795Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T11:13:50.3542857Z with policy(): 2025-12-04T11:13:50.3543183Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T11:13:50.3543257Z raise RuntimeError(msg) 2025-12-04T11:13:50.3544104Z RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_8_in_features_144_out_features_64_cuda_bfloat16! Caching allocator allocated memory was 9216 and is now reported as 18432 on device 0. CUDA driver allocated memory was 274726912 and is now 276824064. 2025-12-04T11:13:50.3544144Z 2025-12-04T11:13:50.3544274Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.3544786Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_8_in_features_144_out_features_64_cuda_bfloat16 2025-12-04T11:13:50.3544790Z 2025-12-04T11:13:50.3544945Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.3545074Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.3545167Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.3545518Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.3545643Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.3545703Z graph_break [] 2025-12-04T11:13:50.3545830Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.3545936Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.3546058Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.3546402Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.3546462Z graph_break [] 2025-12-04T11:13:50.3546549Z =================================== FAILURES =================================== 2025-12-04T11:13:50.3546835Z _ TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_8_in_features_144_out_features_64_cuda_bfloat16 _ 2025-12-04T11:13:50.3546912Z Traceback (most recent call last): 2025-12-04T11:13:50.3547214Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.3547281Z method(*args, **kwargs) 2025-12-04T11:13:50.3547569Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.3547634Z method(*args, **kwargs) 2025-12-04T11:13:50.3547922Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T11:13:50.3547985Z with policy(): 2025-12-04T11:13:50.3548277Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T11:13:50.3548343Z raise RuntimeError(msg) 2025-12-04T11:13:50.3549231Z RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_8_in_features_144_out_features_64_cuda_bfloat16! Caching allocator allocated memory was 18432 and is now reported as 27648 on device 0. CUDA driver allocated memory was 276824064 and is now 278921216. 2025-12-04T11:13:50.3549236Z 2025-12-04T11:13:50.3549362Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.3549885Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_8_in_features_144_out_features_64_cuda_bfloat16 2025-12-04T11:13:50.3549888Z 2025-12-04T11:13:50.3550043Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.3550203Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.3550298Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.3550643Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.3550768Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.3550865Z graph_break [] 2025-12-04T11:13:50.3550991Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.3551083Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.3551202Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.3551556Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.3551616Z graph_break [] 2025-12-04T11:13:50.3551740Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.3551833Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.3551955Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.3552297Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.3552358Z graph_break [] 2025-12-04T11:13:50.3552839Z - generated xml file: /var/lib/jenkins/workspace/test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-cc96100da976ce4e.xml - 2025-12-04T11:13:50.3552941Z =========================== short test summary info ============================ 2025-12-04T11:13:50.3554219Z FAILED [0.4467s] inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_1_mid_dim_8_in_features_144_out_features_64_cuda_bfloat16 - RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_8_in_features_144_out_features_64_cuda_bfloat16! Caching allocator allocated memory was 18432 and is now reported as 27648 on device 0. CUDA driver allocated memory was 276824064 and is now 278921216. 2025-12-04T11:13:50.3554226Z 2025-12-04T11:13:50.3554353Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.3554862Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_8_in_features_144_out_features_64_cuda_bfloat16 2025-12-04T11:13:50.3554865Z 2025-12-04T11:13:50.3555025Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.3555129Z !!!!!!!!!!!!!!!!!!!!!!!!!! stopping after 1 failures !!!!!!!!!!!!!!!!!!!!!!!!!!! 2025-12-04T11:13:50.3555247Z ================== 1 failed, 38 deselected, 2 rerun in 2.79s =================== 2025-12-04T11:13:50.3555308Z Got exit code 1 2025-12-04T11:13:50.3555373Z Retrying single test... 2025-12-04T11:13:50.3555677Z W1204 10:57:58.801000 78152 site-packages/torch/_inductor/utils.py:1703] Not enough SMs to use max_autotune_gemm mode 2025-12-04T11:13:50.3556103Z Test results will be stored in test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-e2ca8845caeab461.xml 2025-12-04T11:13:50.3556202Z ============================= test session starts ============================== 2025-12-04T11:13:50.3556413Z platform linux -- Python 3.10.14, pytest-7.3.2, pluggy-1.6.0 -- /opt/conda/envs/py_3.10/bin/python 2025-12-04T11:13:50.3556481Z cachedir: .pytest_cache 2025-12-04T11:13:50.3556786Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow] 2025-12-04T11:13:50.3556898Z rootdir: /var/lib/jenkins/workspace 2025-12-04T11:13:50.3556967Z configfile: pytest.ini 2025-12-04T11:13:50.3557280Z plugins: hypothesis-6.56.4, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-14.0, subtests-0.13.1, xdist-3.3.1, xdoctest-1.3.0, typeguard-4.3.0 2025-12-04T11:13:50.3557414Z collecting ... collected 58 items / 57 deselected / 1 selected 2025-12-04T11:13:50.3557985Z stepcurrent: skipping 38 already run items. Running only test/inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_1_mid_dim_8_in_features_144_out_features_64_cuda_bfloat16 2025-12-04T11:13:50.3558093Z Running 1 items in this shard 2025-12-04T11:13:50.3558097Z 2025-12-04T11:13:50.3558821Z inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_1_mid_dim_8_in_features_144_out_features_64_cuda_bfloat16 [W1204 10:57:59.900679504 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3558826Z 2025-12-04T11:13:50.3559127Z [W1204 10:58:09.018230554 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3559132Z 2025-12-04T11:13:50.3559419Z [W1204 10:58:09.018471389 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3559423Z 2025-12-04T11:13:50.3559729Z [W1204 10:58:09.024134675 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3559737Z 2025-12-04T11:13:50.3560025Z [W1204 10:58:09.024685087 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3560028Z 2025-12-04T11:13:50.3560314Z [W1204 10:58:09.024849610 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3560317Z 2025-12-04T11:13:50.3560607Z [W1204 10:58:09.030146178 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3560611Z 2025-12-04T11:13:50.3560900Z [W1204 10:58:09.030669159 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3560905Z 2025-12-04T11:13:50.3561194Z [W1204 10:58:09.030829983 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3561199Z 2025-12-04T11:13:50.3561280Z ('RERUN', {'yellow': True}) [11.0112s] [100%] 2025-12-04T11:13:50.3562001Z inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_1_mid_dim_8_in_features_144_out_features_64_cuda_bfloat16 [W1204 10:58:10.217855917 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3562005Z 2025-12-04T11:13:50.3562297Z [W1204 10:58:10.218436940 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3562301Z 2025-12-04T11:13:50.3562638Z [W1204 10:58:10.218578103 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3562692Z 2025-12-04T11:13:50.3562981Z [W1204 10:58:10.221611180 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3562987Z 2025-12-04T11:13:50.3563271Z [W1204 10:58:10.222198073 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3563277Z 2025-12-04T11:13:50.3563561Z [W1204 10:58:10.222338146 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3563564Z 2025-12-04T11:13:50.3563884Z [W1204 10:58:10.226963358 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3563888Z 2025-12-04T11:13:50.3564180Z [W1204 10:58:10.227439489 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3564184Z 2025-12-04T11:13:50.3564471Z [W1204 10:58:10.227579642 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3564509Z 2025-12-04T11:13:50.3564594Z ('RERUN', {'yellow': True}) [0.4246s] [100%] 2025-12-04T11:13:50.3565307Z inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_1_mid_dim_8_in_features_144_out_features_64_cuda_bfloat16 [W1204 10:58:10.642534277 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3565311Z 2025-12-04T11:13:50.3565604Z [W1204 10:58:10.643118580 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3565607Z 2025-12-04T11:13:50.3565894Z [W1204 10:58:10.643257103 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3565899Z 2025-12-04T11:13:50.3566184Z [W1204 10:58:10.646232809 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3566192Z 2025-12-04T11:13:50.3566476Z [W1204 10:58:10.646804142 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3566480Z 2025-12-04T11:13:50.3566763Z [W1204 10:58:10.646941715 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3566766Z 2025-12-04T11:13:50.3567057Z [W1204 10:58:10.651564007 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3567060Z 2025-12-04T11:13:50.3567345Z [W1204 10:58:10.652040158 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3567350Z 2025-12-04T11:13:50.3567641Z [W1204 10:58:10.652174691 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3567646Z 2025-12-04T11:13:50.3567707Z FAILED [0.4204s] [100%] 2025-12-04T11:13:50.3567711Z 2025-12-04T11:13:50.3567798Z ==================================== RERUNS ==================================== 2025-12-04T11:13:50.3568087Z _ TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_8_in_features_144_out_features_64_cuda_bfloat16 _ 2025-12-04T11:13:50.3568162Z Traceback (most recent call last): 2025-12-04T11:13:50.3568474Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.3568540Z method(*args, **kwargs) 2025-12-04T11:13:50.3568833Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.3568941Z method(*args, **kwargs) 2025-12-04T11:13:50.3569262Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T11:13:50.3569329Z with policy(): 2025-12-04T11:13:50.3569630Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T11:13:50.3569696Z raise RuntimeError(msg) 2025-12-04T11:13:50.3570523Z RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_8_in_features_144_out_features_64_cuda_bfloat16! Caching allocator allocated memory was 0 and is now reported as 9216 on device 0. CUDA driver allocated memory was 230686720 and is now 274726912. 2025-12-04T11:13:50.3570528Z 2025-12-04T11:13:50.3570656Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.3571175Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_8_in_features_144_out_features_64_cuda_bfloat16 2025-12-04T11:13:50.3571212Z 2025-12-04T11:13:50.3571369Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.3571502Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.3571599Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.3571956Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.3572090Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.3572150Z graph_break [] 2025-12-04T11:13:50.3572276Z ----------------------------- Captured stderr call ----------------------------- 2025-12-04T11:13:50.3572972Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/constant_folding.py:256: UserWarning: Unsupported unwinding pattern: Address not in range (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/profiler/unwind/unwind.cpp:219.) 2025-12-04T11:13:50.3573044Z if out == self.unknown_value: 2025-12-04T11:13:50.3573332Z _ TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_8_in_features_144_out_features_64_cuda_bfloat16 _ 2025-12-04T11:13:50.3573408Z Traceback (most recent call last): 2025-12-04T11:13:50.3573705Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.3573772Z method(*args, **kwargs) 2025-12-04T11:13:50.3574064Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.3574129Z method(*args, **kwargs) 2025-12-04T11:13:50.3574418Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T11:13:50.3574479Z with policy(): 2025-12-04T11:13:50.3574772Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T11:13:50.3574840Z raise RuntimeError(msg) 2025-12-04T11:13:50.3575644Z RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_8_in_features_144_out_features_64_cuda_bfloat16! Caching allocator allocated memory was 9216 and is now reported as 18432 on device 0. CUDA driver allocated memory was 274726912 and is now 276824064. 2025-12-04T11:13:50.3575650Z 2025-12-04T11:13:50.3575777Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.3576293Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_8_in_features_144_out_features_64_cuda_bfloat16 2025-12-04T11:13:50.3576344Z 2025-12-04T11:13:50.3576542Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.3576669Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.3576766Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.3577110Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.3577234Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.3577297Z graph_break [] 2025-12-04T11:13:50.3577481Z ----------------------------- Captured stderr call ----------------------------- 2025-12-04T11:13:50.3578323Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/constant_folding.py:256: UserWarning: Unsupported unwinding pattern: Address not in range (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/profiler/unwind/unwind.cpp:219.) 2025-12-04T11:13:50.3578399Z if out == self.unknown_value: 2025-12-04T11:13:50.3578523Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.3578688Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.3578813Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.3579159Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.3579224Z graph_break [] 2025-12-04T11:13:50.3579322Z =================================== FAILURES =================================== 2025-12-04T11:13:50.3579627Z _ TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_8_in_features_144_out_features_64_cuda_bfloat16 _ 2025-12-04T11:13:50.3579702Z Traceback (most recent call last): 2025-12-04T11:13:50.3580012Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.3580082Z method(*args, **kwargs) 2025-12-04T11:13:50.3580376Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.3580442Z method(*args, **kwargs) 2025-12-04T11:13:50.3580731Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T11:13:50.3580791Z with policy(): 2025-12-04T11:13:50.3581084Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T11:13:50.3581151Z raise RuntimeError(msg) 2025-12-04T11:13:50.3581962Z RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_8_in_features_144_out_features_64_cuda_bfloat16! Caching allocator allocated memory was 18432 and is now reported as 27648 on device 0. CUDA driver allocated memory was 276824064 and is now 278921216. 2025-12-04T11:13:50.3581972Z 2025-12-04T11:13:50.3582101Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.3582619Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_8_in_features_144_out_features_64_cuda_bfloat16 2025-12-04T11:13:50.3582623Z 2025-12-04T11:13:50.3582787Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.3582916Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.3583014Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.3583362Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.3583605Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.3583727Z graph_break [] 2025-12-04T11:13:50.3583857Z ----------------------------- Captured stderr call ----------------------------- 2025-12-04T11:13:50.3584544Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/constant_folding.py:256: UserWarning: Unsupported unwinding pattern: Address not in range (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/profiler/unwind/unwind.cpp:219.) 2025-12-04T11:13:50.3584616Z if out == self.unknown_value: 2025-12-04T11:13:50.3584738Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.3584833Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.3585003Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.3585349Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.3585413Z graph_break [] 2025-12-04T11:13:50.3585538Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.3585679Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.3585806Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.3586145Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.3586206Z graph_break [] 2025-12-04T11:13:50.3586695Z - generated xml file: /var/lib/jenkins/workspace/test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-e2ca8845caeab461.xml - 2025-12-04T11:13:50.3586797Z =========================== short test summary info ============================ 2025-12-04T11:13:50.3588077Z FAILED [0.4204s] inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_1_mid_dim_8_in_features_144_out_features_64_cuda_bfloat16 - RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_8_in_features_144_out_features_64_cuda_bfloat16! Caching allocator allocated memory was 18432 and is now reported as 27648 on device 0. CUDA driver allocated memory was 276824064 and is now 278921216. 2025-12-04T11:13:50.3588084Z 2025-12-04T11:13:50.3588208Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.3588728Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_8_in_features_144_out_features_64_cuda_bfloat16 2025-12-04T11:13:50.3588731Z 2025-12-04T11:13:50.3588886Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.3588997Z !!!!!!!!!!!!!!!!!!!!!!!!!! stopping after 1 failures !!!!!!!!!!!!!!!!!!!!!!!!!!! 2025-12-04T11:13:50.3589112Z ================== 1 failed, 57 deselected, 2 rerun in 11.88s ================== 2025-12-04T11:13:50.3589174Z Got exit code 1 2025-12-04T11:13:50.3589245Z Retrying single test... 2025-12-04T11:13:50.3589519Z W1204 10:58:17.518000 78338 site-packages/torch/_inductor/utils.py:1703] Not enough SMs to use max_autotune_gemm mode 2025-12-04T11:13:50.3589905Z Test results will be stored in test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-ea9e1f7f9950a9ad.xml 2025-12-04T11:13:50.3590002Z ============================= test session starts ============================== 2025-12-04T11:13:50.3590214Z platform linux -- Python 3.10.14, pytest-7.3.2, pluggy-1.6.0 -- /opt/conda/envs/py_3.10/bin/python 2025-12-04T11:13:50.3590288Z cachedir: .pytest_cache 2025-12-04T11:13:50.3590592Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow] 2025-12-04T11:13:50.3590734Z rootdir: /var/lib/jenkins/workspace 2025-12-04T11:13:50.3590798Z configfile: pytest.ini 2025-12-04T11:13:50.3591144Z plugins: hypothesis-6.56.4, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-14.0, subtests-0.13.1, xdist-3.3.1, xdoctest-1.3.0, typeguard-4.3.0 2025-12-04T11:13:50.3591279Z collecting ... collected 58 items / 57 deselected / 1 selected 2025-12-04T11:13:50.3591841Z stepcurrent: skipping 38 already run items. Running only test/inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_1_mid_dim_8_in_features_144_out_features_64_cuda_bfloat16 2025-12-04T11:13:50.3591911Z Running 1 items in this shard 2025-12-04T11:13:50.3591918Z 2025-12-04T11:13:50.3592676Z inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_1_mid_dim_8_in_features_144_out_features_64_cuda_bfloat16 [W1204 10:58:18.606365266 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3592682Z 2025-12-04T11:13:50.3592982Z [W1204 10:58:27.723013177 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3593019Z 2025-12-04T11:13:50.3593310Z [W1204 10:58:27.723240972 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3593313Z 2025-12-04T11:13:50.3593600Z [W1204 10:58:27.729410849 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3593603Z 2025-12-04T11:13:50.3593893Z [W1204 10:58:27.729957021 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3593897Z 2025-12-04T11:13:50.3594182Z [W1204 10:58:27.730175066 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3594187Z 2025-12-04T11:13:50.3594484Z [W1204 10:58:27.735466903 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3594490Z 2025-12-04T11:13:50.3594782Z [W1204 10:58:27.735986164 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3594786Z 2025-12-04T11:13:50.3595074Z [W1204 10:58:27.736142678 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3595077Z 2025-12-04T11:13:50.3595160Z ('RERUN', {'yellow': True}) [11.0006s] [100%] 2025-12-04T11:13:50.3595876Z inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_1_mid_dim_8_in_features_144_out_features_64_cuda_bfloat16 [W1204 10:58:28.920909938 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3595884Z 2025-12-04T11:13:50.3596171Z [W1204 10:58:28.921497200 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3596176Z 2025-12-04T11:13:50.3596458Z [W1204 10:58:28.921641844 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3596461Z 2025-12-04T11:13:50.3596753Z [W1204 10:58:28.924570748 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3596756Z 2025-12-04T11:13:50.3597040Z [W1204 10:58:28.925151481 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3597046Z 2025-12-04T11:13:50.3597336Z [W1204 10:58:28.925292804 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3597378Z 2025-12-04T11:13:50.3597699Z [W1204 10:58:28.929857605 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3597704Z 2025-12-04T11:13:50.3597994Z [W1204 10:58:28.930353456 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3597997Z 2025-12-04T11:13:50.3598282Z [W1204 10:58:28.930497960 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3598285Z 2025-12-04T11:13:50.3598368Z ('RERUN', {'yellow': True}) [0.4198s] [100%] 2025-12-04T11:13:50.3599112Z inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_1_mid_dim_8_in_features_144_out_features_64_cuda_bfloat16 [W1204 10:58:29.338095407 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3599118Z 2025-12-04T11:13:50.3599406Z [W1204 10:58:29.338691080 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3599446Z 2025-12-04T11:13:50.3599742Z [W1204 10:58:29.338830583 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3599746Z 2025-12-04T11:13:50.3600033Z [W1204 10:58:29.341790788 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3600037Z 2025-12-04T11:13:50.3600327Z [W1204 10:58:29.342367121 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3600332Z 2025-12-04T11:13:50.3600615Z [W1204 10:58:29.342504224 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3600620Z 2025-12-04T11:13:50.3600910Z [W1204 10:58:29.347053185 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3600913Z 2025-12-04T11:13:50.3601200Z [W1204 10:58:29.347527826 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3601203Z 2025-12-04T11:13:50.3601492Z [W1204 10:58:29.347664569 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3601495Z 2025-12-04T11:13:50.3601559Z FAILED [0.4152s] [100%] 2025-12-04T11:13:50.3601562Z 2025-12-04T11:13:50.3601647Z ==================================== RERUNS ==================================== 2025-12-04T11:13:50.3601936Z _ TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_8_in_features_144_out_features_64_cuda_bfloat16 _ 2025-12-04T11:13:50.3602011Z Traceback (most recent call last): 2025-12-04T11:13:50.3602322Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.3602388Z method(*args, **kwargs) 2025-12-04T11:13:50.3602684Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.3602749Z method(*args, **kwargs) 2025-12-04T11:13:50.3603036Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T11:13:50.3603096Z with policy(): 2025-12-04T11:13:50.3603391Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T11:13:50.3603456Z raise RuntimeError(msg) 2025-12-04T11:13:50.3604292Z RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_8_in_features_144_out_features_64_cuda_bfloat16! Caching allocator allocated memory was 0 and is now reported as 9216 on device 0. CUDA driver allocated memory was 230686720 and is now 274726912. 2025-12-04T11:13:50.3604328Z 2025-12-04T11:13:50.3604458Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.3604978Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_8_in_features_144_out_features_64_cuda_bfloat16 2025-12-04T11:13:50.3604982Z 2025-12-04T11:13:50.3605142Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.3605272Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.3605406Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.3605754Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.3605888Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.3605949Z graph_break [] 2025-12-04T11:13:50.3606074Z ----------------------------- Captured stderr call ----------------------------- 2025-12-04T11:13:50.3606800Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/constant_folding.py:256: UserWarning: Unsupported unwinding pattern: Address not in range (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/profiler/unwind/unwind.cpp:219.) 2025-12-04T11:13:50.3606869Z if out == self.unknown_value: 2025-12-04T11:13:50.3607154Z _ TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_8_in_features_144_out_features_64_cuda_bfloat16 _ 2025-12-04T11:13:50.3607235Z Traceback (most recent call last): 2025-12-04T11:13:50.3607532Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.3607602Z method(*args, **kwargs) 2025-12-04T11:13:50.3607892Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.3607955Z method(*args, **kwargs) 2025-12-04T11:13:50.3608249Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T11:13:50.3608308Z with policy(): 2025-12-04T11:13:50.3608603Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T11:13:50.3608669Z raise RuntimeError(msg) 2025-12-04T11:13:50.3609480Z RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_8_in_features_144_out_features_64_cuda_bfloat16! Caching allocator allocated memory was 9216 and is now reported as 18432 on device 0. CUDA driver allocated memory was 274726912 and is now 276824064. 2025-12-04T11:13:50.3609486Z 2025-12-04T11:13:50.3609616Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.3610132Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_8_in_features_144_out_features_64_cuda_bfloat16 2025-12-04T11:13:50.3610137Z 2025-12-04T11:13:50.3610295Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.3610420Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.3610512Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.3610862Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.3610991Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.3611059Z graph_break [] 2025-12-04T11:13:50.3611225Z ----------------------------- Captured stderr call ----------------------------- 2025-12-04T11:13:50.3612031Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/constant_folding.py:256: UserWarning: Unsupported unwinding pattern: Address not in range (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/profiler/unwind/unwind.cpp:219.) 2025-12-04T11:13:50.3612109Z if out == self.unknown_value: 2025-12-04T11:13:50.3612232Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.3612327Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.3612451Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.3612827Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.3612895Z graph_break [] 2025-12-04T11:13:50.3612980Z =================================== FAILURES =================================== 2025-12-04T11:13:50.3613268Z _ TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_8_in_features_144_out_features_64_cuda_bfloat16 _ 2025-12-04T11:13:50.3613398Z Traceback (most recent call last): 2025-12-04T11:13:50.3613696Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.3613764Z method(*args, **kwargs) 2025-12-04T11:13:50.3614057Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.3614124Z method(*args, **kwargs) 2025-12-04T11:13:50.3614416Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T11:13:50.3614477Z with policy(): 2025-12-04T11:13:50.3614767Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T11:13:50.3614836Z raise RuntimeError(msg) 2025-12-04T11:13:50.3615637Z RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_8_in_features_144_out_features_64_cuda_bfloat16! Caching allocator allocated memory was 18432 and is now reported as 27648 on device 0. CUDA driver allocated memory was 276824064 and is now 278921216. 2025-12-04T11:13:50.3615643Z 2025-12-04T11:13:50.3615770Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.3616280Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_8_in_features_144_out_features_64_cuda_bfloat16 2025-12-04T11:13:50.3616285Z 2025-12-04T11:13:50.3616441Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.3616568Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.3616660Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.3617006Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.3617131Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.3617191Z graph_break [] 2025-12-04T11:13:50.3617313Z ----------------------------- Captured stderr call ----------------------------- 2025-12-04T11:13:50.3617995Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/constant_folding.py:256: UserWarning: Unsupported unwinding pattern: Address not in range (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/profiler/unwind/unwind.cpp:219.) 2025-12-04T11:13:50.3618067Z if out == self.unknown_value: 2025-12-04T11:13:50.3618190Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.3618322Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.3618484Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.3618826Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.3618894Z graph_break [] 2025-12-04T11:13:50.3619016Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.3619106Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.3619230Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.3619610Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.3619675Z graph_break [] 2025-12-04T11:13:50.3620159Z - generated xml file: /var/lib/jenkins/workspace/test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-ea9e1f7f9950a9ad.xml - 2025-12-04T11:13:50.3620260Z =========================== short test summary info ============================ 2025-12-04T11:13:50.3621581Z FAILED [0.4152s] inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_1_mid_dim_8_in_features_144_out_features_64_cuda_bfloat16 - RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_8_in_features_144_out_features_64_cuda_bfloat16! Caching allocator allocated memory was 18432 and is now reported as 27648 on device 0. CUDA driver allocated memory was 276824064 and is now 278921216. 2025-12-04T11:13:50.3621585Z 2025-12-04T11:13:50.3621710Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.3622228Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_8_in_features_144_out_features_64_cuda_bfloat16 2025-12-04T11:13:50.3622233Z 2025-12-04T11:13:50.3622391Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.3622495Z !!!!!!!!!!!!!!!!!!!!!!!!!! stopping after 1 failures !!!!!!!!!!!!!!!!!!!!!!!!!!! 2025-12-04T11:13:50.3622609Z ================== 1 failed, 57 deselected, 2 rerun in 11.86s ================== 2025-12-04T11:13:50.3622669Z Got exit code 1 2025-12-04T11:13:50.3623139Z FAILED CONSISTENTLY: test/inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_1_mid_dim_8_in_features_144_out_features_64_cuda_bfloat16 2025-12-04T11:13:50.3623381Z Test failed consistently, continuing with the rest of the tests due to continue-through-error being set 2025-12-04T11:13:50.3623681Z W1204 10:58:36.143000 78524 site-packages/torch/_inductor/utils.py:1703] Not enough SMs to use max_autotune_gemm mode 2025-12-04T11:13:50.3624068Z Test results will be stored in test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-778cc31dcd227b51.xml 2025-12-04T11:13:50.3624164Z ============================= test session starts ============================== 2025-12-04T11:13:50.3624377Z platform linux -- Python 3.10.14, pytest-7.3.2, pluggy-1.6.0 -- /opt/conda/envs/py_3.10/bin/python 2025-12-04T11:13:50.3624446Z cachedir: .pytest_cache 2025-12-04T11:13:50.3624749Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow] 2025-12-04T11:13:50.3624830Z rootdir: /var/lib/jenkins/workspace 2025-12-04T11:13:50.3624900Z configfile: pytest.ini 2025-12-04T11:13:50.3625218Z plugins: hypothesis-6.56.4, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-14.0, subtests-0.13.1, xdist-3.3.1, xdoctest-1.3.0, typeguard-4.3.0 2025-12-04T11:13:50.3625345Z collecting ... collected 58 items / 39 deselected / 19 selected 2025-12-04T11:13:50.3625474Z stepcurrent: skipping 39 already run items. 2025-12-04T11:13:50.3625547Z Running 19 items in this shard 2025-12-04T11:13:50.3625550Z 2025-12-04T11:13:50.3626076Z inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_1_mid_dim_8_in_features_144_out_features_65_cuda_bfloat16 ('RERUN', {'yellow': True}) [1.9659s] [ 5%] 2025-12-04T11:13:50.3626560Z inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_1_mid_dim_8_in_features_144_out_features_65_cuda_bfloat16 ('RERUN', {'yellow': True}) [0.5470s] [ 5%] 2025-12-04T11:13:50.3627001Z inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_1_mid_dim_8_in_features_144_out_features_65_cuda_bfloat16 FAILED [0.5368s] [ 5%] 2025-12-04T11:13:50.3627037Z 2025-12-04T11:13:50.3627122Z ==================================== RERUNS ==================================== 2025-12-04T11:13:50.3627411Z _ TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_8_in_features_144_out_features_65_cuda_bfloat16 _ 2025-12-04T11:13:50.3627487Z Traceback (most recent call last): 2025-12-04T11:13:50.3627795Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.3627898Z method(*args, **kwargs) 2025-12-04T11:13:50.3628191Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.3628259Z method(*args, **kwargs) 2025-12-04T11:13:50.3628548Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T11:13:50.3628607Z with policy(): 2025-12-04T11:13:50.3628906Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T11:13:50.3628970Z raise RuntimeError(msg) 2025-12-04T11:13:50.3629774Z RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_8_in_features_144_out_features_65_cuda_bfloat16! Caching allocator allocated memory was 0 and is now reported as 9728 on device 0. CUDA driver allocated memory was 230686720 and is now 274726912. 2025-12-04T11:13:50.3629780Z 2025-12-04T11:13:50.3629905Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.3630422Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_8_in_features_144_out_features_65_cuda_bfloat16 2025-12-04T11:13:50.3630426Z 2025-12-04T11:13:50.3630584Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.3630714Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.3630812Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.3631361Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('benchmarking.InductorBenchmarker.benchmark_gpu', 2), ('pad_mm_bench', 1), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.3631494Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.3631553Z graph_break [] 2025-12-04T11:13:50.3631836Z _ TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_8_in_features_144_out_features_65_cuda_bfloat16 _ 2025-12-04T11:13:50.3631913Z Traceback (most recent call last): 2025-12-04T11:13:50.3632206Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.3632270Z method(*args, **kwargs) 2025-12-04T11:13:50.3632566Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.3632633Z method(*args, **kwargs) 2025-12-04T11:13:50.3633000Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T11:13:50.3633064Z with policy(): 2025-12-04T11:13:50.3633372Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T11:13:50.3633442Z raise RuntimeError(msg) 2025-12-04T11:13:50.3634246Z RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_8_in_features_144_out_features_65_cuda_bfloat16! Caching allocator allocated memory was 9728 and is now reported as 19456 on device 0. CUDA driver allocated memory was 274726912 and is now 276824064. 2025-12-04T11:13:50.3634250Z 2025-12-04T11:13:50.3634413Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.3634925Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_8_in_features_144_out_features_65_cuda_bfloat16 2025-12-04T11:13:50.3634930Z 2025-12-04T11:13:50.3635091Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.3635253Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.3635347Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.3635888Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('benchmarking.InductorBenchmarker.benchmark_gpu', 2), ('pad_mm_bench', 1), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.3636015Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.3636074Z graph_break [] 2025-12-04T11:13:50.3636202Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.3636289Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.3636411Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.3636953Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('benchmarking.InductorBenchmarker.benchmark_gpu', 2), ('pad_mm_bench', 1), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.3637012Z graph_break [] 2025-12-04T11:13:50.3637110Z =================================== FAILURES =================================== 2025-12-04T11:13:50.3637394Z _ TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_8_in_features_144_out_features_65_cuda_bfloat16 _ 2025-12-04T11:13:50.3637470Z Traceback (most recent call last): 2025-12-04T11:13:50.3637767Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.3637830Z method(*args, **kwargs) 2025-12-04T11:13:50.3638120Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.3638184Z method(*args, **kwargs) 2025-12-04T11:13:50.3638474Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T11:13:50.3638540Z with policy(): 2025-12-04T11:13:50.3638834Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T11:13:50.3638902Z raise RuntimeError(msg) 2025-12-04T11:13:50.3639703Z RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_8_in_features_144_out_features_65_cuda_bfloat16! Caching allocator allocated memory was 19456 and is now reported as 29184 on device 0. CUDA driver allocated memory was 276824064 and is now 278921216. 2025-12-04T11:13:50.3639707Z 2025-12-04T11:13:50.3639832Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.3640425Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_8_in_features_144_out_features_65_cuda_bfloat16 2025-12-04T11:13:50.3640431Z 2025-12-04T11:13:50.3640585Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.3640712Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.3640802Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.3641393Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('benchmarking.InductorBenchmarker.benchmark_gpu', 2), ('pad_mm_bench', 1), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.3641519Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.3641576Z graph_break [] 2025-12-04T11:13:50.3641705Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.3641796Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.3641916Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.3642492Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('benchmarking.InductorBenchmarker.benchmark_gpu', 2), ('pad_mm_bench', 1), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.3642549Z graph_break [] 2025-12-04T11:13:50.3642677Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.3642777Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.3642907Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.3643445Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('benchmarking.InductorBenchmarker.benchmark_gpu', 2), ('pad_mm_bench', 1), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.3643507Z graph_break [] 2025-12-04T11:13:50.3643998Z - generated xml file: /var/lib/jenkins/workspace/test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-778cc31dcd227b51.xml - 2025-12-04T11:13:50.3644098Z =========================== short test summary info ============================ 2025-12-04T11:13:50.3645367Z FAILED [0.5368s] inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_1_mid_dim_8_in_features_144_out_features_65_cuda_bfloat16 - RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_8_in_features_144_out_features_65_cuda_bfloat16! Caching allocator allocated memory was 19456 and is now reported as 29184 on device 0. CUDA driver allocated memory was 276824064 and is now 278921216. 2025-12-04T11:13:50.3645376Z 2025-12-04T11:13:50.3645500Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.3646012Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_8_in_features_144_out_features_65_cuda_bfloat16 2025-12-04T11:13:50.3646018Z 2025-12-04T11:13:50.3646177Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.3646281Z !!!!!!!!!!!!!!!!!!!!!!!!!! stopping after 1 failures !!!!!!!!!!!!!!!!!!!!!!!!!!! 2025-12-04T11:13:50.3646400Z ================== 1 failed, 39 deselected, 2 rerun in 3.08s =================== 2025-12-04T11:13:50.3646458Z Got exit code 1 2025-12-04T11:13:50.3646523Z Retrying single test... 2025-12-04T11:13:50.3646788Z W1204 10:58:46.209000 78706 site-packages/torch/_inductor/utils.py:1703] Not enough SMs to use max_autotune_gemm mode 2025-12-04T11:13:50.3647172Z Test results will be stored in test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-d8517590777959a7.xml 2025-12-04T11:13:50.3647343Z ============================= test session starts ============================== 2025-12-04T11:13:50.3647557Z platform linux -- Python 3.10.14, pytest-7.3.2, pluggy-1.6.0 -- /opt/conda/envs/py_3.10/bin/python 2025-12-04T11:13:50.3647623Z cachedir: .pytest_cache 2025-12-04T11:13:50.3647929Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow] 2025-12-04T11:13:50.3648004Z rootdir: /var/lib/jenkins/workspace 2025-12-04T11:13:50.3648070Z configfile: pytest.ini 2025-12-04T11:13:50.3648420Z plugins: hypothesis-6.56.4, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-14.0, subtests-0.13.1, xdist-3.3.1, xdoctest-1.3.0, typeguard-4.3.0 2025-12-04T11:13:50.3648548Z collecting ... collected 58 items / 57 deselected / 1 selected 2025-12-04T11:13:50.3649113Z stepcurrent: skipping 39 already run items. Running only test/inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_1_mid_dim_8_in_features_144_out_features_65_cuda_bfloat16 2025-12-04T11:13:50.3649186Z Running 1 items in this shard 2025-12-04T11:13:50.3649224Z 2025-12-04T11:13:50.3649949Z inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_1_mid_dim_8_in_features_144_out_features_65_cuda_bfloat16 [W1204 10:58:47.854806102 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3649957Z 2025-12-04T11:13:50.3650257Z [W1204 10:58:57.054393345 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3650261Z 2025-12-04T11:13:50.3650553Z [W1204 10:58:57.054630400 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3650558Z 2025-12-04T11:13:50.3650849Z [W1204 10:58:57.060507390 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3650852Z 2025-12-04T11:13:50.3651140Z [W1204 10:58:57.061095313 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3651143Z 2025-12-04T11:13:50.3651431Z [W1204 10:58:57.061264757 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3651435Z 2025-12-04T11:13:50.3651720Z [W1204 10:58:57.066693557 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3651723Z 2025-12-04T11:13:50.3652012Z [W1204 10:58:57.067254139 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3652015Z 2025-12-04T11:13:50.3652302Z [W1204 10:58:57.067427273 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3652307Z 2025-12-04T11:13:50.3652387Z ('RERUN', {'yellow': True}) [11.1886s] [100%] 2025-12-04T11:13:50.3653105Z inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_1_mid_dim_8_in_features_144_out_features_65_cuda_bfloat16 [W1204 10:58:57.880842527 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3653109Z 2025-12-04T11:13:50.3653397Z [W1204 10:58:57.881421230 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3653400Z 2025-12-04T11:13:50.3653700Z [W1204 10:58:57.881566633 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3653746Z 2025-12-04T11:13:50.3654035Z [W1204 10:58:57.884523358 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3654072Z 2025-12-04T11:13:50.3654360Z [W1204 10:58:57.885012109 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3654365Z 2025-12-04T11:13:50.3654648Z [W1204 10:58:57.885150702 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3654651Z 2025-12-04T11:13:50.3654941Z [W1204 10:58:57.889759084 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3654944Z 2025-12-04T11:13:50.3655262Z [W1204 10:58:57.890293276 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3655266Z 2025-12-04T11:13:50.3655559Z [W1204 10:58:57.890435049 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3655562Z 2025-12-04T11:13:50.3655642Z ('RERUN', {'yellow': True}) [0.5050s] [100%] 2025-12-04T11:13:50.3656390Z inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_1_mid_dim_8_in_features_144_out_features_65_cuda_bfloat16 [W1204 10:58:58.382246433 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3656398Z 2025-12-04T11:13:50.3656686Z [W1204 10:58:58.382806505 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3656689Z 2025-12-04T11:13:50.3656972Z [W1204 10:58:58.382951658 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3656975Z 2025-12-04T11:13:50.3657265Z [W1204 10:58:58.385921214 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3657270Z 2025-12-04T11:13:50.3657555Z [W1204 10:58:58.386393965 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3657560Z 2025-12-04T11:13:50.3657849Z [W1204 10:58:58.386533648 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3657852Z 2025-12-04T11:13:50.3658137Z [W1204 10:58:58.391216272 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3658140Z 2025-12-04T11:13:50.3658430Z [W1204 10:58:58.391690463 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3658433Z 2025-12-04T11:13:50.3658716Z [W1204 10:58:58.391828286 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3658720Z 2025-12-04T11:13:50.3658784Z FAILED [0.4995s] [100%] 2025-12-04T11:13:50.3658792Z 2025-12-04T11:13:50.3658878Z ==================================== RERUNS ==================================== 2025-12-04T11:13:50.3659171Z _ TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_8_in_features_144_out_features_65_cuda_bfloat16 _ 2025-12-04T11:13:50.3659252Z Traceback (most recent call last): 2025-12-04T11:13:50.3659557Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.3659624Z method(*args, **kwargs) 2025-12-04T11:13:50.3659921Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.3659986Z method(*args, **kwargs) 2025-12-04T11:13:50.3660278Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T11:13:50.3660380Z with policy(): 2025-12-04T11:13:50.3660705Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T11:13:50.3660778Z raise RuntimeError(msg) 2025-12-04T11:13:50.3661569Z RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_8_in_features_144_out_features_65_cuda_bfloat16! Caching allocator allocated memory was 0 and is now reported as 9728 on device 0. CUDA driver allocated memory was 230686720 and is now 274726912. 2025-12-04T11:13:50.3661574Z 2025-12-04T11:13:50.3661703Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.3662249Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_8_in_features_144_out_features_65_cuda_bfloat16 2025-12-04T11:13:50.3662255Z 2025-12-04T11:13:50.3662416Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.3662546Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.3662674Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.3663217Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('benchmarking.InductorBenchmarker.benchmark_gpu', 2), ('pad_mm_bench', 1), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.3663344Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.3663404Z graph_break [] 2025-12-04T11:13:50.3663567Z ----------------------------- Captured stderr call ----------------------------- 2025-12-04T11:13:50.3664257Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/constant_folding.py:256: UserWarning: Unsupported unwinding pattern: Address not in range (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/profiler/unwind/unwind.cpp:219.) 2025-12-04T11:13:50.3664334Z if out == self.unknown_value: 2025-12-04T11:13:50.3664620Z _ TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_8_in_features_144_out_features_65_cuda_bfloat16 _ 2025-12-04T11:13:50.3664694Z Traceback (most recent call last): 2025-12-04T11:13:50.3664994Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.3665058Z method(*args, **kwargs) 2025-12-04T11:13:50.3665351Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.3665416Z method(*args, **kwargs) 2025-12-04T11:13:50.3665703Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T11:13:50.3665767Z with policy(): 2025-12-04T11:13:50.3666063Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T11:13:50.3666129Z raise RuntimeError(msg) 2025-12-04T11:13:50.3666935Z RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_8_in_features_144_out_features_65_cuda_bfloat16! Caching allocator allocated memory was 9728 and is now reported as 19456 on device 0. CUDA driver allocated memory was 274726912 and is now 276824064. 2025-12-04T11:13:50.3666939Z 2025-12-04T11:13:50.3667078Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.3667596Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_8_in_features_144_out_features_65_cuda_bfloat16 2025-12-04T11:13:50.3667600Z 2025-12-04T11:13:50.3667801Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.3667982Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.3668075Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.3668620Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('benchmarking.InductorBenchmarker.benchmark_gpu', 2), ('pad_mm_bench', 1), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.3668752Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.3668810Z graph_break [] 2025-12-04T11:13:50.3668933Z ----------------------------- Captured stderr call ----------------------------- 2025-12-04T11:13:50.3669656Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/constant_folding.py:256: UserWarning: Unsupported unwinding pattern: Address not in range (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/profiler/unwind/unwind.cpp:219.) 2025-12-04T11:13:50.3669730Z if out == self.unknown_value: 2025-12-04T11:13:50.3669858Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.3669982Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.3670106Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.3670647Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('benchmarking.InductorBenchmarker.benchmark_gpu', 2), ('pad_mm_bench', 1), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.3670706Z graph_break [] 2025-12-04T11:13:50.3670793Z =================================== FAILURES =================================== 2025-12-04T11:13:50.3671079Z _ TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_8_in_features_144_out_features_65_cuda_bfloat16 _ 2025-12-04T11:13:50.3671154Z Traceback (most recent call last): 2025-12-04T11:13:50.3671456Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.3671520Z method(*args, **kwargs) 2025-12-04T11:13:50.3671817Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.3671879Z method(*args, **kwargs) 2025-12-04T11:13:50.3672170Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T11:13:50.3672232Z with policy(): 2025-12-04T11:13:50.3672524Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T11:13:50.3672591Z raise RuntimeError(msg) 2025-12-04T11:13:50.3673396Z RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_8_in_features_144_out_features_65_cuda_bfloat16! Caching allocator allocated memory was 19456 and is now reported as 29184 on device 0. CUDA driver allocated memory was 276824064 and is now 278921216. 2025-12-04T11:13:50.3673401Z 2025-12-04T11:13:50.3673530Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.3674054Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_8_in_features_144_out_features_65_cuda_bfloat16 2025-12-04T11:13:50.3674058Z 2025-12-04T11:13:50.3674215Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.3674344Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.3674436Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.3674974Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('benchmarking.InductorBenchmarker.benchmark_gpu', 2), ('pad_mm_bench', 1), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.3675182Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.3675244Z graph_break [] 2025-12-04T11:13:50.3675370Z ----------------------------- Captured stderr call ----------------------------- 2025-12-04T11:13:50.3676053Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/constant_folding.py:256: UserWarning: Unsupported unwinding pattern: Address not in range (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/profiler/unwind/unwind.cpp:219.) 2025-12-04T11:13:50.3676124Z if out == self.unknown_value: 2025-12-04T11:13:50.3676284Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.3676378Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.3676499Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.3677043Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('benchmarking.InductorBenchmarker.benchmark_gpu', 2), ('pad_mm_bench', 1), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.3677139Z graph_break [] 2025-12-04T11:13:50.3677268Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.3677357Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.3677478Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.3678192Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('benchmarking.InductorBenchmarker.benchmark_gpu', 2), ('pad_mm_bench', 1), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.3678255Z graph_break [] 2025-12-04T11:13:50.3678761Z - generated xml file: /var/lib/jenkins/workspace/test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-d8517590777959a7.xml - 2025-12-04T11:13:50.3678868Z =========================== short test summary info ============================ 2025-12-04T11:13:50.3680160Z FAILED [0.4995s] inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_1_mid_dim_8_in_features_144_out_features_65_cuda_bfloat16 - RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_8_in_features_144_out_features_65_cuda_bfloat16! Caching allocator allocated memory was 19456 and is now reported as 29184 on device 0. CUDA driver allocated memory was 276824064 and is now 278921216. 2025-12-04T11:13:50.3680165Z 2025-12-04T11:13:50.3680294Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.3680808Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_8_in_features_144_out_features_65_cuda_bfloat16 2025-12-04T11:13:50.3680817Z 2025-12-04T11:13:50.3680973Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.3681079Z !!!!!!!!!!!!!!!!!!!!!!!!!! stopping after 1 failures !!!!!!!!!!!!!!!!!!!!!!!!!!! 2025-12-04T11:13:50.3681197Z ================== 1 failed, 57 deselected, 2 rerun in 12.22s ================== 2025-12-04T11:13:50.3681255Z Got exit code 1 2025-12-04T11:13:50.3681327Z Retrying single test... 2025-12-04T11:13:50.3681599Z W1204 10:59:05.209000 78893 site-packages/torch/_inductor/utils.py:1703] Not enough SMs to use max_autotune_gemm mode 2025-12-04T11:13:50.3681981Z Test results will be stored in test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-7c1369cc4e5815f4.xml 2025-12-04T11:13:50.3682080Z ============================= test session starts ============================== 2025-12-04T11:13:50.3682356Z platform linux -- Python 3.10.14, pytest-7.3.2, pluggy-1.6.0 -- /opt/conda/envs/py_3.10/bin/python 2025-12-04T11:13:50.3682424Z cachedir: .pytest_cache 2025-12-04T11:13:50.3682781Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow] 2025-12-04T11:13:50.3682862Z rootdir: /var/lib/jenkins/workspace 2025-12-04T11:13:50.3682926Z configfile: pytest.ini 2025-12-04T11:13:50.3683245Z plugins: hypothesis-6.56.4, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-14.0, subtests-0.13.1, xdist-3.3.1, xdoctest-1.3.0, typeguard-4.3.0 2025-12-04T11:13:50.3683373Z collecting ... collected 58 items / 57 deselected / 1 selected 2025-12-04T11:13:50.3683989Z stepcurrent: skipping 39 already run items. Running only test/inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_1_mid_dim_8_in_features_144_out_features_65_cuda_bfloat16 2025-12-04T11:13:50.3684062Z Running 1 items in this shard 2025-12-04T11:13:50.3684067Z 2025-12-04T11:13:50.3684792Z inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_1_mid_dim_8_in_features_144_out_features_65_cuda_bfloat16 [W1204 10:59:06.847010930 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3684845Z 2025-12-04T11:13:50.3685142Z [W1204 10:59:15.875902559 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3685146Z 2025-12-04T11:13:50.3685436Z [W1204 10:59:15.876137024 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3685439Z 2025-12-04T11:13:50.3685730Z [W1204 10:59:15.882078526 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3685734Z 2025-12-04T11:13:50.3686021Z [W1204 10:59:15.882631888 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3686026Z 2025-12-04T11:13:50.3686331Z [W1204 10:59:15.882796992 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3686336Z 2025-12-04T11:13:50.3686623Z [W1204 10:59:15.888113009 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3686627Z 2025-12-04T11:13:50.3686912Z [W1204 10:59:15.888655441 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3686915Z 2025-12-04T11:13:50.3687202Z [W1204 10:59:15.888828725 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3687206Z 2025-12-04T11:13:50.3687290Z ('RERUN', {'yellow': True}) [11.0082s] [100%] 2025-12-04T11:13:50.3688005Z inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_1_mid_dim_8_in_features_144_out_features_65_cuda_bfloat16 [W1204 10:59:16.701165901 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3688010Z 2025-12-04T11:13:50.3688297Z [W1204 10:59:16.701712543 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3688303Z 2025-12-04T11:13:50.3688589Z [W1204 10:59:16.701847676 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3688592Z 2025-12-04T11:13:50.3688878Z [W1204 10:59:16.704772371 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3688881Z 2025-12-04T11:13:50.3689170Z [W1204 10:59:16.705229151 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3689213Z 2025-12-04T11:13:50.3689543Z [W1204 10:59:16.705363564 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3689548Z 2025-12-04T11:13:50.3689837Z [W1204 10:59:16.709940405 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3689840Z 2025-12-04T11:13:50.3690127Z [W1204 10:59:16.710457417 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3690130Z 2025-12-04T11:13:50.3690462Z [W1204 10:59:16.710598340 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3690466Z 2025-12-04T11:13:50.3690549Z ('RERUN', {'yellow': True}) [0.5077s] [100%] 2025-12-04T11:13:50.3691264Z inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_1_mid_dim_8_in_features_144_out_features_65_cuda_bfloat16 [W1204 10:59:17.205307233 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3691305Z 2025-12-04T11:13:50.3691591Z [W1204 10:59:17.205847555 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3691595Z 2025-12-04T11:13:50.3691878Z [W1204 10:59:17.205982168 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3691885Z 2025-12-04T11:13:50.3692174Z [W1204 10:59:17.208936193 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3692177Z 2025-12-04T11:13:50.3692462Z [W1204 10:59:17.209396373 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3692467Z 2025-12-04T11:13:50.3692756Z [W1204 10:59:17.209538907 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3692760Z 2025-12-04T11:13:50.3693046Z [W1204 10:59:17.214165119 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3693049Z 2025-12-04T11:13:50.3693339Z [W1204 10:59:17.214631799 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3693342Z 2025-12-04T11:13:50.3693632Z [W1204 10:59:17.214770652 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3693635Z 2025-12-04T11:13:50.3693701Z FAILED [0.5004s] [100%] 2025-12-04T11:13:50.3693704Z 2025-12-04T11:13:50.3693789Z ==================================== RERUNS ==================================== 2025-12-04T11:13:50.3694082Z _ TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_8_in_features_144_out_features_65_cuda_bfloat16 _ 2025-12-04T11:13:50.3694163Z Traceback (most recent call last): 2025-12-04T11:13:50.3694469Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.3694541Z method(*args, **kwargs) 2025-12-04T11:13:50.3694837Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.3694899Z method(*args, **kwargs) 2025-12-04T11:13:50.3695193Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T11:13:50.3695256Z with policy(): 2025-12-04T11:13:50.3695550Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T11:13:50.3695682Z raise RuntimeError(msg) 2025-12-04T11:13:50.3696507Z RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_8_in_features_144_out_features_65_cuda_bfloat16! Caching allocator allocated memory was 0 and is now reported as 9728 on device 0. CUDA driver allocated memory was 230686720 and is now 274726912. 2025-12-04T11:13:50.3696514Z 2025-12-04T11:13:50.3696646Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.3697159Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_8_in_features_144_out_features_65_cuda_bfloat16 2025-12-04T11:13:50.3697162Z 2025-12-04T11:13:50.3697354Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.3697492Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.3697591Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.3698138Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('benchmarking.InductorBenchmarker.benchmark_gpu', 2), ('pad_mm_bench', 1), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.3698303Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.3698363Z graph_break [] 2025-12-04T11:13:50.3698494Z ----------------------------- Captured stderr call ----------------------------- 2025-12-04T11:13:50.3699183Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/constant_folding.py:256: UserWarning: Unsupported unwinding pattern: Address not in range (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/profiler/unwind/unwind.cpp:219.) 2025-12-04T11:13:50.3699257Z if out == self.unknown_value: 2025-12-04T11:13:50.3699545Z _ TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_8_in_features_144_out_features_65_cuda_bfloat16 _ 2025-12-04T11:13:50.3699624Z Traceback (most recent call last): 2025-12-04T11:13:50.3699935Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.3700002Z method(*args, **kwargs) 2025-12-04T11:13:50.3700296Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.3700359Z method(*args, **kwargs) 2025-12-04T11:13:50.3700646Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T11:13:50.3700711Z with policy(): 2025-12-04T11:13:50.3701008Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T11:13:50.3701075Z raise RuntimeError(msg) 2025-12-04T11:13:50.3701882Z RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_8_in_features_144_out_features_65_cuda_bfloat16! Caching allocator allocated memory was 9728 and is now reported as 19456 on device 0. CUDA driver allocated memory was 274726912 and is now 276824064. 2025-12-04T11:13:50.3701888Z 2025-12-04T11:13:50.3702014Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.3702531Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_8_in_features_144_out_features_65_cuda_bfloat16 2025-12-04T11:13:50.3702535Z 2025-12-04T11:13:50.3702693Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.3702830Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.3702928Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.3703587Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('benchmarking.InductorBenchmarker.benchmark_gpu', 2), ('pad_mm_bench', 1), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.3703722Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.3703782Z graph_break [] 2025-12-04T11:13:50.3703908Z ----------------------------- Captured stderr call ----------------------------- 2025-12-04T11:13:50.3704589Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/constant_folding.py:256: UserWarning: Unsupported unwinding pattern: Address not in range (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/profiler/unwind/unwind.cpp:219.) 2025-12-04T11:13:50.3704693Z if out == self.unknown_value: 2025-12-04T11:13:50.3704821Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.3704912Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.3705041Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.3705584Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('benchmarking.InductorBenchmarker.benchmark_gpu', 2), ('pad_mm_bench', 1), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.3705680Z graph_break [] 2025-12-04T11:13:50.3705768Z =================================== FAILURES =================================== 2025-12-04T11:13:50.3706053Z _ TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_8_in_features_144_out_features_65_cuda_bfloat16 _ 2025-12-04T11:13:50.3706128Z Traceback (most recent call last): 2025-12-04T11:13:50.3706426Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.3706492Z method(*args, **kwargs) 2025-12-04T11:13:50.3706785Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.3706862Z method(*args, **kwargs) 2025-12-04T11:13:50.3707153Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T11:13:50.3707220Z with policy(): 2025-12-04T11:13:50.3707512Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T11:13:50.3707577Z raise RuntimeError(msg) 2025-12-04T11:13:50.3708386Z RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_8_in_features_144_out_features_65_cuda_bfloat16! Caching allocator allocated memory was 19456 and is now reported as 29184 on device 0. CUDA driver allocated memory was 276824064 and is now 278921216. 2025-12-04T11:13:50.3708391Z 2025-12-04T11:13:50.3708515Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.3709036Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_8_in_features_144_out_features_65_cuda_bfloat16 2025-12-04T11:13:50.3709041Z 2025-12-04T11:13:50.3709194Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.3709326Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.3709417Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.3709967Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('benchmarking.InductorBenchmarker.benchmark_gpu', 2), ('pad_mm_bench', 1), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.3710095Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.3710154Z graph_break [] 2025-12-04T11:13:50.3710324Z ----------------------------- Captured stderr call ----------------------------- 2025-12-04T11:13:50.3711047Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/constant_folding.py:256: UserWarning: Unsupported unwinding pattern: Address not in range (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/profiler/unwind/unwind.cpp:219.) 2025-12-04T11:13:50.3711120Z if out == self.unknown_value: 2025-12-04T11:13:50.3711245Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.3711336Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.3711463Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.3712033Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('benchmarking.InductorBenchmarker.benchmark_gpu', 2), ('pad_mm_bench', 1), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.3712096Z graph_break [] 2025-12-04T11:13:50.3712224Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.3712316Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.3712471Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.3713008Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('benchmarking.InductorBenchmarker.benchmark_gpu', 2), ('pad_mm_bench', 1), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.3713065Z graph_break [] 2025-12-04T11:13:50.3713558Z - generated xml file: /var/lib/jenkins/workspace/test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-7c1369cc4e5815f4.xml - 2025-12-04T11:13:50.3713659Z =========================== short test summary info ============================ 2025-12-04T11:13:50.3714941Z FAILED [0.5004s] inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_1_mid_dim_8_in_features_144_out_features_65_cuda_bfloat16 - RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_8_in_features_144_out_features_65_cuda_bfloat16! Caching allocator allocated memory was 19456 and is now reported as 29184 on device 0. CUDA driver allocated memory was 276824064 and is now 278921216. 2025-12-04T11:13:50.3714949Z 2025-12-04T11:13:50.3715073Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.3715585Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_1_mid_dim_8_in_features_144_out_features_65_cuda_bfloat16 2025-12-04T11:13:50.3715594Z 2025-12-04T11:13:50.3715752Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.3715857Z !!!!!!!!!!!!!!!!!!!!!!!!!! stopping after 1 failures !!!!!!!!!!!!!!!!!!!!!!!!!!! 2025-12-04T11:13:50.3715978Z ================== 1 failed, 57 deselected, 2 rerun in 12.04s ================== 2025-12-04T11:13:50.3716038Z Got exit code 1 2025-12-04T11:13:50.3716508Z FAILED CONSISTENTLY: test/inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_1_mid_dim_8_in_features_144_out_features_65_cuda_bfloat16 2025-12-04T11:13:50.3716751Z Test failed consistently, continuing with the rest of the tests due to continue-through-error being set 2025-12-04T11:13:50.3717011Z W1204 10:59:24.030000 79080 site-packages/torch/_inductor/utils.py:1703] Not enough SMs to use max_autotune_gemm mode 2025-12-04T11:13:50.3717403Z Test results will be stored in test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-f43c4b9f36164ff6.xml 2025-12-04T11:13:50.3717500Z ============================= test session starts ============================== 2025-12-04T11:13:50.3717752Z platform linux -- Python 3.10.14, pytest-7.3.2, pluggy-1.6.0 -- /opt/conda/envs/py_3.10/bin/python 2025-12-04T11:13:50.3717826Z cachedir: .pytest_cache 2025-12-04T11:13:50.3718166Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow] 2025-12-04T11:13:50.3718248Z rootdir: /var/lib/jenkins/workspace 2025-12-04T11:13:50.3718316Z configfile: pytest.ini 2025-12-04T11:13:50.3718629Z plugins: hypothesis-6.56.4, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-14.0, subtests-0.13.1, xdist-3.3.1, xdoctest-1.3.0, typeguard-4.3.0 2025-12-04T11:13:50.3718771Z collecting ... collected 58 items / 40 deselected / 18 selected 2025-12-04T11:13:50.3718860Z stepcurrent: skipping 40 already run items. 2025-12-04T11:13:50.3718929Z Running 18 items in this shard 2025-12-04T11:13:50.3718973Z 2025-12-04T11:13:50.3719479Z inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_32_mid_dim_1_in_features_1024_out_features_1024_cuda_bfloat16 ('RERUN', {'yellow': True}) [2.0392s] [ 5%] 2025-12-04T11:13:50.3719976Z inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_32_mid_dim_1_in_features_1024_out_features_1024_cuda_bfloat16 ('RERUN', {'yellow': True}) [0.6200s] [ 5%] 2025-12-04T11:13:50.3720478Z inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_32_mid_dim_1_in_features_1024_out_features_1024_cuda_bfloat16 FAILED [0.6207s] [ 5%] 2025-12-04T11:13:50.3720482Z 2025-12-04T11:13:50.3720565Z ==================================== RERUNS ==================================== 2025-12-04T11:13:50.3720865Z _ TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_1_in_features_1024_out_features_1024_cuda_bfloat16 _ 2025-12-04T11:13:50.3720941Z Traceback (most recent call last): 2025-12-04T11:13:50.3721245Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.3721315Z method(*args, **kwargs) 2025-12-04T11:13:50.3721608Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.3721678Z method(*args, **kwargs) 2025-12-04T11:13:50.3721967Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T11:13:50.3722027Z with policy(): 2025-12-04T11:13:50.3722324Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T11:13:50.3722392Z raise RuntimeError(msg) 2025-12-04T11:13:50.3723204Z RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_1_in_features_1024_out_features_1024_cuda_bfloat16! Caching allocator allocated memory was 0 and is now reported as 1048576 on device 0. CUDA driver allocated memory was 230686720 and is now 274726912. 2025-12-04T11:13:50.3723212Z 2025-12-04T11:13:50.3723337Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.3723863Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_1_in_features_1024_out_features_1024_cuda_bfloat16 2025-12-04T11:13:50.3723868Z 2025-12-04T11:13:50.3724027Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.3724153Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.3724251Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.3724602Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.3724730Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.3724921Z graph_break [] 2025-12-04T11:13:50.3725254Z _ TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_1_in_features_1024_out_features_1024_cuda_bfloat16 _ 2025-12-04T11:13:50.3725329Z Traceback (most recent call last): 2025-12-04T11:13:50.3725628Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.3725692Z method(*args, **kwargs) 2025-12-04T11:13:50.3725985Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.3726050Z method(*args, **kwargs) 2025-12-04T11:13:50.3726337Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T11:13:50.3726436Z with policy(): 2025-12-04T11:13:50.3726730Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T11:13:50.3726797Z raise RuntimeError(msg) 2025-12-04T11:13:50.3727636Z RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_1_in_features_1024_out_features_1024_cuda_bfloat16! Caching allocator allocated memory was 1048576 and is now reported as 2097152 on device 0. CUDA driver allocated memory was 274726912 and is now 276824064. 2025-12-04T11:13:50.3727675Z 2025-12-04T11:13:50.3727797Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.3728323Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_1_in_features_1024_out_features_1024_cuda_bfloat16 2025-12-04T11:13:50.3728327Z 2025-12-04T11:13:50.3728484Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.3728612Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.3728706Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.3729052Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.3729184Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.3729240Z graph_break [] 2025-12-04T11:13:50.3729369Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.3729458Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.3729586Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.3729929Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.3729989Z graph_break [] 2025-12-04T11:13:50.3730078Z =================================== FAILURES =================================== 2025-12-04T11:13:50.3730381Z _ TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_1_in_features_1024_out_features_1024_cuda_bfloat16 _ 2025-12-04T11:13:50.3730457Z Traceback (most recent call last): 2025-12-04T11:13:50.3730757Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.3730822Z method(*args, **kwargs) 2025-12-04T11:13:50.3731117Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.3731183Z method(*args, **kwargs) 2025-12-04T11:13:50.3731483Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T11:13:50.3731544Z with policy(): 2025-12-04T11:13:50.3731842Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T11:13:50.3731952Z raise RuntimeError(msg) 2025-12-04T11:13:50.3732820Z RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_1_in_features_1024_out_features_1024_cuda_bfloat16! Caching allocator allocated memory was 2097152 and is now reported as 3145728 on device 0. CUDA driver allocated memory was 276824064 and is now 278921216. 2025-12-04T11:13:50.3732825Z 2025-12-04T11:13:50.3732955Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.3733480Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_1_in_features_1024_out_features_1024_cuda_bfloat16 2025-12-04T11:13:50.3733517Z 2025-12-04T11:13:50.3733674Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.3733800Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.3733902Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.3734252Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.3734421Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.3734485Z graph_break [] 2025-12-04T11:13:50.3734612Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.3734704Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.3734827Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.3735172Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.3735232Z graph_break [] 2025-12-04T11:13:50.3735358Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.3735453Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.3735574Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.3735916Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.3735977Z graph_break [] 2025-12-04T11:13:50.3736466Z - generated xml file: /var/lib/jenkins/workspace/test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-f43c4b9f36164ff6.xml - 2025-12-04T11:13:50.3736569Z =========================== short test summary info ============================ 2025-12-04T11:13:50.3737875Z FAILED [0.6207s] inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_32_mid_dim_1_in_features_1024_out_features_1024_cuda_bfloat16 - RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_1_in_features_1024_out_features_1024_cuda_bfloat16! Caching allocator allocated memory was 2097152 and is now reported as 3145728 on device 0. CUDA driver allocated memory was 276824064 and is now 278921216. 2025-12-04T11:13:50.3737882Z 2025-12-04T11:13:50.3738014Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.3738535Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_1_in_features_1024_out_features_1024_cuda_bfloat16 2025-12-04T11:13:50.3738538Z 2025-12-04T11:13:50.3738696Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.3738806Z !!!!!!!!!!!!!!!!!!!!!!!!!! stopping after 1 failures !!!!!!!!!!!!!!!!!!!!!!!!!!! 2025-12-04T11:13:50.3738921Z ================== 1 failed, 40 deselected, 2 rerun in 3.30s =================== 2025-12-04T11:13:50.3739029Z Got exit code 1 2025-12-04T11:13:50.3739095Z Retrying single test... 2025-12-04T11:13:50.3739397Z W1204 10:59:34.193000 79269 site-packages/torch/_inductor/utils.py:1703] Not enough SMs to use max_autotune_gemm mode 2025-12-04T11:13:50.3739787Z Test results will be stored in test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-d56aa1824afdfad3.xml 2025-12-04T11:13:50.3739884Z ============================= test session starts ============================== 2025-12-04T11:13:50.3740088Z platform linux -- Python 3.10.14, pytest-7.3.2, pluggy-1.6.0 -- /opt/conda/envs/py_3.10/bin/python 2025-12-04T11:13:50.3740156Z cachedir: .pytest_cache 2025-12-04T11:13:50.3740494Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow] 2025-12-04T11:13:50.3740578Z rootdir: /var/lib/jenkins/workspace 2025-12-04T11:13:50.3740646Z configfile: pytest.ini 2025-12-04T11:13:50.3740977Z plugins: hypothesis-6.56.4, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-14.0, subtests-0.13.1, xdist-3.3.1, xdoctest-1.3.0, typeguard-4.3.0 2025-12-04T11:13:50.3741115Z collecting ... collected 58 items / 57 deselected / 1 selected 2025-12-04T11:13:50.3741725Z stepcurrent: skipping 40 already run items. Running only test/inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_32_mid_dim_1_in_features_1024_out_features_1024_cuda_bfloat16 2025-12-04T11:13:50.3741799Z Running 1 items in this shard 2025-12-04T11:13:50.3741803Z 2025-12-04T11:13:50.3742544Z inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_32_mid_dim_1_in_features_1024_out_features_1024_cuda_bfloat16 [W1204 10:59:35.468251298 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3742548Z 2025-12-04T11:13:50.3742845Z [W1204 10:59:44.579759040 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3742852Z 2025-12-04T11:13:50.3743141Z [W1204 10:59:44.579998625 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3743146Z 2025-12-04T11:13:50.3743434Z [W1204 10:59:44.585670311 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3743437Z 2025-12-04T11:13:50.3743763Z [W1204 10:59:44.586240803 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3743767Z 2025-12-04T11:13:50.3744058Z [W1204 10:59:44.586414317 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3744061Z 2025-12-04T11:13:50.3744352Z [W1204 10:59:44.591903599 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3744357Z 2025-12-04T11:13:50.3744646Z [W1204 10:59:44.592458221 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3744651Z 2025-12-04T11:13:50.3744941Z [W1204 10:59:44.592637735 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3744944Z 2025-12-04T11:13:50.3745026Z ('RERUN', {'yellow': True}) [11.1854s] [100%] 2025-12-04T11:13:50.3745769Z inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_32_mid_dim_1_in_features_1024_out_features_1024_cuda_bfloat16 [W1204 10:59:46.957537183 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3745776Z 2025-12-04T11:13:50.3746065Z [W1204 10:59:46.958085775 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3746111Z 2025-12-04T11:13:50.3746429Z [W1204 10:59:46.958222108 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3746437Z 2025-12-04T11:13:50.3746723Z [W1204 10:59:46.961232404 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3746727Z 2025-12-04T11:13:50.3747014Z [W1204 10:59:46.961811917 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3747017Z 2025-12-04T11:13:50.3747355Z [W1204 10:59:46.961947420 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3747359Z 2025-12-04T11:13:50.3747648Z [W1204 10:59:46.966553302 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3747653Z 2025-12-04T11:13:50.3747942Z [W1204 10:59:46.967022673 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3747979Z 2025-12-04T11:13:50.3748266Z [W1204 10:59:46.967158016 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3748269Z 2025-12-04T11:13:50.3748349Z ('RERUN', {'yellow': True}) [0.6033s] [100%] 2025-12-04T11:13:50.3749080Z inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_32_mid_dim_1_in_features_1024_out_features_1024_cuda_bfloat16 [W1204 10:59:46.557825853 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3749084Z 2025-12-04T11:13:50.3749371Z [W1204 10:59:46.558387925 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3749379Z 2025-12-04T11:13:50.3749674Z [W1204 10:59:46.558526198 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3749679Z 2025-12-04T11:13:50.3749964Z [W1204 10:59:46.561549845 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3749967Z 2025-12-04T11:13:50.3750257Z [W1204 10:59:46.562130448 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3750260Z 2025-12-04T11:13:50.3750547Z [W1204 10:59:46.562268181 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3750550Z 2025-12-04T11:13:50.3750839Z [W1204 10:59:46.566916334 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3750844Z 2025-12-04T11:13:50.3751132Z [W1204 10:59:46.567391214 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3751137Z 2025-12-04T11:13:50.3751427Z [W1204 10:59:46.567527977 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3751430Z 2025-12-04T11:13:50.3751491Z FAILED [0.6033s] [100%] 2025-12-04T11:13:50.3751495Z 2025-12-04T11:13:50.3751582Z ==================================== RERUNS ==================================== 2025-12-04T11:13:50.3751885Z _ TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_1_in_features_1024_out_features_1024_cuda_bfloat16 _ 2025-12-04T11:13:50.3751962Z Traceback (most recent call last): 2025-12-04T11:13:50.3752284Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.3752391Z method(*args, **kwargs) 2025-12-04T11:13:50.3752715Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.3752782Z method(*args, **kwargs) 2025-12-04T11:13:50.3753071Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T11:13:50.3753135Z with policy(): 2025-12-04T11:13:50.3753430Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T11:13:50.3753495Z raise RuntimeError(msg) 2025-12-04T11:13:50.3754338Z RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_1_in_features_1024_out_features_1024_cuda_bfloat16! Caching allocator allocated memory was 0 and is now reported as 1048576 on device 0. CUDA driver allocated memory was 230686720 and is now 274726912. 2025-12-04T11:13:50.3754344Z 2025-12-04T11:13:50.3754477Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.3755004Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_1_in_features_1024_out_features_1024_cuda_bfloat16 2025-12-04T11:13:50.3755041Z 2025-12-04T11:13:50.3755200Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.3755329Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.3755426Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.3755776Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.3755907Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.3755967Z graph_break [] 2025-12-04T11:13:50.3756095Z ----------------------------- Captured stderr call ----------------------------- 2025-12-04T11:13:50.3756787Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/constant_folding.py:256: UserWarning: Unsupported unwinding pattern: Address not in range (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/profiler/unwind/unwind.cpp:219.) 2025-12-04T11:13:50.3756860Z if out == self.unknown_value: 2025-12-04T11:13:50.3757160Z _ TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_1_in_features_1024_out_features_1024_cuda_bfloat16 _ 2025-12-04T11:13:50.3757234Z Traceback (most recent call last): 2025-12-04T11:13:50.3757530Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.3757598Z method(*args, **kwargs) 2025-12-04T11:13:50.3757886Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.3757951Z method(*args, **kwargs) 2025-12-04T11:13:50.3758246Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T11:13:50.3758308Z with policy(): 2025-12-04T11:13:50.3758602Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T11:13:50.3758667Z raise RuntimeError(msg) 2025-12-04T11:13:50.3759499Z RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_1_in_features_1024_out_features_1024_cuda_bfloat16! Caching allocator allocated memory was 1048576 and is now reported as 2097152 on device 0. CUDA driver allocated memory was 274726912 and is now 276824064. 2025-12-04T11:13:50.3759507Z 2025-12-04T11:13:50.3759637Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.3760237Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_1_in_features_1024_out_features_1024_cuda_bfloat16 2025-12-04T11:13:50.3760243Z 2025-12-04T11:13:50.3760404Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.3760529Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.3760634Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.3760985Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.3761148Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.3761211Z graph_break [] 2025-12-04T11:13:50.3761335Z ----------------------------- Captured stderr call ----------------------------- 2025-12-04T11:13:50.3762021Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/constant_folding.py:256: UserWarning: Unsupported unwinding pattern: Address not in range (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/profiler/unwind/unwind.cpp:219.) 2025-12-04T11:13:50.3762131Z if out == self.unknown_value: 2025-12-04T11:13:50.3762253Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.3762346Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.3762467Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.3762810Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.3762874Z graph_break [] 2025-12-04T11:13:50.3762959Z =================================== FAILURES =================================== 2025-12-04T11:13:50.3763262Z _ TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_1_in_features_1024_out_features_1024_cuda_bfloat16 _ 2025-12-04T11:13:50.3763338Z Traceback (most recent call last): 2025-12-04T11:13:50.3763636Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.3763706Z method(*args, **kwargs) 2025-12-04T11:13:50.3763997Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.3764063Z method(*args, **kwargs) 2025-12-04T11:13:50.3764354Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T11:13:50.3764414Z with policy(): 2025-12-04T11:13:50.3764902Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T11:13:50.3764976Z raise RuntimeError(msg) 2025-12-04T11:13:50.3765943Z RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_1_in_features_1024_out_features_1024_cuda_bfloat16! Caching allocator allocated memory was 2097152 and is now reported as 3145728 on device 0. CUDA driver allocated memory was 276824064 and is now 278921216. 2025-12-04T11:13:50.3765950Z 2025-12-04T11:13:50.3766192Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.3766749Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_1_in_features_1024_out_features_1024_cuda_bfloat16 2025-12-04T11:13:50.3766754Z 2025-12-04T11:13:50.3766925Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.3767059Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.3767208Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.3771267Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.3771453Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.3771520Z graph_break [] 2025-12-04T11:13:50.3771664Z ----------------------------- Captured stderr call ----------------------------- 2025-12-04T11:13:50.3772384Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/constant_folding.py:256: UserWarning: Unsupported unwinding pattern: Address not in range (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/profiler/unwind/unwind.cpp:219.) 2025-12-04T11:13:50.3772464Z if out == self.unknown_value: 2025-12-04T11:13:50.3772645Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.3772750Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.3772892Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.3773249Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.3773349Z graph_break [] 2025-12-04T11:13:50.3773483Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.3773582Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.3773705Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.3774049Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.3774112Z graph_break [] 2025-12-04T11:13:50.3774615Z - generated xml file: /var/lib/jenkins/workspace/test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-d56aa1824afdfad3.xml - 2025-12-04T11:13:50.3774721Z =========================== short test summary info ============================ 2025-12-04T11:13:50.3776055Z FAILED [0.6033s] inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_32_mid_dim_1_in_features_1024_out_features_1024_cuda_bfloat16 - RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_1_in_features_1024_out_features_1024_cuda_bfloat16! Caching allocator allocated memory was 2097152 and is now reported as 3145728 on device 0. CUDA driver allocated memory was 276824064 and is now 278921216. 2025-12-04T11:13:50.3776063Z 2025-12-04T11:13:50.3776199Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.3776726Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_1_in_features_1024_out_features_1024_cuda_bfloat16 2025-12-04T11:13:50.3776732Z 2025-12-04T11:13:50.3776894Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.3777004Z !!!!!!!!!!!!!!!!!!!!!!!!!! stopping after 1 failures !!!!!!!!!!!!!!!!!!!!!!!!!!! 2025-12-04T11:13:50.3777125Z ================== 1 failed, 57 deselected, 2 rerun in 12.42s ================== 2025-12-04T11:13:50.3777187Z Got exit code 1 2025-12-04T11:13:50.3777258Z Retrying single test... 2025-12-04T11:13:50.3777527Z W1204 10:59:53.411000 79463 site-packages/torch/_inductor/utils.py:1703] Not enough SMs to use max_autotune_gemm mode 2025-12-04T11:13:50.3778281Z Test results will be stored in test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-6a9d289c68c250dc.xml 2025-12-04T11:13:50.3778456Z ============================= test session starts ============================== 2025-12-04T11:13:50.3778794Z platform linux -- Python 3.10.14, pytest-7.3.2, pluggy-1.6.0 -- /opt/conda/envs/py_3.10/bin/python 2025-12-04T11:13:50.3778969Z cachedir: .pytest_cache 2025-12-04T11:13:50.3779338Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow] 2025-12-04T11:13:50.3779425Z rootdir: /var/lib/jenkins/workspace 2025-12-04T11:13:50.3779500Z configfile: pytest.ini 2025-12-04T11:13:50.3779816Z plugins: hypothesis-6.56.4, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-14.0, subtests-0.13.1, xdist-3.3.1, xdoctest-1.3.0, typeguard-4.3.0 2025-12-04T11:13:50.3779955Z collecting ... collected 58 items / 57 deselected / 1 selected 2025-12-04T11:13:50.3780590Z stepcurrent: skipping 40 already run items. Running only test/inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_32_mid_dim_1_in_features_1024_out_features_1024_cuda_bfloat16 2025-12-04T11:13:50.3780664Z Running 1 items in this shard 2025-12-04T11:13:50.3780672Z 2025-12-04T11:13:50.3781418Z inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_32_mid_dim_1_in_features_1024_out_features_1024_cuda_bfloat16 [W1204 10:59:54.695281754 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3781469Z 2025-12-04T11:13:50.3781785Z [W1204 11:00:04.986736471 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3781792Z 2025-12-04T11:13:50.3782094Z [W1204 11:00:04.986978657 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3782097Z 2025-12-04T11:13:50.3782390Z [W1204 11:00:04.992540389 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3782393Z 2025-12-04T11:13:50.3782684Z [W1204 11:00:04.993123472 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3782689Z 2025-12-04T11:13:50.3782976Z [W1204 11:00:04.993295696 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3782981Z 2025-12-04T11:13:50.3783271Z [W1204 11:00:04.998751877 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3783274Z 2025-12-04T11:13:50.3783629Z [W1204 11:00:04.999308759 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3783633Z 2025-12-04T11:13:50.3783928Z [W1204 11:00:04.999482523 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3783931Z 2025-12-04T11:13:50.3784012Z ('RERUN', {'yellow': True}) [11.3703s] [100%] 2025-12-04T11:13:50.3784749Z inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_32_mid_dim_1_in_features_1024_out_features_1024_cuda_bfloat16 [W1204 11:00:05.344788517 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3784759Z 2025-12-04T11:13:50.3785049Z [W1204 11:00:05.345328339 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3785053Z 2025-12-04T11:13:50.3785337Z [W1204 11:00:05.345463922 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3785340Z 2025-12-04T11:13:50.3785628Z [W1204 11:00:05.348415447 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3785631Z 2025-12-04T11:13:50.3785918Z [W1204 11:00:05.349002930 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3785966Z 2025-12-04T11:13:50.3786289Z [W1204 11:00:05.349144383 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3786295Z 2025-12-04T11:13:50.3786579Z [W1204 11:00:05.353738135 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3786582Z 2025-12-04T11:13:50.3786869Z [W1204 11:00:05.354215645 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3786872Z 2025-12-04T11:13:50.3787192Z [W1204 11:00:05.354350799 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3787196Z 2025-12-04T11:13:50.3787280Z ('RERUN', {'yellow': True}) [0.5828s] [100%] 2025-12-04T11:13:50.3788010Z inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_32_mid_dim_1_in_features_1024_out_features_1024_cuda_bfloat16 [W1204 11:00:05.925838829 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3788049Z 2025-12-04T11:13:50.3788336Z [W1204 11:00:05.926376650 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3788343Z 2025-12-04T11:13:50.3788628Z [W1204 11:00:05.926513073 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3788631Z 2025-12-04T11:13:50.3788917Z [W1204 11:00:05.929452838 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3788921Z 2025-12-04T11:13:50.3789209Z [W1204 11:00:05.930039491 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3789214Z 2025-12-04T11:13:50.3789501Z [W1204 11:00:05.930181445 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3789506Z 2025-12-04T11:13:50.3789793Z [W1204 11:00:05.934708535 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3789796Z 2025-12-04T11:13:50.3790082Z [W1204 11:00:05.935178725 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3790085Z 2025-12-04T11:13:50.3790373Z [W1204 11:00:05.935311268 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3790378Z 2025-12-04T11:13:50.3790440Z FAILED [0.5799s] [100%] 2025-12-04T11:13:50.3790444Z 2025-12-04T11:13:50.3790537Z ==================================== RERUNS ==================================== 2025-12-04T11:13:50.3790843Z _ TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_1_in_features_1024_out_features_1024_cuda_bfloat16 _ 2025-12-04T11:13:50.3790922Z Traceback (most recent call last): 2025-12-04T11:13:50.3791250Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.3791317Z method(*args, **kwargs) 2025-12-04T11:13:50.3791608Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.3791683Z method(*args, **kwargs) 2025-12-04T11:13:50.3791970Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T11:13:50.3792037Z with policy(): 2025-12-04T11:13:50.3792331Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T11:13:50.3792440Z raise RuntimeError(msg) 2025-12-04T11:13:50.3793290Z RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_1_in_features_1024_out_features_1024_cuda_bfloat16! Caching allocator allocated memory was 0 and is now reported as 1048576 on device 0. CUDA driver allocated memory was 230686720 and is now 274726912. 2025-12-04T11:13:50.3793296Z 2025-12-04T11:13:50.3793429Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.3793959Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_1_in_features_1024_out_features_1024_cuda_bfloat16 2025-12-04T11:13:50.3793963Z 2025-12-04T11:13:50.3794156Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.3794290Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.3794396Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.3794752Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.3794942Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.3795002Z graph_break [] 2025-12-04T11:13:50.3795127Z ----------------------------- Captured stderr call ----------------------------- 2025-12-04T11:13:50.3795822Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/constant_folding.py:256: UserWarning: Unsupported unwinding pattern: Address not in range (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/profiler/unwind/unwind.cpp:219.) 2025-12-04T11:13:50.3795896Z if out == self.unknown_value: 2025-12-04T11:13:50.3796199Z _ TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_1_in_features_1024_out_features_1024_cuda_bfloat16 _ 2025-12-04T11:13:50.3796279Z Traceback (most recent call last): 2025-12-04T11:13:50.3796576Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.3796643Z method(*args, **kwargs) 2025-12-04T11:13:50.3796927Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.3796990Z method(*args, **kwargs) 2025-12-04T11:13:50.3797280Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T11:13:50.3797339Z with policy(): 2025-12-04T11:13:50.3797644Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T11:13:50.3797708Z raise RuntimeError(msg) 2025-12-04T11:13:50.3798534Z RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_1_in_features_1024_out_features_1024_cuda_bfloat16! Caching allocator allocated memory was 1048576 and is now reported as 2097152 on device 0. CUDA driver allocated memory was 274726912 and is now 276824064. 2025-12-04T11:13:50.3798541Z 2025-12-04T11:13:50.3798675Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.3799196Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_1_in_features_1024_out_features_1024_cuda_bfloat16 2025-12-04T11:13:50.3799200Z 2025-12-04T11:13:50.3799369Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.3799499Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.3799591Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.3799954Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.3800164Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.3800234Z graph_break [] 2025-12-04T11:13:50.3800357Z ----------------------------- Captured stderr call ----------------------------- 2025-12-04T11:13:50.3801042Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/constant_folding.py:256: UserWarning: Unsupported unwinding pattern: Address not in range (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/profiler/unwind/unwind.cpp:219.) 2025-12-04T11:13:50.3801114Z if out == self.unknown_value: 2025-12-04T11:13:50.3801237Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.3801364Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.3801488Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.3801832Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.3801894Z graph_break [] 2025-12-04T11:13:50.3802014Z =================================== FAILURES =================================== 2025-12-04T11:13:50.3802308Z _ TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_1_in_features_1024_out_features_1024_cuda_bfloat16 _ 2025-12-04T11:13:50.3802387Z Traceback (most recent call last): 2025-12-04T11:13:50.3802681Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.3802755Z method(*args, **kwargs) 2025-12-04T11:13:50.3803043Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.3803108Z method(*args, **kwargs) 2025-12-04T11:13:50.3803398Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T11:13:50.3803457Z with policy(): 2025-12-04T11:13:50.3803753Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T11:13:50.3803821Z raise RuntimeError(msg) 2025-12-04T11:13:50.3804648Z RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_1_in_features_1024_out_features_1024_cuda_bfloat16! Caching allocator allocated memory was 2097152 and is now reported as 3145728 on device 0. CUDA driver allocated memory was 276824064 and is now 278921216. 2025-12-04T11:13:50.3804652Z 2025-12-04T11:13:50.3804786Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.3805303Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_1_in_features_1024_out_features_1024_cuda_bfloat16 2025-12-04T11:13:50.3805308Z 2025-12-04T11:13:50.3805468Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.3805597Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.3805687Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.3806033Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.3806159Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.3806221Z graph_break [] 2025-12-04T11:13:50.3806345Z ----------------------------- Captured stderr call ----------------------------- 2025-12-04T11:13:50.3807031Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/constant_folding.py:256: UserWarning: Unsupported unwinding pattern: Address not in range (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/profiler/unwind/unwind.cpp:219.) 2025-12-04T11:13:50.3807180Z if out == self.unknown_value: 2025-12-04T11:13:50.3807304Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.3807398Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.3807520Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.3807864Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.3807925Z graph_break [] 2025-12-04T11:13:50.3808048Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.3808180Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.3808307Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.3808659Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.3808720Z graph_break [] 2025-12-04T11:13:50.3809241Z - generated xml file: /var/lib/jenkins/workspace/test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-6a9d289c68c250dc.xml - 2025-12-04T11:13:50.3809342Z =========================== short test summary info ============================ 2025-12-04T11:13:50.3810655Z FAILED [0.5799s] inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_32_mid_dim_1_in_features_1024_out_features_1024_cuda_bfloat16 - RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_1_in_features_1024_out_features_1024_cuda_bfloat16! Caching allocator allocated memory was 2097152 and is now reported as 3145728 on device 0. CUDA driver allocated memory was 276824064 and is now 278921216. 2025-12-04T11:13:50.3810661Z 2025-12-04T11:13:50.3810787Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.3811313Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_1_in_features_1024_out_features_1024_cuda_bfloat16 2025-12-04T11:13:50.3811318Z 2025-12-04T11:13:50.3811472Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.3811579Z !!!!!!!!!!!!!!!!!!!!!!!!!! stopping after 1 failures !!!!!!!!!!!!!!!!!!!!!!!!!!! 2025-12-04T11:13:50.3811695Z ================== 1 failed, 57 deselected, 2 rerun in 12.56s ================== 2025-12-04T11:13:50.3811755Z Got exit code 1 2025-12-04T11:13:50.3812235Z FAILED CONSISTENTLY: test/inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_32_mid_dim_1_in_features_1024_out_features_1024_cuda_bfloat16 2025-12-04T11:13:50.3812482Z Test failed consistently, continuing with the rest of the tests due to continue-through-error being set 2025-12-04T11:13:50.3812750Z W1204 11:00:16.291000 79657 site-packages/torch/_inductor/utils.py:1703] Not enough SMs to use max_autotune_gemm mode 2025-12-04T11:13:50.3813134Z Test results will be stored in test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-04e5068c64f117f5.xml 2025-12-04T11:13:50.3813231Z ============================= test session starts ============================== 2025-12-04T11:13:50.3813442Z platform linux -- Python 3.10.14, pytest-7.3.2, pluggy-1.6.0 -- /opt/conda/envs/py_3.10/bin/python 2025-12-04T11:13:50.3813509Z cachedir: .pytest_cache 2025-12-04T11:13:50.3813816Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow] 2025-12-04T11:13:50.3813896Z rootdir: /var/lib/jenkins/workspace 2025-12-04T11:13:50.3814004Z configfile: pytest.ini 2025-12-04T11:13:50.3814355Z plugins: hypothesis-6.56.4, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-14.0, subtests-0.13.1, xdist-3.3.1, xdoctest-1.3.0, typeguard-4.3.0 2025-12-04T11:13:50.3814486Z collecting ... collected 58 items / 41 deselected / 17 selected 2025-12-04T11:13:50.3814576Z stepcurrent: skipping 41 already run items. 2025-12-04T11:13:50.3814649Z Running 17 items in this shard 2025-12-04T11:13:50.3814653Z 2025-12-04T11:13:50.3815151Z inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_32_mid_dim_1_in_features_1024_out_features_64_cuda_bfloat16 ('RERUN', {'yellow': True}) [1.8865s] [ 5%] 2025-12-04T11:13:50.3815672Z inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_32_mid_dim_1_in_features_1024_out_features_64_cuda_bfloat16 ('RERUN', {'yellow': True}) [0.4589s] [ 5%] 2025-12-04T11:13:50.3816118Z inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_32_mid_dim_1_in_features_1024_out_features_64_cuda_bfloat16 FAILED [0.4672s] [ 5%] 2025-12-04T11:13:50.3816124Z 2025-12-04T11:13:50.3816211Z ==================================== RERUNS ==================================== 2025-12-04T11:13:50.3816538Z _ TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_1_in_features_1024_out_features_64_cuda_bfloat16 _ 2025-12-04T11:13:50.3816613Z Traceback (most recent call last): 2025-12-04T11:13:50.3816922Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.3816988Z method(*args, **kwargs) 2025-12-04T11:13:50.3817290Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.3817360Z method(*args, **kwargs) 2025-12-04T11:13:50.3817646Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T11:13:50.3817711Z with policy(): 2025-12-04T11:13:50.3818004Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T11:13:50.3818072Z raise RuntimeError(msg) 2025-12-04T11:13:50.3818874Z RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_1_in_features_1024_out_features_64_cuda_bfloat16! Caching allocator allocated memory was 0 and is now reported as 65536 on device 0. CUDA driver allocated memory was 230686720 and is now 274726912. 2025-12-04T11:13:50.3818878Z 2025-12-04T11:13:50.3819003Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.3819530Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_1_in_features_1024_out_features_64_cuda_bfloat16 2025-12-04T11:13:50.3819536Z 2025-12-04T11:13:50.3819696Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.3819825Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.3819926Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.3820273Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.3820404Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.3820463Z graph_break [] 2025-12-04T11:13:50.3820754Z _ TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_1_in_features_1024_out_features_64_cuda_bfloat16 _ 2025-12-04T11:13:50.3820831Z Traceback (most recent call last): 2025-12-04T11:13:50.3821123Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.3821236Z method(*args, **kwargs) 2025-12-04T11:13:50.3821556Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.3821620Z method(*args, **kwargs) 2025-12-04T11:13:50.3821908Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T11:13:50.3821966Z with policy(): 2025-12-04T11:13:50.3822256Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T11:13:50.3822323Z raise RuntimeError(msg) 2025-12-04T11:13:50.3823180Z RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_1_in_features_1024_out_features_64_cuda_bfloat16! Caching allocator allocated memory was 65536 and is now reported as 131072 on device 0. CUDA driver allocated memory was 274726912 and is now 276824064. 2025-12-04T11:13:50.3823186Z 2025-12-04T11:13:50.3823318Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.3823885Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_1_in_features_1024_out_features_64_cuda_bfloat16 2025-12-04T11:13:50.3823927Z 2025-12-04T11:13:50.3824087Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.3824215Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.3824306Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.3824653Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.3824778Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.3824839Z graph_break [] 2025-12-04T11:13:50.3824966Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.3825055Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.3825175Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.3825516Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.3825583Z graph_break [] 2025-12-04T11:13:50.3825673Z =================================== FAILURES =================================== 2025-12-04T11:13:50.3825962Z _ TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_1_in_features_1024_out_features_64_cuda_bfloat16 _ 2025-12-04T11:13:50.3826037Z Traceback (most recent call last): 2025-12-04T11:13:50.3826331Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.3826395Z method(*args, **kwargs) 2025-12-04T11:13:50.3826686Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.3826749Z method(*args, **kwargs) 2025-12-04T11:13:50.3827033Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T11:13:50.3827095Z with policy(): 2025-12-04T11:13:50.3827384Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T11:13:50.3827453Z raise RuntimeError(msg) 2025-12-04T11:13:50.3828264Z RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_1_in_features_1024_out_features_64_cuda_bfloat16! Caching allocator allocated memory was 131072 and is now reported as 196608 on device 0. CUDA driver allocated memory was 276824064 and is now 278921216. 2025-12-04T11:13:50.3828310Z 2025-12-04T11:13:50.3828445Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.3829001Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_1_in_features_1024_out_features_64_cuda_bfloat16 2025-12-04T11:13:50.3829007Z 2025-12-04T11:13:50.3829163Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.3829290Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.3829379Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.3829764Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.3829890Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.3829949Z graph_break [] 2025-12-04T11:13:50.3830075Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.3830164Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.3830318Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.3830659Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.3830716Z graph_break [] 2025-12-04T11:13:50.3830835Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.3830925Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.3831047Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.3831390Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.3831449Z graph_break [] 2025-12-04T11:13:50.3831934Z - generated xml file: /var/lib/jenkins/workspace/test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-04e5068c64f117f5.xml - 2025-12-04T11:13:50.3832040Z =========================== short test summary info ============================ 2025-12-04T11:13:50.3833336Z FAILED [0.4672s] inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_32_mid_dim_1_in_features_1024_out_features_64_cuda_bfloat16 - RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_1_in_features_1024_out_features_64_cuda_bfloat16! Caching allocator allocated memory was 131072 and is now reported as 196608 on device 0. CUDA driver allocated memory was 276824064 and is now 278921216. 2025-12-04T11:13:50.3833341Z 2025-12-04T11:13:50.3833467Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.3833989Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_1_in_features_1024_out_features_64_cuda_bfloat16 2025-12-04T11:13:50.3833994Z 2025-12-04T11:13:50.3834150Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.3834254Z !!!!!!!!!!!!!!!!!!!!!!!!!! stopping after 1 failures !!!!!!!!!!!!!!!!!!!!!!!!!!! 2025-12-04T11:13:50.3834369Z ================== 1 failed, 41 deselected, 2 rerun in 2.84s =================== 2025-12-04T11:13:50.3834431Z Got exit code 1 2025-12-04T11:13:50.3834493Z Retrying single test... 2025-12-04T11:13:50.3834755Z W1204 11:00:26.356000 79845 site-packages/torch/_inductor/utils.py:1703] Not enough SMs to use max_autotune_gemm mode 2025-12-04T11:13:50.3835137Z Test results will be stored in test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-85ab96cf3cf02016.xml 2025-12-04T11:13:50.3835275Z ============================= test session starts ============================== 2025-12-04T11:13:50.3835596Z platform linux -- Python 3.10.14, pytest-7.3.2, pluggy-1.6.0 -- /opt/conda/envs/py_3.10/bin/python 2025-12-04T11:13:50.3835665Z cachedir: .pytest_cache 2025-12-04T11:13:50.3835969Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow] 2025-12-04T11:13:50.3836048Z rootdir: /var/lib/jenkins/workspace 2025-12-04T11:13:50.3836113Z configfile: pytest.ini 2025-12-04T11:13:50.3836430Z plugins: hypothesis-6.56.4, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-14.0, subtests-0.13.1, xdist-3.3.1, xdoctest-1.3.0, typeguard-4.3.0 2025-12-04T11:13:50.3836592Z collecting ... collected 58 items / 57 deselected / 1 selected 2025-12-04T11:13:50.3837166Z stepcurrent: skipping 41 already run items. Running only test/inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_32_mid_dim_1_in_features_1024_out_features_64_cuda_bfloat16 2025-12-04T11:13:50.3837245Z Running 1 items in this shard 2025-12-04T11:13:50.3837251Z 2025-12-04T11:13:50.3837984Z inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_32_mid_dim_1_in_features_1024_out_features_64_cuda_bfloat16 [W1204 11:00:27.449170393 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3838022Z 2025-12-04T11:13:50.3838322Z [W1204 11:00:36.597056849 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3838326Z 2025-12-04T11:13:50.3838614Z [W1204 11:00:36.597288194 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3838618Z 2025-12-04T11:13:50.3838908Z [W1204 11:00:36.602870167 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3838913Z 2025-12-04T11:13:50.3839199Z [W1204 11:00:36.603420770 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3839204Z 2025-12-04T11:13:50.3839503Z [W1204 11:00:36.603589153 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3839507Z 2025-12-04T11:13:50.3839791Z [W1204 11:00:36.609150346 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3839794Z 2025-12-04T11:13:50.3840084Z [W1204 11:00:36.609705809 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3840090Z 2025-12-04T11:13:50.3840372Z [W1204 11:00:36.609877362 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3840376Z 2025-12-04T11:13:50.3840458Z ('RERUN', {'yellow': True}) [11.0347s] [100%] 2025-12-04T11:13:50.3841189Z inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_32_mid_dim_1_in_features_1024_out_features_64_cuda_bfloat16 [W1204 11:00:37.792741042 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3841194Z 2025-12-04T11:13:50.3841484Z [W1204 11:00:37.793304205 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3841487Z 2025-12-04T11:13:50.3841776Z [W1204 11:00:37.793451768 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3841779Z 2025-12-04T11:13:50.3842064Z [W1204 11:00:37.796609948 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3842105Z 2025-12-04T11:13:50.3842429Z [W1204 11:00:37.797206561 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3842434Z 2025-12-04T11:13:50.3842733Z [W1204 11:00:37.797346744 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3842736Z 2025-12-04T11:13:50.3843027Z [W1204 11:00:37.802221492 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3843031Z 2025-12-04T11:13:50.3843347Z [W1204 11:00:37.802701473 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3843351Z 2025-12-04T11:13:50.3843636Z [W1204 11:00:37.802839286 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3843646Z 2025-12-04T11:13:50.3843726Z ('RERUN', {'yellow': True}) [0.4215s] [100%] 2025-12-04T11:13:50.3844444Z inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_32_mid_dim_1_in_features_1024_out_features_64_cuda_bfloat16 [W1204 11:00:38.211296870 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3844482Z 2025-12-04T11:13:50.3844770Z [W1204 11:00:38.211850322 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3844774Z 2025-12-04T11:13:50.3845061Z [W1204 11:00:38.211992775 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3845064Z 2025-12-04T11:13:50.3845351Z [W1204 11:00:38.215120063 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3845356Z 2025-12-04T11:13:50.3845641Z [W1204 11:00:38.215705456 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3845646Z 2025-12-04T11:13:50.3845933Z [W1204 11:00:38.215846539 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3845937Z 2025-12-04T11:13:50.3846221Z [W1204 11:00:38.220766678 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3846225Z 2025-12-04T11:13:50.3846510Z [W1204 11:00:38.221240768 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3846518Z 2025-12-04T11:13:50.3846802Z [W1204 11:00:38.221375421 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3846807Z 2025-12-04T11:13:50.3846867Z FAILED [0.4149s] [100%] 2025-12-04T11:13:50.3846870Z 2025-12-04T11:13:50.3846960Z ==================================== RERUNS ==================================== 2025-12-04T11:13:50.3847254Z _ TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_1_in_features_1024_out_features_64_cuda_bfloat16 _ 2025-12-04T11:13:50.3847332Z Traceback (most recent call last): 2025-12-04T11:13:50.3847637Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.3847700Z method(*args, **kwargs) 2025-12-04T11:13:50.3847990Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.3848055Z method(*args, **kwargs) 2025-12-04T11:13:50.3848341Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T11:13:50.3848449Z with policy(): 2025-12-04T11:13:50.3848794Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T11:13:50.3848861Z raise RuntimeError(msg) 2025-12-04T11:13:50.3849663Z RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_1_in_features_1024_out_features_64_cuda_bfloat16! Caching allocator allocated memory was 0 and is now reported as 65536 on device 0. CUDA driver allocated memory was 230686720 and is now 274726912. 2025-12-04T11:13:50.3849668Z 2025-12-04T11:13:50.3849794Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.3850354Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_1_in_features_1024_out_features_64_cuda_bfloat16 2025-12-04T11:13:50.3850360Z 2025-12-04T11:13:50.3850521Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.3850655Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.3850749Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.3851130Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.3851260Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.3851319Z graph_break [] 2025-12-04T11:13:50.3851448Z ----------------------------- Captured stderr call ----------------------------- 2025-12-04T11:13:50.3852141Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/constant_folding.py:256: UserWarning: Unsupported unwinding pattern: Address not in range (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/profiler/unwind/unwind.cpp:219.) 2025-12-04T11:13:50.3852215Z if out == self.unknown_value: 2025-12-04T11:13:50.3852508Z _ TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_1_in_features_1024_out_features_64_cuda_bfloat16 _ 2025-12-04T11:13:50.3852580Z Traceback (most recent call last): 2025-12-04T11:13:50.3852879Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.3852941Z method(*args, **kwargs) 2025-12-04T11:13:50.3853227Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.3853292Z method(*args, **kwargs) 2025-12-04T11:13:50.3853580Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T11:13:50.3853639Z with policy(): 2025-12-04T11:13:50.3853941Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T11:13:50.3854008Z raise RuntimeError(msg) 2025-12-04T11:13:50.3854821Z RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_1_in_features_1024_out_features_64_cuda_bfloat16! Caching allocator allocated memory was 65536 and is now reported as 131072 on device 0. CUDA driver allocated memory was 274726912 and is now 276824064. 2025-12-04T11:13:50.3854826Z 2025-12-04T11:13:50.3854953Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.3855471Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_1_in_features_1024_out_features_64_cuda_bfloat16 2025-12-04T11:13:50.3855475Z 2025-12-04T11:13:50.3855633Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.3855759Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.3855897Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.3856276Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.3856408Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.3856468Z graph_break [] 2025-12-04T11:13:50.3856591Z ----------------------------- Captured stderr call ----------------------------- 2025-12-04T11:13:50.3857578Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/constant_folding.py:256: UserWarning: Unsupported unwinding pattern: Address not in range (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/profiler/unwind/unwind.cpp:219.) 2025-12-04T11:13:50.3857654Z if out == self.unknown_value: 2025-12-04T11:13:50.3857786Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.3857884Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.3858010Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.3858359Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.3858456Z graph_break [] 2025-12-04T11:13:50.3858540Z =================================== FAILURES =================================== 2025-12-04T11:13:50.3858831Z _ TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_1_in_features_1024_out_features_64_cuda_bfloat16 _ 2025-12-04T11:13:50.3858906Z Traceback (most recent call last): 2025-12-04T11:13:50.3859209Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.3859272Z method(*args, **kwargs) 2025-12-04T11:13:50.3859562Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.3859630Z method(*args, **kwargs) 2025-12-04T11:13:50.3859929Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T11:13:50.3859992Z with policy(): 2025-12-04T11:13:50.3860294Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T11:13:50.3860360Z raise RuntimeError(msg) 2025-12-04T11:13:50.3861177Z RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_1_in_features_1024_out_features_64_cuda_bfloat16! Caching allocator allocated memory was 131072 and is now reported as 196608 on device 0. CUDA driver allocated memory was 276824064 and is now 278921216. 2025-12-04T11:13:50.3861181Z 2025-12-04T11:13:50.3861307Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.3861828Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_1_in_features_1024_out_features_64_cuda_bfloat16 2025-12-04T11:13:50.3861836Z 2025-12-04T11:13:50.3861991Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.3862115Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.3862207Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.3862550Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.3862679Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.3862738Z graph_break [] 2025-12-04T11:13:50.3862861Z ----------------------------- Captured stderr call ----------------------------- 2025-12-04T11:13:50.3863670Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/constant_folding.py:256: UserWarning: Unsupported unwinding pattern: Address not in range (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/profiler/unwind/unwind.cpp:219.) 2025-12-04T11:13:50.3863743Z if out == self.unknown_value: 2025-12-04T11:13:50.3863868Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.3863962Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.3864082Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.3864460Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.3864520Z graph_break [] 2025-12-04T11:13:50.3864644Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.3864739Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.3864858Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.3865195Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.3865313Z graph_break [] 2025-12-04T11:13:50.3865802Z - generated xml file: /var/lib/jenkins/workspace/test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-85ab96cf3cf02016.xml - 2025-12-04T11:13:50.3865909Z =========================== short test summary info ============================ 2025-12-04T11:13:50.3867200Z FAILED [0.4149s] inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_32_mid_dim_1_in_features_1024_out_features_64_cuda_bfloat16 - RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_1_in_features_1024_out_features_64_cuda_bfloat16! Caching allocator allocated memory was 131072 and is now reported as 196608 on device 0. CUDA driver allocated memory was 276824064 and is now 278921216. 2025-12-04T11:13:50.3867208Z 2025-12-04T11:13:50.3867334Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.3867850Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_1_in_features_1024_out_features_64_cuda_bfloat16 2025-12-04T11:13:50.3867855Z 2025-12-04T11:13:50.3868011Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.3868115Z !!!!!!!!!!!!!!!!!!!!!!!!!! stopping after 1 failures !!!!!!!!!!!!!!!!!!!!!!!!!!! 2025-12-04T11:13:50.3868229Z ================== 1 failed, 57 deselected, 2 rerun in 11.90s ================== 2025-12-04T11:13:50.3868292Z Got exit code 1 2025-12-04T11:13:50.3868357Z Retrying single test... 2025-12-04T11:13:50.3868618Z W1204 11:00:45.107000 80038 site-packages/torch/_inductor/utils.py:1703] Not enough SMs to use max_autotune_gemm mode 2025-12-04T11:13:50.3869005Z Test results will be stored in test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-888ba6ece96c2997.xml 2025-12-04T11:13:50.3869102Z ============================= test session starts ============================== 2025-12-04T11:13:50.3869311Z platform linux -- Python 3.10.14, pytest-7.3.2, pluggy-1.6.0 -- /opt/conda/envs/py_3.10/bin/python 2025-12-04T11:13:50.3869377Z cachedir: .pytest_cache 2025-12-04T11:13:50.3869683Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow] 2025-12-04T11:13:50.3869762Z rootdir: /var/lib/jenkins/workspace 2025-12-04T11:13:50.3869835Z configfile: pytest.ini 2025-12-04T11:13:50.3870149Z plugins: hypothesis-6.56.4, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-14.0, subtests-0.13.1, xdist-3.3.1, xdoctest-1.3.0, typeguard-4.3.0 2025-12-04T11:13:50.3870323Z collecting ... collected 58 items / 57 deselected / 1 selected 2025-12-04T11:13:50.3870923Z stepcurrent: skipping 41 already run items. Running only test/inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_32_mid_dim_1_in_features_1024_out_features_64_cuda_bfloat16 2025-12-04T11:13:50.3871001Z Running 1 items in this shard 2025-12-04T11:13:50.3871005Z 2025-12-04T11:13:50.3871742Z inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_32_mid_dim_1_in_features_1024_out_features_64_cuda_bfloat16 [W1204 11:00:46.212751418 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3871780Z 2025-12-04T11:13:50.3872086Z [W1204 11:00:55.482977861 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3872092Z 2025-12-04T11:13:50.3872382Z [W1204 11:00:55.483220786 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3872386Z 2025-12-04T11:13:50.3872706Z [W1204 11:00:55.488970794 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3872710Z 2025-12-04T11:13:50.3872996Z [W1204 11:00:55.489526056 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3873000Z 2025-12-04T11:13:50.3873284Z [W1204 11:00:55.489706190 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3873287Z 2025-12-04T11:13:50.3873580Z [W1204 11:00:55.495269142 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3873585Z 2025-12-04T11:13:50.3873871Z [W1204 11:00:55.495829585 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3873874Z 2025-12-04T11:13:50.3874166Z [W1204 11:00:55.496003519 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3874171Z 2025-12-04T11:13:50.3874251Z ('RERUN', {'yellow': True}) [11.1745s] [100%] 2025-12-04T11:13:50.3874976Z inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_32_mid_dim_1_in_features_1024_out_features_64_cuda_bfloat16 [W1204 11:00:56.686681127 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3874980Z 2025-12-04T11:13:50.3875268Z [W1204 11:00:56.687233899 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3875273Z 2025-12-04T11:13:50.3875561Z [W1204 11:00:56.687376602 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3875565Z 2025-12-04T11:13:50.3875852Z [W1204 11:00:56.690413130 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3875857Z 2025-12-04T11:13:50.3876142Z [W1204 11:00:56.691010013 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3876147Z 2025-12-04T11:13:50.3876430Z [W1204 11:00:56.691148886 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3876433Z 2025-12-04T11:13:50.3876720Z [W1204 11:00:56.695772148 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3876723Z 2025-12-04T11:13:50.3877080Z [W1204 11:00:56.696244728 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3877117Z 2025-12-04T11:13:50.3877404Z [W1204 11:00:56.696380931 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3877409Z 2025-12-04T11:13:50.3877493Z ('RERUN', {'yellow': True}) [0.4219s] [100%] 2025-12-04T11:13:50.3878386Z inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_32_mid_dim_1_in_features_1024_out_features_64_cuda_bfloat16 [W1204 11:00:57.104702848 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3878391Z 2025-12-04T11:13:50.3878753Z [W1204 11:00:57.105255670 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3878757Z 2025-12-04T11:13:50.3879049Z [W1204 11:00:57.105401593 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3879054Z 2025-12-04T11:13:50.3879342Z [W1204 11:00:57.108382029 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3879401Z 2025-12-04T11:13:50.3879700Z [W1204 11:00:57.108980462 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3879703Z 2025-12-04T11:13:50.3879993Z [W1204 11:00:57.109119905 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3879999Z 2025-12-04T11:13:50.3880284Z [W1204 11:00:57.113780729 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3880287Z 2025-12-04T11:13:50.3880576Z [W1204 11:00:57.114259779 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3880581Z 2025-12-04T11:13:50.3880873Z [W1204 11:00:57.114395202 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3880878Z 2025-12-04T11:13:50.3880938Z FAILED [0.4157s] [100%] 2025-12-04T11:13:50.3880942Z 2025-12-04T11:13:50.3881028Z ==================================== RERUNS ==================================== 2025-12-04T11:13:50.3881322Z _ TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_1_in_features_1024_out_features_64_cuda_bfloat16 _ 2025-12-04T11:13:50.3881395Z Traceback (most recent call last): 2025-12-04T11:13:50.3881702Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.3881767Z method(*args, **kwargs) 2025-12-04T11:13:50.3882060Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.3882127Z method(*args, **kwargs) 2025-12-04T11:13:50.3882417Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T11:13:50.3882481Z with policy(): 2025-12-04T11:13:50.3882771Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T11:13:50.3882836Z raise RuntimeError(msg) 2025-12-04T11:13:50.3883642Z RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_1_in_features_1024_out_features_64_cuda_bfloat16! Caching allocator allocated memory was 0 and is now reported as 65536 on device 0. CUDA driver allocated memory was 230686720 and is now 274726912. 2025-12-04T11:13:50.3883646Z 2025-12-04T11:13:50.3883773Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.3884405Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_1_in_features_1024_out_features_64_cuda_bfloat16 2025-12-04T11:13:50.3884412Z 2025-12-04T11:13:50.3884572Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.3884702Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.3884795Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.3885147Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.3885312Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.3885376Z graph_break [] 2025-12-04T11:13:50.3885500Z ----------------------------- Captured stderr call ----------------------------- 2025-12-04T11:13:50.3886192Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/constant_folding.py:256: UserWarning: Unsupported unwinding pattern: Address not in range (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/profiler/unwind/unwind.cpp:219.) 2025-12-04T11:13:50.3886296Z if out == self.unknown_value: 2025-12-04T11:13:50.3886590Z _ TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_1_in_features_1024_out_features_64_cuda_bfloat16 _ 2025-12-04T11:13:50.3886672Z Traceback (most recent call last): 2025-12-04T11:13:50.3886967Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.3887034Z method(*args, **kwargs) 2025-12-04T11:13:50.3887321Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.3887385Z method(*args, **kwargs) 2025-12-04T11:13:50.3887671Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T11:13:50.3887731Z with policy(): 2025-12-04T11:13:50.3888025Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T11:13:50.3888092Z raise RuntimeError(msg) 2025-12-04T11:13:50.3888897Z RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_1_in_features_1024_out_features_64_cuda_bfloat16! Caching allocator allocated memory was 65536 and is now reported as 131072 on device 0. CUDA driver allocated memory was 274726912 and is now 276824064. 2025-12-04T11:13:50.3888901Z 2025-12-04T11:13:50.3889029Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.3889549Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_1_in_features_1024_out_features_64_cuda_bfloat16 2025-12-04T11:13:50.3889557Z 2025-12-04T11:13:50.3889721Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.3889845Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.3889939Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.3890282Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.3890405Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.3890469Z graph_break [] 2025-12-04T11:13:50.3890592Z ----------------------------- Captured stderr call ----------------------------- 2025-12-04T11:13:50.3891274Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/constant_folding.py:256: UserWarning: Unsupported unwinding pattern: Address not in range (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/profiler/unwind/unwind.cpp:219.) 2025-12-04T11:13:50.3891433Z if out == self.unknown_value: 2025-12-04T11:13:50.3891559Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.3891653Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.3891774Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.3892114Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.3892176Z graph_break [] 2025-12-04T11:13:50.3892295Z =================================== FAILURES =================================== 2025-12-04T11:13:50.3892588Z _ TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_1_in_features_1024_out_features_64_cuda_bfloat16 _ 2025-12-04T11:13:50.3892661Z Traceback (most recent call last): 2025-12-04T11:13:50.3892959Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.3893025Z method(*args, **kwargs) 2025-12-04T11:13:50.3893347Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.3893413Z method(*args, **kwargs) 2025-12-04T11:13:50.3893697Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T11:13:50.3893756Z with policy(): 2025-12-04T11:13:50.3894048Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T11:13:50.3894125Z raise RuntimeError(msg) 2025-12-04T11:13:50.3894937Z RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_1_in_features_1024_out_features_64_cuda_bfloat16! Caching allocator allocated memory was 131072 and is now reported as 196608 on device 0. CUDA driver allocated memory was 276824064 and is now 278921216. 2025-12-04T11:13:50.3894949Z 2025-12-04T11:13:50.3895073Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.3895588Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_1_in_features_1024_out_features_64_cuda_bfloat16 2025-12-04T11:13:50.3895591Z 2025-12-04T11:13:50.3895748Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.3895871Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.3895964Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.3896305Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.3896429Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.3896491Z graph_break [] 2025-12-04T11:13:50.3896614Z ----------------------------- Captured stderr call ----------------------------- 2025-12-04T11:13:50.3897298Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/constant_folding.py:256: UserWarning: Unsupported unwinding pattern: Address not in range (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/profiler/unwind/unwind.cpp:219.) 2025-12-04T11:13:50.3897366Z if out == self.unknown_value: 2025-12-04T11:13:50.3897490Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.3897584Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.3897704Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.3898046Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.3898194Z graph_break [] 2025-12-04T11:13:50.3898321Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.3898414Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.3898534Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.3898870Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.3898933Z graph_break [] 2025-12-04T11:13:50.3899459Z - generated xml file: /var/lib/jenkins/workspace/test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-888ba6ece96c2997.xml - 2025-12-04T11:13:50.3899564Z =========================== short test summary info ============================ 2025-12-04T11:13:50.3900849Z FAILED [0.4157s] inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_32_mid_dim_1_in_features_1024_out_features_64_cuda_bfloat16 - RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_1_in_features_1024_out_features_64_cuda_bfloat16! Caching allocator allocated memory was 131072 and is now reported as 196608 on device 0. CUDA driver allocated memory was 276824064 and is now 278921216. 2025-12-04T11:13:50.3900905Z 2025-12-04T11:13:50.3901031Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.3901548Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_1_in_features_1024_out_features_64_cuda_bfloat16 2025-12-04T11:13:50.3901552Z 2025-12-04T11:13:50.3901705Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.3901813Z !!!!!!!!!!!!!!!!!!!!!!!!!! stopping after 1 failures !!!!!!!!!!!!!!!!!!!!!!!!!!! 2025-12-04T11:13:50.3901927Z ================== 1 failed, 57 deselected, 2 rerun in 12.04s ================== 2025-12-04T11:13:50.3901989Z Got exit code 1 2025-12-04T11:13:50.3902465Z FAILED CONSISTENTLY: test/inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_32_mid_dim_1_in_features_1024_out_features_64_cuda_bfloat16 2025-12-04T11:13:50.3902706Z Test failed consistently, continuing with the rest of the tests due to continue-through-error being set 2025-12-04T11:13:50.3902975Z W1204 11:01:03.913000 80231 site-packages/torch/_inductor/utils.py:1703] Not enough SMs to use max_autotune_gemm mode 2025-12-04T11:13:50.3903358Z Test results will be stored in test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-58cf8d0fed87176f.xml 2025-12-04T11:13:50.3903457Z ============================= test session starts ============================== 2025-12-04T11:13:50.3903737Z platform linux -- Python 3.10.14, pytest-7.3.2, pluggy-1.6.0 -- /opt/conda/envs/py_3.10/bin/python 2025-12-04T11:13:50.3903805Z cachedir: .pytest_cache 2025-12-04T11:13:50.3904117Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow] 2025-12-04T11:13:50.3904193Z rootdir: /var/lib/jenkins/workspace 2025-12-04T11:13:50.3904257Z configfile: pytest.ini 2025-12-04T11:13:50.3904577Z plugins: hypothesis-6.56.4, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-14.0, subtests-0.13.1, xdist-3.3.1, xdoctest-1.3.0, typeguard-4.3.0 2025-12-04T11:13:50.3904704Z collecting ... collected 58 items / 42 deselected / 16 selected 2025-12-04T11:13:50.3904796Z stepcurrent: skipping 42 already run items. 2025-12-04T11:13:50.3904871Z Running 16 items in this shard 2025-12-04T11:13:50.3904875Z 2025-12-04T11:13:50.3905372Z inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_32_mid_dim_1_in_features_1024_out_features_65_cuda_bfloat16 ('RERUN', {'yellow': True}) [1.9767s] [ 6%] 2025-12-04T11:13:50.3905940Z inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_32_mid_dim_1_in_features_1024_out_features_65_cuda_bfloat16 ('RERUN', {'yellow': True}) [0.5500s] [ 6%] 2025-12-04T11:13:50.3906386Z inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_32_mid_dim_1_in_features_1024_out_features_65_cuda_bfloat16 FAILED [0.5413s] [ 6%] 2025-12-04T11:13:50.3906390Z 2025-12-04T11:13:50.3906475Z ==================================== RERUNS ==================================== 2025-12-04T11:13:50.3906795Z _ TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_1_in_features_1024_out_features_65_cuda_bfloat16 _ 2025-12-04T11:13:50.3906870Z Traceback (most recent call last): 2025-12-04T11:13:50.3907181Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.3907250Z method(*args, **kwargs) 2025-12-04T11:13:50.3907546Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.3907644Z method(*args, **kwargs) 2025-12-04T11:13:50.3907932Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T11:13:50.3907997Z with policy(): 2025-12-04T11:13:50.3908288Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T11:13:50.3908354Z raise RuntimeError(msg) 2025-12-04T11:13:50.3909154Z RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_1_in_features_1024_out_features_65_cuda_bfloat16! Caching allocator allocated memory was 0 and is now reported as 66560 on device 0. CUDA driver allocated memory was 230686720 and is now 274726912. 2025-12-04T11:13:50.3909160Z 2025-12-04T11:13:50.3909288Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.3909822Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_1_in_features_1024_out_features_65_cuda_bfloat16 2025-12-04T11:13:50.3909825Z 2025-12-04T11:13:50.3909991Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.3910128Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.3910220Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.3910769Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('benchmarking.InductorBenchmarker.benchmark_gpu', 2), ('pad_mm_bench', 1), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.3910904Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.3910963Z graph_break [] 2025-12-04T11:13:50.3911255Z _ TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_1_in_features_1024_out_features_65_cuda_bfloat16 _ 2025-12-04T11:13:50.3911333Z Traceback (most recent call last): 2025-12-04T11:13:50.3911630Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.3911696Z method(*args, **kwargs) 2025-12-04T11:13:50.3911982Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.3912044Z method(*args, **kwargs) 2025-12-04T11:13:50.3912331Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T11:13:50.3912389Z with policy(): 2025-12-04T11:13:50.3912734Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T11:13:50.3912834Z raise RuntimeError(msg) 2025-12-04T11:13:50.3913645Z RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_1_in_features_1024_out_features_65_cuda_bfloat16! Caching allocator allocated memory was 66560 and is now reported as 133120 on device 0. CUDA driver allocated memory was 274726912 and is now 276824064. 2025-12-04T11:13:50.3913652Z 2025-12-04T11:13:50.3913777Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.3914338Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_1_in_features_1024_out_features_65_cuda_bfloat16 2025-12-04T11:13:50.3914342Z 2025-12-04T11:13:50.3914501Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.3914630Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.3914723Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.3915294Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('benchmarking.InductorBenchmarker.benchmark_gpu', 2), ('pad_mm_bench', 1), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.3915419Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.3915480Z graph_break [] 2025-12-04T11:13:50.3915606Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.3915696Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.3915820Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.3916352Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('benchmarking.InductorBenchmarker.benchmark_gpu', 2), ('pad_mm_bench', 1), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.3916422Z graph_break [] 2025-12-04T11:13:50.3916510Z =================================== FAILURES =================================== 2025-12-04T11:13:50.3916798Z _ TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_1_in_features_1024_out_features_65_cuda_bfloat16 _ 2025-12-04T11:13:50.3916875Z Traceback (most recent call last): 2025-12-04T11:13:50.3917166Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.3917229Z method(*args, **kwargs) 2025-12-04T11:13:50.3917516Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.3917579Z method(*args, **kwargs) 2025-12-04T11:13:50.3917867Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T11:13:50.3917926Z with policy(): 2025-12-04T11:13:50.3918217Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T11:13:50.3918286Z raise RuntimeError(msg) 2025-12-04T11:13:50.3919096Z RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_1_in_features_1024_out_features_65_cuda_bfloat16! Caching allocator allocated memory was 133120 and is now reported as 199680 on device 0. CUDA driver allocated memory was 276824064 and is now 278921216. 2025-12-04T11:13:50.3919100Z 2025-12-04T11:13:50.3919228Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.3919743Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_1_in_features_1024_out_features_65_cuda_bfloat16 2025-12-04T11:13:50.3919788Z 2025-12-04T11:13:50.3919979Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.3920106Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.3920195Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.3920730Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('benchmarking.InductorBenchmarker.benchmark_gpu', 2), ('pad_mm_bench', 1), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.3920851Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.3920911Z graph_break [] 2025-12-04T11:13:50.3921075Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.3921163Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.3921288Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.3921823Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('benchmarking.InductorBenchmarker.benchmark_gpu', 2), ('pad_mm_bench', 1), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.3921926Z graph_break [] 2025-12-04T11:13:50.3922052Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.3922138Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.3922262Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.3922796Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('benchmarking.InductorBenchmarker.benchmark_gpu', 2), ('pad_mm_bench', 1), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.3922854Z graph_break [] 2025-12-04T11:13:50.3923347Z - generated xml file: /var/lib/jenkins/workspace/test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-58cf8d0fed87176f.xml - 2025-12-04T11:13:50.3923447Z =========================== short test summary info ============================ 2025-12-04T11:13:50.3924736Z FAILED [0.5413s] inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_32_mid_dim_1_in_features_1024_out_features_65_cuda_bfloat16 - RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_1_in_features_1024_out_features_65_cuda_bfloat16! Caching allocator allocated memory was 133120 and is now reported as 199680 on device 0. CUDA driver allocated memory was 276824064 and is now 278921216. 2025-12-04T11:13:50.3924740Z 2025-12-04T11:13:50.3924865Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.3925387Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_1_in_features_1024_out_features_65_cuda_bfloat16 2025-12-04T11:13:50.3925392Z 2025-12-04T11:13:50.3925547Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.3925651Z !!!!!!!!!!!!!!!!!!!!!!!!!! stopping after 1 failures !!!!!!!!!!!!!!!!!!!!!!!!!!! 2025-12-04T11:13:50.3925769Z ================== 1 failed, 42 deselected, 2 rerun in 3.09s =================== 2025-12-04T11:13:50.3925827Z Got exit code 1 2025-12-04T11:13:50.3925893Z Retrying single test... 2025-12-04T11:13:50.3926153Z W1204 11:01:14.015000 80420 site-packages/torch/_inductor/utils.py:1703] Not enough SMs to use max_autotune_gemm mode 2025-12-04T11:13:50.3926537Z Test results will be stored in test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-0022ebcfefe802db.xml 2025-12-04T11:13:50.3926635Z ============================= test session starts ============================== 2025-12-04T11:13:50.3926917Z platform linux -- Python 3.10.14, pytest-7.3.2, pluggy-1.6.0 -- /opt/conda/envs/py_3.10/bin/python 2025-12-04T11:13:50.3926983Z cachedir: .pytest_cache 2025-12-04T11:13:50.3927293Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow] 2025-12-04T11:13:50.3927379Z rootdir: /var/lib/jenkins/workspace 2025-12-04T11:13:50.3927449Z configfile: pytest.ini 2025-12-04T11:13:50.3927765Z plugins: hypothesis-6.56.4, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-14.0, subtests-0.13.1, xdist-3.3.1, xdoctest-1.3.0, typeguard-4.3.0 2025-12-04T11:13:50.3927892Z collecting ... collected 58 items / 57 deselected / 1 selected 2025-12-04T11:13:50.3928516Z stepcurrent: skipping 42 already run items. Running only test/inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_32_mid_dim_1_in_features_1024_out_features_65_cuda_bfloat16 2025-12-04T11:13:50.3928589Z Running 1 items in this shard 2025-12-04T11:13:50.3928592Z 2025-12-04T11:13:50.3929331Z inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_32_mid_dim_1_in_features_1024_out_features_65_cuda_bfloat16 [W1204 11:01:15.643175396 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3929369Z 2025-12-04T11:13:50.3929671Z [W1204 11:01:24.750807086 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3929674Z 2025-12-04T11:13:50.3929967Z [W1204 11:01:24.751062601 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3929972Z 2025-12-04T11:13:50.3930256Z [W1204 11:01:24.757006183 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3930261Z 2025-12-04T11:13:50.3930547Z [W1204 11:01:24.757641737 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3930554Z 2025-12-04T11:13:50.3930840Z [W1204 11:01:24.757823781 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3930844Z 2025-12-04T11:13:50.3931131Z [W1204 11:01:24.763452235 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3931134Z 2025-12-04T11:13:50.3931421Z [W1204 11:01:24.764006528 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3931425Z 2025-12-04T11:13:50.3931710Z [W1204 11:01:24.764161781 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3931715Z 2025-12-04T11:13:50.3931800Z ('RERUN', {'yellow': True}) [11.0772s] [100%] 2025-12-04T11:13:50.3932518Z inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_32_mid_dim_1_in_features_1024_out_features_65_cuda_bfloat16 [W1204 11:01:25.572881550 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3932524Z 2025-12-04T11:13:50.3932816Z [W1204 11:01:25.573395241 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3932819Z 2025-12-04T11:13:50.3933106Z [W1204 11:01:25.573536235 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3933110Z 2025-12-04T11:13:50.3933401Z [W1204 11:01:25.576443739 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3933443Z 2025-12-04T11:13:50.3933761Z [W1204 11:01:25.576911179 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3933765Z 2025-12-04T11:13:50.3934062Z [W1204 11:01:25.577047132 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3934070Z 2025-12-04T11:13:50.3934356Z [W1204 11:01:25.581653294 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3934360Z 2025-12-04T11:13:50.3934644Z [W1204 11:01:25.582123054 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3934648Z 2025-12-04T11:13:50.3934969Z [W1204 11:01:25.582258348 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3934973Z 2025-12-04T11:13:50.3935052Z ('RERUN', {'yellow': True}) [0.5019s] [100%] 2025-12-04T11:13:50.3935777Z inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_32_mid_dim_1_in_features_1024_out_features_65_cuda_bfloat16 [W1204 11:01:26.071914461 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3935814Z 2025-12-04T11:13:50.3936102Z [W1204 11:01:26.072428762 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3936106Z 2025-12-04T11:13:50.3936393Z [W1204 11:01:26.072571775 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3936397Z 2025-12-04T11:13:50.3936682Z [W1204 11:01:26.075449099 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3936686Z 2025-12-04T11:13:50.3936974Z [W1204 11:01:26.075913489 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3936978Z 2025-12-04T11:13:50.3937262Z [W1204 11:01:26.076050872 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3937266Z 2025-12-04T11:13:50.3937553Z [W1204 11:01:26.080657254 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3937556Z 2025-12-04T11:13:50.3937842Z [W1204 11:01:26.081136054 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3937846Z 2025-12-04T11:13:50.3938132Z [W1204 11:01:26.081272077 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3938135Z 2025-12-04T11:13:50.3938201Z FAILED [0.4966s] [100%] 2025-12-04T11:13:50.3938205Z 2025-12-04T11:13:50.3938288Z ==================================== RERUNS ==================================== 2025-12-04T11:13:50.3938586Z _ TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_1_in_features_1024_out_features_65_cuda_bfloat16 _ 2025-12-04T11:13:50.3938662Z Traceback (most recent call last): 2025-12-04T11:13:50.3938981Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.3939048Z method(*args, **kwargs) 2025-12-04T11:13:50.3939345Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.3939408Z method(*args, **kwargs) 2025-12-04T11:13:50.3939699Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T11:13:50.3939759Z with policy(): 2025-12-04T11:13:50.3940098Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T11:13:50.3940198Z raise RuntimeError(msg) 2025-12-04T11:13:50.3941000Z RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_1_in_features_1024_out_features_65_cuda_bfloat16! Caching allocator allocated memory was 0 and is now reported as 66560 on device 0. CUDA driver allocated memory was 230686720 and is now 274726912. 2025-12-04T11:13:50.3941006Z 2025-12-04T11:13:50.3941135Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.3941684Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_1_in_features_1024_out_features_65_cuda_bfloat16 2025-12-04T11:13:50.3941688Z 2025-12-04T11:13:50.3941846Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.3941974Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.3942068Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.3942613Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('benchmarking.InductorBenchmarker.benchmark_gpu', 2), ('pad_mm_bench', 1), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.3942790Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.3942854Z graph_break [] 2025-12-04T11:13:50.3942977Z ----------------------------- Captured stderr call ----------------------------- 2025-12-04T11:13:50.3943709Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/constant_folding.py:256: UserWarning: Unsupported unwinding pattern: Address not in range (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/profiler/unwind/unwind.cpp:219.) 2025-12-04T11:13:50.3943783Z if out == self.unknown_value: 2025-12-04T11:13:50.3944075Z _ TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_1_in_features_1024_out_features_65_cuda_bfloat16 _ 2025-12-04T11:13:50.3944162Z Traceback (most recent call last): 2025-12-04T11:13:50.3944457Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.3944520Z method(*args, **kwargs) 2025-12-04T11:13:50.3944808Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.3944872Z method(*args, **kwargs) 2025-12-04T11:13:50.3945160Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T11:13:50.3945220Z with policy(): 2025-12-04T11:13:50.3945510Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T11:13:50.3945581Z raise RuntimeError(msg) 2025-12-04T11:13:50.3946389Z RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_1_in_features_1024_out_features_65_cuda_bfloat16! Caching allocator allocated memory was 66560 and is now reported as 133120 on device 0. CUDA driver allocated memory was 274726912 and is now 276824064. 2025-12-04T11:13:50.3946394Z 2025-12-04T11:13:50.3946523Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.3947041Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_1_in_features_1024_out_features_65_cuda_bfloat16 2025-12-04T11:13:50.3947044Z 2025-12-04T11:13:50.3947202Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.3947332Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.3947557Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.3948139Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('benchmarking.InductorBenchmarker.benchmark_gpu', 2), ('pad_mm_bench', 1), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.3948269Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.3948328Z graph_break [] 2025-12-04T11:13:50.3948454Z ----------------------------- Captured stderr call ----------------------------- 2025-12-04T11:13:50.3949175Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/constant_folding.py:256: UserWarning: Unsupported unwinding pattern: Address not in range (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/profiler/unwind/unwind.cpp:219.) 2025-12-04T11:13:50.3949252Z if out == self.unknown_value: 2025-12-04T11:13:50.3949375Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.3949466Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.3949592Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.3950167Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('benchmarking.InductorBenchmarker.benchmark_gpu', 2), ('pad_mm_bench', 1), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.3950227Z graph_break [] 2025-12-04T11:13:50.3950310Z =================================== FAILURES =================================== 2025-12-04T11:13:50.3950614Z _ TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_1_in_features_1024_out_features_65_cuda_bfloat16 _ 2025-12-04T11:13:50.3950694Z Traceback (most recent call last): 2025-12-04T11:13:50.3950989Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.3951053Z method(*args, **kwargs) 2025-12-04T11:13:50.3951345Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.3951408Z method(*args, **kwargs) 2025-12-04T11:13:50.3951696Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T11:13:50.3951753Z with policy(): 2025-12-04T11:13:50.3952044Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T11:13:50.3952113Z raise RuntimeError(msg) 2025-12-04T11:13:50.3952933Z RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_1_in_features_1024_out_features_65_cuda_bfloat16! Caching allocator allocated memory was 133120 and is now reported as 199680 on device 0. CUDA driver allocated memory was 276824064 and is now 278921216. 2025-12-04T11:13:50.3952939Z 2025-12-04T11:13:50.3953067Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.3953588Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_1_in_features_1024_out_features_65_cuda_bfloat16 2025-12-04T11:13:50.3953593Z 2025-12-04T11:13:50.3953749Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.3953876Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.3953965Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.3954507Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('benchmarking.InductorBenchmarker.benchmark_gpu', 2), ('pad_mm_bench', 1), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.3954673Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.3954732Z graph_break [] 2025-12-04T11:13:50.3954911Z ----------------------------- Captured stderr call ----------------------------- 2025-12-04T11:13:50.3955605Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/constant_folding.py:256: UserWarning: Unsupported unwinding pattern: Address not in range (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/profiler/unwind/unwind.cpp:219.) 2025-12-04T11:13:50.3955676Z if out == self.unknown_value: 2025-12-04T11:13:50.3955800Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.3955890Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.3956048Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.3956586Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('benchmarking.InductorBenchmarker.benchmark_gpu', 2), ('pad_mm_bench', 1), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.3956648Z graph_break [] 2025-12-04T11:13:50.3956770Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.3956892Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.3957014Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.3957546Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('benchmarking.InductorBenchmarker.benchmark_gpu', 2), ('pad_mm_bench', 1), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.3957605Z graph_break [] 2025-12-04T11:13:50.3958098Z - generated xml file: /var/lib/jenkins/workspace/test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-0022ebcfefe802db.xml - 2025-12-04T11:13:50.3958195Z =========================== short test summary info ============================ 2025-12-04T11:13:50.3959495Z FAILED [0.4966s] inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_32_mid_dim_1_in_features_1024_out_features_65_cuda_bfloat16 - RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_1_in_features_1024_out_features_65_cuda_bfloat16! Caching allocator allocated memory was 133120 and is now reported as 199680 on device 0. CUDA driver allocated memory was 276824064 and is now 278921216. 2025-12-04T11:13:50.3959500Z 2025-12-04T11:13:50.3959625Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.3960145Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_1_in_features_1024_out_features_65_cuda_bfloat16 2025-12-04T11:13:50.3960150Z 2025-12-04T11:13:50.3960304Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.3960413Z !!!!!!!!!!!!!!!!!!!!!!!!!! stopping after 1 failures !!!!!!!!!!!!!!!!!!!!!!!!!!! 2025-12-04T11:13:50.3960542Z ================== 1 failed, 57 deselected, 2 rerun in 12.10s ================== 2025-12-04T11:13:50.3960602Z Got exit code 1 2025-12-04T11:13:50.3960670Z Retrying single test... 2025-12-04T11:13:50.3960932Z W1204 11:01:33.047000 80614 site-packages/torch/_inductor/utils.py:1703] Not enough SMs to use max_autotune_gemm mode 2025-12-04T11:13:50.3961319Z Test results will be stored in test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-dac6db8ed6c97e37.xml 2025-12-04T11:13:50.3961415Z ============================= test session starts ============================== 2025-12-04T11:13:50.3961622Z platform linux -- Python 3.10.14, pytest-7.3.2, pluggy-1.6.0 -- /opt/conda/envs/py_3.10/bin/python 2025-12-04T11:13:50.3961691Z cachedir: .pytest_cache 2025-12-04T11:13:50.3962040Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow] 2025-12-04T11:13:50.3962149Z rootdir: /var/lib/jenkins/workspace 2025-12-04T11:13:50.3962221Z configfile: pytest.ini 2025-12-04T11:13:50.3962536Z plugins: hypothesis-6.56.4, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-14.0, subtests-0.13.1, xdist-3.3.1, xdoctest-1.3.0, typeguard-4.3.0 2025-12-04T11:13:50.3962665Z collecting ... collected 58 items / 57 deselected / 1 selected 2025-12-04T11:13:50.3963234Z stepcurrent: skipping 42 already run items. Running only test/inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_32_mid_dim_1_in_features_1024_out_features_65_cuda_bfloat16 2025-12-04T11:13:50.3963338Z Running 1 items in this shard 2025-12-04T11:13:50.3963342Z 2025-12-04T11:13:50.3964073Z inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_32_mid_dim_1_in_features_1024_out_features_65_cuda_bfloat16 [W1204 11:01:34.692152667 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3964079Z 2025-12-04T11:13:50.3964376Z [W1204 11:01:43.804247538 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3964421Z 2025-12-04T11:13:50.3964718Z [W1204 11:01:43.804491113 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3964722Z 2025-12-04T11:13:50.3965014Z [W1204 11:01:43.810753152 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3965018Z 2025-12-04T11:13:50.3965311Z [W1204 11:01:43.811330234 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3965314Z 2025-12-04T11:13:50.3965600Z [W1204 11:01:43.811507478 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3965605Z 2025-12-04T11:13:50.3965890Z [W1204 11:01:43.816917528 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3965898Z 2025-12-04T11:13:50.3966182Z [W1204 11:01:43.817444300 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3966185Z 2025-12-04T11:13:50.3966470Z [W1204 11:01:43.817600273 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3966473Z 2025-12-04T11:13:50.3966556Z ('RERUN', {'yellow': True}) [11.1006s] [100%] 2025-12-04T11:13:50.3967277Z inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_32_mid_dim_1_in_features_1024_out_features_65_cuda_bfloat16 [W1204 11:01:44.633614196 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3967284Z 2025-12-04T11:13:50.3967574Z [W1204 11:01:44.634139197 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3967579Z 2025-12-04T11:13:50.3967863Z [W1204 11:01:44.634280621 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3967866Z 2025-12-04T11:13:50.3968155Z [W1204 11:01:44.637273536 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3968158Z 2025-12-04T11:13:50.3968446Z [W1204 11:01:44.637741126 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3968450Z 2025-12-04T11:13:50.3968780Z [W1204 11:01:44.637878489 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3968784Z 2025-12-04T11:13:50.3969103Z [W1204 11:01:44.642583852 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3969109Z 2025-12-04T11:13:50.3969403Z [W1204 11:01:44.643063293 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3969414Z 2025-12-04T11:13:50.3969700Z [W1204 11:01:44.643198136 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3969703Z 2025-12-04T11:13:50.3969780Z ('RERUN', {'yellow': True}) [0.5106s] [100%] 2025-12-04T11:13:50.3970541Z inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_32_mid_dim_1_in_features_1024_out_features_65_cuda_bfloat16 [W1204 11:01:45.142609324 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3970548Z 2025-12-04T11:13:50.3970835Z [W1204 11:01:45.143136836 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3970872Z 2025-12-04T11:13:50.3971162Z [W1204 11:01:45.143272809 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3971165Z 2025-12-04T11:13:50.3971448Z [W1204 11:01:45.146231043 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3971451Z 2025-12-04T11:13:50.3971742Z [W1204 11:01:45.146695624 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3971746Z 2025-12-04T11:13:50.3972030Z [W1204 11:01:45.146831547 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3972034Z 2025-12-04T11:13:50.3972322Z [W1204 11:01:45.151640112 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3972330Z 2025-12-04T11:13:50.3972616Z [W1204 11:01:45.152118132 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3972619Z 2025-12-04T11:13:50.3972906Z [W1204 11:01:45.152256985 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.3972909Z 2025-12-04T11:13:50.3972975Z FAILED [0.5090s] [100%] 2025-12-04T11:13:50.3972979Z 2025-12-04T11:13:50.3973075Z ==================================== RERUNS ==================================== 2025-12-04T11:13:50.3973372Z _ TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_1_in_features_1024_out_features_65_cuda_bfloat16 _ 2025-12-04T11:13:50.3973448Z Traceback (most recent call last): 2025-12-04T11:13:50.3973747Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.3973817Z method(*args, **kwargs) 2025-12-04T11:13:50.3974109Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.3974175Z method(*args, **kwargs) 2025-12-04T11:13:50.3974462Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T11:13:50.3974525Z with policy(): 2025-12-04T11:13:50.3974820Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T11:13:50.3974885Z raise RuntimeError(msg) 2025-12-04T11:13:50.3975720Z RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_1_in_features_1024_out_features_65_cuda_bfloat16! Caching allocator allocated memory was 0 and is now reported as 66560 on device 0. CUDA driver allocated memory was 230686720 and is now 274726912. 2025-12-04T11:13:50.3975766Z 2025-12-04T11:13:50.3975893Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.3976407Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_1_in_features_1024_out_features_65_cuda_bfloat16 2025-12-04T11:13:50.3976411Z 2025-12-04T11:13:50.3976574Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.3976733Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.3976828Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.3977374Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('benchmarking.InductorBenchmarker.benchmark_gpu', 2), ('pad_mm_bench', 1), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.3977536Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.3977598Z graph_break [] 2025-12-04T11:13:50.3977724Z ----------------------------- Captured stderr call ----------------------------- 2025-12-04T11:13:50.3978566Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/constant_folding.py:256: UserWarning: Unsupported unwinding pattern: Address not in range (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/profiler/unwind/unwind.cpp:219.) 2025-12-04T11:13:50.3978637Z if out == self.unknown_value: 2025-12-04T11:13:50.3978929Z _ TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_1_in_features_1024_out_features_65_cuda_bfloat16 _ 2025-12-04T11:13:50.3979003Z Traceback (most recent call last): 2025-12-04T11:13:50.3979299Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.3979362Z method(*args, **kwargs) 2025-12-04T11:13:50.3979656Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.3979719Z method(*args, **kwargs) 2025-12-04T11:13:50.3980008Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T11:13:50.3980080Z with policy(): 2025-12-04T11:13:50.3980371Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T11:13:50.3980442Z raise RuntimeError(msg) 2025-12-04T11:13:50.3981248Z RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_1_in_features_1024_out_features_65_cuda_bfloat16! Caching allocator allocated memory was 66560 and is now reported as 133120 on device 0. CUDA driver allocated memory was 274726912 and is now 276824064. 2025-12-04T11:13:50.3981254Z 2025-12-04T11:13:50.3981382Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.3981897Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_1_in_features_1024_out_features_65_cuda_bfloat16 2025-12-04T11:13:50.3981901Z 2025-12-04T11:13:50.3982060Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.3982188Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.3982284Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.3982825Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('benchmarking.InductorBenchmarker.benchmark_gpu', 2), ('pad_mm_bench', 1), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.3983082Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.3983146Z graph_break [] 2025-12-04T11:13:50.3983278Z ----------------------------- Captured stderr call ----------------------------- 2025-12-04T11:13:50.3984016Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/constant_folding.py:256: UserWarning: Unsupported unwinding pattern: Address not in range (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/profiler/unwind/unwind.cpp:219.) 2025-12-04T11:13:50.3984088Z if out == self.unknown_value: 2025-12-04T11:13:50.3984264Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.3984360Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.3984486Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.3985028Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('benchmarking.InductorBenchmarker.benchmark_gpu', 2), ('pad_mm_bench', 1), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.3985137Z graph_break [] 2025-12-04T11:13:50.3985221Z =================================== FAILURES =================================== 2025-12-04T11:13:50.3985512Z _ TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_1_in_features_1024_out_features_65_cuda_bfloat16 _ 2025-12-04T11:13:50.3985587Z Traceback (most recent call last): 2025-12-04T11:13:50.3985881Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.3985946Z method(*args, **kwargs) 2025-12-04T11:13:50.3986238Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.3986302Z method(*args, **kwargs) 2025-12-04T11:13:50.3986592Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T11:13:50.3986650Z with policy(): 2025-12-04T11:13:50.3986942Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T11:13:50.3987011Z raise RuntimeError(msg) 2025-12-04T11:13:50.3987819Z RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_1_in_features_1024_out_features_65_cuda_bfloat16! Caching allocator allocated memory was 133120 and is now reported as 199680 on device 0. CUDA driver allocated memory was 276824064 and is now 278921216. 2025-12-04T11:13:50.3987825Z 2025-12-04T11:13:50.3987952Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.3988465Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_1_in_features_1024_out_features_65_cuda_bfloat16 2025-12-04T11:13:50.3988471Z 2025-12-04T11:13:50.3988627Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.3988753Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.3988843Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.3989382Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('benchmarking.InductorBenchmarker.benchmark_gpu', 2), ('pad_mm_bench', 1), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.3989508Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.3989568Z graph_break [] 2025-12-04T11:13:50.3989700Z ----------------------------- Captured stderr call ----------------------------- 2025-12-04T11:13:50.3990458Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/constant_folding.py:256: UserWarning: Unsupported unwinding pattern: Address not in range (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/profiler/unwind/unwind.cpp:219.) 2025-12-04T11:13:50.3990533Z if out == self.unknown_value: 2025-12-04T11:13:50.3990657Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.3990745Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.3990869Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.3991442Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('benchmarking.InductorBenchmarker.benchmark_gpu', 2), ('pad_mm_bench', 1), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.3991506Z graph_break [] 2025-12-04T11:13:50.3991628Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.3991717Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.3991843Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.3992373Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('benchmarking.InductorBenchmarker.benchmark_gpu', 2), ('pad_mm_bench', 1), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.3992468Z graph_break [] 2025-12-04T11:13:50.3992958Z - generated xml file: /var/lib/jenkins/workspace/test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-dac6db8ed6c97e37.xml - 2025-12-04T11:13:50.3993057Z =========================== short test summary info ============================ 2025-12-04T11:13:50.3994351Z FAILED [0.5090s] inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_32_mid_dim_1_in_features_1024_out_features_65_cuda_bfloat16 - RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_1_in_features_1024_out_features_65_cuda_bfloat16! Caching allocator allocated memory was 133120 and is now reported as 199680 on device 0. CUDA driver allocated memory was 276824064 and is now 278921216. 2025-12-04T11:13:50.3994358Z 2025-12-04T11:13:50.3994482Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.3995003Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_1_in_features_1024_out_features_65_cuda_bfloat16 2025-12-04T11:13:50.3995007Z 2025-12-04T11:13:50.3995162Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.3995269Z !!!!!!!!!!!!!!!!!!!!!!!!!! stopping after 1 failures !!!!!!!!!!!!!!!!!!!!!!!!!!! 2025-12-04T11:13:50.3995384Z ================== 1 failed, 57 deselected, 2 rerun in 12.15s ================== 2025-12-04T11:13:50.3995444Z Got exit code 1 2025-12-04T11:13:50.3995919Z FAILED CONSISTENTLY: test/inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_32_mid_dim_1_in_features_1024_out_features_65_cuda_bfloat16 2025-12-04T11:13:50.3996161Z Test failed consistently, continuing with the rest of the tests due to continue-through-error being set 2025-12-04T11:13:50.3996424Z W1204 11:01:52.038000 80808 site-packages/torch/_inductor/utils.py:1703] Not enough SMs to use max_autotune_gemm mode 2025-12-04T11:13:50.3996808Z Test results will be stored in test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-040b4092def69522.xml 2025-12-04T11:13:50.3996906Z ============================= test session starts ============================== 2025-12-04T11:13:50.3997114Z platform linux -- Python 3.10.14, pytest-7.3.2, pluggy-1.6.0 -- /opt/conda/envs/py_3.10/bin/python 2025-12-04T11:13:50.3997222Z cachedir: .pytest_cache 2025-12-04T11:13:50.3997562Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow] 2025-12-04T11:13:50.3997641Z rootdir: /var/lib/jenkins/workspace 2025-12-04T11:13:50.3997707Z configfile: pytest.ini 2025-12-04T11:13:50.3998023Z plugins: hypothesis-6.56.4, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-14.0, subtests-0.13.1, xdist-3.3.1, xdoctest-1.3.0, typeguard-4.3.0 2025-12-04T11:13:50.3998154Z collecting ... collected 58 items / 43 deselected / 15 selected 2025-12-04T11:13:50.3998240Z stepcurrent: skipping 43 already run items. 2025-12-04T11:13:50.3998315Z Running 15 items in this shard 2025-12-04T11:13:50.3998319Z 2025-12-04T11:13:50.3998859Z inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_32_mid_dim_1_in_features_128_out_features_1024_cuda_bfloat16 ('RERUN', {'yellow': True}) [1.9134s] [ 6%] 2025-12-04T11:13:50.3999357Z inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_32_mid_dim_1_in_features_128_out_features_1024_cuda_bfloat16 ('RERUN', {'yellow': True}) [0.4942s] [ 6%] 2025-12-04T11:13:50.3999802Z inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_32_mid_dim_1_in_features_128_out_features_1024_cuda_bfloat16 FAILED [0.4988s] [ 6%] 2025-12-04T11:13:50.3999840Z 2025-12-04T11:13:50.3999941Z ==================================== RERUNS ==================================== 2025-12-04T11:13:50.4000240Z _ TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_1_in_features_128_out_features_1024_cuda_bfloat16 _ 2025-12-04T11:13:50.4000315Z Traceback (most recent call last): 2025-12-04T11:13:50.4000623Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.4000690Z method(*args, **kwargs) 2025-12-04T11:13:50.4000980Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.4001054Z method(*args, **kwargs) 2025-12-04T11:13:50.4001344Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T11:13:50.4001408Z with policy(): 2025-12-04T11:13:50.4001700Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T11:13:50.4001764Z raise RuntimeError(msg) 2025-12-04T11:13:50.4002575Z RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_1_in_features_128_out_features_1024_cuda_bfloat16! Caching allocator allocated memory was 0 and is now reported as 131072 on device 0. CUDA driver allocated memory was 230686720 and is now 274726912. 2025-12-04T11:13:50.4002579Z 2025-12-04T11:13:50.4002706Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.4003231Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_1_in_features_128_out_features_1024_cuda_bfloat16 2025-12-04T11:13:50.4003237Z 2025-12-04T11:13:50.4003399Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.4003526Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.4003633Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.4003985Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.4004120Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.4004180Z graph_break [] 2025-12-04T11:13:50.4004471Z _ TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_1_in_features_128_out_features_1024_cuda_bfloat16 _ 2025-12-04T11:13:50.4004590Z Traceback (most recent call last): 2025-12-04T11:13:50.4004925Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.4004994Z method(*args, **kwargs) 2025-12-04T11:13:50.4005282Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.4005346Z method(*args, **kwargs) 2025-12-04T11:13:50.4005636Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T11:13:50.4005694Z with policy(): 2025-12-04T11:13:50.4006019Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T11:13:50.4006091Z raise RuntimeError(msg) 2025-12-04T11:13:50.4006902Z RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_1_in_features_128_out_features_1024_cuda_bfloat16! Caching allocator allocated memory was 131072 and is now reported as 262144 on device 0. CUDA driver allocated memory was 274726912 and is now 276824064. 2025-12-04T11:13:50.4006958Z 2025-12-04T11:13:50.4007086Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.4007601Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_1_in_features_128_out_features_1024_cuda_bfloat16 2025-12-04T11:13:50.4007606Z 2025-12-04T11:13:50.4007767Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.4007896Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.4007988Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.4008346Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.4008473Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.4008534Z graph_break [] 2025-12-04T11:13:50.4008663Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.4008753Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.4008889Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.4009233Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.4009295Z graph_break [] 2025-12-04T11:13:50.4009382Z =================================== FAILURES =================================== 2025-12-04T11:13:50.4009672Z _ TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_1_in_features_128_out_features_1024_cuda_bfloat16 _ 2025-12-04T11:13:50.4009749Z Traceback (most recent call last): 2025-12-04T11:13:50.4010048Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.4010115Z method(*args, **kwargs) 2025-12-04T11:13:50.4010406Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.4010470Z method(*args, **kwargs) 2025-12-04T11:13:50.4010757Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T11:13:50.4010820Z with policy(): 2025-12-04T11:13:50.4011110Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T11:13:50.4011179Z raise RuntimeError(msg) 2025-12-04T11:13:50.4012075Z RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_1_in_features_128_out_features_1024_cuda_bfloat16! Caching allocator allocated memory was 262144 and is now reported as 393216 on device 0. CUDA driver allocated memory was 276824064 and is now 278921216. 2025-12-04T11:13:50.4012082Z 2025-12-04T11:13:50.4012208Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.4012728Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_1_in_features_128_out_features_1024_cuda_bfloat16 2025-12-04T11:13:50.4012733Z 2025-12-04T11:13:50.4012887Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.4013048Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.4013138Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.4013480Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.4013609Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.4013702Z graph_break [] 2025-12-04T11:13:50.4013828Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.4013916Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.4014038Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.4014381Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.4014453Z graph_break [] 2025-12-04T11:13:50.4014579Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.4014674Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.4014795Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.4015140Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.4015199Z graph_break [] 2025-12-04T11:13:50.4015689Z - generated xml file: /var/lib/jenkins/workspace/test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-040b4092def69522.xml - 2025-12-04T11:13:50.4015793Z =========================== short test summary info ============================ 2025-12-04T11:13:50.4017082Z FAILED [0.4988s] inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_32_mid_dim_1_in_features_128_out_features_1024_cuda_bfloat16 - RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_1_in_features_128_out_features_1024_cuda_bfloat16! Caching allocator allocated memory was 262144 and is now reported as 393216 on device 0. CUDA driver allocated memory was 276824064 and is now 278921216. 2025-12-04T11:13:50.4017090Z 2025-12-04T11:13:50.4017215Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.4017730Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_1_in_features_128_out_features_1024_cuda_bfloat16 2025-12-04T11:13:50.4017733Z 2025-12-04T11:13:50.4017892Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.4017997Z !!!!!!!!!!!!!!!!!!!!!!!!!! stopping after 1 failures !!!!!!!!!!!!!!!!!!!!!!!!!!! 2025-12-04T11:13:50.4018112Z ================== 1 failed, 43 deselected, 2 rerun in 2.93s =================== 2025-12-04T11:13:50.4018174Z Got exit code 1 2025-12-04T11:13:50.4018280Z Retrying single test... 2025-12-04T11:13:50.4018546Z W1204 11:02:02.068000 80997 site-packages/torch/_inductor/utils.py:1703] Not enough SMs to use max_autotune_gemm mode 2025-12-04T11:13:50.4018963Z Test results will be stored in test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-084397a772e6c7f7.xml 2025-12-04T11:13:50.4019062Z ============================= test session starts ============================== 2025-12-04T11:13:50.4019276Z platform linux -- Python 3.10.14, pytest-7.3.2, pluggy-1.6.0 -- /opt/conda/envs/py_3.10/bin/python 2025-12-04T11:13:50.4019342Z cachedir: .pytest_cache 2025-12-04T11:13:50.4019645Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow] 2025-12-04T11:13:50.4019760Z rootdir: /var/lib/jenkins/workspace 2025-12-04T11:13:50.4019826Z configfile: pytest.ini 2025-12-04T11:13:50.4020140Z plugins: hypothesis-6.56.4, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-14.0, subtests-0.13.1, xdist-3.3.1, xdoctest-1.3.0, typeguard-4.3.0 2025-12-04T11:13:50.4020272Z collecting ... collected 58 items / 57 deselected / 1 selected 2025-12-04T11:13:50.4020840Z stepcurrent: skipping 43 already run items. Running only test/inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_32_mid_dim_1_in_features_128_out_features_1024_cuda_bfloat16 2025-12-04T11:13:50.4020960Z Running 1 items in this shard 2025-12-04T11:13:50.4020964Z 2025-12-04T11:13:50.4021695Z inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_32_mid_dim_1_in_features_128_out_features_1024_cuda_bfloat16 [W1204 11:02:03.183058547 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.4021701Z 2025-12-04T11:13:50.4022001Z [W1204 11:02:12.293574240 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.4022006Z 2025-12-04T11:13:50.4022297Z [W1204 11:02:12.293807835 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.4022301Z 2025-12-04T11:13:50.4022592Z [W1204 11:02:12.299619424 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.4022596Z 2025-12-04T11:13:50.4022885Z [W1204 11:02:12.300198307 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.4022889Z 2025-12-04T11:13:50.4023180Z [W1204 11:02:12.300372430 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.4023184Z 2025-12-04T11:13:50.4023471Z [W1204 11:02:12.305716959 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.4023476Z 2025-12-04T11:13:50.4023841Z [W1204 11:02:12.306236990 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.4023847Z 2025-12-04T11:13:50.4024134Z [W1204 11:02:12.306388404 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.4024140Z 2025-12-04T11:13:50.4024220Z ('RERUN', {'yellow': True}) [11.0232s] [100%] 2025-12-04T11:13:50.4024953Z inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_32_mid_dim_1_in_features_128_out_features_1024_cuda_bfloat16 [W1204 11:02:13.526969345 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.4024957Z 2025-12-04T11:13:50.4025246Z [W1204 11:02:13.527509677 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.4025292Z 2025-12-04T11:13:50.4025628Z [W1204 11:02:13.527649490 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.4025632Z 2025-12-04T11:13:50.4025920Z [W1204 11:02:13.530610135 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.4025925Z 2025-12-04T11:13:50.4026216Z [W1204 11:02:13.531174868 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.4026219Z 2025-12-04T11:13:50.4026502Z [W1204 11:02:13.531312181 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.4026505Z 2025-12-04T11:13:50.4026822Z [W1204 11:02:13.535824140 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.4026826Z 2025-12-04T11:13:50.4027112Z [W1204 11:02:13.536288340 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.4027117Z 2025-12-04T11:13:50.4027404Z [W1204 11:02:13.536421593 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.4027441Z 2025-12-04T11:13:50.4027524Z ('RERUN', {'yellow': True}) [0.4621s] [100%] 2025-12-04T11:13:50.4028245Z inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_32_mid_dim_1_in_features_128_out_features_1024_cuda_bfloat16 [W1204 11:02:14.986408033 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.4028249Z 2025-12-04T11:13:50.4028541Z [W1204 11:02:14.986945614 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.4028544Z 2025-12-04T11:13:50.4028830Z [W1204 11:02:14.987083888 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.4028835Z 2025-12-04T11:13:50.4029122Z [W1204 11:02:14.990019842 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.4029127Z 2025-12-04T11:13:50.4029411Z [W1204 11:02:14.990587535 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.4029415Z 2025-12-04T11:13:50.4029700Z [W1204 11:02:14.990723738 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.4029703Z 2025-12-04T11:13:50.4029988Z [W1204 11:02:14.995243778 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.4029991Z 2025-12-04T11:13:50.4030277Z [W1204 11:02:14.995706628 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.4030286Z 2025-12-04T11:13:50.4030575Z [W1204 11:02:14.995840571 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.4030579Z 2025-12-04T11:13:50.4030641Z FAILED [0.4540s] [100%] 2025-12-04T11:13:50.4030645Z 2025-12-04T11:13:50.4030735Z ==================================== RERUNS ==================================== 2025-12-04T11:13:50.4031029Z _ TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_1_in_features_128_out_features_1024_cuda_bfloat16 _ 2025-12-04T11:13:50.4031105Z Traceback (most recent call last): 2025-12-04T11:13:50.4031408Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.4031472Z method(*args, **kwargs) 2025-12-04T11:13:50.4031764Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.4031901Z method(*args, **kwargs) 2025-12-04T11:13:50.4032191Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T11:13:50.4032254Z with policy(): 2025-12-04T11:13:50.4032556Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T11:13:50.4032627Z raise RuntimeError(msg) 2025-12-04T11:13:50.4033497Z RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_1_in_features_128_out_features_1024_cuda_bfloat16! Caching allocator allocated memory was 0 and is now reported as 131072 on device 0. CUDA driver allocated memory was 230686720 and is now 274726912. 2025-12-04T11:13:50.4033502Z 2025-12-04T11:13:50.4033632Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.4034156Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_1_in_features_128_out_features_1024_cuda_bfloat16 2025-12-04T11:13:50.4034193Z 2025-12-04T11:13:50.4034352Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.4034483Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.4034579Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.4034926Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.4035058Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.4035117Z graph_break [] 2025-12-04T11:13:50.4035245Z ----------------------------- Captured stderr call ----------------------------- 2025-12-04T11:13:50.4035934Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/constant_folding.py:256: UserWarning: Unsupported unwinding pattern: Address not in range (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/profiler/unwind/unwind.cpp:219.) 2025-12-04T11:13:50.4036007Z if out == self.unknown_value: 2025-12-04T11:13:50.4036303Z _ TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_1_in_features_128_out_features_1024_cuda_bfloat16 _ 2025-12-04T11:13:50.4036377Z Traceback (most recent call last): 2025-12-04T11:13:50.4036677Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.4036741Z method(*args, **kwargs) 2025-12-04T11:13:50.4037031Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.4037095Z method(*args, **kwargs) 2025-12-04T11:13:50.4037383Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T11:13:50.4037448Z with policy(): 2025-12-04T11:13:50.4037739Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T11:13:50.4037807Z raise RuntimeError(msg) 2025-12-04T11:13:50.4038626Z RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_1_in_features_128_out_features_1024_cuda_bfloat16! Caching allocator allocated memory was 131072 and is now reported as 262144 on device 0. CUDA driver allocated memory was 274726912 and is now 276824064. 2025-12-04T11:13:50.4038630Z 2025-12-04T11:13:50.4038757Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.4039285Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_1_in_features_128_out_features_1024_cuda_bfloat16 2025-12-04T11:13:50.4039328Z 2025-12-04T11:13:50.4039520Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.4039649Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.4039744Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.4040093Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.4040221Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.4040281Z graph_break [] 2025-12-04T11:13:50.4040437Z ----------------------------- Captured stderr call ----------------------------- 2025-12-04T11:13:50.4041126Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/constant_folding.py:256: UserWarning: Unsupported unwinding pattern: Address not in range (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/profiler/unwind/unwind.cpp:219.) 2025-12-04T11:13:50.4041202Z if out == self.unknown_value: 2025-12-04T11:13:50.4041324Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.4041454Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.4041580Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.4041924Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.4041992Z graph_break [] 2025-12-04T11:13:50.4042079Z =================================== FAILURES =================================== 2025-12-04T11:13:50.4042376Z _ TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_1_in_features_128_out_features_1024_cuda_bfloat16 _ 2025-12-04T11:13:50.4042453Z Traceback (most recent call last): 2025-12-04T11:13:50.4042751Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.4042821Z method(*args, **kwargs) 2025-12-04T11:13:50.4043108Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.4043173Z method(*args, **kwargs) 2025-12-04T11:13:50.4043460Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T11:13:50.4043519Z with policy(): 2025-12-04T11:13:50.4043811Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T11:13:50.4043878Z raise RuntimeError(msg) 2025-12-04T11:13:50.4044697Z RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_1_in_features_128_out_features_1024_cuda_bfloat16! Caching allocator allocated memory was 262144 and is now reported as 393216 on device 0. CUDA driver allocated memory was 276824064 and is now 278921216. 2025-12-04T11:13:50.4044704Z 2025-12-04T11:13:50.4044829Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.4045348Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_1_in_features_128_out_features_1024_cuda_bfloat16 2025-12-04T11:13:50.4045351Z 2025-12-04T11:13:50.4045508Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.4045632Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.4045730Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.4046072Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.4046286Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.4046346Z graph_break [] 2025-12-04T11:13:50.4046472Z ----------------------------- Captured stderr call ----------------------------- 2025-12-04T11:13:50.4047154Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/constant_folding.py:256: UserWarning: Unsupported unwinding pattern: Address not in range (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/profiler/unwind/unwind.cpp:219.) 2025-12-04T11:13:50.4047224Z if out == self.unknown_value: 2025-12-04T11:13:50.4047347Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.4047473Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.4047599Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.4047944Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.4048005Z graph_break [] 2025-12-04T11:13:50.4048131Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.4048260Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.4048381Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.4048719Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.4048782Z graph_break [] 2025-12-04T11:13:50.4049266Z - generated xml file: /var/lib/jenkins/workspace/test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-084397a772e6c7f7.xml - 2025-12-04T11:13:50.4049378Z =========================== short test summary info ============================ 2025-12-04T11:13:50.4050665Z FAILED [0.4540s] inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_32_mid_dim_1_in_features_128_out_features_1024_cuda_bfloat16 - RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_1_in_features_128_out_features_1024_cuda_bfloat16! Caching allocator allocated memory was 262144 and is now reported as 393216 on device 0. CUDA driver allocated memory was 276824064 and is now 278921216. 2025-12-04T11:13:50.4050671Z 2025-12-04T11:13:50.4050798Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.4051319Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_1_in_features_128_out_features_1024_cuda_bfloat16 2025-12-04T11:13:50.4051322Z 2025-12-04T11:13:50.4051480Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.4051586Z !!!!!!!!!!!!!!!!!!!!!!!!!! stopping after 1 failures !!!!!!!!!!!!!!!!!!!!!!!!!!! 2025-12-04T11:13:50.4051703Z ================== 1 failed, 57 deselected, 2 rerun in 11.96s ================== 2025-12-04T11:13:50.4051766Z Got exit code 1 2025-12-04T11:13:50.4051832Z Retrying single test... 2025-12-04T11:13:50.4052094Z W1204 11:02:20.818000 81190 site-packages/torch/_inductor/utils.py:1703] Not enough SMs to use max_autotune_gemm mode 2025-12-04T11:13:50.4052479Z Test results will be stored in test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-2942f663738ee80d.xml 2025-12-04T11:13:50.4052574Z ============================= test session starts ============================== 2025-12-04T11:13:50.4052785Z platform linux -- Python 3.10.14, pytest-7.3.2, pluggy-1.6.0 -- /opt/conda/envs/py_3.10/bin/python 2025-12-04T11:13:50.4052850Z cachedir: .pytest_cache 2025-12-04T11:13:50.4053152Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow] 2025-12-04T11:13:50.4053321Z rootdir: /var/lib/jenkins/workspace 2025-12-04T11:13:50.4053387Z configfile: pytest.ini 2025-12-04T11:13:50.4053707Z plugins: hypothesis-6.56.4, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-14.0, subtests-0.13.1, xdist-3.3.1, xdoctest-1.3.0, typeguard-4.3.0 2025-12-04T11:13:50.4053840Z collecting ... collected 58 items / 57 deselected / 1 selected 2025-12-04T11:13:50.4054409Z stepcurrent: skipping 43 already run items. Running only test/inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_32_mid_dim_1_in_features_128_out_features_1024_cuda_bfloat16 2025-12-04T11:13:50.4054484Z Running 1 items in this shard 2025-12-04T11:13:50.4054488Z 2025-12-04T11:13:50.4055248Z inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_32_mid_dim_1_in_features_128_out_features_1024_cuda_bfloat16 [W1204 11:02:21.940908169 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.4055255Z 2025-12-04T11:13:50.4055570Z [W1204 11:02:31.138591697 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.4055688Z 2025-12-04T11:13:50.4055983Z [W1204 11:02:31.138827502 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.4055986Z 2025-12-04T11:13:50.4056282Z [W1204 11:02:31.144545318 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.4056285Z 2025-12-04T11:13:50.4056575Z [W1204 11:02:31.145097371 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.4056578Z 2025-12-04T11:13:50.4056863Z [W1204 11:02:31.145272314 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.4056867Z 2025-12-04T11:13:50.4057158Z [W1204 11:02:31.150846408 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.4057163Z 2025-12-04T11:13:50.4057445Z [W1204 11:02:31.151378570 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.4057449Z 2025-12-04T11:13:50.4057738Z [W1204 11:02:31.151538663 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.4057742Z 2025-12-04T11:13:50.4057825Z ('RERUN', {'yellow': True}) [11.1113s] [100%] 2025-12-04T11:13:50.4058565Z inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_32_mid_dim_1_in_features_128_out_features_1024_cuda_bfloat16 [W1204 11:02:32.370294939 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.4058571Z 2025-12-04T11:13:50.4058861Z [W1204 11:02:32.370833321 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.4058865Z 2025-12-04T11:13:50.4059154Z [W1204 11:02:32.370970954 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.4059157Z 2025-12-04T11:13:50.4059443Z [W1204 11:02:32.373901438 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.4059446Z 2025-12-04T11:13:50.4059733Z [W1204 11:02:32.374470001 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.4059744Z 2025-12-04T11:13:50.4060038Z [W1204 11:02:32.374606284 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.4060081Z 2025-12-04T11:13:50.4060420Z [W1204 11:02:32.379077432 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.4060426Z 2025-12-04T11:13:50.4060719Z [W1204 11:02:32.379541772 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.4060722Z 2025-12-04T11:13:50.4061009Z [W1204 11:02:32.379677115 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.4061012Z 2025-12-04T11:13:50.4061095Z ('RERUN', {'yellow': True}) [0.4548s] [100%] 2025-12-04T11:13:50.4061863Z inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_32_mid_dim_1_in_features_128_out_features_1024_cuda_bfloat16 [W1204 11:02:32.821075944 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.4061869Z 2025-12-04T11:13:50.4062166Z [W1204 11:02:32.821614625 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.4062202Z 2025-12-04T11:13:50.4062506Z [W1204 11:02:32.821751678 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.4062510Z 2025-12-04T11:13:50.4062802Z [W1204 11:02:32.824664213 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.4062805Z 2025-12-04T11:13:50.4063093Z [W1204 11:02:32.825219515 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.4063097Z 2025-12-04T11:13:50.4063381Z [W1204 11:02:32.825354838 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.4063392Z 2025-12-04T11:13:50.4063724Z [W1204 11:02:32.829838547 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.4063729Z 2025-12-04T11:13:50.4064013Z [W1204 11:02:32.830318328 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.4064016Z 2025-12-04T11:13:50.4064311Z [W1204 11:02:32.830458631 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.4064314Z 2025-12-04T11:13:50.4064375Z FAILED [0.4487s] [100%] 2025-12-04T11:13:50.4064379Z 2025-12-04T11:13:50.4064471Z ==================================== RERUNS ==================================== 2025-12-04T11:13:50.4064768Z _ TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_1_in_features_128_out_features_1024_cuda_bfloat16 _ 2025-12-04T11:13:50.4064847Z Traceback (most recent call last): 2025-12-04T11:13:50.4065158Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.4065226Z method(*args, **kwargs) 2025-12-04T11:13:50.4065518Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.4065582Z method(*args, **kwargs) 2025-12-04T11:13:50.4065869Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T11:13:50.4065933Z with policy(): 2025-12-04T11:13:50.4066225Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T11:13:50.4066303Z raise RuntimeError(msg) 2025-12-04T11:13:50.4067154Z RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_1_in_features_128_out_features_1024_cuda_bfloat16! Caching allocator allocated memory was 0 and is now reported as 131072 on device 0. CUDA driver allocated memory was 230686720 and is now 274726912. 2025-12-04T11:13:50.4067192Z 2025-12-04T11:13:50.4067326Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.4067856Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_1_in_features_128_out_features_1024_cuda_bfloat16 2025-12-04T11:13:50.4067859Z 2025-12-04T11:13:50.4068021Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.4068187Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.4068283Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.4068631Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.4068763Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.4068859Z graph_break [] 2025-12-04T11:13:50.4068987Z ----------------------------- Captured stderr call ----------------------------- 2025-12-04T11:13:50.4069692Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/constant_folding.py:256: UserWarning: Unsupported unwinding pattern: Address not in range (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/profiler/unwind/unwind.cpp:219.) 2025-12-04T11:13:50.4069763Z if out == self.unknown_value: 2025-12-04T11:13:50.4070064Z _ TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_1_in_features_128_out_features_1024_cuda_bfloat16 _ 2025-12-04T11:13:50.4070137Z Traceback (most recent call last): 2025-12-04T11:13:50.4070436Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.4070507Z method(*args, **kwargs) 2025-12-04T11:13:50.4070799Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.4070868Z method(*args, **kwargs) 2025-12-04T11:13:50.4071155Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T11:13:50.4071214Z with policy(): 2025-12-04T11:13:50.4071509Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T11:13:50.4071574Z raise RuntimeError(msg) 2025-12-04T11:13:50.4072400Z RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_1_in_features_128_out_features_1024_cuda_bfloat16! Caching allocator allocated memory was 131072 and is now reported as 262144 on device 0. CUDA driver allocated memory was 274726912 and is now 276824064. 2025-12-04T11:13:50.4072405Z 2025-12-04T11:13:50.4072533Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.4073056Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_1_in_features_128_out_features_1024_cuda_bfloat16 2025-12-04T11:13:50.4073063Z 2025-12-04T11:13:50.4073222Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.4073349Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.4073446Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.4073797Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.4073926Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.4074031Z graph_break [] 2025-12-04T11:13:50.4074189Z ----------------------------- Captured stderr call ----------------------------- 2025-12-04T11:13:50.4074876Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/constant_folding.py:256: UserWarning: Unsupported unwinding pattern: Address not in range (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/profiler/unwind/unwind.cpp:219.) 2025-12-04T11:13:50.4074949Z if out == self.unknown_value: 2025-12-04T11:13:50.4075074Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.4075172Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.4075295Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.4075674Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.4075739Z graph_break [] 2025-12-04T11:13:50.4075824Z =================================== FAILURES =================================== 2025-12-04T11:13:50.4076122Z _ TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_1_in_features_128_out_features_1024_cuda_bfloat16 _ 2025-12-04T11:13:50.4076229Z Traceback (most recent call last): 2025-12-04T11:13:50.4076525Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.4076592Z method(*args, **kwargs) 2025-12-04T11:13:50.4076882Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.4076952Z method(*args, **kwargs) 2025-12-04T11:13:50.4077241Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T11:13:50.4077300Z with policy(): 2025-12-04T11:13:50.4077594Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T11:13:50.4077662Z raise RuntimeError(msg) 2025-12-04T11:13:50.4078657Z RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_1_in_features_128_out_features_1024_cuda_bfloat16! Caching allocator allocated memory was 262144 and is now reported as 393216 on device 0. CUDA driver allocated memory was 276824064 and is now 278921216. 2025-12-04T11:13:50.4078669Z 2025-12-04T11:13:50.4078799Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.4079332Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_1_in_features_128_out_features_1024_cuda_bfloat16 2025-12-04T11:13:50.4079336Z 2025-12-04T11:13:50.4079496Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.4079624Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.4079722Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.4080066Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.4080190Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.4080253Z graph_break [] 2025-12-04T11:13:50.4080380Z ----------------------------- Captured stderr call ----------------------------- 2025-12-04T11:13:50.4081089Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/constant_folding.py:256: UserWarning: Unsupported unwinding pattern: Address not in range (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/profiler/unwind/unwind.cpp:219.) 2025-12-04T11:13:50.4081160Z if out == self.unknown_value: 2025-12-04T11:13:50.4081375Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.4081524Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.4081654Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.4082001Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.4082065Z graph_break [] 2025-12-04T11:13:50.4082189Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.4082283Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.4082407Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.4082798Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.4082865Z graph_break [] 2025-12-04T11:13:50.4083359Z - generated xml file: /var/lib/jenkins/workspace/test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-2942f663738ee80d.xml - 2025-12-04T11:13:50.4083511Z =========================== short test summary info ============================ 2025-12-04T11:13:50.4084819Z FAILED [0.4487s] inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_32_mid_dim_1_in_features_128_out_features_1024_cuda_bfloat16 - RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_1_in_features_128_out_features_1024_cuda_bfloat16! Caching allocator allocated memory was 262144 and is now reported as 393216 on device 0. CUDA driver allocated memory was 276824064 and is now 278921216. 2025-12-04T11:13:50.4084824Z 2025-12-04T11:13:50.4084954Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.4085476Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_1_in_features_128_out_features_1024_cuda_bfloat16 2025-12-04T11:13:50.4085482Z 2025-12-04T11:13:50.4085641Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.4085751Z !!!!!!!!!!!!!!!!!!!!!!!!!! stopping after 1 failures !!!!!!!!!!!!!!!!!!!!!!!!!!! 2025-12-04T11:13:50.4085865Z ================== 1 failed, 57 deselected, 2 rerun in 12.04s ================== 2025-12-04T11:13:50.4085931Z Got exit code 1 2025-12-04T11:13:50.4086407Z FAILED CONSISTENTLY: test/inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_32_mid_dim_1_in_features_128_out_features_1024_cuda_bfloat16 2025-12-04T11:13:50.4086649Z Test failed consistently, continuing with the rest of the tests due to continue-through-error being set 2025-12-04T11:13:50.4086918Z W1204 11:02:39.638000 81383 site-packages/torch/_inductor/utils.py:1703] Not enough SMs to use max_autotune_gemm mode 2025-12-04T11:13:50.4087307Z Test results will be stored in test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-ac0d8e76747a1a1b.xml 2025-12-04T11:13:50.4087411Z ============================= test session starts ============================== 2025-12-04T11:13:50.4087620Z platform linux -- Python 3.10.14, pytest-7.3.2, pluggy-1.6.0 -- /opt/conda/envs/py_3.10/bin/python 2025-12-04T11:13:50.4087690Z cachedir: .pytest_cache 2025-12-04T11:13:50.4088003Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow] 2025-12-04T11:13:50.4088083Z rootdir: /var/lib/jenkins/workspace 2025-12-04T11:13:50.4088152Z configfile: pytest.ini 2025-12-04T11:13:50.4088473Z plugins: hypothesis-6.56.4, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-14.0, subtests-0.13.1, xdist-3.3.1, xdoctest-1.3.0, typeguard-4.3.0 2025-12-04T11:13:50.4088662Z collecting ... collected 58 items / 44 deselected / 14 selected 2025-12-04T11:13:50.4088787Z stepcurrent: skipping 44 already run items. 2025-12-04T11:13:50.4088859Z Running 14 items in this shard 2025-12-04T11:13:50.4088864Z 2025-12-04T11:13:50.4089365Z inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_32_mid_dim_1_in_features_128_out_features_64_cuda_bfloat16 ('RERUN', {'yellow': True}) [1.8750s] [ 7%] 2025-12-04T11:13:50.4089850Z inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_32_mid_dim_1_in_features_128_out_features_64_cuda_bfloat16 ('RERUN', {'yellow': True}) [0.4623s] [ 7%] 2025-12-04T11:13:50.4090324Z inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_32_mid_dim_1_in_features_128_out_features_64_cuda_bfloat16 FAILED [0.4568s] [ 7%] 2025-12-04T11:13:50.4090329Z 2025-12-04T11:13:50.4090425Z ==================================== RERUNS ==================================== 2025-12-04T11:13:50.4090725Z _ TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_1_in_features_128_out_features_64_cuda_bfloat16 _ 2025-12-04T11:13:50.4090834Z Traceback (most recent call last): 2025-12-04T11:13:50.4091148Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.4091216Z method(*args, **kwargs) 2025-12-04T11:13:50.4091517Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.4091581Z method(*args, **kwargs) 2025-12-04T11:13:50.4091870Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T11:13:50.4091937Z with policy(): 2025-12-04T11:13:50.4092231Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T11:13:50.4092299Z raise RuntimeError(msg) 2025-12-04T11:13:50.4093102Z RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_1_in_features_128_out_features_64_cuda_bfloat16! Caching allocator allocated memory was 0 and is now reported as 8192 on device 0. CUDA driver allocated memory was 230686720 and is now 274726912. 2025-12-04T11:13:50.4093108Z 2025-12-04T11:13:50.4093234Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.4093759Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_1_in_features_128_out_features_64_cuda_bfloat16 2025-12-04T11:13:50.4093765Z 2025-12-04T11:13:50.4093924Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.4094057Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.4094155Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.4094507Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.4094641Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.4094702Z graph_break [] 2025-12-04T11:13:50.4094997Z _ TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_1_in_features_128_out_features_64_cuda_bfloat16 _ 2025-12-04T11:13:50.4095071Z Traceback (most recent call last): 2025-12-04T11:13:50.4095370Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.4095440Z method(*args, **kwargs) 2025-12-04T11:13:50.4095729Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.4095835Z method(*args, **kwargs) 2025-12-04T11:13:50.4096163Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T11:13:50.4096225Z with policy(): 2025-12-04T11:13:50.4096527Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T11:13:50.4096602Z raise RuntimeError(msg) 2025-12-04T11:13:50.4097441Z RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_1_in_features_128_out_features_64_cuda_bfloat16! Caching allocator allocated memory was 8192 and is now reported as 16384 on device 0. CUDA driver allocated memory was 274726912 and is now 276824064. 2025-12-04T11:13:50.4097450Z 2025-12-04T11:13:50.4097578Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.4098094Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_1_in_features_128_out_features_64_cuda_bfloat16 2025-12-04T11:13:50.4098100Z 2025-12-04T11:13:50.4098258Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.4098422Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.4098513Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.4098860Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.4098986Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.4099051Z graph_break [] 2025-12-04T11:13:50.4099176Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.4099266Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.4099399Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.4099740Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.4099804Z graph_break [] 2025-12-04T11:13:50.4099888Z =================================== FAILURES =================================== 2025-12-04T11:13:50.4100177Z _ TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_1_in_features_128_out_features_64_cuda_bfloat16 _ 2025-12-04T11:13:50.4100253Z Traceback (most recent call last): 2025-12-04T11:13:50.4100553Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.4100619Z method(*args, **kwargs) 2025-12-04T11:13:50.4100913Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.4100989Z method(*args, **kwargs) 2025-12-04T11:13:50.4101284Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T11:13:50.4101347Z with policy(): 2025-12-04T11:13:50.4101650Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T11:13:50.4101720Z raise RuntimeError(msg) 2025-12-04T11:13:50.4102526Z RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_1_in_features_128_out_features_64_cuda_bfloat16! Caching allocator allocated memory was 16384 and is now reported as 24576 on device 0. CUDA driver allocated memory was 276824064 and is now 278921216. 2025-12-04T11:13:50.4102530Z 2025-12-04T11:13:50.4102662Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.4103223Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_1_in_features_128_out_features_64_cuda_bfloat16 2025-12-04T11:13:50.4103260Z 2025-12-04T11:13:50.4103420Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.4103586Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.4103677Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.4104029Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.4104168Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.4104266Z graph_break [] 2025-12-04T11:13:50.4104395Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.4104482Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.4104606Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.4104949Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.4105043Z graph_break [] 2025-12-04T11:13:50.4105171Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.4105259Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.4105380Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.4105725Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.4105784Z graph_break [] 2025-12-04T11:13:50.4106278Z - generated xml file: /var/lib/jenkins/workspace/test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-ac0d8e76747a1a1b.xml - 2025-12-04T11:13:50.4106384Z =========================== short test summary info ============================ 2025-12-04T11:13:50.4107659Z FAILED [0.4568s] inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_32_mid_dim_1_in_features_128_out_features_64_cuda_bfloat16 - RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_1_in_features_128_out_features_64_cuda_bfloat16! Caching allocator allocated memory was 16384 and is now reported as 24576 on device 0. CUDA driver allocated memory was 276824064 and is now 278921216. 2025-12-04T11:13:50.4107668Z 2025-12-04T11:13:50.4107794Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.4108313Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_1_in_features_128_out_features_64_cuda_bfloat16 2025-12-04T11:13:50.4108318Z 2025-12-04T11:13:50.4108483Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.4108589Z !!!!!!!!!!!!!!!!!!!!!!!!!! stopping after 1 failures !!!!!!!!!!!!!!!!!!!!!!!!!!! 2025-12-04T11:13:50.4108705Z ================== 1 failed, 44 deselected, 2 rerun in 2.82s =================== 2025-12-04T11:13:50.4108766Z Got exit code 1 2025-12-04T11:13:50.4108833Z Retrying single test... 2025-12-04T11:13:50.4109100Z W1204 11:02:49.770000 81564 site-packages/torch/_inductor/utils.py:1703] Not enough SMs to use max_autotune_gemm mode 2025-12-04T11:13:50.4109485Z Test results will be stored in test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-7f452630b39063ce.xml 2025-12-04T11:13:50.4109581Z ============================= test session starts ============================== 2025-12-04T11:13:50.4109789Z platform linux -- Python 3.10.14, pytest-7.3.2, pluggy-1.6.0 -- /opt/conda/envs/py_3.10/bin/python 2025-12-04T11:13:50.4109899Z cachedir: .pytest_cache 2025-12-04T11:13:50.4110241Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow] 2025-12-04T11:13:50.4110322Z rootdir: /var/lib/jenkins/workspace 2025-12-04T11:13:50.4110388Z configfile: pytest.ini 2025-12-04T11:13:50.4110706Z plugins: hypothesis-6.56.4, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-14.0, subtests-0.13.1, xdist-3.3.1, xdoctest-1.3.0, typeguard-4.3.0 2025-12-04T11:13:50.4110834Z collecting ... collected 58 items / 57 deselected / 1 selected 2025-12-04T11:13:50.4111443Z stepcurrent: skipping 44 already run items. Running only test/inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_32_mid_dim_1_in_features_128_out_features_64_cuda_bfloat16 2025-12-04T11:13:50.4111527Z Running 1 items in this shard 2025-12-04T11:13:50.4111531Z 2025-12-04T11:13:50.4112263Z inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_32_mid_dim_1_in_features_128_out_features_64_cuda_bfloat16 [W1204 11:02:50.870415468 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.4112323Z 2025-12-04T11:13:50.4112625Z [W1204 11:03:00.969981439 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.4112629Z 2025-12-04T11:13:50.4112918Z [W1204 11:03:00.970255805 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.4112921Z 2025-12-04T11:13:50.4113214Z [W1204 11:03:00.975877879 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.4113218Z 2025-12-04T11:13:50.4113507Z [W1204 11:03:00.976424511 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.4113511Z 2025-12-04T11:13:50.4113800Z [W1204 11:03:00.976605746 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.4113805Z 2025-12-04T11:13:50.4114092Z [W1204 11:03:00.982037396 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.4114096Z 2025-12-04T11:13:50.4114382Z [W1204 11:03:00.982598378 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.4114385Z 2025-12-04T11:13:50.4114673Z [W1204 11:03:00.982765792 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.4114676Z 2025-12-04T11:13:50.4114759Z ('RERUN', {'yellow': True}) [10.9968s] [100%] 2025-12-04T11:13:50.4115480Z inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_32_mid_dim_1_in_features_128_out_features_64_cuda_bfloat16 [W1204 11:03:01.169761872 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.4115486Z 2025-12-04T11:13:50.4115778Z [W1204 11:03:01.170373446 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.4115781Z 2025-12-04T11:13:50.4116071Z [W1204 11:03:01.170525859 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.4116075Z 2025-12-04T11:13:50.4116362Z [W1204 11:03:01.173491804 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.4116365Z 2025-12-04T11:13:50.4116655Z [W1204 11:03:01.174070137 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.4116697Z 2025-12-04T11:13:50.4117017Z [W1204 11:03:01.174209311 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.4117022Z 2025-12-04T11:13:50.4117322Z [W1204 11:03:01.178747001 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.4117328Z 2025-12-04T11:13:50.4117618Z [W1204 11:03:01.179220811 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.4117621Z 2025-12-04T11:13:50.4117944Z [W1204 11:03:01.179360084 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.4117948Z 2025-12-04T11:13:50.4118027Z ('RERUN', {'yellow': True}) [0.4199s] [100%] 2025-12-04T11:13:50.4118745Z inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_32_mid_dim_1_in_features_128_out_features_64_cuda_bfloat16 [W1204 11:03:01.585463842 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.4118786Z 2025-12-04T11:13:50.4119075Z [W1204 11:03:01.586055445 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.4119079Z 2025-12-04T11:13:50.4119369Z [W1204 11:03:01.586202398 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.4119372Z 2025-12-04T11:13:50.4119671Z [W1204 11:03:01.589161294 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.4119674Z 2025-12-04T11:13:50.4119962Z [W1204 11:03:01.589734926 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.4119967Z 2025-12-04T11:13:50.4120257Z [W1204 11:03:01.589872209 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.4120261Z 2025-12-04T11:13:50.4120546Z [W1204 11:03:01.594450620 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.4120549Z 2025-12-04T11:13:50.4120841Z [W1204 11:03:01.594923940 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.4120845Z 2025-12-04T11:13:50.4121128Z [W1204 11:03:01.595056623 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.4121133Z 2025-12-04T11:13:50.4121198Z FAILED [0.4137s] [100%] 2025-12-04T11:13:50.4121202Z 2025-12-04T11:13:50.4121287Z ==================================== RERUNS ==================================== 2025-12-04T11:13:50.4121585Z _ TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_1_in_features_128_out_features_64_cuda_bfloat16 _ 2025-12-04T11:13:50.4121664Z Traceback (most recent call last): 2025-12-04T11:13:50.4121972Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.4122036Z method(*args, **kwargs) 2025-12-04T11:13:50.4122332Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.4122396Z method(*args, **kwargs) 2025-12-04T11:13:50.4122700Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T11:13:50.4122763Z with policy(): 2025-12-04T11:13:50.4123061Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T11:13:50.4123174Z raise RuntimeError(msg) 2025-12-04T11:13:50.4124007Z RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_1_in_features_128_out_features_64_cuda_bfloat16! Caching allocator allocated memory was 0 and is now reported as 8192 on device 0. CUDA driver allocated memory was 230686720 and is now 274726912. 2025-12-04T11:13:50.4124013Z 2025-12-04T11:13:50.4124144Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.4124659Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_1_in_features_128_out_features_64_cuda_bfloat16 2025-12-04T11:13:50.4124663Z 2025-12-04T11:13:50.4124856Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.4124988Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.4125083Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.4125434Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.4125596Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.4125656Z graph_break [] 2025-12-04T11:13:50.4125787Z ----------------------------- Captured stderr call ----------------------------- 2025-12-04T11:13:50.4126483Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/constant_folding.py:256: UserWarning: Unsupported unwinding pattern: Address not in range (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/profiler/unwind/unwind.cpp:219.) 2025-12-04T11:13:50.4126558Z if out == self.unknown_value: 2025-12-04T11:13:50.4126850Z _ TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_1_in_features_128_out_features_64_cuda_bfloat16 _ 2025-12-04T11:13:50.4126925Z Traceback (most recent call last): 2025-12-04T11:13:50.4127239Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.4127305Z method(*args, **kwargs) 2025-12-04T11:13:50.4127597Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.4127663Z method(*args, **kwargs) 2025-12-04T11:13:50.4127954Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T11:13:50.4128019Z with policy(): 2025-12-04T11:13:50.4128311Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T11:13:50.4128380Z raise RuntimeError(msg) 2025-12-04T11:13:50.4129187Z RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_1_in_features_128_out_features_64_cuda_bfloat16! Caching allocator allocated memory was 8192 and is now reported as 16384 on device 0. CUDA driver allocated memory was 274726912 and is now 276824064. 2025-12-04T11:13:50.4129194Z 2025-12-04T11:13:50.4129329Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.4129854Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_1_in_features_128_out_features_64_cuda_bfloat16 2025-12-04T11:13:50.4129858Z 2025-12-04T11:13:50.4130018Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.4130149Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.4130242Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.4130627Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.4134286Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.4134374Z graph_break [] 2025-12-04T11:13:50.4134529Z ----------------------------- Captured stderr call ----------------------------- 2025-12-04T11:13:50.4135242Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/constant_folding.py:256: UserWarning: Unsupported unwinding pattern: Address not in range (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/profiler/unwind/unwind.cpp:219.) 2025-12-04T11:13:50.4135320Z if out == self.unknown_value: 2025-12-04T11:13:50.4135472Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.4135624Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.4135762Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.4136122Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.4136189Z graph_break [] 2025-12-04T11:13:50.4136283Z =================================== FAILURES =================================== 2025-12-04T11:13:50.4136621Z _ TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_1_in_features_128_out_features_64_cuda_bfloat16 _ 2025-12-04T11:13:50.4136700Z Traceback (most recent call last): 2025-12-04T11:13:50.4137023Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.4137091Z method(*args, **kwargs) 2025-12-04T11:13:50.4137401Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.4137467Z method(*args, **kwargs) 2025-12-04T11:13:50.4137765Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T11:13:50.4137833Z with policy(): 2025-12-04T11:13:50.4138142Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T11:13:50.4138212Z raise RuntimeError(msg) 2025-12-04T11:13:50.4139035Z RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_1_in_features_128_out_features_64_cuda_bfloat16! Caching allocator allocated memory was 16384 and is now reported as 24576 on device 0. CUDA driver allocated memory was 276824064 and is now 278921216. 2025-12-04T11:13:50.4139041Z 2025-12-04T11:13:50.4139172Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.4139712Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_1_in_features_128_out_features_64_cuda_bfloat16 2025-12-04T11:13:50.4139717Z 2025-12-04T11:13:50.4139882Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.4140015Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.4140121Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.4140469Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.4140603Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.4140663Z graph_break [] 2025-12-04T11:13:50.4140798Z ----------------------------- Captured stderr call ----------------------------- 2025-12-04T11:13:50.4141495Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/constant_folding.py:256: UserWarning: Unsupported unwinding pattern: Address not in range (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/profiler/unwind/unwind.cpp:219.) 2025-12-04T11:13:50.4141620Z if out == self.unknown_value: 2025-12-04T11:13:50.4141803Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.4141903Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.4142032Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.4142393Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.4142453Z graph_break [] 2025-12-04T11:13:50.4142582Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.4142712Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.4142837Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.4143194Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.4143257Z graph_break [] 2025-12-04T11:13:50.4143818Z - generated xml file: /var/lib/jenkins/workspace/test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-7f452630b39063ce.xml - 2025-12-04T11:13:50.4143965Z =========================== short test summary info ============================ 2025-12-04T11:13:50.4145257Z FAILED [0.4137s] inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_32_mid_dim_1_in_features_128_out_features_64_cuda_bfloat16 - RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_1_in_features_128_out_features_64_cuda_bfloat16! Caching allocator allocated memory was 16384 and is now reported as 24576 on device 0. CUDA driver allocated memory was 276824064 and is now 278921216. 2025-12-04T11:13:50.4145264Z 2025-12-04T11:13:50.4145397Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.4145921Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_1_in_features_128_out_features_64_cuda_bfloat16 2025-12-04T11:13:50.4145926Z 2025-12-04T11:13:50.4146091Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.4146200Z !!!!!!!!!!!!!!!!!!!!!!!!!! stopping after 1 failures !!!!!!!!!!!!!!!!!!!!!!!!!!! 2025-12-04T11:13:50.4146317Z ================== 1 failed, 57 deselected, 2 rerun in 11.86s ================== 2025-12-04T11:13:50.4146380Z Got exit code 1 2025-12-04T11:13:50.4146447Z Retrying single test... 2025-12-04T11:13:50.4146727Z W1204 11:03:08.428000 81750 site-packages/torch/_inductor/utils.py:1703] Not enough SMs to use max_autotune_gemm mode 2025-12-04T11:13:50.4147118Z Test results will be stored in test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-bb668a52e682b2dd.xml 2025-12-04T11:13:50.4147219Z ============================= test session starts ============================== 2025-12-04T11:13:50.4147432Z platform linux -- Python 3.10.14, pytest-7.3.2, pluggy-1.6.0 -- /opt/conda/envs/py_3.10/bin/python 2025-12-04T11:13:50.4147498Z cachedir: .pytest_cache 2025-12-04T11:13:50.4147805Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow] 2025-12-04T11:13:50.4147889Z rootdir: /var/lib/jenkins/workspace 2025-12-04T11:13:50.4147955Z configfile: pytest.ini 2025-12-04T11:13:50.4148277Z plugins: hypothesis-6.56.4, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-14.0, subtests-0.13.1, xdist-3.3.1, xdoctest-1.3.0, typeguard-4.3.0 2025-12-04T11:13:50.4148410Z collecting ... collected 58 items / 57 deselected / 1 selected 2025-12-04T11:13:50.4149029Z stepcurrent: skipping 44 already run items. Running only test/inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_32_mid_dim_1_in_features_128_out_features_64_cuda_bfloat16 2025-12-04T11:13:50.4149138Z Running 1 items in this shard 2025-12-04T11:13:50.4149144Z 2025-12-04T11:13:50.4149874Z inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_32_mid_dim_1_in_features_128_out_features_64_cuda_bfloat16 [W1204 11:03:09.516124116 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.4149878Z 2025-12-04T11:13:50.4150183Z [W1204 11:03:18.819928446 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.4150188Z 2025-12-04T11:13:50.4150517Z [W1204 11:03:18.820197892 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.4150523Z 2025-12-04T11:13:50.4150818Z [W1204 11:03:18.825714123 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.4150822Z 2025-12-04T11:13:50.4151115Z [W1204 11:03:18.826253555 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.4151151Z 2025-12-04T11:13:50.4151441Z [W1204 11:03:18.826431029 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.4151445Z 2025-12-04T11:13:50.4151734Z [W1204 11:03:18.831773567 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.4151738Z 2025-12-04T11:13:50.4152024Z [W1204 11:03:18.832320179 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.4152030Z 2025-12-04T11:13:50.4152316Z [W1204 11:03:18.832484462 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.4152321Z 2025-12-04T11:13:50.4152404Z ('RERUN', {'yellow': True}) [11.1874s] [100%] 2025-12-04T11:13:50.4153133Z inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_32_mid_dim_1_in_features_128_out_features_64_cuda_bfloat16 [W1204 11:03:20.016899676 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.4153136Z 2025-12-04T11:13:50.4153426Z [W1204 11:03:20.017483769 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.4153430Z 2025-12-04T11:13:50.4153724Z [W1204 11:03:20.017621062 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.4153728Z 2025-12-04T11:13:50.4154015Z [W1204 11:03:20.020684989 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.4154020Z 2025-12-04T11:13:50.4154306Z [W1204 11:03:20.021262882 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.4154310Z 2025-12-04T11:13:50.4154598Z [W1204 11:03:20.021399835 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.4154601Z 2025-12-04T11:13:50.4154893Z [W1204 11:03:20.026087458 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.4154896Z 2025-12-04T11:13:50.4155183Z [W1204 11:03:20.026555968 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.4155186Z 2025-12-04T11:13:50.4155473Z [W1204 11:03:20.026692801 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.4155516Z 2025-12-04T11:13:50.4155638Z ('RERUN', {'yellow': True}) [0.4224s] [100%] 2025-12-04T11:13:50.4156364Z inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_32_mid_dim_1_in_features_128_out_features_64_cuda_bfloat16 [W1204 11:03:20.434509876 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.4156368Z 2025-12-04T11:13:50.4156655Z [W1204 11:03:20.435097119 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.4156659Z 2025-12-04T11:13:50.4156977Z [W1204 11:03:20.435235982 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.4156982Z 2025-12-04T11:13:50.4157276Z [W1204 11:03:20.438287539 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.4157281Z 2025-12-04T11:13:50.4157573Z [W1204 11:03:20.438853252 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.4157609Z 2025-12-04T11:13:50.4157902Z [W1204 11:03:20.438991295 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.4157906Z 2025-12-04T11:13:50.4158196Z [W1204 11:03:20.443702899 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.4158200Z 2025-12-04T11:13:50.4158487Z [W1204 11:03:20.444175440 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.4158492Z 2025-12-04T11:13:50.4158777Z [W1204 11:03:20.444311203 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.4158782Z 2025-12-04T11:13:50.4158844Z FAILED [0.4159s] [100%] 2025-12-04T11:13:50.4158847Z 2025-12-04T11:13:50.4158936Z ==================================== RERUNS ==================================== 2025-12-04T11:13:50.4159233Z _ TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_1_in_features_128_out_features_64_cuda_bfloat16 _ 2025-12-04T11:13:50.4159312Z Traceback (most recent call last): 2025-12-04T11:13:50.4159633Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.4159702Z method(*args, **kwargs) 2025-12-04T11:13:50.4160000Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.4160062Z method(*args, **kwargs) 2025-12-04T11:13:50.4160351Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T11:13:50.4160417Z with policy(): 2025-12-04T11:13:50.4160714Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T11:13:50.4160784Z raise RuntimeError(msg) 2025-12-04T11:13:50.4161583Z RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_1_in_features_128_out_features_64_cuda_bfloat16! Caching allocator allocated memory was 0 and is now reported as 8192 on device 0. CUDA driver allocated memory was 230686720 and is now 274726912. 2025-12-04T11:13:50.4161587Z 2025-12-04T11:13:50.4161716Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.4162247Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_1_in_features_128_out_features_64_cuda_bfloat16 2025-12-04T11:13:50.4162292Z 2025-12-04T11:13:50.4162485Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.4162618Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.4162714Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.4163063Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.4163195Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.4163257Z graph_break [] 2025-12-04T11:13:50.4163386Z ----------------------------- Captured stderr call ----------------------------- 2025-12-04T11:13:50.4164112Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/constant_folding.py:256: UserWarning: Unsupported unwinding pattern: Address not in range (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/profiler/unwind/unwind.cpp:219.) 2025-12-04T11:13:50.4164186Z if out == self.unknown_value: 2025-12-04T11:13:50.4164485Z _ TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_1_in_features_128_out_features_64_cuda_bfloat16 _ 2025-12-04T11:13:50.4164593Z Traceback (most recent call last): 2025-12-04T11:13:50.4164910Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.4164976Z method(*args, **kwargs) 2025-12-04T11:13:50.4165275Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.4165342Z method(*args, **kwargs) 2025-12-04T11:13:50.4165636Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T11:13:50.4165697Z with policy(): 2025-12-04T11:13:50.4165998Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T11:13:50.4166066Z raise RuntimeError(msg) 2025-12-04T11:13:50.4166886Z RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_1_in_features_128_out_features_64_cuda_bfloat16! Caching allocator allocated memory was 8192 and is now reported as 16384 on device 0. CUDA driver allocated memory was 274726912 and is now 276824064. 2025-12-04T11:13:50.4166892Z 2025-12-04T11:13:50.4167021Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.4167545Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_1_in_features_128_out_features_64_cuda_bfloat16 2025-12-04T11:13:50.4167549Z 2025-12-04T11:13:50.4167712Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.4167846Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.4167946Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.4168299Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.4168431Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.4168491Z graph_break [] 2025-12-04T11:13:50.4168619Z ----------------------------- Captured stderr call ----------------------------- 2025-12-04T11:13:50.4169320Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/constant_folding.py:256: UserWarning: Unsupported unwinding pattern: Address not in range (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/profiler/unwind/unwind.cpp:219.) 2025-12-04T11:13:50.4169392Z if out == self.unknown_value: 2025-12-04T11:13:50.4169578Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.4169677Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.4169920Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.4170273Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.4170333Z graph_break [] 2025-12-04T11:13:50.4170418Z =================================== FAILURES =================================== 2025-12-04T11:13:50.4170715Z _ TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_1_in_features_128_out_features_64_cuda_bfloat16 _ 2025-12-04T11:13:50.4170791Z Traceback (most recent call last): 2025-12-04T11:13:50.4171133Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.4171200Z method(*args, **kwargs) 2025-12-04T11:13:50.4171495Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.4171564Z method(*args, **kwargs) 2025-12-04T11:13:50.4171894Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T11:13:50.4171955Z with policy(): 2025-12-04T11:13:50.4172251Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T11:13:50.4172317Z raise RuntimeError(msg) 2025-12-04T11:13:50.4173133Z RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_1_in_features_128_out_features_64_cuda_bfloat16! Caching allocator allocated memory was 16384 and is now reported as 24576 on device 0. CUDA driver allocated memory was 276824064 and is now 278921216. 2025-12-04T11:13:50.4173141Z 2025-12-04T11:13:50.4173270Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.4173792Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_1_in_features_128_out_features_64_cuda_bfloat16 2025-12-04T11:13:50.4173800Z 2025-12-04T11:13:50.4173959Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.4174088Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.4174184Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.4174533Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.4174661Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.4174724Z graph_break [] 2025-12-04T11:13:50.4174850Z ----------------------------- Captured stderr call ----------------------------- 2025-12-04T11:13:50.4175543Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/constant_folding.py:256: UserWarning: Unsupported unwinding pattern: Address not in range (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/profiler/unwind/unwind.cpp:219.) 2025-12-04T11:13:50.4175613Z if out == self.unknown_value: 2025-12-04T11:13:50.4175738Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.4175835Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.4175958Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.4176303Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.4176362Z graph_break [] 2025-12-04T11:13:50.4176485Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.4176621Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.4176778Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.4177122Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.4177183Z graph_break [] 2025-12-04T11:13:50.4177670Z - generated xml file: /var/lib/jenkins/workspace/test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-bb668a52e682b2dd.xml - 2025-12-04T11:13:50.4177775Z =========================== short test summary info ============================ 2025-12-04T11:13:50.4179367Z FAILED [0.4159s] inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_32_mid_dim_1_in_features_128_out_features_64_cuda_bfloat16 - RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_1_in_features_128_out_features_64_cuda_bfloat16! Caching allocator allocated memory was 16384 and is now reported as 24576 on device 0. CUDA driver allocated memory was 276824064 and is now 278921216. 2025-12-04T11:13:50.4179424Z 2025-12-04T11:13:50.4179571Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.4180101Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_1_in_features_128_out_features_64_cuda_bfloat16 2025-12-04T11:13:50.4180105Z 2025-12-04T11:13:50.4180268Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.4180378Z !!!!!!!!!!!!!!!!!!!!!!!!!! stopping after 1 failures !!!!!!!!!!!!!!!!!!!!!!!!!!! 2025-12-04T11:13:50.4180497Z ================== 1 failed, 57 deselected, 2 rerun in 12.05s ================== 2025-12-04T11:13:50.4180563Z Got exit code 1 2025-12-04T11:13:50.4181044Z FAILED CONSISTENTLY: test/inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_32_mid_dim_1_in_features_128_out_features_64_cuda_bfloat16 2025-12-04T11:13:50.4181295Z Test failed consistently, continuing with the rest of the tests due to continue-through-error being set 2025-12-04T11:13:50.4181560Z W1204 11:03:27.258000 81936 site-packages/torch/_inductor/utils.py:1703] Not enough SMs to use max_autotune_gemm mode 2025-12-04T11:13:50.4181946Z Test results will be stored in test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-60466f6816d813eb.xml 2025-12-04T11:13:50.4182045Z ============================= test session starts ============================== 2025-12-04T11:13:50.4182258Z platform linux -- Python 3.10.14, pytest-7.3.2, pluggy-1.6.0 -- /opt/conda/envs/py_3.10/bin/python 2025-12-04T11:13:50.4182327Z cachedir: .pytest_cache 2025-12-04T11:13:50.4182643Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow] 2025-12-04T11:13:50.4182724Z rootdir: /var/lib/jenkins/workspace 2025-12-04T11:13:50.4182791Z configfile: pytest.ini 2025-12-04T11:13:50.4183105Z plugins: hypothesis-6.56.4, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-14.0, subtests-0.13.1, xdist-3.3.1, xdoctest-1.3.0, typeguard-4.3.0 2025-12-04T11:13:50.4183234Z collecting ... collected 58 items / 45 deselected / 13 selected 2025-12-04T11:13:50.4183324Z stepcurrent: skipping 45 already run items. 2025-12-04T11:13:50.4183405Z Running 13 items in this shard 2025-12-04T11:13:50.4183408Z 2025-12-04T11:13:50.4183948Z inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_32_mid_dim_1_in_features_128_out_features_65_cuda_bfloat16 ('RERUN', {'yellow': True}) [1.9666s] [ 7%] 2025-12-04T11:13:50.4184442Z inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_32_mid_dim_1_in_features_128_out_features_65_cuda_bfloat16 ('RERUN', {'yellow': True}) [0.5469s] [ 7%] 2025-12-04T11:13:50.4185001Z inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_32_mid_dim_1_in_features_128_out_features_65_cuda_bfloat16 FAILED [0.5322s] [ 7%] 2025-12-04T11:13:50.4185008Z 2025-12-04T11:13:50.4185097Z ==================================== RERUNS ==================================== 2025-12-04T11:13:50.4185400Z _ TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_1_in_features_128_out_features_65_cuda_bfloat16 _ 2025-12-04T11:13:50.4185477Z Traceback (most recent call last): 2025-12-04T11:13:50.4185819Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.4185887Z method(*args, **kwargs) 2025-12-04T11:13:50.4186182Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.4186247Z method(*args, **kwargs) 2025-12-04T11:13:50.4186540Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T11:13:50.4186641Z with policy(): 2025-12-04T11:13:50.4186939Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T11:13:50.4187019Z raise RuntimeError(msg) 2025-12-04T11:13:50.4187821Z RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_1_in_features_128_out_features_65_cuda_bfloat16! Caching allocator allocated memory was 0 and is now reported as 8704 on device 0. CUDA driver allocated memory was 230686720 and is now 274726912. 2025-12-04T11:13:50.4187825Z 2025-12-04T11:13:50.4187958Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.4188479Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_1_in_features_128_out_features_65_cuda_bfloat16 2025-12-04T11:13:50.4188482Z 2025-12-04T11:13:50.4188640Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.4188770Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.4188865Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.4189425Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('benchmarking.InductorBenchmarker.benchmark_gpu', 2), ('pad_mm_bench', 1), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.4189554Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.4189612Z graph_break [] 2025-12-04T11:13:50.4189909Z _ TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_1_in_features_128_out_features_65_cuda_bfloat16 _ 2025-12-04T11:13:50.4189984Z Traceback (most recent call last): 2025-12-04T11:13:50.4190280Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.4190348Z method(*args, **kwargs) 2025-12-04T11:13:50.4190636Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.4190702Z method(*args, **kwargs) 2025-12-04T11:13:50.4190992Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T11:13:50.4191053Z with policy(): 2025-12-04T11:13:50.4191353Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T11:13:50.4191418Z raise RuntimeError(msg) 2025-12-04T11:13:50.4192270Z RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_1_in_features_128_out_features_65_cuda_bfloat16! Caching allocator allocated memory was 8704 and is now reported as 17408 on device 0. CUDA driver allocated memory was 274726912 and is now 276824064. 2025-12-04T11:13:50.4192312Z 2025-12-04T11:13:50.4192441Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.4192955Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_1_in_features_128_out_features_65_cuda_bfloat16 2025-12-04T11:13:50.4192961Z 2025-12-04T11:13:50.4193116Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.4193276Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.4193372Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.4193921Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('benchmarking.InductorBenchmarker.benchmark_gpu', 2), ('pad_mm_bench', 1), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.4194100Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.4194161Z graph_break [] 2025-12-04T11:13:50.4194286Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.4194380Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.4194500Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.4195036Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('benchmarking.InductorBenchmarker.benchmark_gpu', 2), ('pad_mm_bench', 1), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.4195097Z graph_break [] 2025-12-04T11:13:50.4195183Z =================================== FAILURES =================================== 2025-12-04T11:13:50.4195479Z _ TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_1_in_features_128_out_features_65_cuda_bfloat16 _ 2025-12-04T11:13:50.4195556Z Traceback (most recent call last): 2025-12-04T11:13:50.4195854Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.4195926Z method(*args, **kwargs) 2025-12-04T11:13:50.4196222Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.4196287Z method(*args, **kwargs) 2025-12-04T11:13:50.4196589Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T11:13:50.4196648Z with policy(): 2025-12-04T11:13:50.4196944Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T11:13:50.4197014Z raise RuntimeError(msg) 2025-12-04T11:13:50.4197826Z RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_1_in_features_128_out_features_65_cuda_bfloat16! Caching allocator allocated memory was 17408 and is now reported as 26112 on device 0. CUDA driver allocated memory was 276824064 and is now 278921216. 2025-12-04T11:13:50.4197831Z 2025-12-04T11:13:50.4197957Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.4198474Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_1_in_features_128_out_features_65_cuda_bfloat16 2025-12-04T11:13:50.4198478Z 2025-12-04T11:13:50.4198639Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.4198763Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.4198898Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.4199473Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('benchmarking.InductorBenchmarker.benchmark_gpu', 2), ('pad_mm_bench', 1), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.4199608Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.4199669Z graph_break [] 2025-12-04T11:13:50.4199792Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.4199883Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.4200039Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.4200581Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('benchmarking.InductorBenchmarker.benchmark_gpu', 2), ('pad_mm_bench', 1), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.4200647Z graph_break [] 2025-12-04T11:13:50.4200774Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.4200901Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.4201022Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.4201556Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('benchmarking.InductorBenchmarker.benchmark_gpu', 2), ('pad_mm_bench', 1), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.4201617Z graph_break [] 2025-12-04T11:13:50.4202107Z - generated xml file: /var/lib/jenkins/workspace/test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-60466f6816d813eb.xml - 2025-12-04T11:13:50.4202219Z =========================== short test summary info ============================ 2025-12-04T11:13:50.4203507Z FAILED [0.5322s] inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_32_mid_dim_1_in_features_128_out_features_65_cuda_bfloat16 - RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_1_in_features_128_out_features_65_cuda_bfloat16! Caching allocator allocated memory was 17408 and is now reported as 26112 on device 0. CUDA driver allocated memory was 276824064 and is now 278921216. 2025-12-04T11:13:50.4203513Z 2025-12-04T11:13:50.4203640Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.4204156Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_1_in_features_128_out_features_65_cuda_bfloat16 2025-12-04T11:13:50.4204160Z 2025-12-04T11:13:50.4204314Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.4204426Z !!!!!!!!!!!!!!!!!!!!!!!!!! stopping after 1 failures !!!!!!!!!!!!!!!!!!!!!!!!!!! 2025-12-04T11:13:50.4204541Z ================== 1 failed, 45 deselected, 2 rerun in 3.07s =================== 2025-12-04T11:13:50.4204603Z Got exit code 1 2025-12-04T11:13:50.4204672Z Retrying single test... 2025-12-04T11:13:50.4204933Z W1204 11:03:37.327000 82118 site-packages/torch/_inductor/utils.py:1703] Not enough SMs to use max_autotune_gemm mode 2025-12-04T11:13:50.4205320Z Test results will be stored in test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-3b52d4f5cebad92d.xml 2025-12-04T11:13:50.4205421Z ============================= test session starts ============================== 2025-12-04T11:13:50.4205630Z platform linux -- Python 3.10.14, pytest-7.3.2, pluggy-1.6.0 -- /opt/conda/envs/py_3.10/bin/python 2025-12-04T11:13:50.4205698Z cachedir: .pytest_cache 2025-12-04T11:13:50.4206046Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow] 2025-12-04T11:13:50.4206155Z rootdir: /var/lib/jenkins/workspace 2025-12-04T11:13:50.4206224Z configfile: pytest.ini 2025-12-04T11:13:50.4206606Z plugins: hypothesis-6.56.4, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-14.0, subtests-0.13.1, xdist-3.3.1, xdoctest-1.3.0, typeguard-4.3.0 2025-12-04T11:13:50.4206836Z collecting ... collected 58 items / 57 deselected / 1 selected 2025-12-04T11:13:50.4207572Z stepcurrent: skipping 45 already run items. Running only test/inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_32_mid_dim_1_in_features_128_out_features_65_cuda_bfloat16 2025-12-04T11:13:50.4207648Z Running 1 items in this shard 2025-12-04T11:13:50.4207712Z 2025-12-04T11:13:50.4208599Z inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_32_mid_dim_1_in_features_128_out_features_65_cuda_bfloat16 [W1204 11:03:39.966109888 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.4208610Z 2025-12-04T11:13:50.4208920Z [W1204 11:03:48.255358814 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.4208981Z 2025-12-04T11:13:50.4209284Z [W1204 11:03:48.255604339 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.4209288Z 2025-12-04T11:13:50.4209576Z [W1204 11:03:48.261442758 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.4209580Z 2025-12-04T11:13:50.4209878Z [W1204 11:03:48.262009600 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.4209881Z 2025-12-04T11:13:50.4210167Z [W1204 11:03:48.262187484 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.4210172Z 2025-12-04T11:13:50.4210457Z [W1204 11:03:48.267604024 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.4210461Z 2025-12-04T11:13:50.4210745Z [W1204 11:03:48.268124086 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.4210748Z 2025-12-04T11:13:50.4211029Z [W1204 11:03:48.268283539 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.4211032Z 2025-12-04T11:13:50.4211119Z ('RERUN', {'yellow': True}) [11.2729s] [100%] 2025-12-04T11:13:50.4211847Z inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_32_mid_dim_1_in_features_128_out_features_65_cuda_bfloat16 [W1204 11:03:49.079760699 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.4211853Z 2025-12-04T11:13:50.4212144Z [W1204 11:03:49.080339362 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.4212148Z 2025-12-04T11:13:50.4212431Z [W1204 11:03:49.080487745 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.4212434Z 2025-12-04T11:13:50.4212732Z [W1204 11:03:49.083435160 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.4212735Z 2025-12-04T11:13:50.4213023Z [W1204 11:03:49.083896930 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.4213027Z 2025-12-04T11:13:50.4213315Z [W1204 11:03:49.084030813 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.4213363Z 2025-12-04T11:13:50.4213687Z [W1204 11:03:49.088564754 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.4213693Z 2025-12-04T11:13:50.4213977Z [W1204 11:03:49.089035324 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.4213985Z 2025-12-04T11:13:50.4214268Z [W1204 11:03:49.089169787 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.4214272Z 2025-12-04T11:13:50.4214351Z ('RERUN', {'yellow': True}) [0.5057s] [100%] 2025-12-04T11:13:50.4215105Z inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_32_mid_dim_1_in_features_128_out_features_65_cuda_bfloat16 [W1204 11:03:49.584520506 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.4215112Z 2025-12-04T11:13:50.4215400Z [W1204 11:03:49.585101199 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.4215437Z 2025-12-04T11:13:50.4215737Z [W1204 11:03:49.585246042 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.4215741Z 2025-12-04T11:13:50.4216028Z [W1204 11:03:49.588135736 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.4216032Z 2025-12-04T11:13:50.4216323Z [W1204 11:03:49.588600916 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.4216327Z 2025-12-04T11:13:50.4216609Z [W1204 11:03:49.588739489 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.4216614Z 2025-12-04T11:13:50.4216908Z [W1204 11:03:49.593316730 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.4216912Z 2025-12-04T11:13:50.4217195Z [W1204 11:03:49.593782280 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.4217198Z 2025-12-04T11:13:50.4217481Z [W1204 11:03:49.593919863 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.4217484Z 2025-12-04T11:13:50.4217551Z FAILED [0.5012s] [100%] 2025-12-04T11:13:50.4217554Z 2025-12-04T11:13:50.4217643Z ==================================== RERUNS ==================================== 2025-12-04T11:13:50.4217942Z _ TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_1_in_features_128_out_features_65_cuda_bfloat16 _ 2025-12-04T11:13:50.4218022Z Traceback (most recent call last): 2025-12-04T11:13:50.4218334Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.4218407Z method(*args, **kwargs) 2025-12-04T11:13:50.4218699Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.4218766Z method(*args, **kwargs) 2025-12-04T11:13:50.4219055Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T11:13:50.4219117Z with policy(): 2025-12-04T11:13:50.4219416Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T11:13:50.4219482Z raise RuntimeError(msg) 2025-12-04T11:13:50.4220317Z RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_1_in_features_128_out_features_65_cuda_bfloat16! Caching allocator allocated memory was 0 and is now reported as 8704 on device 0. CUDA driver allocated memory was 230686720 and is now 274726912. 2025-12-04T11:13:50.4220365Z 2025-12-04T11:13:50.4220497Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.4221020Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_1_in_features_128_out_features_65_cuda_bfloat16 2025-12-04T11:13:50.4221024Z 2025-12-04T11:13:50.4221187Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.4221371Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.4221470Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.4222022Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('benchmarking.InductorBenchmarker.benchmark_gpu', 2), ('pad_mm_bench', 1), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.4222153Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.4222256Z graph_break [] 2025-12-04T11:13:50.4222380Z ----------------------------- Captured stderr call ----------------------------- 2025-12-04T11:13:50.4223073Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/constant_folding.py:256: UserWarning: Unsupported unwinding pattern: Address not in range (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/profiler/unwind/unwind.cpp:219.) 2025-12-04T11:13:50.4223146Z if out == self.unknown_value: 2025-12-04T11:13:50.4223442Z _ TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_1_in_features_128_out_features_65_cuda_bfloat16 _ 2025-12-04T11:13:50.4223603Z Traceback (most recent call last): 2025-12-04T11:13:50.4223907Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.4223973Z method(*args, **kwargs) 2025-12-04T11:13:50.4224265Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.4224331Z method(*args, **kwargs) 2025-12-04T11:13:50.4224625Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T11:13:50.4224689Z with policy(): 2025-12-04T11:13:50.4224980Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T11:13:50.4225047Z raise RuntimeError(msg) 2025-12-04T11:13:50.4225854Z RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_1_in_features_128_out_features_65_cuda_bfloat16! Caching allocator allocated memory was 8704 and is now reported as 17408 on device 0. CUDA driver allocated memory was 274726912 and is now 276824064. 2025-12-04T11:13:50.4225862Z 2025-12-04T11:13:50.4225992Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.4226511Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_1_in_features_128_out_features_65_cuda_bfloat16 2025-12-04T11:13:50.4226515Z 2025-12-04T11:13:50.4226675Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.4226804Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.4226900Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.4227450Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('benchmarking.InductorBenchmarker.benchmark_gpu', 2), ('pad_mm_bench', 1), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.4227662Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.4227722Z graph_break [] 2025-12-04T11:13:50.4227850Z ----------------------------- Captured stderr call ----------------------------- 2025-12-04T11:13:50.4228535Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/constant_folding.py:256: UserWarning: Unsupported unwinding pattern: Address not in range (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/profiler/unwind/unwind.cpp:219.) 2025-12-04T11:13:50.4228607Z if out == self.unknown_value: 2025-12-04T11:13:50.4228729Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.4229104Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.4229240Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.4229781Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('benchmarking.InductorBenchmarker.benchmark_gpu', 2), ('pad_mm_bench', 1), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.4229885Z graph_break [] 2025-12-04T11:13:50.4229969Z =================================== FAILURES =================================== 2025-12-04T11:13:50.4230268Z _ TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_1_in_features_128_out_features_65_cuda_bfloat16 _ 2025-12-04T11:13:50.4230348Z Traceback (most recent call last): 2025-12-04T11:13:50.4230653Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.4230722Z method(*args, **kwargs) 2025-12-04T11:13:50.4231015Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.4231079Z method(*args, **kwargs) 2025-12-04T11:13:50.4231372Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T11:13:50.4231433Z with policy(): 2025-12-04T11:13:50.4231723Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T11:13:50.4231793Z raise RuntimeError(msg) 2025-12-04T11:13:50.4232606Z RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_1_in_features_128_out_features_65_cuda_bfloat16! Caching allocator allocated memory was 17408 and is now reported as 26112 on device 0. CUDA driver allocated memory was 276824064 and is now 278921216. 2025-12-04T11:13:50.4232610Z 2025-12-04T11:13:50.4232742Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.4233257Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_1_in_features_128_out_features_65_cuda_bfloat16 2025-12-04T11:13:50.4233262Z 2025-12-04T11:13:50.4233423Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.4233549Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.4233640Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.4234185Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('benchmarking.InductorBenchmarker.benchmark_gpu', 2), ('pad_mm_bench', 1), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.4234310Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.4234373Z graph_break [] 2025-12-04T11:13:50.4234494Z ----------------------------- Captured stderr call ----------------------------- 2025-12-04T11:13:50.4235225Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/constant_folding.py:256: UserWarning: Unsupported unwinding pattern: Address not in range (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/profiler/unwind/unwind.cpp:219.) 2025-12-04T11:13:50.4235334Z if out == self.unknown_value: 2025-12-04T11:13:50.4235457Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.4235547Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.4235674Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.4236250Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('benchmarking.InductorBenchmarker.benchmark_gpu', 2), ('pad_mm_bench', 1), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.4236314Z graph_break [] 2025-12-04T11:13:50.4236437Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.4236526Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.4236650Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.4237197Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('benchmarking.InductorBenchmarker.benchmark_gpu', 2), ('pad_mm_bench', 1), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.4237293Z graph_break [] 2025-12-04T11:13:50.4237783Z - generated xml file: /var/lib/jenkins/workspace/test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-3b52d4f5cebad92d.xml - 2025-12-04T11:13:50.4237887Z =========================== short test summary info ============================ 2025-12-04T11:13:50.4239173Z FAILED [0.5012s] inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_32_mid_dim_1_in_features_128_out_features_65_cuda_bfloat16 - RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_1_in_features_128_out_features_65_cuda_bfloat16! Caching allocator allocated memory was 17408 and is now reported as 26112 on device 0. CUDA driver allocated memory was 276824064 and is now 278921216. 2025-12-04T11:13:50.4239180Z 2025-12-04T11:13:50.4239305Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.4239832Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_1_in_features_128_out_features_65_cuda_bfloat16 2025-12-04T11:13:50.4239835Z 2025-12-04T11:13:50.4239990Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.4240098Z !!!!!!!!!!!!!!!!!!!!!!!!!! stopping after 1 failures !!!!!!!!!!!!!!!!!!!!!!!!!!! 2025-12-04T11:13:50.4240216Z ================== 1 failed, 57 deselected, 2 rerun in 12.31s ================== 2025-12-04T11:13:50.4240276Z Got exit code 1 2025-12-04T11:13:50.4240350Z Retrying single test... 2025-12-04T11:13:50.4240615Z W1204 11:03:56.443000 82305 site-packages/torch/_inductor/utils.py:1703] Not enough SMs to use max_autotune_gemm mode 2025-12-04T11:13:50.4241005Z Test results will be stored in test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-2ff4d68a3a0ad43c.xml 2025-12-04T11:13:50.4241101Z ============================= test session starts ============================== 2025-12-04T11:13:50.4241314Z platform linux -- Python 3.10.14, pytest-7.3.2, pluggy-1.6.0 -- /opt/conda/envs/py_3.10/bin/python 2025-12-04T11:13:50.4241384Z cachedir: .pytest_cache 2025-12-04T11:13:50.4241693Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow] 2025-12-04T11:13:50.4241772Z rootdir: /var/lib/jenkins/workspace 2025-12-04T11:13:50.4241840Z configfile: pytest.ini 2025-12-04T11:13:50.4242195Z plugins: hypothesis-6.56.4, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-14.0, subtests-0.13.1, xdist-3.3.1, xdoctest-1.3.0, typeguard-4.3.0 2025-12-04T11:13:50.4242369Z collecting ... collected 58 items / 57 deselected / 1 selected 2025-12-04T11:13:50.4242941Z stepcurrent: skipping 45 already run items. Running only test/inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_32_mid_dim_1_in_features_128_out_features_65_cuda_bfloat16 2025-12-04T11:13:50.4243012Z Running 1 items in this shard 2025-12-04T11:13:50.4243015Z 2025-12-04T11:13:50.4243773Z inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_32_mid_dim_1_in_features_128_out_features_65_cuda_bfloat16 [W1204 11:03:58.081287756 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.4243778Z 2025-12-04T11:13:50.4244077Z [W1204 11:04:07.098196623 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.4244082Z 2025-12-04T11:13:50.4244375Z [W1204 11:04:07.098443049 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.4244411Z 2025-12-04T11:13:50.4244696Z [W1204 11:04:07.104460851 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.4244699Z 2025-12-04T11:13:50.4244985Z [W1204 11:04:07.105048235 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.4244989Z 2025-12-04T11:13:50.4245276Z [W1204 11:04:07.105218208 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.4245279Z 2025-12-04T11:13:50.4245568Z [W1204 11:04:07.110715630 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.4245573Z 2025-12-04T11:13:50.4245859Z [W1204 11:04:07.111257462 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.4245864Z 2025-12-04T11:13:50.4246154Z [W1204 11:04:07.111422856 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.4246161Z 2025-12-04T11:13:50.4246243Z ('RERUN', {'yellow': True}) [11.0040s] [100%] 2025-12-04T11:13:50.4246967Z inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_32_mid_dim_1_in_features_128_out_features_65_cuda_bfloat16 [W1204 11:04:07.931233483 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.4246971Z 2025-12-04T11:13:50.4247272Z [W1204 11:04:07.931789445 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.4247277Z 2025-12-04T11:13:50.4247564Z [W1204 11:04:07.931935668 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.4247569Z 2025-12-04T11:13:50.4247857Z [W1204 11:04:07.934900394 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.4247860Z 2025-12-04T11:13:50.4248145Z [W1204 11:04:07.935367335 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.4248149Z 2025-12-04T11:13:50.4248438Z [W1204 11:04:07.935507207 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.4248441Z 2025-12-04T11:13:50.4248730Z [W1204 11:04:07.940101249 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.4248773Z 2025-12-04T11:13:50.4249109Z [W1204 11:04:07.940582800 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.4249115Z 2025-12-04T11:13:50.4249399Z [W1204 11:04:07.940719403 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.4249403Z 2025-12-04T11:13:50.4249480Z ('RERUN', {'yellow': True}) [0.5099s] [100%] 2025-12-04T11:13:50.4250268Z inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_32_mid_dim_1_in_features_128_out_features_65_cuda_bfloat16 [W1204 11:04:08.437876723 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.4250273Z 2025-12-04T11:13:50.4250597Z [W1204 11:04:08.438454296 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.4250602Z 2025-12-04T11:13:50.4250898Z [W1204 11:04:08.438596909 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.4250934Z 2025-12-04T11:13:50.4251220Z [W1204 11:04:08.441623136 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.4251224Z 2025-12-04T11:13:50.4251513Z [W1204 11:04:08.442099267 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.4251516Z 2025-12-04T11:13:50.4251800Z [W1204 11:04:08.442235850 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.4251805Z 2025-12-04T11:13:50.4252093Z [W1204 11:04:08.446773749 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.4252098Z 2025-12-04T11:13:50.4252384Z [W1204 11:04:08.447243249 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.4252387Z 2025-12-04T11:13:50.4252683Z [W1204 11:04:08.447380172 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.4252687Z 2025-12-04T11:13:50.4252750Z FAILED [0.5055s] [100%] 2025-12-04T11:13:50.4252754Z 2025-12-04T11:13:50.4252840Z ==================================== RERUNS ==================================== 2025-12-04T11:13:50.4253139Z _ TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_1_in_features_128_out_features_65_cuda_bfloat16 _ 2025-12-04T11:13:50.4253217Z Traceback (most recent call last): 2025-12-04T11:13:50.4253524Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.4253594Z method(*args, **kwargs) 2025-12-04T11:13:50.4253888Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.4253956Z method(*args, **kwargs) 2025-12-04T11:13:50.4254247Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T11:13:50.4254306Z with policy(): 2025-12-04T11:13:50.4254602Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T11:13:50.4254668Z raise RuntimeError(msg) 2025-12-04T11:13:50.4255466Z RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_1_in_features_128_out_features_65_cuda_bfloat16! Caching allocator allocated memory was 0 and is now reported as 8704 on device 0. CUDA driver allocated memory was 230686720 and is now 274726912. 2025-12-04T11:13:50.4255509Z 2025-12-04T11:13:50.4255641Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.4256198Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_1_in_features_128_out_features_65_cuda_bfloat16 2025-12-04T11:13:50.4256214Z 2025-12-04T11:13:50.4256381Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.4256511Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.4256610Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.4257192Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('benchmarking.InductorBenchmarker.benchmark_gpu', 2), ('pad_mm_bench', 1), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.4257322Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.4257385Z graph_break [] 2025-12-04T11:13:50.4257511Z ----------------------------- Captured stderr call ----------------------------- 2025-12-04T11:13:50.4258204Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/constant_folding.py:256: UserWarning: Unsupported unwinding pattern: Address not in range (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/profiler/unwind/unwind.cpp:219.) 2025-12-04T11:13:50.4258312Z if out == self.unknown_value: 2025-12-04T11:13:50.4258603Z _ TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_1_in_features_128_out_features_65_cuda_bfloat16 _ 2025-12-04T11:13:50.4258683Z Traceback (most recent call last): 2025-12-04T11:13:50.4258980Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.4259047Z method(*args, **kwargs) 2025-12-04T11:13:50.4259336Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.4259400Z method(*args, **kwargs) 2025-12-04T11:13:50.4259691Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T11:13:50.4259752Z with policy(): 2025-12-04T11:13:50.4260045Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T11:13:50.4260114Z raise RuntimeError(msg) 2025-12-04T11:13:50.4260917Z RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_1_in_features_128_out_features_65_cuda_bfloat16! Caching allocator allocated memory was 8704 and is now reported as 17408 on device 0. CUDA driver allocated memory was 274726912 and is now 276824064. 2025-12-04T11:13:50.4260921Z 2025-12-04T11:13:50.4261050Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.4261573Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_1_in_features_128_out_features_65_cuda_bfloat16 2025-12-04T11:13:50.4261578Z 2025-12-04T11:13:50.4261738Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.4261864Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.4261957Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.4262503Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('benchmarking.InductorBenchmarker.benchmark_gpu', 2), ('pad_mm_bench', 1), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.4262628Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.4262691Z graph_break [] 2025-12-04T11:13:50.4262860Z ----------------------------- Captured stderr call ----------------------------- 2025-12-04T11:13:50.4263634Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/constant_folding.py:256: UserWarning: Unsupported unwinding pattern: Address not in range (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/profiler/unwind/unwind.cpp:219.) 2025-12-04T11:13:50.4263713Z if out == self.unknown_value: 2025-12-04T11:13:50.4263839Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.4263934Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.4264063Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.4264637Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('benchmarking.InductorBenchmarker.benchmark_gpu', 2), ('pad_mm_bench', 1), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.4264703Z graph_break [] 2025-12-04T11:13:50.4264787Z =================================== FAILURES =================================== 2025-12-04T11:13:50.4265079Z _ TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_1_in_features_128_out_features_65_cuda_bfloat16 _ 2025-12-04T11:13:50.4265192Z Traceback (most recent call last): 2025-12-04T11:13:50.4265491Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.4265555Z method(*args, **kwargs) 2025-12-04T11:13:50.4265845Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.4265907Z method(*args, **kwargs) 2025-12-04T11:13:50.4266200Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T11:13:50.4266259Z with policy(): 2025-12-04T11:13:50.4266549Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T11:13:50.4266620Z raise RuntimeError(msg) 2025-12-04T11:13:50.4267430Z RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_1_in_features_128_out_features_65_cuda_bfloat16! Caching allocator allocated memory was 17408 and is now reported as 26112 on device 0. CUDA driver allocated memory was 276824064 and is now 278921216. 2025-12-04T11:13:50.4267435Z 2025-12-04T11:13:50.4267562Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.4268079Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_1_in_features_128_out_features_65_cuda_bfloat16 2025-12-04T11:13:50.4268083Z 2025-12-04T11:13:50.4268239Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.4268366Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.4268459Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.4269003Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('benchmarking.InductorBenchmarker.benchmark_gpu', 2), ('pad_mm_bench', 1), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.4269128Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.4269190Z graph_break [] 2025-12-04T11:13:50.4269313Z ----------------------------- Captured stderr call ----------------------------- 2025-12-04T11:13:50.4270002Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/constant_folding.py:256: UserWarning: Unsupported unwinding pattern: Address not in range (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/profiler/unwind/unwind.cpp:219.) 2025-12-04T11:13:50.4270117Z if out == self.unknown_value: 2025-12-04T11:13:50.4270240Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.4270368Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.4270493Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.4271030Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('benchmarking.InductorBenchmarker.benchmark_gpu', 2), ('pad_mm_bench', 1), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.4271092Z graph_break [] 2025-12-04T11:13:50.4271215Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.4271304Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.4271468Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.4272011Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('benchmarking.InductorBenchmarker.benchmark_gpu', 2), ('pad_mm_bench', 1), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.4272074Z graph_break [] 2025-12-04T11:13:50.4272614Z - generated xml file: /var/lib/jenkins/workspace/test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-2ff4d68a3a0ad43c.xml - 2025-12-04T11:13:50.4272713Z =========================== short test summary info ============================ 2025-12-04T11:13:50.4273993Z FAILED [0.5055s] inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_32_mid_dim_1_in_features_128_out_features_65_cuda_bfloat16 - RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_1_in_features_128_out_features_65_cuda_bfloat16! Caching allocator allocated memory was 17408 and is now reported as 26112 on device 0. CUDA driver allocated memory was 276824064 and is now 278921216. 2025-12-04T11:13:50.4273999Z 2025-12-04T11:13:50.4274125Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.4274643Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_1_in_features_128_out_features_65_cuda_bfloat16 2025-12-04T11:13:50.4274648Z 2025-12-04T11:13:50.4274801Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.4274908Z !!!!!!!!!!!!!!!!!!!!!!!!!! stopping after 1 failures !!!!!!!!!!!!!!!!!!!!!!!!!!! 2025-12-04T11:13:50.4275021Z ================== 1 failed, 57 deselected, 2 rerun in 12.04s ================== 2025-12-04T11:13:50.4275080Z Got exit code 1 2025-12-04T11:13:50.4275557Z FAILED CONSISTENTLY: test/inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_32_mid_dim_1_in_features_128_out_features_65_cuda_bfloat16 2025-12-04T11:13:50.4275801Z Test failed consistently, continuing with the rest of the tests due to continue-through-error being set 2025-12-04T11:13:50.4276067Z W1204 11:04:15.310000 82492 site-packages/torch/_inductor/utils.py:1703] Not enough SMs to use max_autotune_gemm mode 2025-12-04T11:13:50.4276449Z Test results will be stored in test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-7240a6bd78b3c1ba.xml 2025-12-04T11:13:50.4276545Z ============================= test session starts ============================== 2025-12-04T11:13:50.4276752Z platform linux -- Python 3.10.14, pytest-7.3.2, pluggy-1.6.0 -- /opt/conda/envs/py_3.10/bin/python 2025-12-04T11:13:50.4276819Z cachedir: .pytest_cache 2025-12-04T11:13:50.4277127Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow] 2025-12-04T11:13:50.4277205Z rootdir: /var/lib/jenkins/workspace 2025-12-04T11:13:50.4277270Z configfile: pytest.ini 2025-12-04T11:13:50.4277682Z plugins: hypothesis-6.56.4, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-14.0, subtests-0.13.1, xdist-3.3.1, xdoctest-1.3.0, typeguard-4.3.0 2025-12-04T11:13:50.4277976Z collecting ... collected 58 items / 46 deselected / 12 selected 2025-12-04T11:13:50.4278073Z stepcurrent: skipping 46 already run items. 2025-12-04T11:13:50.4278146Z Running 12 items in this shard 2025-12-04T11:13:50.4278151Z 2025-12-04T11:13:50.4278659Z inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_32_mid_dim_1_in_features_144_out_features_1024_cuda_bfloat16 ('RERUN', {'yellow': True}) [1.9059s] [ 8%] 2025-12-04T11:13:50.4279226Z inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_32_mid_dim_1_in_features_144_out_features_1024_cuda_bfloat16 ('RERUN', {'yellow': True}) [0.5024s] [ 8%] 2025-12-04T11:13:50.4279677Z inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_32_mid_dim_1_in_features_144_out_features_1024_cuda_bfloat16 FAILED [0.4977s] [ 8%] 2025-12-04T11:13:50.4279683Z 2025-12-04T11:13:50.4279769Z ==================================== RERUNS ==================================== 2025-12-04T11:13:50.4280117Z _ TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_1_in_features_144_out_features_1024_cuda_bfloat16 _ 2025-12-04T11:13:50.4280192Z Traceback (most recent call last): 2025-12-04T11:13:50.4280502Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.4280567Z method(*args, **kwargs) 2025-12-04T11:13:50.4280863Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.4280929Z method(*args, **kwargs) 2025-12-04T11:13:50.4281217Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T11:13:50.4281281Z with policy(): 2025-12-04T11:13:50.4281574Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T11:13:50.4281638Z raise RuntimeError(msg) 2025-12-04T11:13:50.4282455Z RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_1_in_features_144_out_features_1024_cuda_bfloat16! Caching allocator allocated memory was 0 and is now reported as 147456 on device 0. CUDA driver allocated memory was 230686720 and is now 274726912. 2025-12-04T11:13:50.4282459Z 2025-12-04T11:13:50.4282586Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.4283111Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_1_in_features_144_out_features_1024_cuda_bfloat16 2025-12-04T11:13:50.4283116Z 2025-12-04T11:13:50.4283270Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.4283403Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.4283496Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.4283855Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.4283986Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.4284044Z graph_break [] 2025-12-04T11:13:50.4284337Z _ TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_1_in_features_144_out_features_1024_cuda_bfloat16 _ 2025-12-04T11:13:50.4284415Z Traceback (most recent call last): 2025-12-04T11:13:50.4284711Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.4284920Z method(*args, **kwargs) 2025-12-04T11:13:50.4285272Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.4285339Z method(*args, **kwargs) 2025-12-04T11:13:50.4285636Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T11:13:50.4285694Z with policy(): 2025-12-04T11:13:50.4285988Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T11:13:50.4286057Z raise RuntimeError(msg) 2025-12-04T11:13:50.4286916Z RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_1_in_features_144_out_features_1024_cuda_bfloat16! Caching allocator allocated memory was 147456 and is now reported as 294912 on device 0. CUDA driver allocated memory was 274726912 and is now 276824064. 2025-12-04T11:13:50.4286922Z 2025-12-04T11:13:50.4287051Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.4287571Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_1_in_features_144_out_features_1024_cuda_bfloat16 2025-12-04T11:13:50.4287609Z 2025-12-04T11:13:50.4287770Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.4287898Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.4287990Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.4288340Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.4288475Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.4288540Z graph_break [] 2025-12-04T11:13:50.4288663Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.4288753Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.4288880Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.4289224Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.4289281Z graph_break [] 2025-12-04T11:13:50.4289366Z =================================== FAILURES =================================== 2025-12-04T11:13:50.4289662Z _ TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_1_in_features_144_out_features_1024_cuda_bfloat16 _ 2025-12-04T11:13:50.4289741Z Traceback (most recent call last): 2025-12-04T11:13:50.4290042Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.4290107Z method(*args, **kwargs) 2025-12-04T11:13:50.4290404Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.4290467Z method(*args, **kwargs) 2025-12-04T11:13:50.4290755Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T11:13:50.4290820Z with policy(): 2025-12-04T11:13:50.4291110Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T11:13:50.4291178Z raise RuntimeError(msg) 2025-12-04T11:13:50.4292001Z RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_1_in_features_144_out_features_1024_cuda_bfloat16! Caching allocator allocated memory was 294912 and is now reported as 442368 on device 0. CUDA driver allocated memory was 276824064 and is now 278921216. 2025-12-04T11:13:50.4292048Z 2025-12-04T11:13:50.4292174Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.4292730Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_1_in_features_144_out_features_1024_cuda_bfloat16 2025-12-04T11:13:50.4292735Z 2025-12-04T11:13:50.4292891Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.4293023Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.4293113Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.4293493Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.4293617Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.4293677Z graph_break [] 2025-12-04T11:13:50.4293802Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.4293892Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.4294049Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.4294390Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.4294450Z graph_break [] 2025-12-04T11:13:50.4294587Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.4294677Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.4294799Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.4295139Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.4295200Z graph_break [] 2025-12-04T11:13:50.4295688Z - generated xml file: /var/lib/jenkins/workspace/test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-7240a6bd78b3c1ba.xml - 2025-12-04T11:13:50.4295793Z =========================== short test summary info ============================ 2025-12-04T11:13:50.4297095Z FAILED [0.4977s] inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_32_mid_dim_1_in_features_144_out_features_1024_cuda_bfloat16 - RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_1_in_features_144_out_features_1024_cuda_bfloat16! Caching allocator allocated memory was 294912 and is now reported as 442368 on device 0. CUDA driver allocated memory was 276824064 and is now 278921216. 2025-12-04T11:13:50.4297104Z 2025-12-04T11:13:50.4297227Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.4297748Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_1_in_features_144_out_features_1024_cuda_bfloat16 2025-12-04T11:13:50.4297752Z 2025-12-04T11:13:50.4297909Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.4298012Z !!!!!!!!!!!!!!!!!!!!!!!!!! stopping after 1 failures !!!!!!!!!!!!!!!!!!!!!!!!!!! 2025-12-04T11:13:50.4298131Z ================== 1 failed, 46 deselected, 2 rerun in 2.93s =================== 2025-12-04T11:13:50.4298190Z Got exit code 1 2025-12-04T11:13:50.4298256Z Retrying single test... 2025-12-04T11:13:50.4298521Z W1204 11:04:25.358000 82681 site-packages/torch/_inductor/utils.py:1703] Not enough SMs to use max_autotune_gemm mode 2025-12-04T11:13:50.4298906Z Test results will be stored in test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-7cb1dae7cd5b9c6d.xml 2025-12-04T11:13:50.4299044Z ============================= test session starts ============================== 2025-12-04T11:13:50.4299291Z platform linux -- Python 3.10.14, pytest-7.3.2, pluggy-1.6.0 -- /opt/conda/envs/py_3.10/bin/python 2025-12-04T11:13:50.4299361Z cachedir: .pytest_cache 2025-12-04T11:13:50.4299668Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow] 2025-12-04T11:13:50.4299744Z rootdir: /var/lib/jenkins/workspace 2025-12-04T11:13:50.4299808Z configfile: pytest.ini 2025-12-04T11:13:50.4300123Z plugins: hypothesis-6.56.4, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-14.0, subtests-0.13.1, xdist-3.3.1, xdoctest-1.3.0, typeguard-4.3.0 2025-12-04T11:13:50.4300284Z collecting ... collected 58 items / 57 deselected / 1 selected 2025-12-04T11:13:50.4300857Z stepcurrent: skipping 46 already run items. Running only test/inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_32_mid_dim_1_in_features_144_out_features_1024_cuda_bfloat16 2025-12-04T11:13:50.4300944Z Running 1 items in this shard 2025-12-04T11:13:50.4300949Z 2025-12-04T11:13:50.4301686Z inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_32_mid_dim_1_in_features_144_out_features_1024_cuda_bfloat16 [W1204 11:04:26.475739641 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.4301743Z 2025-12-04T11:13:50.4302044Z [W1204 11:04:35.790165091 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.4302048Z 2025-12-04T11:13:50.4302339Z [W1204 11:04:35.790404226 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.4302342Z 2025-12-04T11:13:50.4302633Z [W1204 11:04:35.796062391 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.4302637Z 2025-12-04T11:13:50.4302926Z [W1204 11:04:35.796625914 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.4302930Z 2025-12-04T11:13:50.4303221Z [W1204 11:04:35.796788748 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.4303224Z 2025-12-04T11:13:50.4303547Z [W1204 11:04:35.802045454 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.4303551Z 2025-12-04T11:13:50.4303841Z [W1204 11:04:35.802563085 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.4303844Z 2025-12-04T11:13:50.4304127Z [W1204 11:04:35.802716459 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.4304132Z 2025-12-04T11:13:50.4304212Z ('RERUN', {'yellow': True}) [11.2285s] [100%] 2025-12-04T11:13:50.4304946Z inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_32_mid_dim_1_in_features_144_out_features_1024_cuda_bfloat16 [W1204 11:04:37.024483143 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.4304951Z 2025-12-04T11:13:50.4305240Z [W1204 11:04:37.025031885 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.4305243Z 2025-12-04T11:13:50.4305533Z [W1204 11:04:37.025177948 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.4305536Z 2025-12-04T11:13:50.4305823Z [W1204 11:04:37.028144224 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.4305868Z 2025-12-04T11:13:50.4306191Z [W1204 11:04:37.028731217 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.4306196Z 2025-12-04T11:13:50.4306481Z [W1204 11:04:37.028871210 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.4306484Z 2025-12-04T11:13:50.4306772Z [W1204 11:04:37.033407121 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.4306776Z 2025-12-04T11:13:50.4307096Z [W1204 11:04:37.033876441 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.4307100Z 2025-12-04T11:13:50.4307389Z [W1204 11:04:37.034012274 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.4307394Z 2025-12-04T11:13:50.4307473Z ('RERUN', {'yellow': True}) [0.4546s] [100%] 2025-12-04T11:13:50.4308203Z inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_32_mid_dim_1_in_features_144_out_features_1024_cuda_bfloat16 [W1204 11:04:37.473168754 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.4308241Z 2025-12-04T11:13:50.4308534Z [W1204 11:04:37.473707816 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.4308537Z 2025-12-04T11:13:50.4308823Z [W1204 11:04:37.473853469 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.4308826Z 2025-12-04T11:13:50.4309127Z [W1204 11:04:37.476776374 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.4309131Z 2025-12-04T11:13:50.4309417Z [W1204 11:04:37.477339196 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.4309422Z 2025-12-04T11:13:50.4309718Z [W1204 11:04:37.477474949 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.4309721Z 2025-12-04T11:13:50.4310008Z [W1204 11:04:37.481991839 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.4310011Z 2025-12-04T11:13:50.4310298Z [W1204 11:04:37.482461349 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.4310304Z 2025-12-04T11:13:50.4310588Z [W1204 11:04:37.482596522 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.4310593Z 2025-12-04T11:13:50.4310658Z FAILED [0.4460s] [100%] 2025-12-04T11:13:50.4310661Z 2025-12-04T11:13:50.4310753Z ==================================== RERUNS ==================================== 2025-12-04T11:13:50.4311048Z _ TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_1_in_features_144_out_features_1024_cuda_bfloat16 _ 2025-12-04T11:13:50.4311125Z Traceback (most recent call last): 2025-12-04T11:13:50.4311431Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.4311495Z method(*args, **kwargs) 2025-12-04T11:13:50.4311791Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.4311855Z method(*args, **kwargs) 2025-12-04T11:13:50.4312147Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T11:13:50.4312248Z with policy(): 2025-12-04T11:13:50.4312579Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T11:13:50.4312651Z raise RuntimeError(msg) 2025-12-04T11:13:50.4313462Z RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_1_in_features_144_out_features_1024_cuda_bfloat16! Caching allocator allocated memory was 0 and is now reported as 147456 on device 0. CUDA driver allocated memory was 230686720 and is now 274726912. 2025-12-04T11:13:50.4313466Z 2025-12-04T11:13:50.4313596Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.4314160Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_1_in_features_144_out_features_1024_cuda_bfloat16 2025-12-04T11:13:50.4314166Z 2025-12-04T11:13:50.4314329Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.4314463Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.4314557Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.4314942Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.4315067Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.4315125Z graph_break [] 2025-12-04T11:13:50.4315254Z ----------------------------- Captured stderr call ----------------------------- 2025-12-04T11:13:50.4315945Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/constant_folding.py:256: UserWarning: Unsupported unwinding pattern: Address not in range (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/profiler/unwind/unwind.cpp:219.) 2025-12-04T11:13:50.4316022Z if out == self.unknown_value: 2025-12-04T11:13:50.4316316Z _ TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_1_in_features_144_out_features_1024_cuda_bfloat16 _ 2025-12-04T11:13:50.4316392Z Traceback (most recent call last): 2025-12-04T11:13:50.4316692Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.4316756Z method(*args, **kwargs) 2025-12-04T11:13:50.4317049Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.4317115Z method(*args, **kwargs) 2025-12-04T11:13:50.4317402Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T11:13:50.4317463Z with policy(): 2025-12-04T11:13:50.4317753Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T11:13:50.4317821Z raise RuntimeError(msg) 2025-12-04T11:13:50.4318643Z RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_1_in_features_144_out_features_1024_cuda_bfloat16! Caching allocator allocated memory was 147456 and is now reported as 294912 on device 0. CUDA driver allocated memory was 274726912 and is now 276824064. 2025-12-04T11:13:50.4318649Z 2025-12-04T11:13:50.4318773Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.4319293Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_1_in_features_144_out_features_1024_cuda_bfloat16 2025-12-04T11:13:50.4319298Z 2025-12-04T11:13:50.4319464Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.4319588Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.4319742Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.4320118Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.4320250Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.4320309Z graph_break [] 2025-12-04T11:13:50.4320433Z ----------------------------- Captured stderr call ----------------------------- 2025-12-04T11:13:50.4321158Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/constant_folding.py:256: UserWarning: Unsupported unwinding pattern: Address not in range (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/profiler/unwind/unwind.cpp:219.) 2025-12-04T11:13:50.4321230Z if out == self.unknown_value: 2025-12-04T11:13:50.4321354Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.4321446Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.4321570Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.4321912Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.4322005Z graph_break [] 2025-12-04T11:13:50.4322086Z =================================== FAILURES =================================== 2025-12-04T11:13:50.4322381Z _ TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_1_in_features_144_out_features_1024_cuda_bfloat16 _ 2025-12-04T11:13:50.4322454Z Traceback (most recent call last): 2025-12-04T11:13:50.4322754Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.4322819Z method(*args, **kwargs) 2025-12-04T11:13:50.4323108Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.4323175Z method(*args, **kwargs) 2025-12-04T11:13:50.4323465Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T11:13:50.4323526Z with policy(): 2025-12-04T11:13:50.4323818Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T11:13:50.4323883Z raise RuntimeError(msg) 2025-12-04T11:13:50.4324705Z RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_1_in_features_144_out_features_1024_cuda_bfloat16! Caching allocator allocated memory was 294912 and is now reported as 442368 on device 0. CUDA driver allocated memory was 276824064 and is now 278921216. 2025-12-04T11:13:50.4324709Z 2025-12-04T11:13:50.4324833Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.4325367Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_1_in_features_144_out_features_1024_cuda_bfloat16 2025-12-04T11:13:50.4325372Z 2025-12-04T11:13:50.4325527Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.4325652Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.4325749Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.4326091Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.4326216Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.4326272Z graph_break [] 2025-12-04T11:13:50.4326395Z ----------------------------- Captured stderr call ----------------------------- 2025-12-04T11:13:50.4327155Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/constant_folding.py:256: UserWarning: Unsupported unwinding pattern: Address not in range (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/profiler/unwind/unwind.cpp:219.) 2025-12-04T11:13:50.4327228Z if out == self.unknown_value: 2025-12-04T11:13:50.4327356Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.4327447Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.4327569Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.4327963Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.4328025Z graph_break [] 2025-12-04T11:13:50.4328146Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.4328240Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.4328361Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.4328701Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.4328795Z graph_break [] 2025-12-04T11:13:50.4329283Z - generated xml file: /var/lib/jenkins/workspace/test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-7cb1dae7cd5b9c6d.xml - 2025-12-04T11:13:50.4329385Z =========================== short test summary info ============================ 2025-12-04T11:13:50.4330681Z FAILED [0.4460s] inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_32_mid_dim_1_in_features_144_out_features_1024_cuda_bfloat16 - RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_1_in_features_144_out_features_1024_cuda_bfloat16! Caching allocator allocated memory was 294912 and is now reported as 442368 on device 0. CUDA driver allocated memory was 276824064 and is now 278921216. 2025-12-04T11:13:50.4330688Z 2025-12-04T11:13:50.4330814Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.4331333Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_1_in_features_144_out_features_1024_cuda_bfloat16 2025-12-04T11:13:50.4331336Z 2025-12-04T11:13:50.4331491Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.4331596Z !!!!!!!!!!!!!!!!!!!!!!!!!! stopping after 1 failures !!!!!!!!!!!!!!!!!!!!!!!!!!! 2025-12-04T11:13:50.4331721Z ================== 1 failed, 57 deselected, 2 rerun in 12.15s ================== 2025-12-04T11:13:50.4331786Z Got exit code 1 2025-12-04T11:13:50.4331852Z Retrying single test... 2025-12-04T11:13:50.4332115Z W1204 11:04:44.341000 82874 site-packages/torch/_inductor/utils.py:1703] Not enough SMs to use max_autotune_gemm mode 2025-12-04T11:13:50.4332501Z Test results will be stored in test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-1238d86169b8b87a.xml 2025-12-04T11:13:50.4332598Z ============================= test session starts ============================== 2025-12-04T11:13:50.4332813Z platform linux -- Python 3.10.14, pytest-7.3.2, pluggy-1.6.0 -- /opt/conda/envs/py_3.10/bin/python 2025-12-04T11:13:50.4332878Z cachedir: .pytest_cache 2025-12-04T11:13:50.4333180Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow] 2025-12-04T11:13:50.4333263Z rootdir: /var/lib/jenkins/workspace 2025-12-04T11:13:50.4333330Z configfile: pytest.ini 2025-12-04T11:13:50.4333646Z plugins: hypothesis-6.56.4, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-14.0, subtests-0.13.1, xdist-3.3.1, xdoctest-1.3.0, typeguard-4.3.0 2025-12-04T11:13:50.4333852Z collecting ... collected 58 items / 57 deselected / 1 selected 2025-12-04T11:13:50.4334425Z stepcurrent: skipping 46 already run items. Running only test/inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_32_mid_dim_1_in_features_144_out_features_1024_cuda_bfloat16 2025-12-04T11:13:50.4334503Z Running 1 items in this shard 2025-12-04T11:13:50.4334506Z 2025-12-04T11:13:50.4335270Z inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_32_mid_dim_1_in_features_144_out_features_1024_cuda_bfloat16 [W1204 11:04:45.478045919 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.4335275Z 2025-12-04T11:13:50.4335575Z [W1204 11:04:54.675224767 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.4335580Z 2025-12-04T11:13:50.4335870Z [W1204 11:04:54.675456942 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.4335907Z 2025-12-04T11:13:50.4336197Z [W1204 11:04:54.681206849 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.4336201Z 2025-12-04T11:13:50.4336486Z [W1204 11:04:54.681764551 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.4336490Z 2025-12-04T11:13:50.4336778Z [W1204 11:04:54.681931045 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.4336783Z 2025-12-04T11:13:50.4337070Z [W1204 11:04:54.687369556 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.4337075Z 2025-12-04T11:13:50.4337362Z [W1204 11:04:54.687891217 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.4337369Z 2025-12-04T11:13:50.4337657Z [W1204 11:04:54.688053851 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.4337660Z 2025-12-04T11:13:50.4337740Z ('RERUN', {'yellow': True}) [11.1311s] [100%] 2025-12-04T11:13:50.4338471Z inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_32_mid_dim_1_in_features_144_out_features_1024_cuda_bfloat16 [W1204 11:04:55.917781831 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.4338476Z 2025-12-04T11:13:50.4338766Z [W1204 11:04:55.918323853 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.4338771Z 2025-12-04T11:13:50.4339060Z [W1204 11:04:55.918464386 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.4339063Z 2025-12-04T11:13:50.4339349Z [W1204 11:04:55.921404311 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.4339353Z 2025-12-04T11:13:50.4339643Z [W1204 11:04:55.921975694 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.4339647Z 2025-12-04T11:13:50.4339931Z [W1204 11:04:55.922113657 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.4339934Z 2025-12-04T11:13:50.4340221Z [W1204 11:04:55.926636997 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.4340265Z 2025-12-04T11:13:50.4340587Z [W1204 11:04:55.927103777 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.4340591Z 2025-12-04T11:13:50.4340876Z [W1204 11:04:55.927240971 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.4340881Z 2025-12-04T11:13:50.4340961Z ('RERUN', {'yellow': True}) [0.4632s] [100%] 2025-12-04T11:13:50.4341689Z inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_32_mid_dim_1_in_features_144_out_features_1024_cuda_bfloat16 [W1204 11:04:56.373583959 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.4341693Z 2025-12-04T11:13:50.4342020Z [W1204 11:04:56.374116171 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.4342026Z 2025-12-04T11:13:50.4342312Z [W1204 11:04:56.374252714 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.4342315Z 2025-12-04T11:13:50.4342603Z [W1204 11:04:56.377209339 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.4342640Z 2025-12-04T11:13:50.4342927Z [W1204 11:04:56.377798602 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.4342930Z 2025-12-04T11:13:50.4343219Z [W1204 11:04:56.377934865 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.4343223Z 2025-12-04T11:13:50.4343552Z [W1204 11:04:56.382492455 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.4343556Z 2025-12-04T11:13:50.4343843Z [W1204 11:04:56.382963136 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.4343851Z 2025-12-04T11:13:50.4344135Z [W1204 11:04:56.383098889 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.4344139Z 2025-12-04T11:13:50.4344200Z FAILED [0.4526s] [100%] 2025-12-04T11:13:50.4344204Z 2025-12-04T11:13:50.4344293Z ==================================== RERUNS ==================================== 2025-12-04T11:13:50.4344588Z _ TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_1_in_features_144_out_features_1024_cuda_bfloat16 _ 2025-12-04T11:13:50.4344669Z Traceback (most recent call last): 2025-12-04T11:13:50.4344985Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.4345050Z method(*args, **kwargs) 2025-12-04T11:13:50.4345345Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.4345410Z method(*args, **kwargs) 2025-12-04T11:13:50.4345699Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T11:13:50.4345764Z with policy(): 2025-12-04T11:13:50.4346054Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T11:13:50.4346125Z raise RuntimeError(msg) 2025-12-04T11:13:50.4346929Z RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_1_in_features_144_out_features_1024_cuda_bfloat16! Caching allocator allocated memory was 0 and is now reported as 147456 on device 0. CUDA driver allocated memory was 230686720 and is now 274726912. 2025-12-04T11:13:50.4346933Z 2025-12-04T11:13:50.4347104Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.4347664Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_1_in_features_144_out_features_1024_cuda_bfloat16 2025-12-04T11:13:50.4347670Z 2025-12-04T11:13:50.4347828Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.4347960Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.4348053Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.4348401Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.4348565Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.4348626Z graph_break [] 2025-12-04T11:13:50.4348753Z ----------------------------- Captured stderr call ----------------------------- 2025-12-04T11:13:50.4349443Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/constant_folding.py:256: UserWarning: Unsupported unwinding pattern: Address not in range (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/profiler/unwind/unwind.cpp:219.) 2025-12-04T11:13:50.4349549Z if out == self.unknown_value: 2025-12-04T11:13:50.4349842Z _ TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_1_in_features_144_out_features_1024_cuda_bfloat16 _ 2025-12-04T11:13:50.4349915Z Traceback (most recent call last): 2025-12-04T11:13:50.4350208Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.4350272Z method(*args, **kwargs) 2025-12-04T11:13:50.4350563Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.4350628Z method(*args, **kwargs) 2025-12-04T11:13:50.4350918Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T11:13:50.4350978Z with policy(): 2025-12-04T11:13:50.4351273Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T11:13:50.4351340Z raise RuntimeError(msg) 2025-12-04T11:13:50.4352158Z RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_1_in_features_144_out_features_1024_cuda_bfloat16! Caching allocator allocated memory was 147456 and is now reported as 294912 on device 0. CUDA driver allocated memory was 274726912 and is now 276824064. 2025-12-04T11:13:50.4352162Z 2025-12-04T11:13:50.4352285Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.4352805Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_1_in_features_144_out_features_1024_cuda_bfloat16 2025-12-04T11:13:50.4352812Z 2025-12-04T11:13:50.4352967Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.4353092Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.4353187Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.4353530Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.4353659Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.4353719Z graph_break [] 2025-12-04T11:13:50.4353841Z ----------------------------- Captured stderr call ----------------------------- 2025-12-04T11:13:50.4354571Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/constant_folding.py:256: UserWarning: Unsupported unwinding pattern: Address not in range (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/profiler/unwind/unwind.cpp:219.) 2025-12-04T11:13:50.4354694Z if out == self.unknown_value: 2025-12-04T11:13:50.4354819Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.4354912Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.4355036Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.4355381Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.4355439Z graph_break [] 2025-12-04T11:13:50.4355560Z =================================== FAILURES =================================== 2025-12-04T11:13:50.4355856Z _ TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_1_in_features_144_out_features_1024_cuda_bfloat16 _ 2025-12-04T11:13:50.4355930Z Traceback (most recent call last): 2025-12-04T11:13:50.4356243Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.4356349Z method(*args, **kwargs) 2025-12-04T11:13:50.4356639Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.4356706Z method(*args, **kwargs) 2025-12-04T11:13:50.4356997Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T11:13:50.4357057Z with policy(): 2025-12-04T11:13:50.4357353Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T11:13:50.4357420Z raise RuntimeError(msg) 2025-12-04T11:13:50.4358242Z RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_1_in_features_144_out_features_1024_cuda_bfloat16! Caching allocator allocated memory was 294912 and is now reported as 442368 on device 0. CUDA driver allocated memory was 276824064 and is now 278921216. 2025-12-04T11:13:50.4358249Z 2025-12-04T11:13:50.4358373Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.4358890Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_1_in_features_144_out_features_1024_cuda_bfloat16 2025-12-04T11:13:50.4358897Z 2025-12-04T11:13:50.4359051Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.4359176Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.4359268Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.4359613Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.4359741Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.4359804Z graph_break [] 2025-12-04T11:13:50.4359930Z ----------------------------- Captured stderr call ----------------------------- 2025-12-04T11:13:50.4360617Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/constant_folding.py:256: UserWarning: Unsupported unwinding pattern: Address not in range (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/profiler/unwind/unwind.cpp:219.) 2025-12-04T11:13:50.4360686Z if out == self.unknown_value: 2025-12-04T11:13:50.4360808Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.4360901Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.4361027Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.4361444Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.4361505Z graph_break [] 2025-12-04T11:13:50.4361628Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.4361723Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.4361841Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.4362178Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.4362239Z graph_break [] 2025-12-04T11:13:50.4362754Z - generated xml file: /var/lib/jenkins/workspace/test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-1238d86169b8b87a.xml - 2025-12-04T11:13:50.4362855Z =========================== short test summary info ============================ 2025-12-04T11:13:50.4364162Z FAILED [0.4526s] inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_32_mid_dim_1_in_features_144_out_features_1024_cuda_bfloat16 - RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_1_in_features_144_out_features_1024_cuda_bfloat16! Caching allocator allocated memory was 294912 and is now reported as 442368 on device 0. CUDA driver allocated memory was 276824064 and is now 278921216. 2025-12-04T11:13:50.4364200Z 2025-12-04T11:13:50.4364329Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.4364853Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_1_in_features_144_out_features_1024_cuda_bfloat16 2025-12-04T11:13:50.4364857Z 2025-12-04T11:13:50.4365016Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.4365122Z !!!!!!!!!!!!!!!!!!!!!!!!!! stopping after 1 failures !!!!!!!!!!!!!!!!!!!!!!!!!!! 2025-12-04T11:13:50.4365238Z ================== 1 failed, 57 deselected, 2 rerun in 12.07s ================== 2025-12-04T11:13:50.4365300Z Got exit code 1 2025-12-04T11:13:50.4365775Z FAILED CONSISTENTLY: test/inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_32_mid_dim_1_in_features_144_out_features_1024_cuda_bfloat16 2025-12-04T11:13:50.4366017Z Test failed consistently, continuing with the rest of the tests due to continue-through-error being set 2025-12-04T11:13:50.4366278Z W1204 11:05:03.218000 83067 site-packages/torch/_inductor/utils.py:1703] Not enough SMs to use max_autotune_gemm mode 2025-12-04T11:13:50.4366660Z Test results will be stored in test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-86e4cbaa0e57366b.xml 2025-12-04T11:13:50.4366759Z ============================= test session starts ============================== 2025-12-04T11:13:50.4366966Z platform linux -- Python 3.10.14, pytest-7.3.2, pluggy-1.6.0 -- /opt/conda/envs/py_3.10/bin/python 2025-12-04T11:13:50.4367034Z cachedir: .pytest_cache 2025-12-04T11:13:50.4367338Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow] 2025-12-04T11:13:50.4367412Z rootdir: /var/lib/jenkins/workspace 2025-12-04T11:13:50.4367479Z configfile: pytest.ini 2025-12-04T11:13:50.4367790Z plugins: hypothesis-6.56.4, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-14.0, subtests-0.13.1, xdist-3.3.1, xdoctest-1.3.0, typeguard-4.3.0 2025-12-04T11:13:50.4367915Z collecting ... collected 58 items / 47 deselected / 11 selected 2025-12-04T11:13:50.4368006Z stepcurrent: skipping 47 already run items. 2025-12-04T11:13:50.4368076Z Running 11 items in this shard 2025-12-04T11:13:50.4368079Z 2025-12-04T11:13:50.4368621Z inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_32_mid_dim_1_in_features_144_out_features_64_cuda_bfloat16 ('RERUN', {'yellow': True}) [1.8712s] [ 9%] 2025-12-04T11:13:50.4369154Z inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_32_mid_dim_1_in_features_144_out_features_64_cuda_bfloat16 ('RERUN', {'yellow': True}) [0.4530s] [ 9%] 2025-12-04T11:13:50.4369600Z inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_32_mid_dim_1_in_features_144_out_features_64_cuda_bfloat16 FAILED [0.4563s] [ 9%] 2025-12-04T11:13:50.4369604Z 2025-12-04T11:13:50.4369690Z ==================================== RERUNS ==================================== 2025-12-04T11:13:50.4370012Z _ TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_1_in_features_144_out_features_64_cuda_bfloat16 _ 2025-12-04T11:13:50.4370090Z Traceback (most recent call last): 2025-12-04T11:13:50.4370407Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.4370475Z method(*args, **kwargs) 2025-12-04T11:13:50.4370771Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.4370871Z method(*args, **kwargs) 2025-12-04T11:13:50.4371158Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T11:13:50.4371220Z with policy(): 2025-12-04T11:13:50.4371515Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T11:13:50.4371583Z raise RuntimeError(msg) 2025-12-04T11:13:50.4372377Z RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_1_in_features_144_out_features_64_cuda_bfloat16! Caching allocator allocated memory was 0 and is now reported as 9216 on device 0. CUDA driver allocated memory was 230686720 and is now 274726912. 2025-12-04T11:13:50.4372382Z 2025-12-04T11:13:50.4372507Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.4373028Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_1_in_features_144_out_features_64_cuda_bfloat16 2025-12-04T11:13:50.4373032Z 2025-12-04T11:13:50.4373189Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.4373320Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.4373413Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.4373764Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.4373892Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.4373950Z graph_break [] 2025-12-04T11:13:50.4374246Z _ TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_1_in_features_144_out_features_64_cuda_bfloat16 _ 2025-12-04T11:13:50.4374321Z Traceback (most recent call last): 2025-12-04T11:13:50.4374615Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.4374683Z method(*args, **kwargs) 2025-12-04T11:13:50.4374975Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.4375042Z method(*args, **kwargs) 2025-12-04T11:13:50.4375331Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T11:13:50.4375392Z with policy(): 2025-12-04T11:13:50.4375696Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T11:13:50.4375805Z raise RuntimeError(msg) 2025-12-04T11:13:50.4376660Z RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_1_in_features_144_out_features_64_cuda_bfloat16! Caching allocator allocated memory was 9216 and is now reported as 18432 on device 0. CUDA driver allocated memory was 274726912 and is now 276824064. 2025-12-04T11:13:50.4376670Z 2025-12-04T11:13:50.4376797Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.4377351Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_1_in_features_144_out_features_64_cuda_bfloat16 2025-12-04T11:13:50.4377355Z 2025-12-04T11:13:50.4377519Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.4377647Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.4377747Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.4378250Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.4378444Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.4378508Z graph_break [] 2025-12-04T11:13:50.4378634Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.4378728Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.4378853Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.4379198Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.4379259Z graph_break [] 2025-12-04T11:13:50.4379345Z =================================== FAILURES =================================== 2025-12-04T11:13:50.4379639Z _ TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_1_in_features_144_out_features_64_cuda_bfloat16 _ 2025-12-04T11:13:50.4379732Z Traceback (most recent call last): 2025-12-04T11:13:50.4380040Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.4380107Z method(*args, **kwargs) 2025-12-04T11:13:50.4380400Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.4380464Z method(*args, **kwargs) 2025-12-04T11:13:50.4380756Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T11:13:50.4380816Z with policy(): 2025-12-04T11:13:50.4381114Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T11:13:50.4381183Z raise RuntimeError(msg) 2025-12-04T11:13:50.4381996Z RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_1_in_features_144_out_features_64_cuda_bfloat16! Caching allocator allocated memory was 18432 and is now reported as 27648 on device 0. CUDA driver allocated memory was 276824064 and is now 278921216. 2025-12-04T11:13:50.4382001Z 2025-12-04T11:13:50.4382132Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.4382651Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_1_in_features_144_out_features_64_cuda_bfloat16 2025-12-04T11:13:50.4382655Z 2025-12-04T11:13:50.4382814Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.4383020Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.4383159Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.4383505Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.4383670Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.4383729Z graph_break [] 2025-12-04T11:13:50.4383855Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.4383943Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.4384069Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.4384474Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.4384540Z graph_break [] 2025-12-04T11:13:50.4384671Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.4384766Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.4384928Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.4385272Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.4385331Z graph_break [] 2025-12-04T11:13:50.4385830Z - generated xml file: /var/lib/jenkins/workspace/test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-86e4cbaa0e57366b.xml - 2025-12-04T11:13:50.4385932Z =========================== short test summary info ============================ 2025-12-04T11:13:50.4387221Z FAILED [0.4563s] inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_32_mid_dim_1_in_features_144_out_features_64_cuda_bfloat16 - RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_1_in_features_144_out_features_64_cuda_bfloat16! Caching allocator allocated memory was 18432 and is now reported as 27648 on device 0. CUDA driver allocated memory was 276824064 and is now 278921216. 2025-12-04T11:13:50.4387228Z 2025-12-04T11:13:50.4387355Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.4387871Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_1_in_features_144_out_features_64_cuda_bfloat16 2025-12-04T11:13:50.4387878Z 2025-12-04T11:13:50.4388037Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.4388142Z !!!!!!!!!!!!!!!!!!!!!!!!!! stopping after 1 failures !!!!!!!!!!!!!!!!!!!!!!!!!!! 2025-12-04T11:13:50.4388265Z ================== 1 failed, 47 deselected, 2 rerun in 2.81s =================== 2025-12-04T11:13:50.4388324Z Got exit code 1 2025-12-04T11:13:50.4388391Z Retrying single test... 2025-12-04T11:13:50.4388657Z W1204 11:05:13.237000 83248 site-packages/torch/_inductor/utils.py:1703] Not enough SMs to use max_autotune_gemm mode 2025-12-04T11:13:50.4389044Z Test results will be stored in test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-1c1918cf02456db6.xml 2025-12-04T11:13:50.4389143Z ============================= test session starts ============================== 2025-12-04T11:13:50.4389354Z platform linux -- Python 3.10.14, pytest-7.3.2, pluggy-1.6.0 -- /opt/conda/envs/py_3.10/bin/python 2025-12-04T11:13:50.4389421Z cachedir: .pytest_cache 2025-12-04T11:13:50.4389733Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow] 2025-12-04T11:13:50.4389809Z rootdir: /var/lib/jenkins/workspace 2025-12-04T11:13:50.4389917Z configfile: pytest.ini 2025-12-04T11:13:50.4390266Z plugins: hypothesis-6.56.4, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-14.0, subtests-0.13.1, xdist-3.3.1, xdoctest-1.3.0, typeguard-4.3.0 2025-12-04T11:13:50.4390396Z collecting ... collected 58 items / 57 deselected / 1 selected 2025-12-04T11:13:50.4390965Z stepcurrent: skipping 47 already run items. Running only test/inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_32_mid_dim_1_in_features_144_out_features_64_cuda_bfloat16 2025-12-04T11:13:50.4391036Z Running 1 items in this shard 2025-12-04T11:13:50.4391039Z 2025-12-04T11:13:50.4391808Z inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_32_mid_dim_1_in_features_144_out_features_64_cuda_bfloat16 [W1204 11:05:14.332372623 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.4391815Z 2025-12-04T11:13:50.4392116Z [W1204 11:05:23.385987171 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.4392119Z 2025-12-04T11:13:50.4392405Z [W1204 11:05:23.386235977 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.4392529Z 2025-12-04T11:13:50.4392819Z [W1204 11:05:23.391962693 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.4392822Z 2025-12-04T11:13:50.4393110Z [W1204 11:05:23.392519155 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.4393113Z 2025-12-04T11:13:50.4393403Z [W1204 11:05:23.392694939 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.4393406Z 2025-12-04T11:13:50.4393696Z [W1204 11:05:23.398201560 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.4393701Z 2025-12-04T11:13:50.4393989Z [W1204 11:05:23.398757902 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.4393994Z 2025-12-04T11:13:50.4394280Z [W1204 11:05:23.398923606 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.4394283Z 2025-12-04T11:13:50.4394372Z ('RERUN', {'yellow': True}) [10.9485s] [100%] 2025-12-04T11:13:50.4395100Z inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_32_mid_dim_1_in_features_144_out_features_64_cuda_bfloat16 [W1204 11:05:24.588566131 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.4395104Z 2025-12-04T11:13:50.4395394Z [W1204 11:05:24.589161524 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.4395398Z 2025-12-04T11:13:50.4395685Z [W1204 11:05:24.589299107 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.4395690Z 2025-12-04T11:13:50.4395975Z [W1204 11:05:24.592376985 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.4395978Z 2025-12-04T11:13:50.4396270Z [W1204 11:05:24.592970078 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.4396273Z 2025-12-04T11:13:50.4396561Z [W1204 11:05:24.593109142 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.4396564Z 2025-12-04T11:13:50.4396891Z [W1204 11:05:24.597850536 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.4396895Z 2025-12-04T11:13:50.4397213Z [W1204 11:05:24.598324167 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.4397218Z 2025-12-04T11:13:50.4397507Z [W1204 11:05:24.598459370 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.4397510Z 2025-12-04T11:13:50.4397590Z ('RERUN', {'yellow': True}) [0.4241s] [100%] 2025-12-04T11:13:50.4398343Z inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_32_mid_dim_1_in_features_144_out_features_64_cuda_bfloat16 [W1204 11:05:25.009258183 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.4398347Z 2025-12-04T11:13:50.4398637Z [W1204 11:05:25.009832995 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.4398641Z 2025-12-04T11:13:50.4398927Z [W1204 11:05:25.009970418 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.4398975Z 2025-12-04T11:13:50.4399268Z [W1204 11:05:25.013027306 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.4399273Z 2025-12-04T11:13:50.4399559Z [W1204 11:05:25.013610129 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.4399563Z 2025-12-04T11:13:50.4399851Z [W1204 11:05:25.013752872 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.4399854Z 2025-12-04T11:13:50.4400138Z [W1204 11:05:25.018424205 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.4400142Z 2025-12-04T11:13:50.4400433Z [W1204 11:05:25.018893176 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.4400438Z 2025-12-04T11:13:50.4400725Z [W1204 11:05:25.019027659 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.4400728Z 2025-12-04T11:13:50.4400792Z FAILED [0.4195s] [100%] 2025-12-04T11:13:50.4400796Z 2025-12-04T11:13:50.4400881Z ==================================== RERUNS ==================================== 2025-12-04T11:13:50.4401173Z _ TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_1_in_features_144_out_features_64_cuda_bfloat16 _ 2025-12-04T11:13:50.4401252Z Traceback (most recent call last): 2025-12-04T11:13:50.4401560Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.4401631Z method(*args, **kwargs) 2025-12-04T11:13:50.4401927Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.4401992Z method(*args, **kwargs) 2025-12-04T11:13:50.4402293Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T11:13:50.4402356Z with policy(): 2025-12-04T11:13:50.4402653Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T11:13:50.4402722Z raise RuntimeError(msg) 2025-12-04T11:13:50.4403518Z RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_1_in_features_144_out_features_64_cuda_bfloat16! Caching allocator allocated memory was 0 and is now reported as 9216 on device 0. CUDA driver allocated memory was 230686720 and is now 274726912. 2025-12-04T11:13:50.4403562Z 2025-12-04T11:13:50.4403727Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.4404249Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_1_in_features_144_out_features_64_cuda_bfloat16 2025-12-04T11:13:50.4404253Z 2025-12-04T11:13:50.4404417Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.4404543Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.4404636Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.4405020Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.4405151Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.4405215Z graph_break [] 2025-12-04T11:13:50.4405341Z ----------------------------- Captured stderr call ----------------------------- 2025-12-04T11:13:50.4406031Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/constant_folding.py:256: UserWarning: Unsupported unwinding pattern: Address not in range (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/profiler/unwind/unwind.cpp:219.) 2025-12-04T11:13:50.4406161Z if out == self.unknown_value: 2025-12-04T11:13:50.4406452Z _ TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_1_in_features_144_out_features_64_cuda_bfloat16 _ 2025-12-04T11:13:50.4406528Z Traceback (most recent call last): 2025-12-04T11:13:50.4406831Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.4406896Z method(*args, **kwargs) 2025-12-04T11:13:50.4407195Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.4407260Z method(*args, **kwargs) 2025-12-04T11:13:50.4407550Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T11:13:50.4407615Z with policy(): 2025-12-04T11:13:50.4407911Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T11:13:50.4407977Z raise RuntimeError(msg) 2025-12-04T11:13:50.4408785Z RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_1_in_features_144_out_features_64_cuda_bfloat16! Caching allocator allocated memory was 9216 and is now reported as 18432 on device 0. CUDA driver allocated memory was 274726912 and is now 276824064. 2025-12-04T11:13:50.4408790Z 2025-12-04T11:13:50.4408917Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.4409441Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_1_in_features_144_out_features_64_cuda_bfloat16 2025-12-04T11:13:50.4409447Z 2025-12-04T11:13:50.4409605Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.4409734Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.4409837Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.4410185Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.4410319Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.4410378Z graph_break [] 2025-12-04T11:13:50.4410506Z ----------------------------- Captured stderr call ----------------------------- 2025-12-04T11:13:50.4411270Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/constant_folding.py:256: UserWarning: Unsupported unwinding pattern: Address not in range (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/profiler/unwind/unwind.cpp:219.) 2025-12-04T11:13:50.4411342Z if out == self.unknown_value: 2025-12-04T11:13:50.4411468Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.4411559Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.4411681Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.4412057Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.4412118Z graph_break [] 2025-12-04T11:13:50.4412204Z =================================== FAILURES =================================== 2025-12-04T11:13:50.4412495Z _ TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_1_in_features_144_out_features_64_cuda_bfloat16 _ 2025-12-04T11:13:50.4412571Z Traceback (most recent call last): 2025-12-04T11:13:50.4412869Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.4412978Z method(*args, **kwargs) 2025-12-04T11:13:50.4413274Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.4413336Z method(*args, **kwargs) 2025-12-04T11:13:50.4413626Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T11:13:50.4413691Z with policy(): 2025-12-04T11:13:50.4413988Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T11:13:50.4414057Z raise RuntimeError(msg) 2025-12-04T11:13:50.4414869Z RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_1_in_features_144_out_features_64_cuda_bfloat16! Caching allocator allocated memory was 18432 and is now reported as 27648 on device 0. CUDA driver allocated memory was 276824064 and is now 278921216. 2025-12-04T11:13:50.4414874Z 2025-12-04T11:13:50.4414998Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.4415520Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_1_in_features_144_out_features_64_cuda_bfloat16 2025-12-04T11:13:50.4415523Z 2025-12-04T11:13:50.4415679Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.4415807Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.4415900Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.4416245Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.4416374Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.4416434Z graph_break [] 2025-12-04T11:13:50.4416559Z ----------------------------- Captured stderr call ----------------------------- 2025-12-04T11:13:50.4417246Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/constant_folding.py:256: UserWarning: Unsupported unwinding pattern: Address not in range (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/profiler/unwind/unwind.cpp:219.) 2025-12-04T11:13:50.4417314Z if out == self.unknown_value: 2025-12-04T11:13:50.4417446Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.4417536Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.4417710Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.4418086Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.4418147Z graph_break [] 2025-12-04T11:13:50.4418273Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.4418363Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.4418485Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.4418826Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.4418917Z graph_break [] 2025-12-04T11:13:50.4419408Z - generated xml file: /var/lib/jenkins/workspace/test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-1c1918cf02456db6.xml - 2025-12-04T11:13:50.4419510Z =========================== short test summary info ============================ 2025-12-04T11:13:50.4420793Z FAILED [0.4195s] inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_32_mid_dim_1_in_features_144_out_features_64_cuda_bfloat16 - RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_1_in_features_144_out_features_64_cuda_bfloat16! Caching allocator allocated memory was 18432 and is now reported as 27648 on device 0. CUDA driver allocated memory was 276824064 and is now 278921216. 2025-12-04T11:13:50.4420838Z 2025-12-04T11:13:50.4420964Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.4421482Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_1_in_features_144_out_features_64_cuda_bfloat16 2025-12-04T11:13:50.4421487Z 2025-12-04T11:13:50.4421652Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.4421756Z !!!!!!!!!!!!!!!!!!!!!!!!!! stopping after 1 failures !!!!!!!!!!!!!!!!!!!!!!!!!!! 2025-12-04T11:13:50.4421876Z ================== 1 failed, 57 deselected, 2 rerun in 11.82s ================== 2025-12-04T11:13:50.4421936Z Got exit code 1 2025-12-04T11:13:50.4422000Z Retrying single test... 2025-12-04T11:13:50.4422266Z W1204 11:05:31.821000 83434 site-packages/torch/_inductor/utils.py:1703] Not enough SMs to use max_autotune_gemm mode 2025-12-04T11:13:50.4422652Z Test results will be stored in test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-4d7a6e50e4cb57cd.xml 2025-12-04T11:13:50.4422748Z ============================= test session starts ============================== 2025-12-04T11:13:50.4422958Z platform linux -- Python 3.10.14, pytest-7.3.2, pluggy-1.6.0 -- /opt/conda/envs/py_3.10/bin/python 2025-12-04T11:13:50.4423026Z cachedir: .pytest_cache 2025-12-04T11:13:50.4423339Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow] 2025-12-04T11:13:50.4423423Z rootdir: /var/lib/jenkins/workspace 2025-12-04T11:13:50.4423489Z configfile: pytest.ini 2025-12-04T11:13:50.4423845Z plugins: hypothesis-6.56.4, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-14.0, subtests-0.13.1, xdist-3.3.1, xdoctest-1.3.0, typeguard-4.3.0 2025-12-04T11:13:50.4423971Z collecting ... collected 58 items / 57 deselected / 1 selected 2025-12-04T11:13:50.4424549Z stepcurrent: skipping 47 already run items. Running only test/inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_32_mid_dim_1_in_features_144_out_features_64_cuda_bfloat16 2025-12-04T11:13:50.4424620Z Running 1 items in this shard 2025-12-04T11:13:50.4424624Z 2025-12-04T11:13:50.4425390Z inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_32_mid_dim_1_in_features_144_out_features_64_cuda_bfloat16 [W1204 11:05:32.915949488 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.4425432Z 2025-12-04T11:13:50.4425733Z [W1204 11:05:42.109689342 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.4425736Z 2025-12-04T11:13:50.4426028Z [W1204 11:05:42.109924877 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.4426031Z 2025-12-04T11:13:50.4426352Z [W1204 11:05:42.115612993 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.4426356Z 2025-12-04T11:13:50.4426641Z [W1204 11:05:42.116168445 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.4426646Z 2025-12-04T11:13:50.4426938Z [W1204 11:05:42.116338169 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.4426978Z 2025-12-04T11:13:50.4427267Z [W1204 11:05:42.121924792 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.4427271Z 2025-12-04T11:13:50.4427560Z [W1204 11:05:42.122489775 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.4427563Z 2025-12-04T11:13:50.4427850Z [W1204 11:05:42.122658419 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.4427854Z 2025-12-04T11:13:50.4427934Z ('RERUN', {'yellow': True}) [11.0776s] [100%] 2025-12-04T11:13:50.4428660Z inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_32_mid_dim_1_in_features_144_out_features_64_cuda_bfloat16 [W1204 11:05:43.299548395 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.4428666Z 2025-12-04T11:13:50.4428956Z [W1204 11:05:43.300149408 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.4428959Z 2025-12-04T11:13:50.4429247Z [W1204 11:05:43.300292052 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.4429250Z 2025-12-04T11:13:50.4429544Z [W1204 11:05:43.303285348 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.4429547Z 2025-12-04T11:13:50.4429837Z [W1204 11:05:43.303860900 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.4429842Z 2025-12-04T11:13:50.4430130Z [W1204 11:05:43.303997834 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.4430135Z 2025-12-04T11:13:50.4430425Z [W1204 11:05:43.308551314 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.4430428Z 2025-12-04T11:13:50.4430710Z [W1204 11:05:43.309033625 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.4430713Z 2025-12-04T11:13:50.4431003Z [W1204 11:05:43.309171718 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.4431008Z 2025-12-04T11:13:50.4431099Z ('RERUN', {'yellow': True}) [0.4369s] [100%] 2025-12-04T11:13:50.4431853Z inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_32_mid_dim_1_in_features_144_out_features_64_cuda_bfloat16 [W1204 11:05:43.746743773 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.4431895Z 2025-12-04T11:13:50.4432186Z [W1204 11:05:43.747309846 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.4432190Z 2025-12-04T11:13:50.4432474Z [W1204 11:05:43.747444618 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.4432477Z 2025-12-04T11:13:50.4432767Z [W1204 11:05:43.750453855 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.4432822Z 2025-12-04T11:13:50.4433109Z [W1204 11:05:43.751024968 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.4433114Z 2025-12-04T11:13:50.4433404Z [W1204 11:05:43.751161041 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.4433407Z 2025-12-04T11:13:50.4433726Z [W1204 11:05:43.755743052 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.4433729Z 2025-12-04T11:13:50.4434016Z [W1204 11:05:43.756208172 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.4434019Z 2025-12-04T11:13:50.4434306Z [W1204 11:05:43.756344015 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.4434310Z 2025-12-04T11:13:50.4434373Z FAILED [0.4232s] [100%] 2025-12-04T11:13:50.4434380Z 2025-12-04T11:13:50.4434463Z ==================================== RERUNS ==================================== 2025-12-04T11:13:50.4434751Z _ TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_1_in_features_144_out_features_64_cuda_bfloat16 _ 2025-12-04T11:13:50.4434831Z Traceback (most recent call last): 2025-12-04T11:13:50.4435135Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.4435200Z method(*args, **kwargs) 2025-12-04T11:13:50.4435497Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.4435560Z method(*args, **kwargs) 2025-12-04T11:13:50.4435856Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T11:13:50.4435924Z with policy(): 2025-12-04T11:13:50.4436221Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T11:13:50.4436291Z raise RuntimeError(msg) 2025-12-04T11:13:50.4437086Z RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_1_in_features_144_out_features_64_cuda_bfloat16! Caching allocator allocated memory was 0 and is now reported as 9216 on device 0. CUDA driver allocated memory was 230686720 and is now 274726912. 2025-12-04T11:13:50.4437091Z 2025-12-04T11:13:50.4437222Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.4437740Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_1_in_features_144_out_features_64_cuda_bfloat16 2025-12-04T11:13:50.4437744Z 2025-12-04T11:13:50.4437903Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.4438033Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.4438173Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.4438556Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.4438695Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.4438754Z graph_break [] 2025-12-04T11:13:50.4438883Z ----------------------------- Captured stderr call ----------------------------- 2025-12-04T11:13:50.4439575Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/constant_folding.py:256: UserWarning: Unsupported unwinding pattern: Address not in range (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/profiler/unwind/unwind.cpp:219.) 2025-12-04T11:13:50.4439682Z if out == self.unknown_value: 2025-12-04T11:13:50.4439993Z _ TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_1_in_features_144_out_features_64_cuda_bfloat16 _ 2025-12-04T11:13:50.4440070Z Traceback (most recent call last): 2025-12-04T11:13:50.4440372Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.4440435Z method(*args, **kwargs) 2025-12-04T11:13:50.4440761Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.4440830Z method(*args, **kwargs) 2025-12-04T11:13:50.4441117Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T11:13:50.4441182Z with policy(): 2025-12-04T11:13:50.4441486Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T11:13:50.4441555Z raise RuntimeError(msg) 2025-12-04T11:13:50.4442366Z RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_1_in_features_144_out_features_64_cuda_bfloat16! Caching allocator allocated memory was 9216 and is now reported as 18432 on device 0. CUDA driver allocated memory was 274726912 and is now 276824064. 2025-12-04T11:13:50.4442373Z 2025-12-04T11:13:50.4442500Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.4443024Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_1_in_features_144_out_features_64_cuda_bfloat16 2025-12-04T11:13:50.4443027Z 2025-12-04T11:13:50.4443181Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.4443306Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.4443404Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.4443750Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.4443885Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.4443944Z graph_break [] 2025-12-04T11:13:50.4444069Z ----------------------------- Captured stderr call ----------------------------- 2025-12-04T11:13:50.4444758Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/constant_folding.py:256: UserWarning: Unsupported unwinding pattern: Address not in range (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/profiler/unwind/unwind.cpp:219.) 2025-12-04T11:13:50.4444828Z if out == self.unknown_value: 2025-12-04T11:13:50.4444951Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.4445046Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.4445168Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.4445517Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.4445655Z graph_break [] 2025-12-04T11:13:50.4445739Z =================================== FAILURES =================================== 2025-12-04T11:13:50.4446034Z _ TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_1_in_features_144_out_features_64_cuda_bfloat16 _ 2025-12-04T11:13:50.4446107Z Traceback (most recent call last): 2025-12-04T11:13:50.4446408Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.4446472Z method(*args, **kwargs) 2025-12-04T11:13:50.4446801Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.4446869Z method(*args, **kwargs) 2025-12-04T11:13:50.4447159Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T11:13:50.4447224Z with policy(): 2025-12-04T11:13:50.4447515Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T11:13:50.4447615Z raise RuntimeError(msg) 2025-12-04T11:13:50.4448424Z RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_1_in_features_144_out_features_64_cuda_bfloat16! Caching allocator allocated memory was 18432 and is now reported as 27648 on device 0. CUDA driver allocated memory was 276824064 and is now 278921216. 2025-12-04T11:13:50.4448428Z 2025-12-04T11:13:50.4448552Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.4449073Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_1_in_features_144_out_features_64_cuda_bfloat16 2025-12-04T11:13:50.4449078Z 2025-12-04T11:13:50.4449240Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.4449374Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.4449470Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.4449809Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.4449943Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.4450004Z graph_break [] 2025-12-04T11:13:50.4450130Z ----------------------------- Captured stderr call ----------------------------- 2025-12-04T11:13:50.4450822Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/constant_folding.py:256: UserWarning: Unsupported unwinding pattern: Address not in range (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/profiler/unwind/unwind.cpp:219.) 2025-12-04T11:13:50.4450892Z if out == self.unknown_value: 2025-12-04T11:13:50.4451019Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.4451111Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.4451233Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.4451578Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.4451639Z graph_break [] 2025-12-04T11:13:50.4451760Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.4451854Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.4451978Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.4452321Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.4452436Z graph_break [] 2025-12-04T11:13:50.4452957Z - generated xml file: /var/lib/jenkins/workspace/test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-4d7a6e50e4cb57cd.xml - 2025-12-04T11:13:50.4453065Z =========================== short test summary info ============================ 2025-12-04T11:13:50.4454376Z FAILED [0.4232s] inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_32_mid_dim_1_in_features_144_out_features_64_cuda_bfloat16 - RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_1_in_features_144_out_features_64_cuda_bfloat16! Caching allocator allocated memory was 18432 and is now reported as 27648 on device 0. CUDA driver allocated memory was 276824064 and is now 278921216. 2025-12-04T11:13:50.4454382Z 2025-12-04T11:13:50.4454511Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.4455025Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_1_in_features_144_out_features_64_cuda_bfloat16 2025-12-04T11:13:50.4455063Z 2025-12-04T11:13:50.4455223Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.4455327Z !!!!!!!!!!!!!!!!!!!!!!!!!! stopping after 1 failures !!!!!!!!!!!!!!!!!!!!!!!!!!! 2025-12-04T11:13:50.4455443Z ================== 1 failed, 57 deselected, 2 rerun in 11.96s ================== 2025-12-04T11:13:50.4455506Z Got exit code 1 2025-12-04T11:13:50.4455978Z FAILED CONSISTENTLY: test/inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_32_mid_dim_1_in_features_144_out_features_64_cuda_bfloat16 2025-12-04T11:13:50.4456224Z Test failed consistently, continuing with the rest of the tests due to continue-through-error being set 2025-12-04T11:13:50.4456486Z W1204 11:05:50.816000 83620 site-packages/torch/_inductor/utils.py:1703] Not enough SMs to use max_autotune_gemm mode 2025-12-04T11:13:50.4456870Z Test results will be stored in test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-cd0db30d38ee28ec.xml 2025-12-04T11:13:50.4456967Z ============================= test session starts ============================== 2025-12-04T11:13:50.4457172Z platform linux -- Python 3.10.14, pytest-7.3.2, pluggy-1.6.0 -- /opt/conda/envs/py_3.10/bin/python 2025-12-04T11:13:50.4457240Z cachedir: .pytest_cache 2025-12-04T11:13:50.4457547Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow] 2025-12-04T11:13:50.4457624Z rootdir: /var/lib/jenkins/workspace 2025-12-04T11:13:50.4457693Z configfile: pytest.ini 2025-12-04T11:13:50.4458005Z plugins: hypothesis-6.56.4, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-14.0, subtests-0.13.1, xdist-3.3.1, xdoctest-1.3.0, typeguard-4.3.0 2025-12-04T11:13:50.4458135Z collecting ... collected 58 items / 48 deselected / 10 selected 2025-12-04T11:13:50.4458224Z stepcurrent: skipping 48 already run items. 2025-12-04T11:13:50.4458297Z Running 10 items in this shard 2025-12-04T11:13:50.4458301Z 2025-12-04T11:13:50.4458797Z inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_32_mid_dim_1_in_features_144_out_features_65_cuda_bfloat16 ('RERUN', {'yellow': True}) [1.9716s] [ 10%] 2025-12-04T11:13:50.4459283Z inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_32_mid_dim_1_in_features_144_out_features_65_cuda_bfloat16 ('RERUN', {'yellow': True}) [0.5422s] [ 10%] 2025-12-04T11:13:50.4459734Z inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_32_mid_dim_1_in_features_144_out_features_65_cuda_bfloat16 FAILED [0.5330s] [ 10%] 2025-12-04T11:13:50.4459782Z 2025-12-04T11:13:50.4459937Z ==================================== RERUNS ==================================== 2025-12-04T11:13:50.4460229Z _ TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_1_in_features_144_out_features_65_cuda_bfloat16 _ 2025-12-04T11:13:50.4460314Z Traceback (most recent call last): 2025-12-04T11:13:50.4460617Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.4460682Z method(*args, **kwargs) 2025-12-04T11:13:50.4460983Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.4461046Z method(*args, **kwargs) 2025-12-04T11:13:50.4461368Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T11:13:50.4461430Z with policy(): 2025-12-04T11:13:50.4461727Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T11:13:50.4461798Z raise RuntimeError(msg) 2025-12-04T11:13:50.4462592Z RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_1_in_features_144_out_features_65_cuda_bfloat16! Caching allocator allocated memory was 0 and is now reported as 9728 on device 0. CUDA driver allocated memory was 230686720 and is now 274726912. 2025-12-04T11:13:50.4462631Z 2025-12-04T11:13:50.4462756Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.4463271Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_1_in_features_144_out_features_65_cuda_bfloat16 2025-12-04T11:13:50.4463275Z 2025-12-04T11:13:50.4463432Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.4463612Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.4463710Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.4464263Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('benchmarking.InductorBenchmarker.benchmark_gpu', 2), ('pad_mm_bench', 1), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.4464391Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.4464450Z graph_break [] 2025-12-04T11:13:50.4464742Z _ TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_1_in_features_144_out_features_65_cuda_bfloat16 _ 2025-12-04T11:13:50.4464817Z Traceback (most recent call last): 2025-12-04T11:13:50.4465118Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.4465184Z method(*args, **kwargs) 2025-12-04T11:13:50.4465476Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.4465544Z method(*args, **kwargs) 2025-12-04T11:13:50.4465833Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T11:13:50.4465892Z with policy(): 2025-12-04T11:13:50.4466188Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T11:13:50.4466254Z raise RuntimeError(msg) 2025-12-04T11:13:50.4467058Z RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_1_in_features_144_out_features_65_cuda_bfloat16! Caching allocator allocated memory was 9728 and is now reported as 19456 on device 0. CUDA driver allocated memory was 274726912 and is now 276824064. 2025-12-04T11:13:50.4467104Z 2025-12-04T11:13:50.4467229Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.4467795Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_1_in_features_144_out_features_65_cuda_bfloat16 2025-12-04T11:13:50.4467801Z 2025-12-04T11:13:50.4467958Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.4468083Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.4468178Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.4468754Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('benchmarking.InductorBenchmarker.benchmark_gpu', 2), ('pad_mm_bench', 1), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.4468884Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.4468946Z graph_break [] 2025-12-04T11:13:50.4469070Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.4469164Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.4469320Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.4469865Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('benchmarking.InductorBenchmarker.benchmark_gpu', 2), ('pad_mm_bench', 1), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.4469927Z graph_break [] 2025-12-04T11:13:50.4470009Z =================================== FAILURES =================================== 2025-12-04T11:13:50.4470300Z _ TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_1_in_features_144_out_features_65_cuda_bfloat16 _ 2025-12-04T11:13:50.4470374Z Traceback (most recent call last): 2025-12-04T11:13:50.4470673Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.4470742Z method(*args, **kwargs) 2025-12-04T11:13:50.4471040Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.4471110Z method(*args, **kwargs) 2025-12-04T11:13:50.4471400Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T11:13:50.4471462Z with policy(): 2025-12-04T11:13:50.4471756Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T11:13:50.4471822Z raise RuntimeError(msg) 2025-12-04T11:13:50.4472626Z RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_1_in_features_144_out_features_65_cuda_bfloat16! Caching allocator allocated memory was 19456 and is now reported as 29184 on device 0. CUDA driver allocated memory was 276824064 and is now 278921216. 2025-12-04T11:13:50.4472638Z 2025-12-04T11:13:50.4472764Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.4473279Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_1_in_features_144_out_features_65_cuda_bfloat16 2025-12-04T11:13:50.4473283Z 2025-12-04T11:13:50.4473440Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.4473564Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.4473657Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.4474195Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('benchmarking.InductorBenchmarker.benchmark_gpu', 2), ('pad_mm_bench', 1), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.4474400Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.4474463Z graph_break [] 2025-12-04T11:13:50.4474589Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.4474678Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.4474802Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.4475337Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('benchmarking.InductorBenchmarker.benchmark_gpu', 2), ('pad_mm_bench', 1), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.4475434Z graph_break [] 2025-12-04T11:13:50.4475558Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.4475646Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.4475787Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.4476321Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('benchmarking.InductorBenchmarker.benchmark_gpu', 2), ('pad_mm_bench', 1), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.4476420Z graph_break [] 2025-12-04T11:13:50.4476908Z - generated xml file: /var/lib/jenkins/workspace/test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-cd0db30d38ee28ec.xml - 2025-12-04T11:13:50.4477008Z =========================== short test summary info ============================ 2025-12-04T11:13:50.4478426Z FAILED [0.5330s] inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_32_mid_dim_1_in_features_144_out_features_65_cuda_bfloat16 - RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_1_in_features_144_out_features_65_cuda_bfloat16! Caching allocator allocated memory was 19456 and is now reported as 29184 on device 0. CUDA driver allocated memory was 276824064 and is now 278921216. 2025-12-04T11:13:50.4478434Z 2025-12-04T11:13:50.4478562Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.4479101Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_1_in_features_144_out_features_65_cuda_bfloat16 2025-12-04T11:13:50.4479105Z 2025-12-04T11:13:50.4479264Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.4479373Z !!!!!!!!!!!!!!!!!!!!!!!!!! stopping after 1 failures !!!!!!!!!!!!!!!!!!!!!!!!!!! 2025-12-04T11:13:50.4479495Z ================== 1 failed, 48 deselected, 2 rerun in 3.07s =================== 2025-12-04T11:13:50.4479556Z Got exit code 1 2025-12-04T11:13:50.4479626Z Retrying single test... 2025-12-04T11:13:50.4479893Z W1204 11:06:00.888000 83802 site-packages/torch/_inductor/utils.py:1703] Not enough SMs to use max_autotune_gemm mode 2025-12-04T11:13:50.4480281Z Test results will be stored in test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-97331d68f5e395ac.xml 2025-12-04T11:13:50.4480378Z ============================= test session starts ============================== 2025-12-04T11:13:50.4480584Z platform linux -- Python 3.10.14, pytest-7.3.2, pluggy-1.6.0 -- /opt/conda/envs/py_3.10/bin/python 2025-12-04T11:13:50.4480657Z cachedir: .pytest_cache 2025-12-04T11:13:50.4480962Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow] 2025-12-04T11:13:50.4481041Z rootdir: /var/lib/jenkins/workspace 2025-12-04T11:13:50.4481111Z configfile: pytest.ini 2025-12-04T11:13:50.4481423Z plugins: hypothesis-6.56.4, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-14.0, subtests-0.13.1, xdist-3.3.1, xdoctest-1.3.0, typeguard-4.3.0 2025-12-04T11:13:50.4481673Z collecting ... collected 58 items / 57 deselected / 1 selected 2025-12-04T11:13:50.4482240Z stepcurrent: skipping 48 already run items. Running only test/inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_32_mid_dim_1_in_features_144_out_features_65_cuda_bfloat16 2025-12-04T11:13:50.4482314Z Running 1 items in this shard 2025-12-04T11:13:50.4482318Z 2025-12-04T11:13:50.4483090Z inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_32_mid_dim_1_in_features_144_out_features_65_cuda_bfloat16 [W1204 11:06:02.523289702 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.4483094Z 2025-12-04T11:13:50.4483397Z [W1204 11:06:11.495311369 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.4483402Z 2025-12-04T11:13:50.4483697Z [W1204 11:06:11.495555995 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.4483746Z 2025-12-04T11:13:50.4484045Z [W1204 11:06:11.501567197 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.4484049Z 2025-12-04T11:13:50.4484344Z [W1204 11:06:11.502145360 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.4484347Z 2025-12-04T11:13:50.4484632Z [W1204 11:06:11.502319654 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.4484637Z 2025-12-04T11:13:50.4484926Z [W1204 11:06:11.507809126 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.4484931Z 2025-12-04T11:13:50.4485218Z [W1204 11:06:11.508342358 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.4485222Z 2025-12-04T11:13:50.4485514Z [W1204 11:06:11.508500971 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.4485517Z 2025-12-04T11:13:50.4485598Z ('RERUN', {'yellow': True}) [10.9490s] [100%] 2025-12-04T11:13:50.4486322Z inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_32_mid_dim_1_in_features_144_out_features_65_cuda_bfloat16 [W1204 11:06:12.316476250 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.4486328Z 2025-12-04T11:13:50.4486621Z [W1204 11:06:12.317059172 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.4486626Z 2025-12-04T11:13:50.4486911Z [W1204 11:06:12.317207326 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.4486914Z 2025-12-04T11:13:50.4487203Z [W1204 11:06:12.320172311 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.4487206Z 2025-12-04T11:13:50.4487492Z [W1204 11:06:12.320649692 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.4487496Z 2025-12-04T11:13:50.4487787Z [W1204 11:06:12.320787785 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.4487790Z 2025-12-04T11:13:50.4488075Z [W1204 11:06:12.325393056 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.4488134Z 2025-12-04T11:13:50.4488458Z [W1204 11:06:12.325859476 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.4488462Z 2025-12-04T11:13:50.4488747Z [W1204 11:06:12.325995789 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.4488752Z 2025-12-04T11:13:50.4488830Z ('RERUN', {'yellow': True}) [0.4998s] [100%] 2025-12-04T11:13:50.4489552Z inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_32_mid_dim_1_in_features_144_out_features_65_cuda_bfloat16 [W1204 11:06:12.812832584 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.4489597Z 2025-12-04T11:13:50.4489895Z [W1204 11:06:12.813387626 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.4489901Z 2025-12-04T11:13:50.4490189Z [W1204 11:06:12.813529729 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.4490192Z 2025-12-04T11:13:50.4490513Z [W1204 11:06:12.816500435 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.4490516Z 2025-12-04T11:13:50.4490805Z [W1204 11:06:12.816978746 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.4490809Z 2025-12-04T11:13:50.4491095Z [W1204 11:06:12.817114119 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.4491098Z 2025-12-04T11:13:50.4491391Z [W1204 11:06:12.821778182 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.4491397Z 2025-12-04T11:13:50.4491687Z [W1204 11:06:12.822252952 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.4491692Z 2025-12-04T11:13:50.4491983Z [W1204 11:06:12.822388775 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.4491988Z 2025-12-04T11:13:50.4492051Z FAILED [0.4953s] [100%] 2025-12-04T11:13:50.4492054Z 2025-12-04T11:13:50.4492142Z ==================================== RERUNS ==================================== 2025-12-04T11:13:50.4492440Z _ TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_1_in_features_144_out_features_65_cuda_bfloat16 _ 2025-12-04T11:13:50.4492520Z Traceback (most recent call last): 2025-12-04T11:13:50.4492840Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.4492909Z method(*args, **kwargs) 2025-12-04T11:13:50.4493239Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.4493364Z method(*args, **kwargs) 2025-12-04T11:13:50.4493723Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T11:13:50.4493812Z with policy(): 2025-12-04T11:13:50.4494205Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T11:13:50.4494278Z raise RuntimeError(msg) 2025-12-04T11:13:50.4495192Z RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_1_in_features_144_out_features_65_cuda_bfloat16! Caching allocator allocated memory was 0 and is now reported as 9728 on device 0. CUDA driver allocated memory was 230686720 and is now 274726912. 2025-12-04T11:13:50.4495199Z 2025-12-04T11:13:50.4498646Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.4499293Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_1_in_features_144_out_features_65_cuda_bfloat16 2025-12-04T11:13:50.4499305Z 2025-12-04T11:13:50.4499492Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.4499635Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.4499742Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.4500343Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('benchmarking.InductorBenchmarker.benchmark_gpu', 2), ('pad_mm_bench', 1), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.4500485Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.4500552Z graph_break [] 2025-12-04T11:13:50.4500688Z ----------------------------- Captured stderr call ----------------------------- 2025-12-04T11:13:50.4501413Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/constant_folding.py:256: UserWarning: Unsupported unwinding pattern: Address not in range (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/profiler/unwind/unwind.cpp:219.) 2025-12-04T11:13:50.4501526Z if out == self.unknown_value: 2025-12-04T11:13:50.4501827Z _ TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_1_in_features_144_out_features_65_cuda_bfloat16 _ 2025-12-04T11:13:50.4501908Z Traceback (most recent call last): 2025-12-04T11:13:50.4502222Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.4502293Z method(*args, **kwargs) 2025-12-04T11:13:50.4502586Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.4502650Z method(*args, **kwargs) 2025-12-04T11:13:50.4502944Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T11:13:50.4503009Z with policy(): 2025-12-04T11:13:50.4503315Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T11:13:50.4503384Z raise RuntimeError(msg) 2025-12-04T11:13:50.4504293Z RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_1_in_features_144_out_features_65_cuda_bfloat16! Caching allocator allocated memory was 9728 and is now reported as 19456 on device 0. CUDA driver allocated memory was 274726912 and is now 276824064. 2025-12-04T11:13:50.4504298Z 2025-12-04T11:13:50.4504438Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.4504967Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_1_in_features_144_out_features_65_cuda_bfloat16 2025-12-04T11:13:50.4504971Z 2025-12-04T11:13:50.4505138Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.4505273Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.4505372Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.4505923Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('benchmarking.InductorBenchmarker.benchmark_gpu', 2), ('pad_mm_bench', 1), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.4506056Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.4506120Z graph_break [] 2025-12-04T11:13:50.4506247Z ----------------------------- Captured stderr call ----------------------------- 2025-12-04T11:13:50.4507051Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/constant_folding.py:256: UserWarning: Unsupported unwinding pattern: Address not in range (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/profiler/unwind/unwind.cpp:219.) 2025-12-04T11:13:50.4507131Z if out == self.unknown_value: 2025-12-04T11:13:50.4507255Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.4507347Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.4507475Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.4508146Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('benchmarking.InductorBenchmarker.benchmark_gpu', 2), ('pad_mm_bench', 1), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.4508213Z graph_break [] 2025-12-04T11:13:50.4508300Z =================================== FAILURES =================================== 2025-12-04T11:13:50.4508605Z _ TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_1_in_features_144_out_features_65_cuda_bfloat16 _ 2025-12-04T11:13:50.4508720Z Traceback (most recent call last): 2025-12-04T11:13:50.4509035Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.4509104Z method(*args, **kwargs) 2025-12-04T11:13:50.4509407Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.4509472Z method(*args, **kwargs) 2025-12-04T11:13:50.4509765Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T11:13:50.4509827Z with policy(): 2025-12-04T11:13:50.4510122Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T11:13:50.4510194Z raise RuntimeError(msg) 2025-12-04T11:13:50.4511014Z RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_1_in_features_144_out_features_65_cuda_bfloat16! Caching allocator allocated memory was 19456 and is now reported as 29184 on device 0. CUDA driver allocated memory was 276824064 and is now 278921216. 2025-12-04T11:13:50.4511020Z 2025-12-04T11:13:50.4511155Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.4511682Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_1_in_features_144_out_features_65_cuda_bfloat16 2025-12-04T11:13:50.4511688Z 2025-12-04T11:13:50.4511853Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.4511981Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.4512078Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.4512629Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('benchmarking.InductorBenchmarker.benchmark_gpu', 2), ('pad_mm_bench', 1), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.4512758Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.4512821Z graph_break [] 2025-12-04T11:13:50.4512948Z ----------------------------- Captured stderr call ----------------------------- 2025-12-04T11:13:50.4513645Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/constant_folding.py:256: UserWarning: Unsupported unwinding pattern: Address not in range (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/profiler/unwind/unwind.cpp:219.) 2025-12-04T11:13:50.4513721Z if out == self.unknown_value: 2025-12-04T11:13:50.4513893Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.4513989Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.4514147Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.4514692Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('benchmarking.InductorBenchmarker.benchmark_gpu', 2), ('pad_mm_bench', 1), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.4514756Z graph_break [] 2025-12-04T11:13:50.4514882Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.4514974Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.4515151Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.4515693Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('benchmarking.InductorBenchmarker.benchmark_gpu', 2), ('pad_mm_bench', 1), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.4515756Z graph_break [] 2025-12-04T11:13:50.4516252Z - generated xml file: /var/lib/jenkins/workspace/test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-97331d68f5e395ac.xml - 2025-12-04T11:13:50.4516412Z =========================== short test summary info ============================ 2025-12-04T11:13:50.4517712Z FAILED [0.4953s] inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_32_mid_dim_1_in_features_144_out_features_65_cuda_bfloat16 - RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_1_in_features_144_out_features_65_cuda_bfloat16! Caching allocator allocated memory was 19456 and is now reported as 29184 on device 0. CUDA driver allocated memory was 276824064 and is now 278921216. 2025-12-04T11:13:50.4517718Z 2025-12-04T11:13:50.4517848Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.4518370Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_1_in_features_144_out_features_65_cuda_bfloat16 2025-12-04T11:13:50.4518375Z 2025-12-04T11:13:50.4518536Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.4518646Z !!!!!!!!!!!!!!!!!!!!!!!!!! stopping after 1 failures !!!!!!!!!!!!!!!!!!!!!!!!!!! 2025-12-04T11:13:50.4518762Z ================== 1 failed, 57 deselected, 2 rerun in 11.97s ================== 2025-12-04T11:13:50.4518822Z Got exit code 1 2025-12-04T11:13:50.4518896Z Retrying single test... 2025-12-04T11:13:50.4519162Z W1204 11:06:19.655000 83989 site-packages/torch/_inductor/utils.py:1703] Not enough SMs to use max_autotune_gemm mode 2025-12-04T11:13:50.4519558Z Test results will be stored in test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-266b0df1c83533e9.xml 2025-12-04T11:13:50.4519659Z ============================= test session starts ============================== 2025-12-04T11:13:50.4519872Z platform linux -- Python 3.10.14, pytest-7.3.2, pluggy-1.6.0 -- /opt/conda/envs/py_3.10/bin/python 2025-12-04T11:13:50.4519946Z cachedir: .pytest_cache 2025-12-04T11:13:50.4520252Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow] 2025-12-04T11:13:50.4520331Z rootdir: /var/lib/jenkins/workspace 2025-12-04T11:13:50.4520399Z configfile: pytest.ini 2025-12-04T11:13:50.4520715Z plugins: hypothesis-6.56.4, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-14.0, subtests-0.13.1, xdist-3.3.1, xdoctest-1.3.0, typeguard-4.3.0 2025-12-04T11:13:50.4520852Z collecting ... collected 58 items / 57 deselected / 1 selected 2025-12-04T11:13:50.4521462Z stepcurrent: skipping 48 already run items. Running only test/inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_32_mid_dim_1_in_features_144_out_features_65_cuda_bfloat16 2025-12-04T11:13:50.4521571Z Running 1 items in this shard 2025-12-04T11:13:50.4521576Z 2025-12-04T11:13:50.4522304Z inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_32_mid_dim_1_in_features_144_out_features_65_cuda_bfloat16 [W1204 11:06:21.290506700 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.4522308Z 2025-12-04T11:13:50.4522610Z [W1204 11:06:30.487735184 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.4522614Z 2025-12-04T11:13:50.4522939Z [W1204 11:06:30.487981250 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.4522944Z 2025-12-04T11:13:50.4523241Z [W1204 11:06:30.493689105 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.4523246Z 2025-12-04T11:13:50.4523537Z [W1204 11:06:30.494260608 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.4523574Z 2025-12-04T11:13:50.4523860Z [W1204 11:06:30.494432581 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.4523864Z 2025-12-04T11:13:50.4524158Z [W1204 11:06:30.499773489 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.4524162Z 2025-12-04T11:13:50.4524447Z [W1204 11:06:30.500332481 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.4524450Z 2025-12-04T11:13:50.4524742Z [W1204 11:06:30.500499485 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.4524745Z 2025-12-04T11:13:50.4524830Z ('RERUN', {'yellow': True}) [11.1746s] [100%] 2025-12-04T11:13:50.4525551Z inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_32_mid_dim_1_in_features_144_out_features_65_cuda_bfloat16 [W1204 11:06:31.312246227 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.4525558Z 2025-12-04T11:13:50.4525849Z [W1204 11:06:31.312825630 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.4525853Z 2025-12-04T11:13:50.4526140Z [W1204 11:06:31.312970873 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.4526143Z 2025-12-04T11:13:50.4526431Z [W1204 11:06:31.315902108 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.4526435Z 2025-12-04T11:13:50.4526720Z [W1204 11:06:31.316374928 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.4526725Z 2025-12-04T11:13:50.4527251Z [W1204 11:06:31.316513711 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.4527257Z 2025-12-04T11:13:50.4527549Z [W1204 11:06:31.321235736 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.4527553Z 2025-12-04T11:13:50.4527842Z [W1204 11:06:31.321720497 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.4527846Z 2025-12-04T11:13:50.4528131Z [W1204 11:06:31.321856330 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.4528188Z 2025-12-04T11:13:50.4528305Z ('RERUN', {'yellow': True}) [0.5052s] [100%] 2025-12-04T11:13:50.4529034Z inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_32_mid_dim_1_in_features_144_out_features_65_cuda_bfloat16 [W1204 11:06:31.816108869 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.4529038Z 2025-12-04T11:13:50.4529336Z [W1204 11:06:31.816694402 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.4529339Z 2025-12-04T11:13:50.4529660Z [W1204 11:06:31.816837725 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.4529663Z 2025-12-04T11:13:50.4529952Z [W1204 11:06:31.819752090 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.4529957Z 2025-12-04T11:13:50.4530245Z [W1204 11:06:31.820244970 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.4530282Z 2025-12-04T11:13:50.4530569Z [W1204 11:06:31.820388273 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.4530572Z 2025-12-04T11:13:50.4530864Z [W1204 11:06:31.825018165 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.4530868Z 2025-12-04T11:13:50.4531154Z [W1204 11:06:31.825495116 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.4531157Z 2025-12-04T11:13:50.4531443Z [W1204 11:06:31.825630999 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.4531448Z 2025-12-04T11:13:50.4531513Z FAILED [0.5037s] [100%] 2025-12-04T11:13:50.4531516Z 2025-12-04T11:13:50.4531606Z ==================================== RERUNS ==================================== 2025-12-04T11:13:50.4531904Z _ TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_1_in_features_144_out_features_65_cuda_bfloat16 _ 2025-12-04T11:13:50.4531981Z Traceback (most recent call last): 2025-12-04T11:13:50.4532292Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.4532358Z method(*args, **kwargs) 2025-12-04T11:13:50.4532653Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.4532720Z method(*args, **kwargs) 2025-12-04T11:13:50.4533009Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T11:13:50.4533070Z with policy(): 2025-12-04T11:13:50.4533366Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T11:13:50.4533435Z raise RuntimeError(msg) 2025-12-04T11:13:50.4534236Z RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_1_in_features_144_out_features_65_cuda_bfloat16! Caching allocator allocated memory was 0 and is now reported as 9728 on device 0. CUDA driver allocated memory was 230686720 and is now 274726912. 2025-12-04T11:13:50.4534241Z 2025-12-04T11:13:50.4534370Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.4534897Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_1_in_features_144_out_features_65_cuda_bfloat16 2025-12-04T11:13:50.4534941Z 2025-12-04T11:13:50.4535138Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.4535269Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.4535369Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.4535914Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('benchmarking.InductorBenchmarker.benchmark_gpu', 2), ('pad_mm_bench', 1), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.4536046Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.4536106Z graph_break [] 2025-12-04T11:13:50.4536263Z ----------------------------- Captured stderr call ----------------------------- 2025-12-04T11:13:50.4536963Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/constant_folding.py:256: UserWarning: Unsupported unwinding pattern: Address not in range (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/profiler/unwind/unwind.cpp:219.) 2025-12-04T11:13:50.4537037Z if out == self.unknown_value: 2025-12-04T11:13:50.4537327Z _ TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_1_in_features_144_out_features_65_cuda_bfloat16 _ 2025-12-04T11:13:50.4537437Z Traceback (most recent call last): 2025-12-04T11:13:50.4537838Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.4537952Z method(*args, **kwargs) 2025-12-04T11:13:50.4538464Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.4538543Z method(*args, **kwargs) 2025-12-04T11:13:50.4538847Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T11:13:50.4538910Z with policy(): 2025-12-04T11:13:50.4539207Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T11:13:50.4539273Z raise RuntimeError(msg) 2025-12-04T11:13:50.4540082Z RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_1_in_features_144_out_features_65_cuda_bfloat16! Caching allocator allocated memory was 9728 and is now reported as 19456 on device 0. CUDA driver allocated memory was 274726912 and is now 276824064. 2025-12-04T11:13:50.4540087Z 2025-12-04T11:13:50.4540225Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.4540746Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_1_in_features_144_out_features_65_cuda_bfloat16 2025-12-04T11:13:50.4540750Z 2025-12-04T11:13:50.4540916Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.4541052Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.4541148Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.4541702Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('benchmarking.InductorBenchmarker.benchmark_gpu', 2), ('pad_mm_bench', 1), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.4541831Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.4541894Z graph_break [] 2025-12-04T11:13:50.4542022Z ----------------------------- Captured stderr call ----------------------------- 2025-12-04T11:13:50.4542711Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/constant_folding.py:256: UserWarning: Unsupported unwinding pattern: Address not in range (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/profiler/unwind/unwind.cpp:219.) 2025-12-04T11:13:50.4542846Z if out == self.unknown_value: 2025-12-04T11:13:50.4543008Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.4543108Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.4543229Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.4543838Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('benchmarking.InductorBenchmarker.benchmark_gpu', 2), ('pad_mm_bench', 1), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.4543903Z graph_break [] 2025-12-04T11:13:50.4543987Z =================================== FAILURES =================================== 2025-12-04T11:13:50.4544349Z _ TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_1_in_features_144_out_features_65_cuda_bfloat16 _ 2025-12-04T11:13:50.4544428Z Traceback (most recent call last): 2025-12-04T11:13:50.4544731Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.4544802Z method(*args, **kwargs) 2025-12-04T11:13:50.4545128Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.4545191Z method(*args, **kwargs) 2025-12-04T11:13:50.4545484Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T11:13:50.4545545Z with policy(): 2025-12-04T11:13:50.4545839Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T11:13:50.4545908Z raise RuntimeError(msg) 2025-12-04T11:13:50.4546718Z RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_1_in_features_144_out_features_65_cuda_bfloat16! Caching allocator allocated memory was 19456 and is now reported as 29184 on device 0. CUDA driver allocated memory was 276824064 and is now 278921216. 2025-12-04T11:13:50.4546724Z 2025-12-04T11:13:50.4546863Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.4547382Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_1_in_features_144_out_features_65_cuda_bfloat16 2025-12-04T11:13:50.4547386Z 2025-12-04T11:13:50.4547552Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.4547680Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.4547773Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.4548322Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('benchmarking.InductorBenchmarker.benchmark_gpu', 2), ('pad_mm_bench', 1), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.4548463Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.4548527Z graph_break [] 2025-12-04T11:13:50.4548652Z ----------------------------- Captured stderr call ----------------------------- 2025-12-04T11:13:50.4549333Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/constant_folding.py:256: UserWarning: Unsupported unwinding pattern: Address not in range (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/profiler/unwind/unwind.cpp:219.) 2025-12-04T11:13:50.4549408Z if out == self.unknown_value: 2025-12-04T11:13:50.4549532Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.4549624Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.4549748Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.4550365Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('benchmarking.InductorBenchmarker.benchmark_gpu', 2), ('pad_mm_bench', 1), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.4550431Z graph_break [] 2025-12-04T11:13:50.4550552Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.4550645Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.4550765Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.4551332Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('benchmarking.InductorBenchmarker.benchmark_gpu', 2), ('pad_mm_bench', 1), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.4551394Z graph_break [] 2025-12-04T11:13:50.4551878Z - generated xml file: /var/lib/jenkins/workspace/test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-266b0df1c83533e9.xml - 2025-12-04T11:13:50.4551986Z =========================== short test summary info ============================ 2025-12-04T11:13:50.4553268Z FAILED [0.5037s] inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_32_mid_dim_1_in_features_144_out_features_65_cuda_bfloat16 - RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_1_in_features_144_out_features_65_cuda_bfloat16! Caching allocator allocated memory was 19456 and is now reported as 29184 on device 0. CUDA driver allocated memory was 276824064 and is now 278921216. 2025-12-04T11:13:50.4553307Z 2025-12-04T11:13:50.4553440Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.4553965Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_1_in_features_144_out_features_65_cuda_bfloat16 2025-12-04T11:13:50.4553971Z 2025-12-04T11:13:50.4554133Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.4554243Z !!!!!!!!!!!!!!!!!!!!!!!!!! stopping after 1 failures !!!!!!!!!!!!!!!!!!!!!!!!!!! 2025-12-04T11:13:50.4554360Z ================== 1 failed, 57 deselected, 2 rerun in 12.21s ================== 2025-12-04T11:13:50.4554423Z Got exit code 1 2025-12-04T11:13:50.4554894Z FAILED CONSISTENTLY: test/inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_32_mid_dim_1_in_features_144_out_features_65_cuda_bfloat16 2025-12-04T11:13:50.4555136Z Test failed consistently, continuing with the rest of the tests due to continue-through-error being set 2025-12-04T11:13:50.4555404Z W1204 11:06:38.713000 84176 site-packages/torch/_inductor/utils.py:1703] Not enough SMs to use max_autotune_gemm mode 2025-12-04T11:13:50.4555785Z Test results will be stored in test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-0728186d53d0914c.xml 2025-12-04T11:13:50.4555886Z ============================= test session starts ============================== 2025-12-04T11:13:50.4556096Z platform linux -- Python 3.10.14, pytest-7.3.2, pluggy-1.6.0 -- /opt/conda/envs/py_3.10/bin/python 2025-12-04T11:13:50.4556166Z cachedir: .pytest_cache 2025-12-04T11:13:50.4556474Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow] 2025-12-04T11:13:50.4556551Z rootdir: /var/lib/jenkins/workspace 2025-12-04T11:13:50.4556617Z configfile: pytest.ini 2025-12-04T11:13:50.4556940Z plugins: hypothesis-6.56.4, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-14.0, subtests-0.13.1, xdist-3.3.1, xdoctest-1.3.0, typeguard-4.3.0 2025-12-04T11:13:50.4557071Z collecting ... collected 58 items / 49 deselected / 9 selected 2025-12-04T11:13:50.4557160Z stepcurrent: skipping 49 already run items. 2025-12-04T11:13:50.4557280Z Running 9 items in this shard 2025-12-04T11:13:50.4557285Z 2025-12-04T11:13:50.4557824Z inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_32_mid_dim_8_in_features_1024_out_features_1024_cuda_bfloat16 ('RERUN', {'yellow': True}) [2.0298s] [ 11%] 2025-12-04T11:13:50.4558322Z inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_32_mid_dim_8_in_features_1024_out_features_1024_cuda_bfloat16 ('RERUN', {'yellow': True}) [0.6182s] [ 11%] 2025-12-04T11:13:50.4558772Z inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_32_mid_dim_8_in_features_1024_out_features_1024_cuda_bfloat16 FAILED [0.6305s] [ 11%] 2025-12-04T11:13:50.4558776Z 2025-12-04T11:13:50.4558898Z ==================================== RERUNS ==================================== 2025-12-04T11:13:50.4559198Z _ TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_8_in_features_1024_out_features_1024_cuda_bfloat16 _ 2025-12-04T11:13:50.4559276Z Traceback (most recent call last): 2025-12-04T11:13:50.4559584Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.4559683Z method(*args, **kwargs) 2025-12-04T11:13:50.4559979Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.4560041Z method(*args, **kwargs) 2025-12-04T11:13:50.4560330Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T11:13:50.4560393Z with policy(): 2025-12-04T11:13:50.4560691Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T11:13:50.4560755Z raise RuntimeError(msg) 2025-12-04T11:13:50.4561568Z RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_8_in_features_1024_out_features_1024_cuda_bfloat16! Caching allocator allocated memory was 0 and is now reported as 1048576 on device 0. CUDA driver allocated memory was 230686720 and is now 274726912. 2025-12-04T11:13:50.4561575Z 2025-12-04T11:13:50.4561703Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.4562234Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_8_in_features_1024_out_features_1024_cuda_bfloat16 2025-12-04T11:13:50.4562237Z 2025-12-04T11:13:50.4562404Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.4562536Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.4562632Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.4562982Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.4563115Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.4563177Z graph_break [] 2025-12-04T11:13:50.4563474Z _ TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_8_in_features_1024_out_features_1024_cuda_bfloat16 _ 2025-12-04T11:13:50.4563549Z Traceback (most recent call last): 2025-12-04T11:13:50.4563846Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.4563915Z method(*args, **kwargs) 2025-12-04T11:13:50.4564203Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.4564265Z method(*args, **kwargs) 2025-12-04T11:13:50.4564555Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T11:13:50.4564663Z with policy(): 2025-12-04T11:13:50.4564992Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T11:13:50.4565059Z raise RuntimeError(msg) 2025-12-04T11:13:50.4565884Z RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_8_in_features_1024_out_features_1024_cuda_bfloat16! Caching allocator allocated memory was 1048576 and is now reported as 2097152 on device 0. CUDA driver allocated memory was 274726912 and is now 276824064. 2025-12-04T11:13:50.4565888Z 2025-12-04T11:13:50.4566017Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.4566575Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_8_in_features_1024_out_features_1024_cuda_bfloat16 2025-12-04T11:13:50.4566581Z 2025-12-04T11:13:50.4566743Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.4566869Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.4567000Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.4567350Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.4567477Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.4567539Z graph_break [] 2025-12-04T11:13:50.4567663Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.4567753Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.4567877Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.4568223Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.4568283Z graph_break [] 2025-12-04T11:13:50.4568372Z =================================== FAILURES =================================== 2025-12-04T11:13:50.4568668Z _ TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_8_in_features_1024_out_features_1024_cuda_bfloat16 _ 2025-12-04T11:13:50.4568744Z Traceback (most recent call last): 2025-12-04T11:13:50.4569049Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.4569115Z method(*args, **kwargs) 2025-12-04T11:13:50.4569421Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.4569485Z method(*args, **kwargs) 2025-12-04T11:13:50.4569776Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T11:13:50.4569838Z with policy(): 2025-12-04T11:13:50.4570130Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T11:13:50.4570198Z raise RuntimeError(msg) 2025-12-04T11:13:50.4571021Z RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_8_in_features_1024_out_features_1024_cuda_bfloat16! Caching allocator allocated memory was 2097152 and is now reported as 3145728 on device 0. CUDA driver allocated memory was 276824064 and is now 278921216. 2025-12-04T11:13:50.4571025Z 2025-12-04T11:13:50.4571154Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.4571679Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_8_in_features_1024_out_features_1024_cuda_bfloat16 2025-12-04T11:13:50.4571723Z 2025-12-04T11:13:50.4571930Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.4572058Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.4572147Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.4572491Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.4572615Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.4572671Z graph_break [] 2025-12-04T11:13:50.4572831Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.4572922Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.4573043Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.4573387Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.4573479Z graph_break [] 2025-12-04T11:13:50.4573603Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.4573691Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.4573810Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.4574151Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.4574210Z graph_break [] 2025-12-04T11:13:50.4574698Z - generated xml file: /var/lib/jenkins/workspace/test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-0728186d53d0914c.xml - 2025-12-04T11:13:50.4574799Z =========================== short test summary info ============================ 2025-12-04T11:13:50.4576102Z FAILED [0.6305s] inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_32_mid_dim_8_in_features_1024_out_features_1024_cuda_bfloat16 - RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_8_in_features_1024_out_features_1024_cuda_bfloat16! Caching allocator allocated memory was 2097152 and is now reported as 3145728 on device 0. CUDA driver allocated memory was 276824064 and is now 278921216. 2025-12-04T11:13:50.4576110Z 2025-12-04T11:13:50.4576234Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.4576766Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_8_in_features_1024_out_features_1024_cuda_bfloat16 2025-12-04T11:13:50.4576771Z 2025-12-04T11:13:50.4576932Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.4577035Z !!!!!!!!!!!!!!!!!!!!!!!!!! stopping after 1 failures !!!!!!!!!!!!!!!!!!!!!!!!!!! 2025-12-04T11:13:50.4577158Z ================== 1 failed, 49 deselected, 2 rerun in 3.30s =================== 2025-12-04T11:13:50.4577216Z Got exit code 1 2025-12-04T11:13:50.4577282Z Retrying single test... 2025-12-04T11:13:50.4577548Z W1204 11:06:48.935000 84365 site-packages/torch/_inductor/utils.py:1703] Not enough SMs to use max_autotune_gemm mode 2025-12-04T11:13:50.4578120Z Test results will be stored in test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-ace5a67d4c3c2a01.xml 2025-12-04T11:13:50.4578222Z ============================= test session starts ============================== 2025-12-04T11:13:50.4578437Z platform linux -- Python 3.10.14, pytest-7.3.2, pluggy-1.6.0 -- /opt/conda/envs/py_3.10/bin/python 2025-12-04T11:13:50.4578582Z cachedir: .pytest_cache 2025-12-04T11:13:50.4578948Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow] 2025-12-04T11:13:50.4579032Z rootdir: /var/lib/jenkins/workspace 2025-12-04T11:13:50.4579098Z configfile: pytest.ini 2025-12-04T11:13:50.4579425Z plugins: hypothesis-6.56.4, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-14.0, subtests-0.13.1, xdist-3.3.1, xdoctest-1.3.0, typeguard-4.3.0 2025-12-04T11:13:50.4579556Z collecting ... collected 58 items / 57 deselected / 1 selected 2025-12-04T11:13:50.4580139Z stepcurrent: skipping 49 already run items. Running only test/inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_32_mid_dim_8_in_features_1024_out_features_1024_cuda_bfloat16 2025-12-04T11:13:50.4580269Z Running 1 items in this shard 2025-12-04T11:13:50.4580274Z 2025-12-04T11:13:50.4581018Z inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_32_mid_dim_8_in_features_1024_out_features_1024_cuda_bfloat16 [W1204 11:06:50.209188651 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.4581029Z 2025-12-04T11:13:50.4581379Z [W1204 11:06:59.179138293 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.4581383Z 2025-12-04T11:13:50.4581671Z [W1204 11:06:59.179374768 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.4581674Z 2025-12-04T11:13:50.4581963Z [W1204 11:06:59.185124325 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.4581967Z 2025-12-04T11:13:50.4582256Z [W1204 11:06:59.185677277 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.4582261Z 2025-12-04T11:13:50.4582549Z [W1204 11:06:59.185842061 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.4582553Z 2025-12-04T11:13:50.4582839Z [W1204 11:06:59.191265461 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.4582844Z 2025-12-04T11:13:50.4583129Z [W1204 11:06:59.191797562 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.4583132Z 2025-12-04T11:13:50.4583416Z [W1204 11:06:59.192003507 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.4583420Z 2025-12-04T11:13:50.4583556Z ('RERUN', {'yellow': True}) [11.0421s] [100%] 2025-12-04T11:13:50.4584305Z inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_32_mid_dim_8_in_features_1024_out_features_1024_cuda_bfloat16 [W1204 11:07:00.557367226 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.4584311Z 2025-12-04T11:13:50.4584600Z [W1204 11:07:00.557905107 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.4584603Z 2025-12-04T11:13:50.4584889Z [W1204 11:07:00.558043390 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.4584893Z 2025-12-04T11:13:50.4585180Z [W1204 11:07:00.560974115 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.4585183Z 2025-12-04T11:13:50.4585472Z [W1204 11:07:00.561538578 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.4585519Z 2025-12-04T11:13:50.4585806Z [W1204 11:07:00.561675291 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.4585843Z 2025-12-04T11:13:50.4586132Z [W1204 11:07:00.566224531 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.4586137Z 2025-12-04T11:13:50.4586420Z [W1204 11:07:00.566694812 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.4586423Z 2025-12-04T11:13:50.4586711Z [W1204 11:07:00.566829965 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.4586714Z 2025-12-04T11:13:50.4586827Z ('RERUN', {'yellow': True}) [0.6034s] [100%] 2025-12-04T11:13:50.4587560Z inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_32_mid_dim_8_in_features_1024_out_features_1024_cuda_bfloat16 [W1204 11:07:01.159426119 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.4587568Z 2025-12-04T11:13:50.4587856Z [W1204 11:07:01.159961461 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.4587893Z 2025-12-04T11:13:50.4588185Z [W1204 11:07:01.160121294 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.4588188Z 2025-12-04T11:13:50.4588479Z [W1204 11:07:01.163048858 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.4588481Z 2025-12-04T11:13:50.4588769Z [W1204 11:07:01.163604721 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.4588772Z 2025-12-04T11:13:50.4589061Z [W1204 11:07:01.163741854 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.4589066Z 2025-12-04T11:13:50.4589348Z [W1204 11:07:01.168323334 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.4589353Z 2025-12-04T11:13:50.4589650Z [W1204 11:07:01.168799175 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.4589654Z 2025-12-04T11:13:50.4589938Z [W1204 11:07:01.168935048 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.4589942Z 2025-12-04T11:13:50.4590006Z FAILED [0.6039s] [100%] 2025-12-04T11:13:50.4590011Z 2025-12-04T11:13:50.4590095Z ==================================== RERUNS ==================================== 2025-12-04T11:13:50.4590393Z _ TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_8_in_features_1024_out_features_1024_cuda_bfloat16 _ 2025-12-04T11:13:50.4590473Z Traceback (most recent call last): 2025-12-04T11:13:50.4590782Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.4590854Z method(*args, **kwargs) 2025-12-04T11:13:50.4591145Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.4591207Z method(*args, **kwargs) 2025-12-04T11:13:50.4591498Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T11:13:50.4591560Z with policy(): 2025-12-04T11:13:50.4591854Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T11:13:50.4591923Z raise RuntimeError(msg) 2025-12-04T11:13:50.4592811Z RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_8_in_features_1024_out_features_1024_cuda_bfloat16! Caching allocator allocated memory was 0 and is now reported as 1048576 on device 0. CUDA driver allocated memory was 230686720 and is now 274726912. 2025-12-04T11:13:50.4593731Z 2025-12-04T11:13:50.4593862Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.4594611Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_8_in_features_1024_out_features_1024_cuda_bfloat16 2025-12-04T11:13:50.4595213Z 2025-12-04T11:13:50.4595380Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.4595786Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.4596091Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.4596622Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.4597174Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.4597475Z graph_break [] 2025-12-04T11:13:50.4597695Z ----------------------------- Captured stderr call ----------------------------- 2025-12-04T11:13:50.4598595Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/constant_folding.py:256: UserWarning: Unsupported unwinding pattern: Address not in range (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/profiler/unwind/unwind.cpp:219.) 2025-12-04T11:13:50.4599431Z if out == self.unknown_value: 2025-12-04T11:13:50.4599875Z _ TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_8_in_features_1024_out_features_1024_cuda_bfloat16 _ 2025-12-04T11:13:50.4600323Z Traceback (most recent call last): 2025-12-04T11:13:50.4600768Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.4601205Z method(*args, **kwargs) 2025-12-04T11:13:50.4601611Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.4602043Z method(*args, **kwargs) 2025-12-04T11:13:50.4602445Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T11:13:50.4602900Z with policy(): 2025-12-04T11:13:50.4603295Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T11:13:50.4603732Z raise RuntimeError(msg) 2025-12-04T11:13:50.4604693Z RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_8_in_features_1024_out_features_1024_cuda_bfloat16! Caching allocator allocated memory was 1048576 and is now reported as 2097152 on device 0. CUDA driver allocated memory was 274726912 and is now 276824064. 2025-12-04T11:13:50.4605601Z 2025-12-04T11:13:50.4605735Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.4606465Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_8_in_features_1024_out_features_1024_cuda_bfloat16 2025-12-04T11:13:50.4607064Z 2025-12-04T11:13:50.4607224Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.4607587Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.4607888Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.4608405Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.4609019Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.4609318Z graph_break [] 2025-12-04T11:13:50.4609537Z ----------------------------- Captured stderr call ----------------------------- 2025-12-04T11:13:50.4610433Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/constant_folding.py:256: UserWarning: Unsupported unwinding pattern: Address not in range (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/profiler/unwind/unwind.cpp:219.) 2025-12-04T11:13:50.4611253Z if out == self.unknown_value: 2025-12-04T11:13:50.4611503Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.4611794Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.4612394Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.4612953Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.4613434Z graph_break [] 2025-12-04T11:13:50.4613607Z =================================== FAILURES =================================== 2025-12-04T11:13:50.4614117Z _ TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_8_in_features_1024_out_features_1024_cuda_bfloat16 _ 2025-12-04T11:13:50.4614566Z Traceback (most recent call last): 2025-12-04T11:13:50.4615007Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.4615441Z method(*args, **kwargs) 2025-12-04T11:13:50.4615879Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.4616323Z method(*args, **kwargs) 2025-12-04T11:13:50.4616728Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T11:13:50.4617157Z with policy(): 2025-12-04T11:13:50.4617556Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T11:13:50.4617996Z raise RuntimeError(msg) 2025-12-04T11:13:50.4618946Z RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_8_in_features_1024_out_features_1024_cuda_bfloat16! Caching allocator allocated memory was 2097152 and is now reported as 3145728 on device 0. CUDA driver allocated memory was 276824064 and is now 278921216. 2025-12-04T11:13:50.4619853Z 2025-12-04T11:13:50.4619981Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.4620715Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_8_in_features_1024_out_features_1024_cuda_bfloat16 2025-12-04T11:13:50.4621313Z 2025-12-04T11:13:50.4621479Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.4621840Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.4622142Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.4622659Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.4623212Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.4623475Z graph_break [] 2025-12-04T11:13:50.4623745Z ----------------------------- Captured stderr call ----------------------------- 2025-12-04T11:13:50.4624638Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/constant_folding.py:256: UserWarning: Unsupported unwinding pattern: Address not in range (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/profiler/unwind/unwind.cpp:219.) 2025-12-04T11:13:50.4625512Z if out == self.unknown_value: 2025-12-04T11:13:50.4625792Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.4626089Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.4626383Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.4626927Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.4627405Z graph_break [] 2025-12-04T11:13:50.4627618Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.4627909Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.4628239Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.4628783Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.4629259Z graph_break [] 2025-12-04T11:13:50.4629844Z - generated xml file: /var/lib/jenkins/workspace/test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-ace5a67d4c3c2a01.xml - 2025-12-04T11:13:50.4630541Z =========================== short test summary info ============================ 2025-12-04T11:13:50.4632039Z FAILED [0.6039s] inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_32_mid_dim_8_in_features_1024_out_features_1024_cuda_bfloat16 - RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_8_in_features_1024_out_features_1024_cuda_bfloat16! Caching allocator allocated memory was 2097152 and is now reported as 3145728 on device 0. CUDA driver allocated memory was 276824064 and is now 278921216. 2025-12-04T11:13:50.4633411Z 2025-12-04T11:13:50.4633538Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.4634265Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_8_in_features_1024_out_features_1024_cuda_bfloat16 2025-12-04T11:13:50.4634865Z 2025-12-04T11:13:50.4635026Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.4635359Z !!!!!!!!!!!!!!!!!!!!!!!!!! stopping after 1 failures !!!!!!!!!!!!!!!!!!!!!!!!!!! 2025-12-04T11:13:50.4635658Z ================== 1 failed, 57 deselected, 2 rerun in 12.27s ================== 2025-12-04T11:13:50.4635909Z Got exit code 1 2025-12-04T11:13:50.4636067Z Retrying single test... 2025-12-04T11:13:50.4636437Z W1204 11:07:08.009000 84559 site-packages/torch/_inductor/utils.py:1703] Not enough SMs to use max_autotune_gemm mode 2025-12-04T11:13:50.4637157Z Test results will be stored in test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-eeeef84645c1bb52.xml 2025-12-04T11:13:50.4637714Z ============================= test session starts ============================== 2025-12-04T11:13:50.4638094Z platform linux -- Python 3.10.14, pytest-7.3.2, pluggy-1.6.0 -- /opt/conda/envs/py_3.10/bin/python 2025-12-04T11:13:50.4638455Z cachedir: .pytest_cache 2025-12-04T11:13:50.4638868Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow] 2025-12-04T11:13:50.4639317Z rootdir: /var/lib/jenkins/workspace 2025-12-04T11:13:50.4639518Z configfile: pytest.ini 2025-12-04T11:13:50.4639941Z plugins: hypothesis-6.56.4, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-14.0, subtests-0.13.1, xdist-3.3.1, xdoctest-1.3.0, typeguard-4.3.0 2025-12-04T11:13:50.4640457Z collecting ... collected 58 items / 57 deselected / 1 selected 2025-12-04T11:13:50.4641236Z stepcurrent: skipping 49 already run items. Running only test/inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_32_mid_dim_8_in_features_1024_out_features_1024_cuda_bfloat16 2025-12-04T11:13:50.4642043Z Running 1 items in this shard 2025-12-04T11:13:50.4642174Z 2025-12-04T11:13:50.4642916Z inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_32_mid_dim_8_in_features_1024_out_features_1024_cuda_bfloat16 [W1204 11:07:09.284832023 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.4643728Z 2025-12-04T11:13:50.4644026Z [W1204 11:07:18.554344531 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.4644396Z 2025-12-04T11:13:50.4644724Z [W1204 11:07:18.554572446 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.4645089Z 2025-12-04T11:13:50.4645380Z [W1204 11:07:18.560259372 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.4645746Z 2025-12-04T11:13:50.4646035Z [W1204 11:07:18.560819594 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.4646441Z 2025-12-04T11:13:50.4646728Z [W1204 11:07:18.560989108 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.4647099Z 2025-12-04T11:13:50.4647387Z [W1204 11:07:18.566244644 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.4647753Z 2025-12-04T11:13:50.4648047Z [W1204 11:07:18.566766905 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.4648411Z 2025-12-04T11:13:50.4648707Z [W1204 11:07:18.566954249 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.4649260Z 2025-12-04T11:13:50.4649355Z ('RERUN', {'yellow': True}) [11.3427s] [100%] 2025-12-04T11:13:50.4650258Z inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_32_mid_dim_8_in_features_1024_out_features_1024_cuda_bfloat16 [W1204 11:07:19.931321107 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.4651064Z 2025-12-04T11:13:50.4651353Z [W1204 11:07:19.931856139 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.4651719Z 2025-12-04T11:13:50.4652011Z [W1204 11:07:19.931994142 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.4652376Z 2025-12-04T11:13:50.4652667Z [W1204 11:07:19.934915666 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.4653035Z 2025-12-04T11:13:50.4653325Z [W1204 11:07:19.935480899 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.4653694Z 2025-12-04T11:13:50.4653981Z [W1204 11:07:19.935620022 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.4654350Z 2025-12-04T11:13:50.4654638Z [W1204 11:07:19.940221674 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.4655001Z 2025-12-04T11:13:50.4655292Z [W1204 11:07:19.940702434 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.4655657Z 2025-12-04T11:13:50.4655944Z [W1204 11:07:19.940843858 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.4656378Z 2025-12-04T11:13:50.4656574Z ('RERUN', {'yellow': True}) [0.6055s] [100%] 2025-12-04T11:13:50.4657458Z inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_32_mid_dim_8_in_features_1024_out_features_1024_cuda_bfloat16 [W1204 11:07:20.533062010 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.4658270Z 2025-12-04T11:13:50.4658559Z [W1204 11:07:20.533599691 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.4658928Z 2025-12-04T11:13:50.4659250Z [W1204 11:07:20.533737304 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.4659647Z 2025-12-04T11:13:50.4660134Z [W1204 11:07:20.536704830 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.4660520Z 2025-12-04T11:13:50.4660814Z [W1204 11:07:20.537268282 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.4661255Z 2025-12-04T11:13:50.4661548Z [W1204 11:07:20.537405715 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.4661915Z 2025-12-04T11:13:50.4662201Z [W1204 11:07:20.542026767 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.4662569Z 2025-12-04T11:13:50.4662856Z [W1204 11:07:20.542496137 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.4663221Z 2025-12-04T11:13:50.4663597Z [W1204 11:07:20.542630710 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.4663963Z 2025-12-04T11:13:50.4664029Z FAILED [0.6092s] [100%] 2025-12-04T11:13:50.4664136Z 2025-12-04T11:13:50.4664225Z ==================================== RERUNS ==================================== 2025-12-04T11:13:50.4664698Z _ TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_8_in_features_1024_out_features_1024_cuda_bfloat16 _ 2025-12-04T11:13:50.4665147Z Traceback (most recent call last): 2025-12-04T11:13:50.4665591Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.4666027Z method(*args, **kwargs) 2025-12-04T11:13:50.4666437Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.4666865Z method(*args, **kwargs) 2025-12-04T11:13:50.4667277Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T11:13:50.4667706Z with policy(): 2025-12-04T11:13:50.4668096Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T11:13:50.4668532Z raise RuntimeError(msg) 2025-12-04T11:13:50.4669469Z RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_8_in_features_1024_out_features_1024_cuda_bfloat16! Caching allocator allocated memory was 0 and is now reported as 1048576 on device 0. CUDA driver allocated memory was 230686720 and is now 274726912. 2025-12-04T11:13:50.4670350Z 2025-12-04T11:13:50.4670478Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.4671209Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_8_in_features_1024_out_features_1024_cuda_bfloat16 2025-12-04T11:13:50.4671855Z 2025-12-04T11:13:50.4672056Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.4672421Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.4672724Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.4673250Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.4673806Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.4674068Z graph_break [] 2025-12-04T11:13:50.4674289Z ----------------------------- Captured stderr call ----------------------------- 2025-12-04T11:13:50.4675224Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/constant_folding.py:256: UserWarning: Unsupported unwinding pattern: Address not in range (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/profiler/unwind/unwind.cpp:219.) 2025-12-04T11:13:50.4676049Z if out == self.unknown_value: 2025-12-04T11:13:50.4676472Z _ TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_8_in_features_1024_out_features_1024_cuda_bfloat16 _ 2025-12-04T11:13:50.4676956Z Traceback (most recent call last): 2025-12-04T11:13:50.4677393Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.4678007Z method(*args, **kwargs) 2025-12-04T11:13:50.4678421Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.4678851Z method(*args, **kwargs) 2025-12-04T11:13:50.4679255Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T11:13:50.4679683Z with policy(): 2025-12-04T11:13:50.4680074Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T11:13:50.4680513Z raise RuntimeError(msg) 2025-12-04T11:13:50.4681465Z RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_8_in_features_1024_out_features_1024_cuda_bfloat16! Caching allocator allocated memory was 1048576 and is now reported as 2097152 on device 0. CUDA driver allocated memory was 274726912 and is now 276824064. 2025-12-04T11:13:50.4682359Z 2025-12-04T11:13:50.4682485Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.4683212Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_8_in_features_1024_out_features_1024_cuda_bfloat16 2025-12-04T11:13:50.4683812Z 2025-12-04T11:13:50.4683970Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.4684340Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.4684642Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.4685163Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.4685709Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.4685970Z graph_break [] 2025-12-04T11:13:50.4686185Z ----------------------------- Captured stderr call ----------------------------- 2025-12-04T11:13:50.4687078Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/constant_folding.py:256: UserWarning: Unsupported unwinding pattern: Address not in range (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/profiler/unwind/unwind.cpp:219.) 2025-12-04T11:13:50.4687914Z if out == self.unknown_value: 2025-12-04T11:13:50.4688253Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.4688549Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.4688893Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.4689444Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.4689921Z graph_break [] 2025-12-04T11:13:50.4690092Z =================================== FAILURES =================================== 2025-12-04T11:13:50.4690555Z _ TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_8_in_features_1024_out_features_1024_cuda_bfloat16 _ 2025-12-04T11:13:50.4691004Z Traceback (most recent call last): 2025-12-04T11:13:50.4691505Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.4691949Z method(*args, **kwargs) 2025-12-04T11:13:50.4692355Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.4692784Z method(*args, **kwargs) 2025-12-04T11:13:50.4693244Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T11:13:50.4693674Z with policy(): 2025-12-04T11:13:50.4694063Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T11:13:50.4694502Z raise RuntimeError(msg) 2025-12-04T11:13:50.4695471Z RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_8_in_features_1024_out_features_1024_cuda_bfloat16! Caching allocator allocated memory was 2097152 and is now reported as 3145728 on device 0. CUDA driver allocated memory was 276824064 and is now 278921216. 2025-12-04T11:13:50.4696377Z 2025-12-04T11:13:50.4696507Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.4697246Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_8_in_features_1024_out_features_1024_cuda_bfloat16 2025-12-04T11:13:50.4697847Z 2025-12-04T11:13:50.4698010Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.4698376Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.4698677Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.4699198Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.4699776Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.4700042Z graph_break [] 2025-12-04T11:13:50.4700263Z ----------------------------- Captured stderr call ----------------------------- 2025-12-04T11:13:50.4701167Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/constant_folding.py:256: UserWarning: Unsupported unwinding pattern: Address not in range (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/profiler/unwind/unwind.cpp:219.) 2025-12-04T11:13:50.4702002Z if out == self.unknown_value: 2025-12-04T11:13:50.4702256Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.4702595Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.4702891Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.4703438Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.4703971Z graph_break [] 2025-12-04T11:13:50.4704182Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.4704544Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.4704868Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.4705450Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.4705928Z graph_break [] 2025-12-04T11:13:50.4706507Z - generated xml file: /var/lib/jenkins/workspace/test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-eeeef84645c1bb52.xml - 2025-12-04T11:13:50.4707167Z =========================== short test summary info ============================ 2025-12-04T11:13:50.4708706Z FAILED [0.6092s] inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_32_mid_dim_8_in_features_1024_out_features_1024_cuda_bfloat16 - RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_8_in_features_1024_out_features_1024_cuda_bfloat16! Caching allocator allocated memory was 2097152 and is now reported as 3145728 on device 0. CUDA driver allocated memory was 276824064 and is now 278921216. 2025-12-04T11:13:50.4710113Z 2025-12-04T11:13:50.4710243Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.4710971Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_8_in_features_1024_out_features_1024_cuda_bfloat16 2025-12-04T11:13:50.4711571Z 2025-12-04T11:13:50.4711728Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.4712075Z !!!!!!!!!!!!!!!!!!!!!!!!!! stopping after 1 failures !!!!!!!!!!!!!!!!!!!!!!!!!!! 2025-12-04T11:13:50.4712377Z ================== 1 failed, 57 deselected, 2 rerun in 12.58s ================== 2025-12-04T11:13:50.4712629Z Got exit code 1 2025-12-04T11:13:50.4713208Z FAILED CONSISTENTLY: test/inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_32_mid_dim_8_in_features_1024_out_features_1024_cuda_bfloat16 2025-12-04T11:13:50.4714002Z Test failed consistently, continuing with the rest of the tests due to continue-through-error being set 2025-12-04T11:13:50.4714587Z W1204 11:07:27.866000 84753 site-packages/torch/_inductor/utils.py:1703] Not enough SMs to use max_autotune_gemm mode 2025-12-04T11:13:50.4715344Z Test results will be stored in test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-ffbbb699e0b6f991.xml 2025-12-04T11:13:50.4715907Z ============================= test session starts ============================== 2025-12-04T11:13:50.4716292Z platform linux -- Python 3.10.14, pytest-7.3.2, pluggy-1.6.0 -- /opt/conda/envs/py_3.10/bin/python 2025-12-04T11:13:50.4716668Z cachedir: .pytest_cache 2025-12-04T11:13:50.4717089Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow] 2025-12-04T11:13:50.4717537Z rootdir: /var/lib/jenkins/workspace 2025-12-04T11:13:50.4717748Z configfile: pytest.ini 2025-12-04T11:13:50.4718172Z plugins: hypothesis-6.56.4, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-14.0, subtests-0.13.1, xdist-3.3.1, xdoctest-1.3.0, typeguard-4.3.0 2025-12-04T11:13:50.4718688Z collecting ... collected 58 items / 50 deselected / 8 selected 2025-12-04T11:13:50.4718974Z stepcurrent: skipping 50 already run items. 2025-12-04T11:13:50.4719204Z Running 8 items in this shard 2025-12-04T11:13:50.4719337Z 2025-12-04T11:13:50.4719905Z inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_32_mid_dim_8_in_features_1024_out_features_64_cuda_bfloat16 ('RERUN', {'yellow': True}) [1.8825s] [ 12%] 2025-12-04T11:13:50.4721015Z inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_32_mid_dim_8_in_features_1024_out_features_64_cuda_bfloat16 ('RERUN', {'yellow': True}) [0.4593s] [ 12%] 2025-12-04T11:13:50.4722127Z inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_32_mid_dim_8_in_features_1024_out_features_64_cuda_bfloat16 FAILED [0.4555s] [ 12%] 2025-12-04T11:13:50.4722655Z 2025-12-04T11:13:50.4722740Z ==================================== RERUNS ==================================== 2025-12-04T11:13:50.4723200Z _ TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_8_in_features_1024_out_features_64_cuda_bfloat16 _ 2025-12-04T11:13:50.4723642Z Traceback (most recent call last): 2025-12-04T11:13:50.4724146Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.4724589Z method(*args, **kwargs) 2025-12-04T11:13:50.4725000Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.4725431Z method(*args, **kwargs) 2025-12-04T11:13:50.4725865Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T11:13:50.4726339Z with policy(): 2025-12-04T11:13:50.4726728Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T11:13:50.4727194Z raise RuntimeError(msg) 2025-12-04T11:13:50.4728116Z RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_8_in_features_1024_out_features_64_cuda_bfloat16! Caching allocator allocated memory was 0 and is now reported as 65536 on device 0. CUDA driver allocated memory was 230686720 and is now 274726912. 2025-12-04T11:13:50.4728990Z 2025-12-04T11:13:50.4729117Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.4729880Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_8_in_features_1024_out_features_64_cuda_bfloat16 2025-12-04T11:13:50.4730477Z 2025-12-04T11:13:50.4730634Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.4731000Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.4731303Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.4731825Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.4732536Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.4732853Z graph_break [] 2025-12-04T11:13:50.4733254Z _ TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_8_in_features_1024_out_features_64_cuda_bfloat16 _ 2025-12-04T11:13:50.4733704Z Traceback (most recent call last): 2025-12-04T11:13:50.4734158Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.4734604Z method(*args, **kwargs) 2025-12-04T11:13:50.4735014Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.4735458Z method(*args, **kwargs) 2025-12-04T11:13:50.4735867Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T11:13:50.4736291Z with policy(): 2025-12-04T11:13:50.4736686Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T11:13:50.4737118Z raise RuntimeError(msg) 2025-12-04T11:13:50.4738113Z RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_8_in_features_1024_out_features_64_cuda_bfloat16! Caching allocator allocated memory was 65536 and is now reported as 131072 on device 0. CUDA driver allocated memory was 274726912 and is now 276824064. 2025-12-04T11:13:50.4739078Z 2025-12-04T11:13:50.4739209Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.4739942Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_8_in_features_1024_out_features_64_cuda_bfloat16 2025-12-04T11:13:50.4740536Z 2025-12-04T11:13:50.4740697Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.4741104Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.4741408Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.4741928Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.4742477Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.4742780Z graph_break [] 2025-12-04T11:13:50.4743003Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.4743327Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.4743678Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.4744225Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.4744801Z graph_break [] 2025-12-04T11:13:50.4744979Z =================================== FAILURES =================================== 2025-12-04T11:13:50.4745442Z _ TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_8_in_features_1024_out_features_64_cuda_bfloat16 _ 2025-12-04T11:13:50.4745889Z Traceback (most recent call last): 2025-12-04T11:13:50.4746331Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.4746765Z method(*args, **kwargs) 2025-12-04T11:13:50.4747173Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.4747604Z method(*args, **kwargs) 2025-12-04T11:13:50.4748010Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T11:13:50.4748435Z with policy(): 2025-12-04T11:13:50.4748831Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T11:13:50.4749265Z raise RuntimeError(msg) 2025-12-04T11:13:50.4750205Z RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_8_in_features_1024_out_features_64_cuda_bfloat16! Caching allocator allocated memory was 131072 and is now reported as 196608 on device 0. CUDA driver allocated memory was 276824064 and is now 278921216. 2025-12-04T11:13:50.4751103Z 2025-12-04T11:13:50.4751231Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.4751954Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_8_in_features_1024_out_features_64_cuda_bfloat16 2025-12-04T11:13:50.4752548Z 2025-12-04T11:13:50.4752717Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.4753084Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.4753384Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.4753964Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.4754644Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.4754912Z graph_break [] 2025-12-04T11:13:50.4755133Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.4755434Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.4755723Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.4756261Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.4756737Z graph_break [] 2025-12-04T11:13:50.4756991Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.4757285Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.4757649Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.4758201Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.4758730Z graph_break [] 2025-12-04T11:13:50.4759472Z - generated xml file: /var/lib/jenkins/workspace/test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-ffbbb699e0b6f991.xml - 2025-12-04T11:13:50.4760276Z =========================== short test summary info ============================ 2025-12-04T11:13:50.4761904Z FAILED [0.4555s] inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_32_mid_dim_8_in_features_1024_out_features_64_cuda_bfloat16 - RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_8_in_features_1024_out_features_64_cuda_bfloat16! Caching allocator allocated memory was 131072 and is now reported as 196608 on device 0. CUDA driver allocated memory was 276824064 and is now 278921216. 2025-12-04T11:13:50.4763265Z 2025-12-04T11:13:50.4763393Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.4764122Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_8_in_features_1024_out_features_64_cuda_bfloat16 2025-12-04T11:13:50.4764717Z 2025-12-04T11:13:50.4764876Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.4765221Z !!!!!!!!!!!!!!!!!!!!!!!!!! stopping after 1 failures !!!!!!!!!!!!!!!!!!!!!!!!!!! 2025-12-04T11:13:50.4765525Z ================== 1 failed, 50 deselected, 2 rerun in 2.82s =================== 2025-12-04T11:13:50.4765779Z Got exit code 1 2025-12-04T11:13:50.4765938Z Retrying single test... 2025-12-04T11:13:50.4766314Z W1204 11:07:37.956000 84941 site-packages/torch/_inductor/utils.py:1703] Not enough SMs to use max_autotune_gemm mode 2025-12-04T11:13:50.4767041Z Test results will be stored in test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-90e4a91c9bbb810f.xml 2025-12-04T11:13:50.4767603Z ============================= test session starts ============================== 2025-12-04T11:13:50.4767989Z platform linux -- Python 3.10.14, pytest-7.3.2, pluggy-1.6.0 -- /opt/conda/envs/py_3.10/bin/python 2025-12-04T11:13:50.4768338Z cachedir: .pytest_cache 2025-12-04T11:13:50.4768753Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow] 2025-12-04T11:13:50.4769204Z rootdir: /var/lib/jenkins/workspace 2025-12-04T11:13:50.4769415Z configfile: pytest.ini 2025-12-04T11:13:50.4769917Z plugins: hypothesis-6.56.4, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-14.0, subtests-0.13.1, xdist-3.3.1, xdoctest-1.3.0, typeguard-4.3.0 2025-12-04T11:13:50.4770512Z collecting ... collected 58 items / 57 deselected / 1 selected 2025-12-04T11:13:50.4771324Z stepcurrent: skipping 50 already run items. Running only test/inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_32_mid_dim_8_in_features_1024_out_features_64_cuda_bfloat16 2025-12-04T11:13:50.4772039Z Running 1 items in this shard 2025-12-04T11:13:50.4772167Z 2025-12-04T11:13:50.4772907Z inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_32_mid_dim_8_in_features_1024_out_features_64_cuda_bfloat16 [W1204 11:07:39.057046763 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.4773708Z 2025-12-04T11:13:50.4774048Z [W1204 11:07:48.205947054 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.4774429Z 2025-12-04T11:13:50.4774729Z [W1204 11:07:48.206233090 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.4775103Z 2025-12-04T11:13:50.4775396Z [W1204 11:07:48.212508338 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.4775804Z 2025-12-04T11:13:50.4776090Z [W1204 11:07:48.213075621 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.4776459Z 2025-12-04T11:13:50.4776748Z [W1204 11:07:48.213261005 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.4777116Z 2025-12-04T11:13:50.4777423Z [W1204 11:07:48.218733826 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.4777952Z 2025-12-04T11:13:50.4778249Z [W1204 11:07:48.219263408 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.4778621Z 2025-12-04T11:13:50.4778912Z [W1204 11:07:48.219453332 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.4779283Z 2025-12-04T11:13:50.4779368Z ('RERUN', {'yellow': True}) [11.0482s] [100%] 2025-12-04T11:13:50.4780261Z inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_32_mid_dim_8_in_features_1024_out_features_64_cuda_bfloat16 [W1204 11:07:49.406839959 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.4781059Z 2025-12-04T11:13:50.4781363Z [W1204 11:07:49.407388091 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.4781731Z 2025-12-04T11:13:50.4782030Z [W1204 11:07:49.407530064 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.4782396Z 2025-12-04T11:13:50.4782686Z [W1204 11:07:49.410486720 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.4783059Z 2025-12-04T11:13:50.4783347Z [W1204 11:07:49.411059582 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.4783812Z 2025-12-04T11:13:50.4784104Z [W1204 11:07:49.411199685 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.4784473Z 2025-12-04T11:13:50.4784772Z [W1204 11:07:49.415763615 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.4785142Z 2025-12-04T11:13:50.4785433Z [W1204 11:07:49.416242196 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.4785883Z 2025-12-04T11:13:50.4786244Z [W1204 11:07:49.416377439 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.4786618Z 2025-12-04T11:13:50.4786699Z ('RERUN', {'yellow': True}) [0.4209s] [100%] 2025-12-04T11:13:50.4787585Z inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_32_mid_dim_8_in_features_1024_out_features_64_cuda_bfloat16 [W1204 11:07:49.824098554 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.4788389Z 2025-12-04T11:13:50.4788740Z [W1204 11:07:49.824656816 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.4789112Z 2025-12-04T11:13:50.4789406Z [W1204 11:07:49.824801220 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.4789776Z 2025-12-04T11:13:50.4790082Z [W1204 11:07:49.827775585 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.4790506Z 2025-12-04T11:13:50.4790794Z [W1204 11:07:49.828356568 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.4791166Z 2025-12-04T11:13:50.4791455Z [W1204 11:07:49.828494181 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.4791828Z 2025-12-04T11:13:50.4792120Z [W1204 11:07:49.833121224 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.4792485Z 2025-12-04T11:13:50.4792777Z [W1204 11:07:49.833599944 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.4793145Z 2025-12-04T11:13:50.4793438Z [W1204 11:07:49.833736727 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.4793806Z 2025-12-04T11:13:50.4793869Z FAILED [0.4169s] [100%] 2025-12-04T11:13:50.4793979Z 2025-12-04T11:13:50.4794069Z ==================================== RERUNS ==================================== 2025-12-04T11:13:50.4794537Z _ TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_8_in_features_1024_out_features_64_cuda_bfloat16 _ 2025-12-04T11:13:50.4794987Z Traceback (most recent call last): 2025-12-04T11:13:50.4795438Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.4795886Z method(*args, **kwargs) 2025-12-04T11:13:50.4796298Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.4796733Z method(*args, **kwargs) 2025-12-04T11:13:50.4797154Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T11:13:50.4797584Z with policy(): 2025-12-04T11:13:50.4797981Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T11:13:50.4798420Z raise RuntimeError(msg) 2025-12-04T11:13:50.4799349Z RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_8_in_features_1024_out_features_64_cuda_bfloat16! Caching allocator allocated memory was 0 and is now reported as 65536 on device 0. CUDA driver allocated memory was 230686720 and is now 274726912. 2025-12-04T11:13:50.4800235Z 2025-12-04T11:13:50.4800366Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.4801184Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_8_in_features_1024_out_features_64_cuda_bfloat16 2025-12-04T11:13:50.4801783Z 2025-12-04T11:13:50.4801947Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.4802319Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.4802639Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.4803166Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.4803757Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.4804029Z graph_break [] 2025-12-04T11:13:50.4804249Z ----------------------------- Captured stderr call ----------------------------- 2025-12-04T11:13:50.4805153Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/constant_folding.py:256: UserWarning: Unsupported unwinding pattern: Address not in range (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/profiler/unwind/unwind.cpp:219.) 2025-12-04T11:13:50.4806019Z if out == self.unknown_value: 2025-12-04T11:13:50.4806465Z _ TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_8_in_features_1024_out_features_64_cuda_bfloat16 _ 2025-12-04T11:13:50.4806917Z Traceback (most recent call last): 2025-12-04T11:13:50.4807360Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.4807807Z method(*args, **kwargs) 2025-12-04T11:13:50.4808221Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.4808658Z method(*args, **kwargs) 2025-12-04T11:13:50.4809059Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T11:13:50.4809492Z with policy(): 2025-12-04T11:13:50.4809891Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T11:13:50.4810335Z raise RuntimeError(msg) 2025-12-04T11:13:50.4811270Z RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_8_in_features_1024_out_features_64_cuda_bfloat16! Caching allocator allocated memory was 65536 and is now reported as 131072 on device 0. CUDA driver allocated memory was 274726912 and is now 276824064. 2025-12-04T11:13:50.4812158Z 2025-12-04T11:13:50.4812286Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.4813013Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_8_in_features_1024_out_features_64_cuda_bfloat16 2025-12-04T11:13:50.4813610Z 2025-12-04T11:13:50.4813775Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.4814132Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.4814443Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.4814964Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.4815513Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.4815773Z graph_break [] 2025-12-04T11:13:50.4815991Z ----------------------------- Captured stderr call ----------------------------- 2025-12-04T11:13:50.4816891Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/constant_folding.py:256: UserWarning: Unsupported unwinding pattern: Address not in range (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/profiler/unwind/unwind.cpp:219.) 2025-12-04T11:13:50.4817770Z if out == self.unknown_value: 2025-12-04T11:13:50.4818067Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.4818373Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.4818669Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.4819217Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.4819703Z graph_break [] 2025-12-04T11:13:50.4819881Z =================================== FAILURES =================================== 2025-12-04T11:13:50.4820384Z _ TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_8_in_features_1024_out_features_64_cuda_bfloat16 _ 2025-12-04T11:13:50.4820830Z Traceback (most recent call last): 2025-12-04T11:13:50.4821271Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.4821709Z method(*args, **kwargs) 2025-12-04T11:13:50.4822122Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.4822687Z method(*args, **kwargs) 2025-12-04T11:13:50.4823097Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T11:13:50.4823584Z with policy(): 2025-12-04T11:13:50.4823986Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T11:13:50.4824439Z raise RuntimeError(msg) 2025-12-04T11:13:50.4825385Z RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_8_in_features_1024_out_features_64_cuda_bfloat16! Caching allocator allocated memory was 131072 and is now reported as 196608 on device 0. CUDA driver allocated memory was 276824064 and is now 278921216. 2025-12-04T11:13:50.4826276Z 2025-12-04T11:13:50.4826405Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.4827133Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_8_in_features_1024_out_features_64_cuda_bfloat16 2025-12-04T11:13:50.4827731Z 2025-12-04T11:13:50.4827891Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.4828344Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.4828653Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.4829164Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.4829729Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.4829998Z graph_break [] 2025-12-04T11:13:50.4830219Z ----------------------------- Captured stderr call ----------------------------- 2025-12-04T11:13:50.4831114Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/constant_folding.py:256: UserWarning: Unsupported unwinding pattern: Address not in range (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/profiler/unwind/unwind.cpp:219.) 2025-12-04T11:13:50.4831946Z if out == self.unknown_value: 2025-12-04T11:13:50.4832199Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.4832501Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.4832791Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.4833347Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.4833890Z graph_break [] 2025-12-04T11:13:50.4834144Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.4834439Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.4834726Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.4835269Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.4835743Z graph_break [] 2025-12-04T11:13:50.4836360Z - generated xml file: /var/lib/jenkins/workspace/test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-90e4a91c9bbb810f.xml - 2025-12-04T11:13:50.4837024Z =========================== short test summary info ============================ 2025-12-04T11:13:50.4838511Z FAILED [0.4169s] inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_32_mid_dim_8_in_features_1024_out_features_64_cuda_bfloat16 - RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_8_in_features_1024_out_features_64_cuda_bfloat16! Caching allocator allocated memory was 131072 and is now reported as 196608 on device 0. CUDA driver allocated memory was 276824064 and is now 278921216. 2025-12-04T11:13:50.4839914Z 2025-12-04T11:13:50.4840045Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.4840767Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_8_in_features_1024_out_features_64_cuda_bfloat16 2025-12-04T11:13:50.4841368Z 2025-12-04T11:13:50.4841526Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.4841870Z !!!!!!!!!!!!!!!!!!!!!!!!!! stopping after 1 failures !!!!!!!!!!!!!!!!!!!!!!!!!!! 2025-12-04T11:13:50.4842174Z ================== 1 failed, 57 deselected, 2 rerun in 11.91s ================== 2025-12-04T11:13:50.4842427Z Got exit code 1 2025-12-04T11:13:50.4842591Z Retrying single test... 2025-12-04T11:13:50.4842968Z W1204 11:07:56.630000 85134 site-packages/torch/_inductor/utils.py:1703] Not enough SMs to use max_autotune_gemm mode 2025-12-04T11:13:50.4843696Z Test results will be stored in test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-2ee8d3487d6282f0.xml 2025-12-04T11:13:50.4844262Z ============================= test session starts ============================== 2025-12-04T11:13:50.4844653Z platform linux -- Python 3.10.14, pytest-7.3.2, pluggy-1.6.0 -- /opt/conda/envs/py_3.10/bin/python 2025-12-04T11:13:50.4845013Z cachedir: .pytest_cache 2025-12-04T11:13:50.4845427Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow] 2025-12-04T11:13:50.4845887Z rootdir: /var/lib/jenkins/workspace 2025-12-04T11:13:50.4846098Z configfile: pytest.ini 2025-12-04T11:13:50.4846526Z plugins: hypothesis-6.56.4, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-14.0, subtests-0.13.1, xdist-3.3.1, xdoctest-1.3.0, typeguard-4.3.0 2025-12-04T11:13:50.4847044Z collecting ... collected 58 items / 57 deselected / 1 selected 2025-12-04T11:13:50.4847829Z stepcurrent: skipping 50 already run items. Running only test/inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_32_mid_dim_8_in_features_1024_out_features_64_cuda_bfloat16 2025-12-04T11:13:50.4848536Z Running 1 items in this shard 2025-12-04T11:13:50.4848663Z 2025-12-04T11:13:50.4849407Z inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_32_mid_dim_8_in_features_1024_out_features_64_cuda_bfloat16 [W1204 11:07:57.735346439 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.4850289Z 2025-12-04T11:13:50.4850637Z [W1204 11:08:06.770407785 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.4851016Z 2025-12-04T11:13:50.4851310Z [W1204 11:08:06.770655310 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.4851686Z 2025-12-04T11:13:50.4851978Z [W1204 11:08:06.776398437 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.4852352Z 2025-12-04T11:13:50.4852683Z [W1204 11:08:06.776960760 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.4853058Z 2025-12-04T11:13:50.4853350Z [W1204 11:08:06.777133974 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.4853719Z 2025-12-04T11:13:50.4854018Z [W1204 11:08:06.782571194 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.4854420Z 2025-12-04T11:13:50.4854711Z [W1204 11:08:06.783102295 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.4855084Z 2025-12-04T11:13:50.4855370Z [W1204 11:08:06.783292540 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.4855746Z 2025-12-04T11:13:50.4855831Z ('RERUN', {'yellow': True}) [10.9340s] [100%] 2025-12-04T11:13:50.4856714Z inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_32_mid_dim_8_in_features_1024_out_features_64_cuda_bfloat16 [W1204 11:08:08.964623579 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.4857514Z 2025-12-04T11:13:50.4857812Z [W1204 11:08:08.965157091 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.4858181Z 2025-12-04T11:13:50.4858473Z [W1204 11:08:08.965294044 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.4858848Z 2025-12-04T11:13:50.4859139Z [W1204 11:08:08.968273370 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.4859518Z 2025-12-04T11:13:50.4859811Z [W1204 11:08:08.968859923 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.4860178Z 2025-12-04T11:13:50.4860479Z [W1204 11:08:08.968994846 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.4860854Z 2025-12-04T11:13:50.4861153Z [W1204 11:08:08.973650689 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.4861521Z 2025-12-04T11:13:50.4861813Z [W1204 11:08:08.974125490 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.4862186Z 2025-12-04T11:13:50.4862476Z [W1204 11:08:08.974260233 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.4862849Z 2025-12-04T11:13:50.4862930Z ('RERUN', {'yellow': True}) [0.4192s] [100%] 2025-12-04T11:13:50.4863885Z inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_32_mid_dim_8_in_features_1024_out_features_64_cuda_bfloat16 [W1204 11:08:08.379772138 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.4864732Z 2025-12-04T11:13:50.4865064Z [W1204 11:08:08.380332130 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.4865439Z 2025-12-04T11:13:50.4865731Z [W1204 11:08:08.380475353 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.4866097Z 2025-12-04T11:13:50.4866387Z [W1204 11:08:08.383498160 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.4866758Z 2025-12-04T11:13:50.4867047Z [W1204 11:08:08.384062753 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.4867461Z 2025-12-04T11:13:50.4867752Z [W1204 11:08:08.384199336 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.4868119Z 2025-12-04T11:13:50.4868413Z [W1204 11:08:08.388813268 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.4868782Z 2025-12-04T11:13:50.4869114Z [W1204 11:08:08.389278928 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.4869491Z 2025-12-04T11:13:50.4869791Z [W1204 11:08:08.389419241 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.4870170Z 2025-12-04T11:13:50.4870232Z FAILED [0.4122s] [100%] 2025-12-04T11:13:50.4870342Z 2025-12-04T11:13:50.4870430Z ==================================== RERUNS ==================================== 2025-12-04T11:13:50.4870900Z _ TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_8_in_features_1024_out_features_64_cuda_bfloat16 _ 2025-12-04T11:13:50.4871348Z Traceback (most recent call last): 2025-12-04T11:13:50.4871803Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.4872249Z method(*args, **kwargs) 2025-12-04T11:13:50.4872662Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.4873097Z method(*args, **kwargs) 2025-12-04T11:13:50.4873508Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T11:13:50.4873572Z with policy(): 2025-12-04T11:13:50.4873876Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T11:13:50.4873946Z raise RuntimeError(msg) 2025-12-04T11:13:50.4874777Z RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_8_in_features_1024_out_features_64_cuda_bfloat16! Caching allocator allocated memory was 0 and is now reported as 65536 on device 0. CUDA driver allocated memory was 230686720 and is now 274726912. 2025-12-04T11:13:50.4874783Z 2025-12-04T11:13:50.4874921Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.4875448Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_8_in_features_1024_out_features_64_cuda_bfloat16 2025-12-04T11:13:50.4875458Z 2025-12-04T11:13:50.4875619Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.4875752Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.4875854Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.4876204Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.4876382Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.4876485Z graph_break [] 2025-12-04T11:13:50.4876615Z ----------------------------- Captured stderr call ----------------------------- 2025-12-04T11:13:50.4877317Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/constant_folding.py:256: UserWarning: Unsupported unwinding pattern: Address not in range (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/profiler/unwind/unwind.cpp:219.) 2025-12-04T11:13:50.4877390Z if out == self.unknown_value: 2025-12-04T11:13:50.4877682Z _ TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_8_in_features_1024_out_features_64_cuda_bfloat16 _ 2025-12-04T11:13:50.4877986Z Traceback (most recent call last): 2025-12-04T11:13:50.4878517Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.4878642Z method(*args, **kwargs) 2025-12-04T11:13:50.4879157Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.4879228Z method(*args, **kwargs) 2025-12-04T11:13:50.4879635Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T11:13:50.4879704Z with policy(): 2025-12-04T11:13:50.4880012Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T11:13:50.4880089Z raise RuntimeError(msg) 2025-12-04T11:13:50.4880917Z RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_8_in_features_1024_out_features_64_cuda_bfloat16! Caching allocator allocated memory was 65536 and is now reported as 131072 on device 0. CUDA driver allocated memory was 274726912 and is now 276824064. 2025-12-04T11:13:50.4880923Z 2025-12-04T11:13:50.4881058Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.4889426Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_8_in_features_1024_out_features_64_cuda_bfloat16 2025-12-04T11:13:50.4889440Z 2025-12-04T11:13:50.4889673Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.4889822Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.4889927Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.4890297Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.4890430Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.4890504Z graph_break [] 2025-12-04T11:13:50.4890641Z ----------------------------- Captured stderr call ----------------------------- 2025-12-04T11:13:50.4891374Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/constant_folding.py:256: UserWarning: Unsupported unwinding pattern: Address not in range (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/profiler/unwind/unwind.cpp:219.) 2025-12-04T11:13:50.4891456Z if out == self.unknown_value: 2025-12-04T11:13:50.4891588Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.4891689Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.4891830Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.4892182Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.4892248Z graph_break [] 2025-12-04T11:13:50.4892459Z =================================== FAILURES =================================== 2025-12-04T11:13:50.4892830Z _ TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_8_in_features_1024_out_features_64_cuda_bfloat16 _ 2025-12-04T11:13:50.4892921Z Traceback (most recent call last): 2025-12-04T11:13:50.4893241Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.4893322Z method(*args, **kwargs) 2025-12-04T11:13:50.4893639Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.4893705Z method(*args, **kwargs) 2025-12-04T11:13:50.4894074Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T11:13:50.4894138Z with policy(): 2025-12-04T11:13:50.4894443Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T11:13:50.4894516Z raise RuntimeError(msg) 2025-12-04T11:13:50.4895347Z RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_8_in_features_1024_out_features_64_cuda_bfloat16! Caching allocator allocated memory was 131072 and is now reported as 196608 on device 0. CUDA driver allocated memory was 276824064 and is now 278921216. 2025-12-04T11:13:50.4895391Z 2025-12-04T11:13:50.4895532Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.4896058Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_8_in_features_1024_out_features_64_cuda_bfloat16 2025-12-04T11:13:50.4896063Z 2025-12-04T11:13:50.4896230Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.4896362Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.4896460Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.4896819Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.4896951Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.4897012Z graph_break [] 2025-12-04T11:13:50.4897143Z ----------------------------- Captured stderr call ----------------------------- 2025-12-04T11:13:50.4897838Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/constant_folding.py:256: UserWarning: Unsupported unwinding pattern: Address not in range (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/profiler/unwind/unwind.cpp:219.) 2025-12-04T11:13:50.4897915Z if out == self.unknown_value: 2025-12-04T11:13:50.4898041Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.4898139Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.4898279Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.4898627Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.4898693Z graph_break [] 2025-12-04T11:13:50.4898818Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.4898909Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.4899049Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.4899400Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.4899459Z graph_break [] 2025-12-04T11:13:50.4900125Z - generated xml file: /var/lib/jenkins/workspace/test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-2ee8d3487d6282f0.xml - 2025-12-04T11:13:50.4900232Z =========================== short test summary info ============================ 2025-12-04T11:13:50.4901541Z FAILED [0.4122s] inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_32_mid_dim_8_in_features_1024_out_features_64_cuda_bfloat16 - RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_8_in_features_1024_out_features_64_cuda_bfloat16! Caching allocator allocated memory was 131072 and is now reported as 196608 on device 0. CUDA driver allocated memory was 276824064 and is now 278921216. 2025-12-04T11:13:50.4901580Z 2025-12-04T11:13:50.4901712Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.4902241Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_8_in_features_1024_out_features_64_cuda_bfloat16 2025-12-04T11:13:50.4902248Z 2025-12-04T11:13:50.4902406Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.4902551Z !!!!!!!!!!!!!!!!!!!!!!!!!! stopping after 1 failures !!!!!!!!!!!!!!!!!!!!!!!!!!! 2025-12-04T11:13:50.4902669Z ================== 1 failed, 57 deselected, 2 rerun in 11.79s ================== 2025-12-04T11:13:50.4902730Z Got exit code 1 2025-12-04T11:13:50.4903209Z FAILED CONSISTENTLY: test/inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_32_mid_dim_8_in_features_1024_out_features_64_cuda_bfloat16 2025-12-04T11:13:50.4903457Z Test failed consistently, continuing with the rest of the tests due to continue-through-error being set 2025-12-04T11:13:50.4903834Z W1204 11:08:15.178000 85327 site-packages/torch/_inductor/utils.py:1703] Not enough SMs to use max_autotune_gemm mode 2025-12-04T11:13:50.4904231Z Test results will be stored in test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-2ab4541f9392b10c.xml 2025-12-04T11:13:50.4904330Z ============================= test session starts ============================== 2025-12-04T11:13:50.4904546Z platform linux -- Python 3.10.14, pytest-7.3.2, pluggy-1.6.0 -- /opt/conda/envs/py_3.10/bin/python 2025-12-04T11:13:50.4904615Z cachedir: .pytest_cache 2025-12-04T11:13:50.4904923Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow] 2025-12-04T11:13:50.4905006Z rootdir: /var/lib/jenkins/workspace 2025-12-04T11:13:50.4905074Z configfile: pytest.ini 2025-12-04T11:13:50.4905394Z plugins: hypothesis-6.56.4, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-14.0, subtests-0.13.1, xdist-3.3.1, xdoctest-1.3.0, typeguard-4.3.0 2025-12-04T11:13:50.4905526Z collecting ... collected 58 items / 51 deselected / 7 selected 2025-12-04T11:13:50.4905615Z stepcurrent: skipping 51 already run items. 2025-12-04T11:13:50.4905693Z Running 7 items in this shard 2025-12-04T11:13:50.4905699Z 2025-12-04T11:13:50.4906204Z inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_32_mid_dim_8_in_features_1024_out_features_65_cuda_bfloat16 ('RERUN', {'yellow': True}) [1.9879s] [ 14%] 2025-12-04T11:13:50.4906696Z inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_32_mid_dim_8_in_features_1024_out_features_65_cuda_bfloat16 ('RERUN', {'yellow': True}) [0.5552s] [ 14%] 2025-12-04T11:13:50.4907139Z inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_32_mid_dim_8_in_features_1024_out_features_65_cuda_bfloat16 FAILED [0.5492s] [ 14%] 2025-12-04T11:13:50.4907143Z 2025-12-04T11:13:50.4907226Z ==================================== RERUNS ==================================== 2025-12-04T11:13:50.4907576Z _ TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_8_in_features_1024_out_features_65_cuda_bfloat16 _ 2025-12-04T11:13:50.4907687Z Traceback (most recent call last): 2025-12-04T11:13:50.4908014Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.4908082Z method(*args, **kwargs) 2025-12-04T11:13:50.4908375Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.4908444Z method(*args, **kwargs) 2025-12-04T11:13:50.4908734Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T11:13:50.4908794Z with policy(): 2025-12-04T11:13:50.4909130Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T11:13:50.4909197Z raise RuntimeError(msg) 2025-12-04T11:13:50.4910018Z RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_8_in_features_1024_out_features_65_cuda_bfloat16! Caching allocator allocated memory was 0 and is now reported as 66560 on device 0. CUDA driver allocated memory was 230686720 and is now 274726912. 2025-12-04T11:13:50.4910056Z 2025-12-04T11:13:50.4910187Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.4910710Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_8_in_features_1024_out_features_65_cuda_bfloat16 2025-12-04T11:13:50.4910714Z 2025-12-04T11:13:50.4910875Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.4911007Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.4911107Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.4911656Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('benchmarking.InductorBenchmarker.benchmark_gpu', 2), ('pad_mm_bench', 1), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.4911791Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.4911851Z graph_break [] 2025-12-04T11:13:50.4912140Z _ TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_8_in_features_1024_out_features_65_cuda_bfloat16 _ 2025-12-04T11:13:50.4912220Z Traceback (most recent call last): 2025-12-04T11:13:50.4912517Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.4912581Z method(*args, **kwargs) 2025-12-04T11:13:50.4912874Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.4912939Z method(*args, **kwargs) 2025-12-04T11:13:50.4913234Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T11:13:50.4913297Z with policy(): 2025-12-04T11:13:50.4913695Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T11:13:50.4913815Z raise RuntimeError(msg) 2025-12-04T11:13:50.4914699Z RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_8_in_features_1024_out_features_65_cuda_bfloat16! Caching allocator allocated memory was 66560 and is now reported as 133120 on device 0. CUDA driver allocated memory was 274726912 and is now 276824064. 2025-12-04T11:13:50.4914704Z 2025-12-04T11:13:50.4914837Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.4915409Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_8_in_features_1024_out_features_65_cuda_bfloat16 2025-12-04T11:13:50.4915449Z 2025-12-04T11:13:50.4915619Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.4915750Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.4915845Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.4916395Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('benchmarking.InductorBenchmarker.benchmark_gpu', 2), ('pad_mm_bench', 1), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.4916580Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.4916642Z graph_break [] 2025-12-04T11:13:50.4916770Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.4916862Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.4916989Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.4917526Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('benchmarking.InductorBenchmarker.benchmark_gpu', 2), ('pad_mm_bench', 1), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.4917621Z graph_break [] 2025-12-04T11:13:50.4917713Z =================================== FAILURES =================================== 2025-12-04T11:13:50.4918003Z _ TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_8_in_features_1024_out_features_65_cuda_bfloat16 _ 2025-12-04T11:13:50.4918083Z Traceback (most recent call last): 2025-12-04T11:13:50.4918388Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.4918456Z method(*args, **kwargs) 2025-12-04T11:13:50.4918755Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.4918818Z method(*args, **kwargs) 2025-12-04T11:13:50.4919111Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T11:13:50.4919177Z with policy(): 2025-12-04T11:13:50.4919480Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T11:13:50.4919550Z raise RuntimeError(msg) 2025-12-04T11:13:50.4920362Z RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_8_in_features_1024_out_features_65_cuda_bfloat16! Caching allocator allocated memory was 133120 and is now reported as 199680 on device 0. CUDA driver allocated memory was 276824064 and is now 278921216. 2025-12-04T11:13:50.4920368Z 2025-12-04T11:13:50.4920496Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.4921025Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_8_in_features_1024_out_features_65_cuda_bfloat16 2025-12-04T11:13:50.4921030Z 2025-12-04T11:13:50.4921189Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.4921320Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.4921416Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.4921964Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('benchmarking.InductorBenchmarker.benchmark_gpu', 2), ('pad_mm_bench', 1), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.4922090Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.4922192Z graph_break [] 2025-12-04T11:13:50.4922366Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.4922460Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.4922586Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.4923128Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('benchmarking.InductorBenchmarker.benchmark_gpu', 2), ('pad_mm_bench', 1), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.4923186Z graph_break [] 2025-12-04T11:13:50.4923314Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.4923436Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.4923559Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.4924094Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('benchmarking.InductorBenchmarker.benchmark_gpu', 2), ('pad_mm_bench', 1), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.4924187Z graph_break [] 2025-12-04T11:13:50.4924676Z - generated xml file: /var/lib/jenkins/workspace/test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-2ab4541f9392b10c.xml - 2025-12-04T11:13:50.4924777Z =========================== short test summary info ============================ 2025-12-04T11:13:50.4926074Z FAILED [0.5492s] inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_32_mid_dim_8_in_features_1024_out_features_65_cuda_bfloat16 - RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_8_in_features_1024_out_features_65_cuda_bfloat16! Caching allocator allocated memory was 133120 and is now reported as 199680 on device 0. CUDA driver allocated memory was 276824064 and is now 278921216. 2025-12-04T11:13:50.4926085Z 2025-12-04T11:13:50.4926213Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.4926732Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_8_in_features_1024_out_features_65_cuda_bfloat16 2025-12-04T11:13:50.4926736Z 2025-12-04T11:13:50.4926896Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.4927000Z !!!!!!!!!!!!!!!!!!!!!!!!!! stopping after 1 failures !!!!!!!!!!!!!!!!!!!!!!!!!!! 2025-12-04T11:13:50.4927120Z ================== 1 failed, 51 deselected, 2 rerun in 3.12s =================== 2025-12-04T11:13:50.4927183Z Got exit code 1 2025-12-04T11:13:50.4927250Z Retrying single test... 2025-12-04T11:13:50.4927525Z W1204 11:08:25.218000 85516 site-packages/torch/_inductor/utils.py:1703] Not enough SMs to use max_autotune_gemm mode 2025-12-04T11:13:50.4927924Z Test results will be stored in test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-ce43d3c4fee34ad0.xml 2025-12-04T11:13:50.4928026Z ============================= test session starts ============================== 2025-12-04T11:13:50.4928239Z platform linux -- Python 3.10.14, pytest-7.3.2, pluggy-1.6.0 -- /opt/conda/envs/py_3.10/bin/python 2025-12-04T11:13:50.4928307Z cachedir: .pytest_cache 2025-12-04T11:13:50.4928616Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow] 2025-12-04T11:13:50.4928693Z rootdir: /var/lib/jenkins/workspace 2025-12-04T11:13:50.4928761Z configfile: pytest.ini 2025-12-04T11:13:50.4929078Z plugins: hypothesis-6.56.4, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-14.0, subtests-0.13.1, xdist-3.3.1, xdoctest-1.3.0, typeguard-4.3.0 2025-12-04T11:13:50.4929207Z collecting ... collected 58 items / 57 deselected / 1 selected 2025-12-04T11:13:50.4929861Z stepcurrent: skipping 51 already run items. Running only test/inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_32_mid_dim_8_in_features_1024_out_features_65_cuda_bfloat16 2025-12-04T11:13:50.4929939Z Running 1 items in this shard 2025-12-04T11:13:50.4929942Z 2025-12-04T11:13:50.4930670Z inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_32_mid_dim_8_in_features_1024_out_features_65_cuda_bfloat16 [W1204 11:08:26.864160331 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.4930679Z 2025-12-04T11:13:50.4931017Z [W1204 11:08:35.725364415 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.4931022Z 2025-12-04T11:13:50.4931311Z [W1204 11:08:35.725597040 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.4931316Z 2025-12-04T11:13:50.4931608Z [W1204 11:08:35.731381258 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.4931646Z 2025-12-04T11:13:50.4931932Z [W1204 11:08:35.731938690 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.4931935Z 2025-12-04T11:13:50.4932226Z [W1204 11:08:35.732125594 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.4932229Z 2025-12-04T11:13:50.4932517Z [W1204 11:08:35.737460302 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.4932521Z 2025-12-04T11:13:50.4932825Z [W1204 11:08:35.737987004 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.4932830Z 2025-12-04T11:13:50.4933119Z [W1204 11:08:35.738153748 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.4933124Z 2025-12-04T11:13:50.4933208Z ('RERUN', {'yellow': True}) [10.8480s] [100%] 2025-12-04T11:13:50.4933932Z inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_32_mid_dim_8_in_features_1024_out_features_65_cuda_bfloat16 [W1204 11:08:36.553039019 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.4933936Z 2025-12-04T11:13:50.4934274Z [W1204 11:08:36.553571111 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.4934282Z 2025-12-04T11:13:50.4934571Z [W1204 11:08:36.553712714 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.4934576Z 2025-12-04T11:13:50.4934864Z [W1204 11:08:36.556657419 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.4934869Z 2025-12-04T11:13:50.4935160Z [W1204 11:08:36.557125900 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.4935163Z 2025-12-04T11:13:50.4935448Z [W1204 11:08:36.557264123 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.4935452Z 2025-12-04T11:13:50.4935754Z [W1204 11:08:36.562001048 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.4935758Z 2025-12-04T11:13:50.4936045Z [W1204 11:08:36.562488528 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.4936089Z 2025-12-04T11:13:50.4936415Z [W1204 11:08:36.562623901 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.4936420Z 2025-12-04T11:13:50.4936503Z ('RERUN', {'yellow': True}) [0.5113s] [100%] 2025-12-04T11:13:50.4937224Z inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_32_mid_dim_8_in_features_1024_out_features_65_cuda_bfloat16 [W1204 11:08:37.060489929 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.4937230Z 2025-12-04T11:13:50.4937552Z [W1204 11:08:37.061027421 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.4937556Z 2025-12-04T11:13:50.4937843Z [W1204 11:08:37.061169614 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.4937848Z 2025-12-04T11:13:50.4938138Z [W1204 11:08:37.064134269 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.4938174Z 2025-12-04T11:13:50.4938461Z [W1204 11:08:37.064614360 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.4938464Z 2025-12-04T11:13:50.4938768Z [W1204 11:08:37.064753183 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.4938772Z 2025-12-04T11:13:50.4939062Z [W1204 11:08:37.069418697 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.4939067Z 2025-12-04T11:13:50.4939367Z [W1204 11:08:37.069896427 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.4939372Z 2025-12-04T11:13:50.4939658Z [W1204 11:08:37.070085851 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.4939661Z 2025-12-04T11:13:50.4939729Z FAILED [0.5052s] [100%] 2025-12-04T11:13:50.4939732Z 2025-12-04T11:13:50.4939816Z ==================================== RERUNS ==================================== 2025-12-04T11:13:50.4940112Z _ TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_8_in_features_1024_out_features_65_cuda_bfloat16 _ 2025-12-04T11:13:50.4940192Z Traceback (most recent call last): 2025-12-04T11:13:50.4940497Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.4940566Z method(*args, **kwargs) 2025-12-04T11:13:50.4940862Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.4940929Z method(*args, **kwargs) 2025-12-04T11:13:50.4941222Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T11:13:50.4941282Z with policy(): 2025-12-04T11:13:50.4941578Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T11:13:50.4941650Z raise RuntimeError(msg) 2025-12-04T11:13:50.4942453Z RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_8_in_features_1024_out_features_65_cuda_bfloat16! Caching allocator allocated memory was 0 and is now reported as 66560 on device 0. CUDA driver allocated memory was 230686720 and is now 274726912. 2025-12-04T11:13:50.4942458Z 2025-12-04T11:13:50.4942591Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.4943115Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_8_in_features_1024_out_features_65_cuda_bfloat16 2025-12-04T11:13:50.4943209Z 2025-12-04T11:13:50.4943381Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.4943574Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.4943672Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.4944222Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('benchmarking.InductorBenchmarker.benchmark_gpu', 2), ('pad_mm_bench', 1), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.4944390Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.4944452Z graph_break [] 2025-12-04T11:13:50.4944585Z ----------------------------- Captured stderr call ----------------------------- 2025-12-04T11:13:50.4945276Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/constant_folding.py:256: UserWarning: Unsupported unwinding pattern: Address not in range (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/profiler/unwind/unwind.cpp:219.) 2025-12-04T11:13:50.4945386Z if out == self.unknown_value: 2025-12-04T11:13:50.4945679Z _ TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_8_in_features_1024_out_features_65_cuda_bfloat16 _ 2025-12-04T11:13:50.4945754Z Traceback (most recent call last): 2025-12-04T11:13:50.4946064Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.4946134Z method(*args, **kwargs) 2025-12-04T11:13:50.4946436Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.4946500Z method(*args, **kwargs) 2025-12-04T11:13:50.4946791Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T11:13:50.4946861Z with policy(): 2025-12-04T11:13:50.4947158Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T11:13:50.4947228Z raise RuntimeError(msg) 2025-12-04T11:13:50.4948045Z RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_8_in_features_1024_out_features_65_cuda_bfloat16! Caching allocator allocated memory was 66560 and is now reported as 133120 on device 0. CUDA driver allocated memory was 274726912 and is now 276824064. 2025-12-04T11:13:50.4948050Z 2025-12-04T11:13:50.4948184Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.4948709Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_8_in_features_1024_out_features_65_cuda_bfloat16 2025-12-04T11:13:50.4948714Z 2025-12-04T11:13:50.4948878Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.4949011Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.4949110Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.4949662Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('benchmarking.InductorBenchmarker.benchmark_gpu', 2), ('pad_mm_bench', 1), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.4949799Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.4949858Z graph_break [] 2025-12-04T11:13:50.4949990Z ----------------------------- Captured stderr call ----------------------------- 2025-12-04T11:13:50.4950717Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/constant_folding.py:256: UserWarning: Unsupported unwinding pattern: Address not in range (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/profiler/unwind/unwind.cpp:219.) 2025-12-04T11:13:50.4950826Z if out == self.unknown_value: 2025-12-04T11:13:50.4950957Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.4951049Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.4951177Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.4951717Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('benchmarking.InductorBenchmarker.benchmark_gpu', 2), ('pad_mm_bench', 1), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.4951778Z graph_break [] 2025-12-04T11:13:50.4951902Z =================================== FAILURES =================================== 2025-12-04T11:13:50.4952197Z _ TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_8_in_features_1024_out_features_65_cuda_bfloat16 _ 2025-12-04T11:13:50.4952274Z Traceback (most recent call last): 2025-12-04T11:13:50.4952576Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.4952681Z method(*args, **kwargs) 2025-12-04T11:13:50.4952974Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.4953039Z method(*args, **kwargs) 2025-12-04T11:13:50.4953327Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T11:13:50.4953404Z with policy(): 2025-12-04T11:13:50.4953702Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T11:13:50.4953776Z raise RuntimeError(msg) 2025-12-04T11:13:50.4954594Z RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_8_in_features_1024_out_features_65_cuda_bfloat16! Caching allocator allocated memory was 133120 and is now reported as 199680 on device 0. CUDA driver allocated memory was 276824064 and is now 278921216. 2025-12-04T11:13:50.4954601Z 2025-12-04T11:13:50.4954730Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.4955256Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_8_in_features_1024_out_features_65_cuda_bfloat16 2025-12-04T11:13:50.4955260Z 2025-12-04T11:13:50.4955416Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.4955557Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.4955648Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.4956186Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('benchmarking.InductorBenchmarker.benchmark_gpu', 2), ('pad_mm_bench', 1), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.4956316Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.4956375Z graph_break [] 2025-12-04T11:13:50.4956501Z ----------------------------- Captured stderr call ----------------------------- 2025-12-04T11:13:50.4957184Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/constant_folding.py:256: UserWarning: Unsupported unwinding pattern: Address not in range (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/profiler/unwind/unwind.cpp:219.) 2025-12-04T11:13:50.4957257Z if out == self.unknown_value: 2025-12-04T11:13:50.4957384Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.4957472Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.4957658Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.4958231Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('benchmarking.InductorBenchmarker.benchmark_gpu', 2), ('pad_mm_bench', 1), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.4958292Z graph_break [] 2025-12-04T11:13:50.4958418Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.4958506Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.4958633Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.4959201Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('benchmarking.InductorBenchmarker.benchmark_gpu', 2), ('pad_mm_bench', 1), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.4959262Z graph_break [] 2025-12-04T11:13:50.4959758Z - generated xml file: /var/lib/jenkins/workspace/test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-ce43d3c4fee34ad0.xml - 2025-12-04T11:13:50.4959892Z =========================== short test summary info ============================ 2025-12-04T11:13:50.4961194Z FAILED [0.5052s] inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_32_mid_dim_8_in_features_1024_out_features_65_cuda_bfloat16 - RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_8_in_features_1024_out_features_65_cuda_bfloat16! Caching allocator allocated memory was 133120 and is now reported as 199680 on device 0. CUDA driver allocated memory was 276824064 and is now 278921216. 2025-12-04T11:13:50.4961199Z 2025-12-04T11:13:50.4961324Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.4961849Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_8_in_features_1024_out_features_65_cuda_bfloat16 2025-12-04T11:13:50.4961854Z 2025-12-04T11:13:50.4962011Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.4962118Z !!!!!!!!!!!!!!!!!!!!!!!!!! stopping after 1 failures !!!!!!!!!!!!!!!!!!!!!!!!!!! 2025-12-04T11:13:50.4962240Z ================== 1 failed, 57 deselected, 2 rerun in 11.89s ================== 2025-12-04T11:13:50.4962301Z Got exit code 1 2025-12-04T11:13:50.4962366Z Retrying single test... 2025-12-04T11:13:50.4962637Z W1204 11:08:44.064000 85710 site-packages/torch/_inductor/utils.py:1703] Not enough SMs to use max_autotune_gemm mode 2025-12-04T11:13:50.4963023Z Test results will be stored in test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-d04b19868e6e9971.xml 2025-12-04T11:13:50.4963127Z ============================= test session starts ============================== 2025-12-04T11:13:50.4963338Z platform linux -- Python 3.10.14, pytest-7.3.2, pluggy-1.6.0 -- /opt/conda/envs/py_3.10/bin/python 2025-12-04T11:13:50.4963406Z cachedir: .pytest_cache 2025-12-04T11:13:50.4963717Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow] 2025-12-04T11:13:50.4963793Z rootdir: /var/lib/jenkins/workspace 2025-12-04T11:13:50.4963863Z configfile: pytest.ini 2025-12-04T11:13:50.4964181Z plugins: hypothesis-6.56.4, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-14.0, subtests-0.13.1, xdist-3.3.1, xdoctest-1.3.0, typeguard-4.3.0 2025-12-04T11:13:50.4964312Z collecting ... collected 58 items / 57 deselected / 1 selected 2025-12-04T11:13:50.4964892Z stepcurrent: skipping 51 already run items. Running only test/inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_32_mid_dim_8_in_features_1024_out_features_65_cuda_bfloat16 2025-12-04T11:13:50.4965006Z Running 1 items in this shard 2025-12-04T11:13:50.4965009Z 2025-12-04T11:13:50.4965777Z inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_32_mid_dim_8_in_features_1024_out_features_65_cuda_bfloat16 [W1204 11:08:45.703571064 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.4965783Z 2025-12-04T11:13:50.4966087Z [W1204 11:08:54.628480053 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.4966090Z 2025-12-04T11:13:50.4966420Z [W1204 11:08:54.628727668 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.4966423Z 2025-12-04T11:13:50.4966715Z [W1204 11:08:54.634671559 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.4966720Z 2025-12-04T11:13:50.4967006Z [W1204 11:08:54.635241162 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.4967046Z 2025-12-04T11:13:50.4967335Z [W1204 11:08:54.635419596 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.4967338Z 2025-12-04T11:13:50.4967628Z [W1204 11:08:54.640769604 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.4967631Z 2025-12-04T11:13:50.4967928Z [W1204 11:08:54.641290166 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.4967933Z 2025-12-04T11:13:50.4968223Z [W1204 11:08:54.641450999 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.4968228Z 2025-12-04T11:13:50.4968310Z ('RERUN', {'yellow': True}) [10.9074s] [100%] 2025-12-04T11:13:50.4969041Z inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_32_mid_dim_8_in_features_1024_out_features_65_cuda_bfloat16 [W1204 11:08:55.453894439 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.4969046Z 2025-12-04T11:13:50.4969337Z [W1204 11:08:55.454409311 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.4969340Z 2025-12-04T11:13:50.4969632Z [W1204 11:08:55.454547984 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.4969637Z 2025-12-04T11:13:50.4969925Z [W1204 11:08:55.457480248 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.4969930Z 2025-12-04T11:13:50.4970229Z [W1204 11:08:55.457951128 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.4970234Z 2025-12-04T11:13:50.4970531Z [W1204 11:08:55.458087691 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.4970534Z 2025-12-04T11:13:50.4970820Z [W1204 11:08:55.462799106 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.4970827Z 2025-12-04T11:13:50.4971113Z [W1204 11:08:55.463271626 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.4971116Z 2025-12-04T11:13:50.4971406Z [W1204 11:08:55.463407619 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.4971466Z 2025-12-04T11:13:50.4971551Z ('RERUN', {'yellow': True}) [0.5064s] [100%] 2025-12-04T11:13:50.4972306Z inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_32_mid_dim_8_in_features_1024_out_features_65_cuda_bfloat16 [W1204 11:08:56.956941023 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.4972312Z 2025-12-04T11:13:50.4972604Z [W1204 11:08:56.957455564 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.4972608Z 2025-12-04T11:13:50.4972895Z [W1204 11:08:56.957595317 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.4972898Z 2025-12-04T11:13:50.4973219Z [W1204 11:08:56.960545222 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.4973224Z 2025-12-04T11:13:50.4973512Z [W1204 11:08:56.961020662 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.4973515Z 2025-12-04T11:13:50.4973803Z [W1204 11:08:56.961157546 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.4973844Z 2025-12-04T11:13:50.4974131Z [W1204 11:08:56.965768598 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.4974135Z 2025-12-04T11:13:50.4974421Z [W1204 11:08:56.966234548 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.4974424Z 2025-12-04T11:13:50.4974717Z [W1204 11:08:56.966369191 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.4974721Z 2025-12-04T11:13:50.4974783Z FAILED [0.5051s] [100%] 2025-12-04T11:13:50.4974787Z 2025-12-04T11:13:50.4974886Z ==================================== RERUNS ==================================== 2025-12-04T11:13:50.4975184Z _ TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_8_in_features_1024_out_features_65_cuda_bfloat16 _ 2025-12-04T11:13:50.4975262Z Traceback (most recent call last): 2025-12-04T11:13:50.4975569Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.4975635Z method(*args, **kwargs) 2025-12-04T11:13:50.4975932Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.4975996Z method(*args, **kwargs) 2025-12-04T11:13:50.4976284Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T11:13:50.4976347Z with policy(): 2025-12-04T11:13:50.4976638Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T11:13:50.4976707Z raise RuntimeError(msg) 2025-12-04T11:13:50.4977516Z RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_8_in_features_1024_out_features_65_cuda_bfloat16! Caching allocator allocated memory was 0 and is now reported as 66560 on device 0. CUDA driver allocated memory was 230686720 and is now 274726912. 2025-12-04T11:13:50.4977522Z 2025-12-04T11:13:50.4977650Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.4978363Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_8_in_features_1024_out_features_65_cuda_bfloat16 2025-12-04T11:13:50.4978368Z 2025-12-04T11:13:50.4978528Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.4978732Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.4978875Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.4979429Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('benchmarking.InductorBenchmarker.benchmark_gpu', 2), ('pad_mm_bench', 1), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.4979571Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.4979632Z graph_break [] 2025-12-04T11:13:50.4979763Z ----------------------------- Captured stderr call ----------------------------- 2025-12-04T11:13:50.4980514Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/constant_folding.py:256: UserWarning: Unsupported unwinding pattern: Address not in range (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/profiler/unwind/unwind.cpp:219.) 2025-12-04T11:13:50.4980594Z if out == self.unknown_value: 2025-12-04T11:13:50.4980892Z _ TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_8_in_features_1024_out_features_65_cuda_bfloat16 _ 2025-12-04T11:13:50.4981020Z Traceback (most recent call last): 2025-12-04T11:13:50.4981326Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.4981390Z method(*args, **kwargs) 2025-12-04T11:13:50.4981683Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.4981750Z method(*args, **kwargs) 2025-12-04T11:13:50.4982045Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T11:13:50.4982104Z with policy(): 2025-12-04T11:13:50.4982400Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T11:13:50.4982467Z raise RuntimeError(msg) 2025-12-04T11:13:50.4983297Z RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_8_in_features_1024_out_features_65_cuda_bfloat16! Caching allocator allocated memory was 66560 and is now reported as 133120 on device 0. CUDA driver allocated memory was 274726912 and is now 276824064. 2025-12-04T11:13:50.4983303Z 2025-12-04T11:13:50.4983436Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.4984032Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_8_in_features_1024_out_features_65_cuda_bfloat16 2025-12-04T11:13:50.4984038Z 2025-12-04T11:13:50.4984195Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.4984322Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.4984420Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.4984970Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('benchmarking.InductorBenchmarker.benchmark_gpu', 2), ('pad_mm_bench', 1), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.4985101Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.4985158Z graph_break [] 2025-12-04T11:13:50.4985281Z ----------------------------- Captured stderr call ----------------------------- 2025-12-04T11:13:50.4985969Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/constant_folding.py:256: UserWarning: Unsupported unwinding pattern: Address not in range (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/profiler/unwind/unwind.cpp:219.) 2025-12-04T11:13:50.4986039Z if out == self.unknown_value: 2025-12-04T11:13:50.4986224Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.4986351Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.4986478Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.4987019Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('benchmarking.InductorBenchmarker.benchmark_gpu', 2), ('pad_mm_bench', 1), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.4987079Z graph_break [] 2025-12-04T11:13:50.4987164Z =================================== FAILURES =================================== 2025-12-04T11:13:50.4987498Z _ TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_8_in_features_1024_out_features_65_cuda_bfloat16 _ 2025-12-04T11:13:50.4987572Z Traceback (most recent call last): 2025-12-04T11:13:50.4987882Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.4987951Z method(*args, **kwargs) 2025-12-04T11:13:50.4988244Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.4988352Z method(*args, **kwargs) 2025-12-04T11:13:50.4988647Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T11:13:50.4988751Z with policy(): 2025-12-04T11:13:50.4989100Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T11:13:50.4989166Z raise RuntimeError(msg) 2025-12-04T11:13:50.4989990Z RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_8_in_features_1024_out_features_65_cuda_bfloat16! Caching allocator allocated memory was 133120 and is now reported as 199680 on device 0. CUDA driver allocated memory was 276824064 and is now 278921216. 2025-12-04T11:13:50.4989996Z 2025-12-04T11:13:50.4990122Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.4990646Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_8_in_features_1024_out_features_65_cuda_bfloat16 2025-12-04T11:13:50.4990652Z 2025-12-04T11:13:50.4990807Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.4990933Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.4991027Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.4991569Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('benchmarking.InductorBenchmarker.benchmark_gpu', 2), ('pad_mm_bench', 1), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.4991696Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.4991757Z graph_break [] 2025-12-04T11:13:50.4991881Z ----------------------------- Captured stderr call ----------------------------- 2025-12-04T11:13:50.4992566Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/constant_folding.py:256: UserWarning: Unsupported unwinding pattern: Address not in range (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/profiler/unwind/unwind.cpp:219.) 2025-12-04T11:13:50.4992636Z if out == self.unknown_value: 2025-12-04T11:13:50.4992760Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.4992850Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.4992973Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.4993514Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('benchmarking.InductorBenchmarker.benchmark_gpu', 2), ('pad_mm_bench', 1), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.4993683Z graph_break [] 2025-12-04T11:13:50.4993807Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.4993900Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.4994020Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.4994554Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('benchmarking.InductorBenchmarker.benchmark_gpu', 2), ('pad_mm_bench', 1), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.4994615Z graph_break [] 2025-12-04T11:13:50.4995141Z - generated xml file: /var/lib/jenkins/workspace/test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-d04b19868e6e9971.xml - 2025-12-04T11:13:50.4995247Z =========================== short test summary info ============================ 2025-12-04T11:13:50.4996549Z FAILED [0.5051s] inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_32_mid_dim_8_in_features_1024_out_features_65_cuda_bfloat16 - RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_8_in_features_1024_out_features_65_cuda_bfloat16! Caching allocator allocated memory was 133120 and is now reported as 199680 on device 0. CUDA driver allocated memory was 276824064 and is now 278921216. 2025-12-04T11:13:50.4996610Z 2025-12-04T11:13:50.4996736Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.4997258Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_8_in_features_1024_out_features_65_cuda_bfloat16 2025-12-04T11:13:50.4997263Z 2025-12-04T11:13:50.4997424Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.4997530Z !!!!!!!!!!!!!!!!!!!!!!!!!! stopping after 1 failures !!!!!!!!!!!!!!!!!!!!!!!!!!! 2025-12-04T11:13:50.4997646Z ================== 1 failed, 57 deselected, 2 rerun in 11.94s ================== 2025-12-04T11:13:50.4997709Z Got exit code 1 2025-12-04T11:13:50.4998190Z FAILED CONSISTENTLY: test/inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_32_mid_dim_8_in_features_1024_out_features_65_cuda_bfloat16 2025-12-04T11:13:50.4998437Z Test failed consistently, continuing with the rest of the tests due to continue-through-error being set 2025-12-04T11:13:50.4998701Z W1204 11:09:02.789000 85904 site-packages/torch/_inductor/utils.py:1703] Not enough SMs to use max_autotune_gemm mode 2025-12-04T11:13:50.4999089Z Test results will be stored in test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-cf6ace4ea5db8dfb.xml 2025-12-04T11:13:50.4999191Z ============================= test session starts ============================== 2025-12-04T11:13:50.4999402Z platform linux -- Python 3.10.14, pytest-7.3.2, pluggy-1.6.0 -- /opt/conda/envs/py_3.10/bin/python 2025-12-04T11:13:50.4999473Z cachedir: .pytest_cache 2025-12-04T11:13:50.4999778Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow] 2025-12-04T11:13:50.4999854Z rootdir: /var/lib/jenkins/workspace 2025-12-04T11:13:50.4999925Z configfile: pytest.ini 2025-12-04T11:13:50.5000239Z plugins: hypothesis-6.56.4, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-14.0, subtests-0.13.1, xdist-3.3.1, xdoctest-1.3.0, typeguard-4.3.0 2025-12-04T11:13:50.5000372Z collecting ... collected 58 items / 52 deselected / 6 selected 2025-12-04T11:13:50.5000461Z stepcurrent: skipping 52 already run items. 2025-12-04T11:13:50.5000531Z Running 6 items in this shard 2025-12-04T11:13:50.5000535Z 2025-12-04T11:13:50.5001113Z inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_32_mid_dim_8_in_features_128_out_features_1024_cuda_bfloat16 ('RERUN', {'yellow': True}) [1.9043s] [ 16%] 2025-12-04T11:13:50.5001614Z inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_32_mid_dim_8_in_features_128_out_features_1024_cuda_bfloat16 ('RERUN', {'yellow': True}) [0.4986s] [ 16%] 2025-12-04T11:13:50.5002061Z inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_32_mid_dim_8_in_features_128_out_features_1024_cuda_bfloat16 FAILED [0.4868s] [ 16%] 2025-12-04T11:13:50.5002066Z 2025-12-04T11:13:50.5002147Z ==================================== RERUNS ==================================== 2025-12-04T11:13:50.5002478Z _ TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_8_in_features_128_out_features_1024_cuda_bfloat16 _ 2025-12-04T11:13:50.5002560Z Traceback (most recent call last): 2025-12-04T11:13:50.5002873Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.5002942Z method(*args, **kwargs) 2025-12-04T11:13:50.5003272Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.5003335Z method(*args, **kwargs) 2025-12-04T11:13:50.5003625Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T11:13:50.5003685Z with policy(): 2025-12-04T11:13:50.5003985Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T11:13:50.5004057Z raise RuntimeError(msg) 2025-12-04T11:13:50.5004864Z RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_8_in_features_128_out_features_1024_cuda_bfloat16! Caching allocator allocated memory was 0 and is now reported as 131072 on device 0. CUDA driver allocated memory was 230686720 and is now 274726912. 2025-12-04T11:13:50.5004869Z 2025-12-04T11:13:50.5005003Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.5005528Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_8_in_features_128_out_features_1024_cuda_bfloat16 2025-12-04T11:13:50.5005532Z 2025-12-04T11:13:50.5005695Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.5005821Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.5005916Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.5006269Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.5006399Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.5006465Z graph_break [] 2025-12-04T11:13:50.5006757Z _ TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_8_in_features_128_out_features_1024_cuda_bfloat16 _ 2025-12-04T11:13:50.5006832Z Traceback (most recent call last): 2025-12-04T11:13:50.5007133Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.5007197Z method(*args, **kwargs) 2025-12-04T11:13:50.5007488Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.5007554Z method(*args, **kwargs) 2025-12-04T11:13:50.5007843Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T11:13:50.5007947Z with policy(): 2025-12-04T11:13:50.5008271Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T11:13:50.5008340Z raise RuntimeError(msg) 2025-12-04T11:13:50.5009163Z RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_8_in_features_128_out_features_1024_cuda_bfloat16! Caching allocator allocated memory was 131072 and is now reported as 262144 on device 0. CUDA driver allocated memory was 274726912 and is now 276824064. 2025-12-04T11:13:50.5009168Z 2025-12-04T11:13:50.5009293Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.5009853Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_8_in_features_128_out_features_1024_cuda_bfloat16 2025-12-04T11:13:50.5009857Z 2025-12-04T11:13:50.5010014Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.5010142Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.5010239Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.5010709Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.5010841Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.5010903Z graph_break [] 2025-12-04T11:13:50.5011024Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.5011115Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.5011236Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.5011579Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.5011638Z graph_break [] 2025-12-04T11:13:50.5011725Z =================================== FAILURES =================================== 2025-12-04T11:13:50.5012033Z _ TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_8_in_features_128_out_features_1024_cuda_bfloat16 _ 2025-12-04T11:13:50.5012159Z Traceback (most recent call last): 2025-12-04T11:13:50.5012578Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.5012652Z method(*args, **kwargs) 2025-12-04T11:13:50.5013039Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.5013145Z method(*args, **kwargs) 2025-12-04T11:13:50.5013438Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T11:13:50.5013499Z with policy(): 2025-12-04T11:13:50.5013864Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T11:13:50.5013982Z raise RuntimeError(msg) 2025-12-04T11:13:50.5014825Z RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_8_in_features_128_out_features_1024_cuda_bfloat16! Caching allocator allocated memory was 262144 and is now reported as 393216 on device 0. CUDA driver allocated memory was 276824064 and is now 278921216. 2025-12-04T11:13:50.5014830Z 2025-12-04T11:13:50.5014954Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.5015478Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_8_in_features_128_out_features_1024_cuda_bfloat16 2025-12-04T11:13:50.5015482Z 2025-12-04T11:13:50.5015708Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.5015871Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.5015969Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.5016317Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.5016444Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.5016507Z graph_break [] 2025-12-04T11:13:50.5016633Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.5016727Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.5016883Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.5017226Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.5017290Z graph_break [] 2025-12-04T11:13:50.5017413Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.5017535Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.5017660Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.5017998Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.5018057Z graph_break [] 2025-12-04T11:13:50.5018561Z - generated xml file: /var/lib/jenkins/workspace/test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-cf6ace4ea5db8dfb.xml - 2025-12-04T11:13:50.5018663Z =========================== short test summary info ============================ 2025-12-04T11:13:50.5019971Z FAILED [0.4868s] inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_32_mid_dim_8_in_features_128_out_features_1024_cuda_bfloat16 - RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_8_in_features_128_out_features_1024_cuda_bfloat16! Caching allocator allocated memory was 262144 and is now reported as 393216 on device 0. CUDA driver allocated memory was 276824064 and is now 278921216. 2025-12-04T11:13:50.5019979Z 2025-12-04T11:13:50.5020103Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.5020630Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_8_in_features_128_out_features_1024_cuda_bfloat16 2025-12-04T11:13:50.5020634Z 2025-12-04T11:13:50.5020791Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.5020898Z !!!!!!!!!!!!!!!!!!!!!!!!!! stopping after 1 failures !!!!!!!!!!!!!!!!!!!!!!!!!!! 2025-12-04T11:13:50.5021018Z ================== 1 failed, 52 deselected, 2 rerun in 2.91s =================== 2025-12-04T11:13:50.5021076Z Got exit code 1 2025-12-04T11:13:50.5021145Z Retrying single test... 2025-12-04T11:13:50.5021406Z W1204 11:09:12.899000 86093 site-packages/torch/_inductor/utils.py:1703] Not enough SMs to use max_autotune_gemm mode 2025-12-04T11:13:50.5021792Z Test results will be stored in test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-8b65a3dabcd39e70.xml 2025-12-04T11:13:50.5021895Z ============================= test session starts ============================== 2025-12-04T11:13:50.5022103Z platform linux -- Python 3.10.14, pytest-7.3.2, pluggy-1.6.0 -- /opt/conda/envs/py_3.10/bin/python 2025-12-04T11:13:50.5022173Z cachedir: .pytest_cache 2025-12-04T11:13:50.5022480Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow] 2025-12-04T11:13:50.5022600Z rootdir: /var/lib/jenkins/workspace 2025-12-04T11:13:50.5022669Z configfile: pytest.ini 2025-12-04T11:13:50.5023016Z plugins: hypothesis-6.56.4, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-14.0, subtests-0.13.1, xdist-3.3.1, xdoctest-1.3.0, typeguard-4.3.0 2025-12-04T11:13:50.5023161Z collecting ... collected 58 items / 57 deselected / 1 selected 2025-12-04T11:13:50.5023839Z stepcurrent: skipping 52 already run items. Running only test/inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_32_mid_dim_8_in_features_128_out_features_1024_cuda_bfloat16 2025-12-04T11:13:50.5023911Z Running 1 items in this shard 2025-12-04T11:13:50.5023915Z 2025-12-04T11:13:50.5024713Z inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_32_mid_dim_8_in_features_128_out_features_1024_cuda_bfloat16 [W1204 11:09:14.027362544 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.5024719Z 2025-12-04T11:13:50.5025024Z [W1204 11:09:23.307396267 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.5025070Z 2025-12-04T11:13:50.5025366Z [W1204 11:09:23.307635852 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.5025370Z 2025-12-04T11:13:50.5025658Z [W1204 11:09:23.313403380 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.5025662Z 2025-12-04T11:13:50.5025955Z [W1204 11:09:23.313962512 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.5025958Z 2025-12-04T11:13:50.5026247Z [W1204 11:09:23.314121915 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.5026252Z 2025-12-04T11:13:50.5026544Z [W1204 11:09:23.319422833 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.5026549Z 2025-12-04T11:13:50.5026835Z [W1204 11:09:23.319939864 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.5026838Z 2025-12-04T11:13:50.5027124Z [W1204 11:09:23.320119778 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.5027131Z 2025-12-04T11:13:50.5027213Z ('RERUN', {'yellow': True}) [11.2050s] [100%] 2025-12-04T11:13:50.5027944Z inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_32_mid_dim_8_in_features_128_out_features_1024_cuda_bfloat16 [W1204 11:09:24.544822580 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.5027950Z 2025-12-04T11:13:50.5028260Z [W1204 11:09:24.545367742 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.5028265Z 2025-12-04T11:13:50.5028551Z [W1204 11:09:24.545504335 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.5028554Z 2025-12-04T11:13:50.5028854Z [W1204 11:09:24.548493191 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.5028858Z 2025-12-04T11:13:50.5029147Z [W1204 11:09:24.549085054 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.5029150Z 2025-12-04T11:13:50.5029442Z [W1204 11:09:24.549224887 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.5029485Z 2025-12-04T11:13:50.5029808Z [W1204 11:09:24.553895240 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.5029814Z 2025-12-04T11:13:50.5030108Z [W1204 11:09:24.554379730 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.5030111Z 2025-12-04T11:13:50.5030400Z [W1204 11:09:24.554515333 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.5030403Z 2025-12-04T11:13:50.5030481Z ('RERUN', {'yellow': True}) [0.4636s] [100%] 2025-12-04T11:13:50.5031243Z inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_32_mid_dim_8_in_features_128_out_features_1024_cuda_bfloat16 [W1204 11:09:25.006123126 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.5031249Z 2025-12-04T11:13:50.5031537Z [W1204 11:09:25.006678899 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.5031573Z 2025-12-04T11:13:50.5031866Z [W1204 11:09:25.006819872 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.5031869Z 2025-12-04T11:13:50.5032155Z [W1204 11:09:25.009815648 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.5032158Z 2025-12-04T11:13:50.5032446Z [W1204 11:09:25.010416661 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.5032450Z 2025-12-04T11:13:50.5032733Z [W1204 11:09:25.010558244 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.5032738Z 2025-12-04T11:13:50.5033031Z [W1204 11:09:25.015185117 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.5033034Z 2025-12-04T11:13:50.5033323Z [W1204 11:09:25.015665227 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.5033327Z 2025-12-04T11:13:50.5033614Z [W1204 11:09:25.015802250 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.5033621Z 2025-12-04T11:13:50.5033682Z FAILED [0.4580s] [100%] 2025-12-04T11:13:50.5033686Z 2025-12-04T11:13:50.5033770Z ==================================== RERUNS ==================================== 2025-12-04T11:13:50.5034066Z _ TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_8_in_features_128_out_features_1024_cuda_bfloat16 _ 2025-12-04T11:13:50.5034143Z Traceback (most recent call last): 2025-12-04T11:13:50.5034462Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.5034532Z method(*args, **kwargs) 2025-12-04T11:13:50.5034825Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.5034891Z method(*args, **kwargs) 2025-12-04T11:13:50.5035179Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T11:13:50.5035241Z with policy(): 2025-12-04T11:13:50.5035538Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T11:13:50.5035608Z raise RuntimeError(msg) 2025-12-04T11:13:50.5036453Z RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_8_in_features_128_out_features_1024_cuda_bfloat16! Caching allocator allocated memory was 0 and is now reported as 131072 on device 0. CUDA driver allocated memory was 230686720 and is now 274726912. 2025-12-04T11:13:50.5036492Z 2025-12-04T11:13:50.5036621Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.5037143Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_8_in_features_128_out_features_1024_cuda_bfloat16 2025-12-04T11:13:50.5037151Z 2025-12-04T11:13:50.5037309Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.5037436Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.5037566Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.5037915Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.5038044Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.5038108Z graph_break [] 2025-12-04T11:13:50.5038234Z ----------------------------- Captured stderr call ----------------------------- 2025-12-04T11:13:50.5038962Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/constant_folding.py:256: UserWarning: Unsupported unwinding pattern: Address not in range (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/profiler/unwind/unwind.cpp:219.) 2025-12-04T11:13:50.5039032Z if out == self.unknown_value: 2025-12-04T11:13:50.5039331Z _ TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_8_in_features_128_out_features_1024_cuda_bfloat16 _ 2025-12-04T11:13:50.5039412Z Traceback (most recent call last): 2025-12-04T11:13:50.5039707Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.5039775Z method(*args, **kwargs) 2025-12-04T11:13:50.5040076Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.5040142Z method(*args, **kwargs) 2025-12-04T11:13:50.5040436Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T11:13:50.5040496Z with policy(): 2025-12-04T11:13:50.5040791Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T11:13:50.5040862Z raise RuntimeError(msg) 2025-12-04T11:13:50.5041678Z RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_8_in_features_128_out_features_1024_cuda_bfloat16! Caching allocator allocated memory was 131072 and is now reported as 262144 on device 0. CUDA driver allocated memory was 274726912 and is now 276824064. 2025-12-04T11:13:50.5041683Z 2025-12-04T11:13:50.5041810Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.5042336Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_8_in_features_128_out_features_1024_cuda_bfloat16 2025-12-04T11:13:50.5042342Z 2025-12-04T11:13:50.5042499Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.5042622Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.5042718Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.5043071Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.5043200Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.5043301Z graph_break [] 2025-12-04T11:13:50.5043428Z ----------------------------- Captured stderr call ----------------------------- 2025-12-04T11:13:50.5044144Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/constant_folding.py:256: UserWarning: Unsupported unwinding pattern: Address not in range (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/profiler/unwind/unwind.cpp:219.) 2025-12-04T11:13:50.5044221Z if out == self.unknown_value: 2025-12-04T11:13:50.5044343Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.5044433Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.5044557Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.5044933Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.5044995Z graph_break [] 2025-12-04T11:13:50.5045081Z =================================== FAILURES =================================== 2025-12-04T11:13:50.5045452Z _ TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_8_in_features_128_out_features_1024_cuda_bfloat16 _ 2025-12-04T11:13:50.5045613Z Traceback (most recent call last): 2025-12-04T11:13:50.5045936Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.5046003Z method(*args, **kwargs) 2025-12-04T11:13:50.5046301Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.5046364Z method(*args, **kwargs) 2025-12-04T11:13:50.5046656Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T11:13:50.5046717Z with policy(): 2025-12-04T11:13:50.5047012Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T11:13:50.5047082Z raise RuntimeError(msg) 2025-12-04T11:13:50.5047912Z RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_8_in_features_128_out_features_1024_cuda_bfloat16! Caching allocator allocated memory was 262144 and is now reported as 393216 on device 0. CUDA driver allocated memory was 276824064 and is now 278921216. 2025-12-04T11:13:50.5047918Z 2025-12-04T11:13:50.5048055Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.5048576Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_8_in_features_128_out_features_1024_cuda_bfloat16 2025-12-04T11:13:50.5048580Z 2025-12-04T11:13:50.5051010Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.5051165Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.5051266Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.5051627Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.5051765Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.5051830Z graph_break [] 2025-12-04T11:13:50.5051972Z ----------------------------- Captured stderr call ----------------------------- 2025-12-04T11:13:50.5052687Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/constant_folding.py:256: UserWarning: Unsupported unwinding pattern: Address not in range (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/profiler/unwind/unwind.cpp:219.) 2025-12-04T11:13:50.5052786Z if out == self.unknown_value: 2025-12-04T11:13:50.5052919Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.5053111Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.5053283Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.5053645Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.5053708Z graph_break [] 2025-12-04T11:13:50.5053836Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.5053935Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.5054063Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.5054450Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.5054515Z graph_break [] 2025-12-04T11:13:50.5055028Z - generated xml file: /var/lib/jenkins/workspace/test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-8b65a3dabcd39e70.xml - 2025-12-04T11:13:50.5055139Z =========================== short test summary info ============================ 2025-12-04T11:13:50.5056455Z FAILED [0.4580s] inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_32_mid_dim_8_in_features_128_out_features_1024_cuda_bfloat16 - RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_8_in_features_128_out_features_1024_cuda_bfloat16! Caching allocator allocated memory was 262144 and is now reported as 393216 on device 0. CUDA driver allocated memory was 276824064 and is now 278921216. 2025-12-04T11:13:50.5056461Z 2025-12-04T11:13:50.5056599Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.5057135Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_8_in_features_128_out_features_1024_cuda_bfloat16 2025-12-04T11:13:50.5057142Z 2025-12-04T11:13:50.5057309Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.5057421Z !!!!!!!!!!!!!!!!!!!!!!!!!! stopping after 1 failures !!!!!!!!!!!!!!!!!!!!!!!!!!! 2025-12-04T11:13:50.5057542Z ================== 1 failed, 57 deselected, 2 rerun in 12.15s ================== 2025-12-04T11:13:50.5057607Z Got exit code 1 2025-12-04T11:13:50.5057677Z Retrying single test... 2025-12-04T11:13:50.5057950Z W1204 11:09:31.985000 86286 site-packages/torch/_inductor/utils.py:1703] Not enough SMs to use max_autotune_gemm mode 2025-12-04T11:13:50.5058338Z Test results will be stored in test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-c85748e91a458eac.xml 2025-12-04T11:13:50.5058514Z ============================= test session starts ============================== 2025-12-04T11:13:50.5058739Z platform linux -- Python 3.10.14, pytest-7.3.2, pluggy-1.6.0 -- /opt/conda/envs/py_3.10/bin/python 2025-12-04T11:13:50.5058808Z cachedir: .pytest_cache 2025-12-04T11:13:50.5059117Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow] 2025-12-04T11:13:50.5059200Z rootdir: /var/lib/jenkins/workspace 2025-12-04T11:13:50.5059268Z configfile: pytest.ini 2025-12-04T11:13:50.5059602Z plugins: hypothesis-6.56.4, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-14.0, subtests-0.13.1, xdist-3.3.1, xdoctest-1.3.0, typeguard-4.3.0 2025-12-04T11:13:50.5059736Z collecting ... collected 58 items / 57 deselected / 1 selected 2025-12-04T11:13:50.5060314Z stepcurrent: skipping 52 already run items. Running only test/inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_32_mid_dim_8_in_features_128_out_features_1024_cuda_bfloat16 2025-12-04T11:13:50.5060433Z Running 1 items in this shard 2025-12-04T11:13:50.5060437Z 2025-12-04T11:13:50.5061211Z inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_32_mid_dim_8_in_features_128_out_features_1024_cuda_bfloat16 [W1204 11:09:33.107390720 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.5061216Z 2025-12-04T11:13:50.5061523Z [W1204 11:09:42.196469282 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.5061526Z 2025-12-04T11:13:50.5061824Z [W1204 11:09:42.196716017 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.5061827Z 2025-12-04T11:13:50.5062157Z [W1204 11:09:42.202444204 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.5062165Z 2025-12-04T11:13:50.5062455Z [W1204 11:09:42.202983696 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.5062459Z 2025-12-04T11:13:50.5062752Z [W1204 11:09:42.203140700 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.5062755Z 2025-12-04T11:13:50.5063043Z [W1204 11:09:42.208555890 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.5063047Z 2025-12-04T11:13:50.5063335Z [W1204 11:09:42.209092251 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.5063344Z 2025-12-04T11:13:50.5063702Z [W1204 11:09:42.209250875 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.5063707Z 2025-12-04T11:13:50.5063795Z ('RERUN', {'yellow': True}) [11.0026s] [100%] 2025-12-04T11:13:50.5064529Z inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_32_mid_dim_8_in_features_128_out_features_1024_cuda_bfloat16 [W1204 11:09:43.427215023 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.5064534Z 2025-12-04T11:13:50.5064822Z [W1204 11:09:43.427807006 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.5064825Z 2025-12-04T11:13:50.5065117Z [W1204 11:09:43.427945119 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.5065120Z 2025-12-04T11:13:50.5065407Z [W1204 11:09:43.431013197 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.5065467Z 2025-12-04T11:13:50.5065774Z [W1204 11:09:43.431597630 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.5065780Z 2025-12-04T11:13:50.5066070Z [W1204 11:09:43.431733413 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.5066073Z 2025-12-04T11:13:50.5066367Z [W1204 11:09:43.436406316 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.5066370Z 2025-12-04T11:13:50.5066658Z [W1204 11:09:43.436889717 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.5066661Z 2025-12-04T11:13:50.5066953Z [W1204 11:09:43.437024620 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.5066962Z 2025-12-04T11:13:50.5067082Z ('RERUN', {'yellow': True}) [0.4628s] [100%] 2025-12-04T11:13:50.5067844Z inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_32_mid_dim_8_in_features_128_out_features_1024_cuda_bfloat16 [W1204 11:09:43.885940798 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.5067849Z 2025-12-04T11:13:50.5068142Z [W1204 11:09:43.886471200 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.5068146Z 2025-12-04T11:13:50.5068434Z [W1204 11:09:43.886609252 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.5068437Z 2025-12-04T11:13:50.5068757Z [W1204 11:09:43.889539877 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.5068763Z 2025-12-04T11:13:50.5069054Z [W1204 11:09:43.890133150 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.5069059Z 2025-12-04T11:13:50.5069352Z [W1204 11:09:43.890274924 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.5069356Z 2025-12-04T11:13:50.5069654Z [W1204 11:09:43.894810234 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.5069657Z 2025-12-04T11:13:50.5069945Z [W1204 11:09:43.895272444 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.5069952Z 2025-12-04T11:13:50.5070240Z [W1204 11:09:43.895406907 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.5070245Z 2025-12-04T11:13:50.5070307Z FAILED [0.4495s] [100%] 2025-12-04T11:13:50.5070312Z 2025-12-04T11:13:50.5070405Z ==================================== RERUNS ==================================== 2025-12-04T11:13:50.5070708Z _ TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_8_in_features_128_out_features_1024_cuda_bfloat16 _ 2025-12-04T11:13:50.5070790Z Traceback (most recent call last): 2025-12-04T11:13:50.5071103Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.5071170Z method(*args, **kwargs) 2025-12-04T11:13:50.5071472Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.5071538Z method(*args, **kwargs) 2025-12-04T11:13:50.5071831Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T11:13:50.5071945Z with policy(): 2025-12-04T11:13:50.5072242Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T11:13:50.5072316Z raise RuntimeError(msg) 2025-12-04T11:13:50.5073126Z RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_8_in_features_128_out_features_1024_cuda_bfloat16! Caching allocator allocated memory was 0 and is now reported as 131072 on device 0. CUDA driver allocated memory was 230686720 and is now 274726912. 2025-12-04T11:13:50.5073130Z 2025-12-04T11:13:50.5073262Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.5073793Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_8_in_features_128_out_features_1024_cuda_bfloat16 2025-12-04T11:13:50.5073798Z 2025-12-04T11:13:50.5073960Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.5074150Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.5074287Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.5074645Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.5074781Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.5074842Z graph_break [] 2025-12-04T11:13:50.5074977Z ----------------------------- Captured stderr call ----------------------------- 2025-12-04T11:13:50.5075713Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/constant_folding.py:256: UserWarning: Unsupported unwinding pattern: Address not in range (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/profiler/unwind/unwind.cpp:219.) 2025-12-04T11:13:50.5075789Z if out == self.unknown_value: 2025-12-04T11:13:50.5076097Z _ TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_8_in_features_128_out_features_1024_cuda_bfloat16 _ 2025-12-04T11:13:50.5076179Z Traceback (most recent call last): 2025-12-04T11:13:50.5076493Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.5076564Z method(*args, **kwargs) 2025-12-04T11:13:50.5076859Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.5076928Z method(*args, **kwargs) 2025-12-04T11:13:50.5077226Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T11:13:50.5077288Z with policy(): 2025-12-04T11:13:50.5077592Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T11:13:50.5077663Z raise RuntimeError(msg) 2025-12-04T11:13:50.5078709Z RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_8_in_features_128_out_features_1024_cuda_bfloat16! Caching allocator allocated memory was 131072 and is now reported as 262144 on device 0. CUDA driver allocated memory was 274726912 and is now 276824064. 2025-12-04T11:13:50.5078714Z 2025-12-04T11:13:50.5078848Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.5079378Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_8_in_features_128_out_features_1024_cuda_bfloat16 2025-12-04T11:13:50.5079382Z 2025-12-04T11:13:50.5079545Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.5079674Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.5079876Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.5080233Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.5080372Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.5080437Z graph_break [] 2025-12-04T11:13:50.5080566Z ----------------------------- Captured stderr call ----------------------------- 2025-12-04T11:13:50.5081269Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/constant_folding.py:256: UserWarning: Unsupported unwinding pattern: Address not in range (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/profiler/unwind/unwind.cpp:219.) 2025-12-04T11:13:50.5081341Z if out == self.unknown_value: 2025-12-04T11:13:50.5081472Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.5081572Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.5081767Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.5082167Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.5082231Z graph_break [] 2025-12-04T11:13:50.5082333Z =================================== FAILURES =================================== 2025-12-04T11:13:50.5082644Z _ TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_8_in_features_128_out_features_1024_cuda_bfloat16 _ 2025-12-04T11:13:50.5082723Z Traceback (most recent call last): 2025-12-04T11:13:50.5083037Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.5083152Z method(*args, **kwargs) 2025-12-04T11:13:50.5083457Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.5083534Z method(*args, **kwargs) 2025-12-04T11:13:50.5083826Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T11:13:50.5083890Z with policy(): 2025-12-04T11:13:50.5084188Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T11:13:50.5084253Z raise RuntimeError(msg) 2025-12-04T11:13:50.5085081Z RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_8_in_features_128_out_features_1024_cuda_bfloat16! Caching allocator allocated memory was 262144 and is now reported as 393216 on device 0. CUDA driver allocated memory was 276824064 and is now 278921216. 2025-12-04T11:13:50.5085086Z 2025-12-04T11:13:50.5085217Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.5085749Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_8_in_features_128_out_features_1024_cuda_bfloat16 2025-12-04T11:13:50.5085754Z 2025-12-04T11:13:50.5085914Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.5086045Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.5086145Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.5086497Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.5086632Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.5086695Z graph_break [] 2025-12-04T11:13:50.5086822Z ----------------------------- Captured stderr call ----------------------------- 2025-12-04T11:13:50.5087557Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/constant_folding.py:256: UserWarning: Unsupported unwinding pattern: Address not in range (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/profiler/unwind/unwind.cpp:219.) 2025-12-04T11:13:50.5087629Z if out == self.unknown_value: 2025-12-04T11:13:50.5087754Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.5087856Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.5087980Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.5088335Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.5088398Z graph_break [] 2025-12-04T11:13:50.5088522Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.5088620Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.5088791Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.5089180Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.5089245Z graph_break [] 2025-12-04T11:13:50.5089738Z - generated xml file: /var/lib/jenkins/workspace/test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-c85748e91a458eac.xml - 2025-12-04T11:13:50.5089855Z =========================== short test summary info ============================ 2025-12-04T11:13:50.5091194Z FAILED [0.4495s] inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_32_mid_dim_8_in_features_128_out_features_1024_cuda_bfloat16 - RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_8_in_features_128_out_features_1024_cuda_bfloat16! Caching allocator allocated memory was 262144 and is now reported as 393216 on device 0. CUDA driver allocated memory was 276824064 and is now 278921216. 2025-12-04T11:13:50.5091204Z 2025-12-04T11:13:50.5091334Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.5091857Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_8_in_features_128_out_features_1024_cuda_bfloat16 2025-12-04T11:13:50.5091861Z 2025-12-04T11:13:50.5092024Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.5092133Z !!!!!!!!!!!!!!!!!!!!!!!!!! stopping after 1 failures !!!!!!!!!!!!!!!!!!!!!!!!!!! 2025-12-04T11:13:50.5092256Z ================== 1 failed, 57 deselected, 2 rerun in 11.94s ================== 2025-12-04T11:13:50.5092324Z Got exit code 1 2025-12-04T11:13:50.5092807Z FAILED CONSISTENTLY: test/inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_32_mid_dim_8_in_features_128_out_features_1024_cuda_bfloat16 2025-12-04T11:13:50.5093056Z Test failed consistently, continuing with the rest of the tests due to continue-through-error being set 2025-12-04T11:13:50.5093325Z W1204 11:09:50.738000 86479 site-packages/torch/_inductor/utils.py:1703] Not enough SMs to use max_autotune_gemm mode 2025-12-04T11:13:50.5093711Z Test results will be stored in test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-85bca917f8457708.xml 2025-12-04T11:13:50.5093814Z ============================= test session starts ============================== 2025-12-04T11:13:50.5094026Z platform linux -- Python 3.10.14, pytest-7.3.2, pluggy-1.6.0 -- /opt/conda/envs/py_3.10/bin/python 2025-12-04T11:13:50.5094096Z cachedir: .pytest_cache 2025-12-04T11:13:50.5094409Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow] 2025-12-04T11:13:50.5094532Z rootdir: /var/lib/jenkins/workspace 2025-12-04T11:13:50.5094603Z configfile: pytest.ini 2025-12-04T11:13:50.5094923Z plugins: hypothesis-6.56.4, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-14.0, subtests-0.13.1, xdist-3.3.1, xdoctest-1.3.0, typeguard-4.3.0 2025-12-04T11:13:50.5095054Z collecting ... collected 58 items / 53 deselected / 5 selected 2025-12-04T11:13:50.5095149Z stepcurrent: skipping 53 already run items. 2025-12-04T11:13:50.5095221Z Running 5 items in this shard 2025-12-04T11:13:50.5095226Z 2025-12-04T11:13:50.5095724Z inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_32_mid_dim_8_in_features_128_out_features_64_cuda_bfloat16 ('RERUN', {'yellow': True}) [1.8655s] [ 20%] 2025-12-04T11:13:50.5096220Z inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_32_mid_dim_8_in_features_128_out_features_64_cuda_bfloat16 ('RERUN', {'yellow': True}) [0.4539s] [ 20%] 2025-12-04T11:13:50.5096731Z inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_32_mid_dim_8_in_features_128_out_features_64_cuda_bfloat16 FAILED [0.4442s] [ 20%] 2025-12-04T11:13:50.5096735Z 2025-12-04T11:13:50.5096827Z ==================================== RERUNS ==================================== 2025-12-04T11:13:50.5097121Z _ TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_8_in_features_128_out_features_64_cuda_bfloat16 _ 2025-12-04T11:13:50.5097199Z Traceback (most recent call last): 2025-12-04T11:13:50.5097507Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.5097574Z method(*args, **kwargs) 2025-12-04T11:13:50.5097909Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.5097978Z method(*args, **kwargs) 2025-12-04T11:13:50.5098270Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T11:13:50.5098337Z with policy(): 2025-12-04T11:13:50.5098633Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T11:13:50.5098705Z raise RuntimeError(msg) 2025-12-04T11:13:50.5099501Z RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_8_in_features_128_out_features_64_cuda_bfloat16! Caching allocator allocated memory was 0 and is now reported as 8192 on device 0. CUDA driver allocated memory was 230686720 and is now 274726912. 2025-12-04T11:13:50.5099506Z 2025-12-04T11:13:50.5099638Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.5100155Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_8_in_features_128_out_features_64_cuda_bfloat16 2025-12-04T11:13:50.5100162Z 2025-12-04T11:13:50.5100321Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.5100456Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.5100552Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.5100910Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.5101041Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.5101103Z graph_break [] 2025-12-04T11:13:50.5101418Z _ TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_8_in_features_128_out_features_64_cuda_bfloat16 _ 2025-12-04T11:13:50.5101496Z Traceback (most recent call last): 2025-12-04T11:13:50.5101840Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.5101913Z method(*args, **kwargs) 2025-12-04T11:13:50.5102206Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.5102276Z method(*args, **kwargs) 2025-12-04T11:13:50.5102567Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T11:13:50.5102632Z with policy(): 2025-12-04T11:13:50.5102931Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T11:13:50.5102999Z raise RuntimeError(msg) 2025-12-04T11:13:50.5103874Z RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_8_in_features_128_out_features_64_cuda_bfloat16! Caching allocator allocated memory was 8192 and is now reported as 16384 on device 0. CUDA driver allocated memory was 274726912 and is now 276824064. 2025-12-04T11:13:50.5103920Z 2025-12-04T11:13:50.5104082Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.5104603Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_8_in_features_128_out_features_64_cuda_bfloat16 2025-12-04T11:13:50.5104607Z 2025-12-04T11:13:50.5104770Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.5104898Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.5104996Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.5105396Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.5105528Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.5105596Z graph_break [] 2025-12-04T11:13:50.5105723Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.5105816Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.5105938Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.5106280Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.5106345Z graph_break [] 2025-12-04T11:13:50.5106432Z =================================== FAILURES =================================== 2025-12-04T11:13:50.5106725Z _ TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_8_in_features_128_out_features_64_cuda_bfloat16 _ 2025-12-04T11:13:50.5106803Z Traceback (most recent call last): 2025-12-04T11:13:50.5107118Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.5107194Z method(*args, **kwargs) 2025-12-04T11:13:50.5107491Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.5107555Z method(*args, **kwargs) 2025-12-04T11:13:50.5107846Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T11:13:50.5107906Z with policy(): 2025-12-04T11:13:50.5108200Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T11:13:50.5108273Z raise RuntimeError(msg) 2025-12-04T11:13:50.5109082Z RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_8_in_features_128_out_features_64_cuda_bfloat16! Caching allocator allocated memory was 16384 and is now reported as 24576 on device 0. CUDA driver allocated memory was 276824064 and is now 278921216. 2025-12-04T11:13:50.5109131Z 2025-12-04T11:13:50.5109263Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.5109778Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_8_in_features_128_out_features_64_cuda_bfloat16 2025-12-04T11:13:50.5109782Z 2025-12-04T11:13:50.5109942Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.5110069Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.5110161Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.5110512Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.5110648Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.5110753Z graph_break [] 2025-12-04T11:13:50.5110916Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.5111007Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.5111129Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.5111470Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.5111530Z graph_break [] 2025-12-04T11:13:50.5111658Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.5111748Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.5111907Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.5112249Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.5112310Z graph_break [] 2025-12-04T11:13:50.5112804Z - generated xml file: /var/lib/jenkins/workspace/test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-85bca917f8457708.xml - 2025-12-04T11:13:50.5112907Z =========================== short test summary info ============================ 2025-12-04T11:13:50.5114192Z FAILED [0.4442s] inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_32_mid_dim_8_in_features_128_out_features_64_cuda_bfloat16 - RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_8_in_features_128_out_features_64_cuda_bfloat16! Caching allocator allocated memory was 16384 and is now reported as 24576 on device 0. CUDA driver allocated memory was 276824064 and is now 278921216. 2025-12-04T11:13:50.5114199Z 2025-12-04T11:13:50.5114324Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.5114844Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_8_in_features_128_out_features_64_cuda_bfloat16 2025-12-04T11:13:50.5114848Z 2025-12-04T11:13:50.5115005Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.5115111Z !!!!!!!!!!!!!!!!!!!!!!!!!! stopping after 1 failures !!!!!!!!!!!!!!!!!!!!!!!!!!! 2025-12-04T11:13:50.5115230Z ================== 1 failed, 53 deselected, 2 rerun in 2.79s =================== 2025-12-04T11:13:50.5115291Z Got exit code 1 2025-12-04T11:13:50.5115358Z Retrying single test... 2025-12-04T11:13:50.5115624Z W1204 11:10:00.828000 86660 site-packages/torch/_inductor/utils.py:1703] Not enough SMs to use max_autotune_gemm mode 2025-12-04T11:13:50.5116053Z Test results will be stored in test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-6a70392d2e5080cc.xml 2025-12-04T11:13:50.5116157Z ============================= test session starts ============================== 2025-12-04T11:13:50.5116366Z platform linux -- Python 3.10.14, pytest-7.3.2, pluggy-1.6.0 -- /opt/conda/envs/py_3.10/bin/python 2025-12-04T11:13:50.5116432Z cachedir: .pytest_cache 2025-12-04T11:13:50.5116742Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow] 2025-12-04T11:13:50.5116818Z rootdir: /var/lib/jenkins/workspace 2025-12-04T11:13:50.5116891Z configfile: pytest.ini 2025-12-04T11:13:50.5117207Z plugins: hypothesis-6.56.4, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-14.0, subtests-0.13.1, xdist-3.3.1, xdoctest-1.3.0, typeguard-4.3.0 2025-12-04T11:13:50.5117338Z collecting ... collected 58 items / 57 deselected / 1 selected 2025-12-04T11:13:50.5117908Z stepcurrent: skipping 53 already run items. Running only test/inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_32_mid_dim_8_in_features_128_out_features_64_cuda_bfloat16 2025-12-04T11:13:50.5118053Z Running 1 items in this shard 2025-12-04T11:13:50.5118057Z 2025-12-04T11:13:50.5118792Z inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_32_mid_dim_8_in_features_128_out_features_64_cuda_bfloat16 [W1204 11:10:01.916110567 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.5118797Z 2025-12-04T11:13:50.5119097Z [W1204 11:10:11.113566584 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.5119101Z 2025-12-04T11:13:50.5119433Z [W1204 11:10:11.113800689 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.5119446Z 2025-12-04T11:13:50.5119747Z [W1204 11:10:11.120069277 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.5119753Z 2025-12-04T11:13:50.5120068Z [W1204 11:10:11.120633600 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.5120072Z 2025-12-04T11:13:50.5120368Z [W1204 11:10:11.120794073 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.5120371Z 2025-12-04T11:13:50.5120660Z [W1204 11:10:11.126210483 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.5120664Z 2025-12-04T11:13:50.5120961Z [W1204 11:10:11.126724765 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.5120966Z 2025-12-04T11:13:50.5121255Z [W1204 11:10:11.126881008 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.5121260Z 2025-12-04T11:13:50.5121350Z ('RERUN', {'yellow': True}) [11.0836s] [100%] 2025-12-04T11:13:50.5122077Z inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_32_mid_dim_8_in_features_128_out_features_64_cuda_bfloat16 [W1204 11:10:12.311723430 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.5122081Z 2025-12-04T11:13:50.5122376Z [W1204 11:10:12.312302723 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.5122379Z 2025-12-04T11:13:50.5122669Z [W1204 11:10:12.312440696 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.5122715Z 2025-12-04T11:13:50.5123008Z [W1204 11:10:12.315403362 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.5123017Z 2025-12-04T11:13:50.5123305Z [W1204 11:10:12.315977024 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.5123308Z 2025-12-04T11:13:50.5123595Z [W1204 11:10:12.316113907 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.5123599Z 2025-12-04T11:13:50.5123891Z [W1204 11:10:12.320683608 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.5123894Z 2025-12-04T11:13:50.5124184Z [W1204 11:10:12.321155429 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.5124189Z 2025-12-04T11:13:50.5124482Z [W1204 11:10:12.321291572 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.5124523Z 2025-12-04T11:13:50.5124719Z ('RERUN', {'yellow': True}) [0.4175s] [100%] 2025-12-04T11:13:50.5125443Z inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_32_mid_dim_8_in_features_128_out_features_64_cuda_bfloat16 [W1204 11:10:12.725844559 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.5125447Z 2025-12-04T11:13:50.5125736Z [W1204 11:10:12.726438242 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.5125740Z 2025-12-04T11:13:50.5126067Z [W1204 11:10:12.726587636 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.5126073Z 2025-12-04T11:13:50.5126364Z [W1204 11:10:12.729557811 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.5126369Z 2025-12-04T11:13:50.5126667Z [W1204 11:10:12.730156224 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.5126670Z 2025-12-04T11:13:50.5126967Z [W1204 11:10:12.730300048 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.5126971Z 2025-12-04T11:13:50.5127259Z [W1204 11:10:12.734837358 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.5127262Z 2025-12-04T11:13:50.5127558Z [W1204 11:10:12.735308258 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.5127563Z 2025-12-04T11:13:50.5127850Z [W1204 11:10:12.735444252 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.5127855Z 2025-12-04T11:13:50.5127924Z FAILED [0.4146s] [100%] 2025-12-04T11:13:50.5127928Z 2025-12-04T11:13:50.5128025Z ==================================== RERUNS ==================================== 2025-12-04T11:13:50.5128325Z _ TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_8_in_features_128_out_features_64_cuda_bfloat16 _ 2025-12-04T11:13:50.5128407Z Traceback (most recent call last): 2025-12-04T11:13:50.5128722Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.5128794Z method(*args, **kwargs) 2025-12-04T11:13:50.5129091Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.5129157Z method(*args, **kwargs) 2025-12-04T11:13:50.5129516Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T11:13:50.5129581Z with policy(): 2025-12-04T11:13:50.5129882Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T11:13:50.5129953Z raise RuntimeError(msg) 2025-12-04T11:13:50.5130754Z RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_8_in_features_128_out_features_64_cuda_bfloat16! Caching allocator allocated memory was 0 and is now reported as 8192 on device 0. CUDA driver allocated memory was 230686720 and is now 274726912. 2025-12-04T11:13:50.5130759Z 2025-12-04T11:13:50.5130898Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.5131422Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_8_in_features_128_out_features_64_cuda_bfloat16 2025-12-04T11:13:50.5131465Z 2025-12-04T11:13:50.5131630Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.5131820Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.5131919Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.5132276Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.5132409Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.5132477Z graph_break [] 2025-12-04T11:13:50.5132602Z ----------------------------- Captured stderr call ----------------------------- 2025-12-04T11:13:50.5133323Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/constant_folding.py:256: UserWarning: Unsupported unwinding pattern: Address not in range (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/profiler/unwind/unwind.cpp:219.) 2025-12-04T11:13:50.5133406Z if out == self.unknown_value: 2025-12-04T11:13:50.5133700Z _ TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_8_in_features_128_out_features_64_cuda_bfloat16 _ 2025-12-04T11:13:50.5133776Z Traceback (most recent call last): 2025-12-04T11:13:50.5134083Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.5134149Z method(*args, **kwargs) 2025-12-04T11:13:50.5134443Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.5134507Z method(*args, **kwargs) 2025-12-04T11:13:50.5134804Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T11:13:50.5134875Z with policy(): 2025-12-04T11:13:50.5135176Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T11:13:50.5135254Z raise RuntimeError(msg) 2025-12-04T11:13:50.5136058Z RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_8_in_features_128_out_features_64_cuda_bfloat16! Caching allocator allocated memory was 8192 and is now reported as 16384 on device 0. CUDA driver allocated memory was 274726912 and is now 276824064. 2025-12-04T11:13:50.5136063Z 2025-12-04T11:13:50.5136195Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.5136722Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_8_in_features_128_out_features_64_cuda_bfloat16 2025-12-04T11:13:50.5136726Z 2025-12-04T11:13:50.5136932Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.5137066Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.5137164Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.5137511Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.5137645Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.5137707Z graph_break [] 2025-12-04T11:13:50.5137838Z ----------------------------- Captured stderr call ----------------------------- 2025-12-04T11:13:50.5138524Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/constant_folding.py:256: UserWarning: Unsupported unwinding pattern: Address not in range (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/profiler/unwind/unwind.cpp:219.) 2025-12-04T11:13:50.5138599Z if out == self.unknown_value: 2025-12-04T11:13:50.5138728Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.5138860Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.5139512Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.5139869Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.5139931Z graph_break [] 2025-12-04T11:13:50.5140019Z =================================== FAILURES =================================== 2025-12-04T11:13:50.5140317Z _ TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_8_in_features_128_out_features_64_cuda_bfloat16 _ 2025-12-04T11:13:50.5140393Z Traceback (most recent call last): 2025-12-04T11:13:50.5140744Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.5140814Z method(*args, **kwargs) 2025-12-04T11:13:50.5141110Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.5141177Z method(*args, **kwargs) 2025-12-04T11:13:50.5141470Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T11:13:50.5141537Z with policy(): 2025-12-04T11:13:50.5141831Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T11:13:50.5141899Z raise RuntimeError(msg) 2025-12-04T11:13:50.5142716Z RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_8_in_features_128_out_features_64_cuda_bfloat16! Caching allocator allocated memory was 16384 and is now reported as 24576 on device 0. CUDA driver allocated memory was 276824064 and is now 278921216. 2025-12-04T11:13:50.5142722Z 2025-12-04T11:13:50.5142853Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.5143380Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_8_in_features_128_out_features_64_cuda_bfloat16 2025-12-04T11:13:50.5143384Z 2025-12-04T11:13:50.5143596Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.5143730Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.5143824Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.5144176Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.5144308Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.5144423Z graph_break [] 2025-12-04T11:13:50.5144553Z ----------------------------- Captured stderr call ----------------------------- 2025-12-04T11:13:50.5145244Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/constant_folding.py:256: UserWarning: Unsupported unwinding pattern: Address not in range (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/profiler/unwind/unwind.cpp:219.) 2025-12-04T11:13:50.5145319Z if out == self.unknown_value: 2025-12-04T11:13:50.5145442Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.5145535Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.5145679Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.5146023Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.5146092Z graph_break [] 2025-12-04T11:13:50.5146221Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.5146355Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.5146516Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.5146857Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.5146923Z graph_break [] 2025-12-04T11:13:50.5147412Z - generated xml file: /var/lib/jenkins/workspace/test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-6a70392d2e5080cc.xml - 2025-12-04T11:13:50.5147516Z =========================== short test summary info ============================ 2025-12-04T11:13:50.5148843Z FAILED [0.4146s] inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_32_mid_dim_8_in_features_128_out_features_64_cuda_bfloat16 - RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_8_in_features_128_out_features_64_cuda_bfloat16! Caching allocator allocated memory was 16384 and is now reported as 24576 on device 0. CUDA driver allocated memory was 276824064 and is now 278921216. 2025-12-04T11:13:50.5148851Z 2025-12-04T11:13:50.5148980Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.5149505Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_8_in_features_128_out_features_64_cuda_bfloat16 2025-12-04T11:13:50.5149509Z 2025-12-04T11:13:50.5149667Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.5149782Z !!!!!!!!!!!!!!!!!!!!!!!!!! stopping after 1 failures !!!!!!!!!!!!!!!!!!!!!!!!!!! 2025-12-04T11:13:50.5149903Z ================== 1 failed, 57 deselected, 2 rerun in 11.94s ================== 2025-12-04T11:13:50.5149967Z Got exit code 1 2025-12-04T11:13:50.5150038Z Retrying single test... 2025-12-04T11:13:50.5150308Z W1204 11:10:19.578000 86846 site-packages/torch/_inductor/utils.py:1703] Not enough SMs to use max_autotune_gemm mode 2025-12-04T11:13:50.5150692Z Test results will be stored in test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-1b506e26b1410ad4.xml 2025-12-04T11:13:50.5150803Z ============================= test session starts ============================== 2025-12-04T11:13:50.5151016Z platform linux -- Python 3.10.14, pytest-7.3.2, pluggy-1.6.0 -- /opt/conda/envs/py_3.10/bin/python 2025-12-04T11:13:50.5151093Z cachedir: .pytest_cache 2025-12-04T11:13:50.5151401Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow] 2025-12-04T11:13:50.5151482Z rootdir: /var/lib/jenkins/workspace 2025-12-04T11:13:50.5151601Z configfile: pytest.ini 2025-12-04T11:13:50.5151920Z plugins: hypothesis-6.56.4, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-14.0, subtests-0.13.1, xdist-3.3.1, xdoctest-1.3.0, typeguard-4.3.0 2025-12-04T11:13:50.5152064Z collecting ... collected 58 items / 57 deselected / 1 selected 2025-12-04T11:13:50.5152645Z stepcurrent: skipping 53 already run items. Running only test/inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_32_mid_dim_8_in_features_128_out_features_64_cuda_bfloat16 2025-12-04T11:13:50.5152720Z Running 1 items in this shard 2025-12-04T11:13:50.5152724Z 2025-12-04T11:13:50.5153464Z inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_32_mid_dim_8_in_features_128_out_features_64_cuda_bfloat16 [W1204 11:10:20.675884651 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.5153469Z 2025-12-04T11:13:50.5153772Z [W1204 11:10:29.798792690 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.5153815Z 2025-12-04T11:13:50.5154147Z [W1204 11:10:29.799039586 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.5154152Z 2025-12-04T11:13:50.5154442Z [W1204 11:10:29.804719262 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.5154445Z 2025-12-04T11:13:50.5154741Z [W1204 11:10:29.805255073 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.5154745Z 2025-12-04T11:13:50.5155030Z [W1204 11:10:29.805417047 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.5155068Z 2025-12-04T11:13:50.5155364Z [W1204 11:10:29.810745475 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.5155371Z 2025-12-04T11:13:50.5155663Z [W1204 11:10:29.811277117 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.5155667Z 2025-12-04T11:13:50.5155954Z [W1204 11:10:29.811430300 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.5155962Z 2025-12-04T11:13:50.5156048Z ('RERUN', {'yellow': True}) [11.0173s] [100%] 2025-12-04T11:13:50.5156768Z inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_32_mid_dim_8_in_features_128_out_features_64_cuda_bfloat16 [W1204 11:10:31.000747761 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.5156772Z 2025-12-04T11:13:50.5157070Z [W1204 11:10:31.001325534 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.5157076Z 2025-12-04T11:13:50.5157366Z [W1204 11:10:31.001472567 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.5157369Z 2025-12-04T11:13:50.5157658Z [W1204 11:10:31.004455293 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.5157661Z 2025-12-04T11:13:50.5157950Z [W1204 11:10:31.005050937 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.5157954Z 2025-12-04T11:13:50.5158251Z [W1204 11:10:31.005193650 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.5158255Z 2025-12-04T11:13:50.5158544Z [W1204 11:10:31.009771751 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.5158590Z 2025-12-04T11:13:50.5158880Z [W1204 11:10:31.010285082 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.5158887Z 2025-12-04T11:13:50.5159176Z [W1204 11:10:31.010428565 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.5159179Z 2025-12-04T11:13:50.5159260Z ('RERUN', {'yellow': True}) [0.4216s] [100%] 2025-12-04T11:13:50.5159987Z inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_32_mid_dim_8_in_features_128_out_features_64_cuda_bfloat16 [W1204 11:10:31.421042117 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.5159993Z 2025-12-04T11:13:50.5160284Z [W1204 11:10:31.421612219 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.5160350Z 2025-12-04T11:13:50.5160681Z [W1204 11:10:31.421761542 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.5160685Z 2025-12-04T11:13:50.5160973Z [W1204 11:10:31.424736828 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.5160977Z 2025-12-04T11:13:50.5161273Z [W1204 11:10:31.425327391 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.5161276Z 2025-12-04T11:13:50.5161566Z [W1204 11:10:31.425472134 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.5161569Z 2025-12-04T11:13:50.5161894Z [W1204 11:10:31.430083925 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.5161901Z 2025-12-04T11:13:50.5162195Z [W1204 11:10:31.430569656 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.5162199Z 2025-12-04T11:13:50.5162485Z [W1204 11:10:31.430707959 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.5162493Z 2025-12-04T11:13:50.5162558Z FAILED [0.4206s] [100%] 2025-12-04T11:13:50.5162561Z 2025-12-04T11:13:50.5162649Z ==================================== RERUNS ==================================== 2025-12-04T11:13:50.5162953Z _ TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_8_in_features_128_out_features_64_cuda_bfloat16 _ 2025-12-04T11:13:50.5163030Z Traceback (most recent call last): 2025-12-04T11:13:50.5163344Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.5163419Z method(*args, **kwargs) 2025-12-04T11:13:50.5163719Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.5163794Z method(*args, **kwargs) 2025-12-04T11:13:50.5164096Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T11:13:50.5164160Z with policy(): 2025-12-04T11:13:50.5164461Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T11:13:50.5164530Z raise RuntimeError(msg) 2025-12-04T11:13:50.5165334Z RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_8_in_features_128_out_features_64_cuda_bfloat16! Caching allocator allocated memory was 0 and is now reported as 8192 on device 0. CUDA driver allocated memory was 230686720 and is now 274726912. 2025-12-04T11:13:50.5165383Z 2025-12-04T11:13:50.5165516Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.5166037Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_8_in_features_128_out_features_64_cuda_bfloat16 2025-12-04T11:13:50.5166041Z 2025-12-04T11:13:50.5166207Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.5166336Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.5166442Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.5166800Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.5166929Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.5166997Z graph_break [] 2025-12-04T11:13:50.5167127Z ----------------------------- Captured stderr call ----------------------------- 2025-12-04T11:13:50.5167896Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/constant_folding.py:256: UserWarning: Unsupported unwinding pattern: Address not in range (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/profiler/unwind/unwind.cpp:219.) 2025-12-04T11:13:50.5167971Z if out == self.unknown_value: 2025-12-04T11:13:50.5168264Z _ TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_8_in_features_128_out_features_64_cuda_bfloat16 _ 2025-12-04T11:13:50.5168347Z Traceback (most recent call last): 2025-12-04T11:13:50.5168647Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.5168713Z method(*args, **kwargs) 2025-12-04T11:13:50.5169057Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.5169132Z method(*args, **kwargs) 2025-12-04T11:13:50.5169431Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T11:13:50.5169494Z with policy(): 2025-12-04T11:13:50.5169793Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T11:13:50.5169867Z raise RuntimeError(msg) 2025-12-04T11:13:50.5170676Z RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_8_in_features_128_out_features_64_cuda_bfloat16! Caching allocator allocated memory was 8192 and is now reported as 16384 on device 0. CUDA driver allocated memory was 274726912 and is now 276824064. 2025-12-04T11:13:50.5170679Z 2025-12-04T11:13:50.5170817Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.5171337Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_8_in_features_128_out_features_64_cuda_bfloat16 2025-12-04T11:13:50.5171344Z 2025-12-04T11:13:50.5171511Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.5171636Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.5171732Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.5172087Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.5172218Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.5172278Z graph_break [] 2025-12-04T11:13:50.5172412Z ----------------------------- Captured stderr call ----------------------------- 2025-12-04T11:13:50.5173157Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/constant_folding.py:256: UserWarning: Unsupported unwinding pattern: Address not in range (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/profiler/unwind/unwind.cpp:219.) 2025-12-04T11:13:50.5173239Z if out == self.unknown_value: 2025-12-04T11:13:50.5173363Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.5173458Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.5173586Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.5173929Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.5173995Z graph_break [] 2025-12-04T11:13:50.5174081Z =================================== FAILURES =================================== 2025-12-04T11:13:50.5174371Z _ TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_8_in_features_128_out_features_64_cuda_bfloat16 _ 2025-12-04T11:13:50.5174493Z Traceback (most recent call last): 2025-12-04T11:13:50.5174828Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.5174895Z method(*args, **kwargs) 2025-12-04T11:13:50.5175195Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.5175259Z method(*args, **kwargs) 2025-12-04T11:13:50.5175560Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T11:13:50.5175621Z with policy(): 2025-12-04T11:13:50.5175949Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T11:13:50.5176024Z raise RuntimeError(msg) 2025-12-04T11:13:50.5176838Z RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_8_in_features_128_out_features_64_cuda_bfloat16! Caching allocator allocated memory was 16384 and is now reported as 24576 on device 0. CUDA driver allocated memory was 276824064 and is now 278921216. 2025-12-04T11:13:50.5176843Z 2025-12-04T11:13:50.5176989Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.5177508Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_8_in_features_128_out_features_64_cuda_bfloat16 2025-12-04T11:13:50.5177512Z 2025-12-04T11:13:50.5177670Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.5177953Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.5178056Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.5178415Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.5178545Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.5178605Z graph_break [] 2025-12-04T11:13:50.5178736Z ----------------------------- Captured stderr call ----------------------------- 2025-12-04T11:13:50.5179436Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/constant_folding.py:256: UserWarning: Unsupported unwinding pattern: Address not in range (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/profiler/unwind/unwind.cpp:219.) 2025-12-04T11:13:50.5179515Z if out == self.unknown_value: 2025-12-04T11:13:50.5179641Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.5179735Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.5179940Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.5180289Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.5180353Z graph_break [] 2025-12-04T11:13:50.5180484Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.5180574Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.5180706Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.5181050Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.5181110Z graph_break [] 2025-12-04T11:13:50.5181606Z - generated xml file: /var/lib/jenkins/workspace/test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-1b506e26b1410ad4.xml - 2025-12-04T11:13:50.5181710Z =========================== short test summary info ============================ 2025-12-04T11:13:50.5183113Z FAILED [0.4206s] inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_32_mid_dim_8_in_features_128_out_features_64_cuda_bfloat16 - RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_8_in_features_128_out_features_64_cuda_bfloat16! Caching allocator allocated memory was 16384 and is now reported as 24576 on device 0. CUDA driver allocated memory was 276824064 and is now 278921216. 2025-12-04T11:13:50.5183118Z 2025-12-04T11:13:50.5183250Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.5183871Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_8_in_features_128_out_features_64_cuda_bfloat16 2025-12-04T11:13:50.5183878Z 2025-12-04T11:13:50.5184038Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.5184150Z !!!!!!!!!!!!!!!!!!!!!!!!!! stopping after 1 failures !!!!!!!!!!!!!!!!!!!!!!!!!!! 2025-12-04T11:13:50.5184278Z ================== 1 failed, 57 deselected, 2 rerun in 11.88s ================== 2025-12-04T11:13:50.5184340Z Got exit code 1 2025-12-04T11:13:50.5184821Z FAILED CONSISTENTLY: test/inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_32_mid_dim_8_in_features_128_out_features_64_cuda_bfloat16 2025-12-04T11:13:50.5185068Z Test failed consistently, continuing with the rest of the tests due to continue-through-error being set 2025-12-04T11:13:50.5185338Z W1204 11:10:38.236000 87032 site-packages/torch/_inductor/utils.py:1703] Not enough SMs to use max_autotune_gemm mode 2025-12-04T11:13:50.5185738Z Test results will be stored in test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-f7f9c3afca4dce49.xml 2025-12-04T11:13:50.5185837Z ============================= test session starts ============================== 2025-12-04T11:13:50.5186058Z platform linux -- Python 3.10.14, pytest-7.3.2, pluggy-1.6.0 -- /opt/conda/envs/py_3.10/bin/python 2025-12-04T11:13:50.5186129Z cachedir: .pytest_cache 2025-12-04T11:13:50.5186437Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow] 2025-12-04T11:13:50.5186521Z rootdir: /var/lib/jenkins/workspace 2025-12-04T11:13:50.5186589Z configfile: pytest.ini 2025-12-04T11:13:50.5186906Z plugins: hypothesis-6.56.4, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-14.0, subtests-0.13.1, xdist-3.3.1, xdoctest-1.3.0, typeguard-4.3.0 2025-12-04T11:13:50.5187042Z collecting ... collected 58 items / 54 deselected / 4 selected 2025-12-04T11:13:50.5187134Z stepcurrent: skipping 54 already run items. 2025-12-04T11:13:50.5187215Z Running 4 items in this shard 2025-12-04T11:13:50.5187280Z 2025-12-04T11:13:50.5187775Z inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_32_mid_dim_8_in_features_128_out_features_65_cuda_bfloat16 ('RERUN', {'yellow': True}) [1.9352s] [ 25%] 2025-12-04T11:13:50.5188266Z inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_32_mid_dim_8_in_features_128_out_features_65_cuda_bfloat16 ('RERUN', {'yellow': True}) [0.4636s] [ 25%] 2025-12-04T11:13:50.5188718Z inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_32_mid_dim_8_in_features_128_out_features_65_cuda_bfloat16 FAILED [0.4441s] [ 25%] 2025-12-04T11:13:50.5188722Z 2025-12-04T11:13:50.5188808Z ==================================== RERUNS ==================================== 2025-12-04T11:13:50.5189111Z _ TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_8_in_features_128_out_features_65_cuda_bfloat16 _ 2025-12-04T11:13:50.5189203Z Traceback (most recent call last): 2025-12-04T11:13:50.5189515Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.5189625Z method(*args, **kwargs) 2025-12-04T11:13:50.5189953Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.5190027Z method(*args, **kwargs) 2025-12-04T11:13:50.5190321Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T11:13:50.5190384Z with policy(): 2025-12-04T11:13:50.5190686Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T11:13:50.5190758Z raise RuntimeError(msg) 2025-12-04T11:13:50.5191595Z RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_8_in_features_128_out_features_65_cuda_bfloat16! Caching allocator allocated memory was 0 and is now reported as 8704 on device 0. CUDA driver allocated memory was 230686720 and is now 274726912. 2025-12-04T11:13:50.5191604Z 2025-12-04T11:13:50.5191737Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.5192257Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_8_in_features_128_out_features_65_cuda_bfloat16 2025-12-04T11:13:50.5192266Z 2025-12-04T11:13:50.5192429Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.5192559Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.5192663Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.5193020Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.5193151Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.5193219Z graph_break [] 2025-12-04T11:13:50.5193513Z _ TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_8_in_features_128_out_features_65_cuda_bfloat16 _ 2025-12-04T11:13:50.5193597Z Traceback (most recent call last): 2025-12-04T11:13:50.5193899Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.5193965Z method(*args, **kwargs) 2025-12-04T11:13:50.5194263Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.5194332Z method(*args, **kwargs) 2025-12-04T11:13:50.5194625Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T11:13:50.5194730Z with policy(): 2025-12-04T11:13:50.5195026Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T11:13:50.5195101Z raise RuntimeError(msg) 2025-12-04T11:13:50.5195912Z RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_8_in_features_128_out_features_65_cuda_bfloat16! Caching allocator allocated memory was 8704 and is now reported as 17408 on device 0. CUDA driver allocated memory was 274726912 and is now 276824064. 2025-12-04T11:13:50.5195917Z 2025-12-04T11:13:50.5196049Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.5196573Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_8_in_features_128_out_features_65_cuda_bfloat16 2025-12-04T11:13:50.5196577Z 2025-12-04T11:13:50.5196736Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.5196905Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.5197037Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.5197386Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.5197530Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.5197593Z graph_break [] 2025-12-04T11:13:50.5197726Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.5197819Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.5197943Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.5198325Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.5198389Z graph_break [] 2025-12-04T11:13:50.5198481Z =================================== FAILURES =================================== 2025-12-04T11:13:50.5198774Z _ TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_8_in_features_128_out_features_65_cuda_bfloat16 _ 2025-12-04T11:13:50.5198852Z Traceback (most recent call last): 2025-12-04T11:13:50.5199154Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.5199222Z method(*args, **kwargs) 2025-12-04T11:13:50.5199521Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.5199590Z method(*args, **kwargs) 2025-12-04T11:13:50.5199880Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T11:13:50.5199948Z with policy(): 2025-12-04T11:13:50.5200242Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T11:13:50.5200311Z raise RuntimeError(msg) 2025-12-04T11:13:50.5201126Z RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_8_in_features_128_out_features_65_cuda_bfloat16! Caching allocator allocated memory was 17408 and is now reported as 26112 on device 0. CUDA driver allocated memory was 276824064 and is now 278921216. 2025-12-04T11:13:50.5201130Z 2025-12-04T11:13:50.5201257Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.5201780Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_8_in_features_128_out_features_65_cuda_bfloat16 2025-12-04T11:13:50.5201783Z 2025-12-04T11:13:50.5201992Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.5202121Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.5202218Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.5202565Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.5202704Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.5202766Z graph_break [] 2025-12-04T11:13:50.5202901Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.5203002Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.5203131Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.5203481Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.5203582Z graph_break [] 2025-12-04T11:13:50.5203710Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.5203843Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.5203969Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.5204313Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.5204383Z graph_break [] 2025-12-04T11:13:50.5204875Z - generated xml file: /var/lib/jenkins/workspace/test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-f7f9c3afca4dce49.xml - 2025-12-04T11:13:50.5205017Z =========================== short test summary info ============================ 2025-12-04T11:13:50.5206293Z FAILED [0.4441s] inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_32_mid_dim_8_in_features_128_out_features_65_cuda_bfloat16 - RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_8_in_features_128_out_features_65_cuda_bfloat16! Caching allocator allocated memory was 17408 and is now reported as 26112 on device 0. CUDA driver allocated memory was 276824064 and is now 278921216. 2025-12-04T11:13:50.5206301Z 2025-12-04T11:13:50.5206437Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.5206955Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_8_in_features_128_out_features_65_cuda_bfloat16 2025-12-04T11:13:50.5206959Z 2025-12-04T11:13:50.5207132Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.5207248Z !!!!!!!!!!!!!!!!!!!!!!!!!! stopping after 1 failures !!!!!!!!!!!!!!!!!!!!!!!!!!! 2025-12-04T11:13:50.5207373Z ================== 1 failed, 54 deselected, 2 rerun in 2.87s =================== 2025-12-04T11:13:50.5207439Z Got exit code 1 2025-12-04T11:13:50.5207512Z Retrying single test... 2025-12-04T11:13:50.5207780Z W1204 11:10:48.222000 87213 site-packages/torch/_inductor/utils.py:1703] Not enough SMs to use max_autotune_gemm mode 2025-12-04T11:13:50.5208175Z Test results will be stored in test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-c41d68e69339995e.xml 2025-12-04T11:13:50.5208274Z ============================= test session starts ============================== 2025-12-04T11:13:50.5208491Z platform linux -- Python 3.10.14, pytest-7.3.2, pluggy-1.6.0 -- /opt/conda/envs/py_3.10/bin/python 2025-12-04T11:13:50.5208559Z cachedir: .pytest_cache 2025-12-04T11:13:50.5208866Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow] 2025-12-04T11:13:50.5208987Z rootdir: /var/lib/jenkins/workspace 2025-12-04T11:13:50.5209056Z configfile: pytest.ini 2025-12-04T11:13:50.5209380Z plugins: hypothesis-6.56.4, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-14.0, subtests-0.13.1, xdist-3.3.1, xdoctest-1.3.0, typeguard-4.3.0 2025-12-04T11:13:50.5209525Z collecting ... collected 58 items / 57 deselected / 1 selected 2025-12-04T11:13:50.5210094Z stepcurrent: skipping 54 already run items. Running only test/inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_32_mid_dim_8_in_features_128_out_features_65_cuda_bfloat16 2025-12-04T11:13:50.5210174Z Running 1 items in this shard 2025-12-04T11:13:50.5210178Z 2025-12-04T11:13:50.5210914Z inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_32_mid_dim_8_in_features_128_out_features_65_cuda_bfloat16 [W1204 11:10:49.548289436 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.5210957Z 2025-12-04T11:13:50.5211266Z [W1204 11:10:58.857023251 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.5211304Z 2025-12-04T11:13:50.5211597Z [W1204 11:10:58.857263206 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.5211600Z 2025-12-04T11:13:50.5211891Z [W1204 11:10:58.863000003 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.5211901Z 2025-12-04T11:13:50.5212190Z [W1204 11:10:58.863556935 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.5212193Z 2025-12-04T11:13:50.5212533Z [W1204 11:10:58.863724599 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.5212538Z 2025-12-04T11:13:50.5212840Z [W1204 11:10:58.869162448 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.5212844Z 2025-12-04T11:13:50.5213135Z [W1204 11:10:58.869695290 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.5213138Z 2025-12-04T11:13:50.5213431Z [W1204 11:10:58.869856133 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.5213435Z 2025-12-04T11:13:50.5213521Z ('RERUN', {'yellow': True}) [11.2487s] [100%] 2025-12-04T11:13:50.5214247Z inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_32_mid_dim_8_in_features_128_out_features_65_cuda_bfloat16 [W1204 11:10:59.872811005 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.5214252Z 2025-12-04T11:13:50.5214543Z [W1204 11:10:59.873396608 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.5214548Z 2025-12-04T11:13:50.5214835Z [W1204 11:10:59.873538111 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.5214844Z 2025-12-04T11:13:50.5215134Z [W1204 11:10:59.876507287 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.5215138Z 2025-12-04T11:13:50.5215426Z [W1204 11:10:59.877092810 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.5215430Z 2025-12-04T11:13:50.5215725Z [W1204 11:10:59.877231673 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.5215766Z 2025-12-04T11:13:50.5216056Z [W1204 11:10:59.881840985 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.5216061Z 2025-12-04T11:13:50.5216355Z [W1204 11:10:59.882320635 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.5216359Z 2025-12-04T11:13:50.5216649Z [W1204 11:10:59.882456178 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.5216652Z 2025-12-04T11:13:50.5216742Z ('RERUN', {'yellow': True}) [0.4235s] [100%] 2025-12-04T11:13:50.5217465Z inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_32_mid_dim_8_in_features_128_out_features_65_cuda_bfloat16 [W1204 11:11:00.293990831 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.5217470Z 2025-12-04T11:13:50.5217769Z [W1204 11:11:00.294568583 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.5217809Z 2025-12-04T11:13:50.5218136Z [W1204 11:11:00.294706596 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.5218140Z 2025-12-04T11:13:50.5218432Z [W1204 11:11:00.297661572 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.5218441Z 2025-12-04T11:13:50.5218737Z [W1204 11:11:00.298230704 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.5218740Z 2025-12-04T11:13:50.5219060Z [W1204 11:11:00.298367227 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.5219065Z 2025-12-04T11:13:50.5219377Z [W1204 11:11:00.302989440 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.5219382Z 2025-12-04T11:13:50.5219671Z [W1204 11:11:00.303465870 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.5219674Z 2025-12-04T11:13:50.5219966Z [W1204 11:11:00.303599553 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.5219969Z 2025-12-04T11:13:50.5220032Z FAILED [0.4196s] [100%] 2025-12-04T11:13:50.5220036Z 2025-12-04T11:13:50.5220128Z ==================================== RERUNS ==================================== 2025-12-04T11:13:50.5220425Z _ TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_8_in_features_128_out_features_65_cuda_bfloat16 _ 2025-12-04T11:13:50.5220503Z Traceback (most recent call last): 2025-12-04T11:13:50.5220821Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.5220892Z method(*args, **kwargs) 2025-12-04T11:13:50.5221191Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.5221262Z method(*args, **kwargs) 2025-12-04T11:13:50.5221558Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T11:13:50.5221625Z with policy(): 2025-12-04T11:13:50.5221924Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T11:13:50.5221992Z raise RuntimeError(msg) 2025-12-04T11:13:50.5222799Z RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_8_in_features_128_out_features_65_cuda_bfloat16! Caching allocator allocated memory was 0 and is now reported as 8704 on device 0. CUDA driver allocated memory was 230686720 and is now 274726912. 2025-12-04T11:13:50.5222852Z 2025-12-04T11:13:50.5222989Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.5223552Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_8_in_features_128_out_features_65_cuda_bfloat16 2025-12-04T11:13:50.5223556Z 2025-12-04T11:13:50.5223718Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.5223851Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.5223956Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.5224308Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.5224443Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.5224546Z graph_break [] 2025-12-04T11:13:50.5224708Z ----------------------------- Captured stderr call ----------------------------- 2025-12-04T11:13:50.5225409Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/constant_folding.py:256: UserWarning: Unsupported unwinding pattern: Address not in range (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/profiler/unwind/unwind.cpp:219.) 2025-12-04T11:13:50.5225482Z if out == self.unknown_value: 2025-12-04T11:13:50.5225780Z _ TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_8_in_features_128_out_features_65_cuda_bfloat16 _ 2025-12-04T11:13:50.5225858Z Traceback (most recent call last): 2025-12-04T11:13:50.5226192Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.5226266Z method(*args, **kwargs) 2025-12-04T11:13:50.5226560Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.5226633Z method(*args, **kwargs) 2025-12-04T11:13:50.5226929Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T11:13:50.5226991Z with policy(): 2025-12-04T11:13:50.5227294Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T11:13:50.5227363Z raise RuntimeError(msg) 2025-12-04T11:13:50.5228167Z RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_8_in_features_128_out_features_65_cuda_bfloat16! Caching allocator allocated memory was 8704 and is now reported as 17408 on device 0. CUDA driver allocated memory was 274726912 and is now 276824064. 2025-12-04T11:13:50.5228179Z 2025-12-04T11:13:50.5228312Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.5228834Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_8_in_features_128_out_features_65_cuda_bfloat16 2025-12-04T11:13:50.5228838Z 2025-12-04T11:13:50.5229010Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.5229137Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.5229245Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.5229612Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.5229746Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.5229813Z graph_break [] 2025-12-04T11:13:50.5229984Z ----------------------------- Captured stderr call ----------------------------- 2025-12-04T11:13:50.5230683Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/constant_folding.py:256: UserWarning: Unsupported unwinding pattern: Address not in range (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/profiler/unwind/unwind.cpp:219.) 2025-12-04T11:13:50.5230762Z if out == self.unknown_value: 2025-12-04T11:13:50.5230888Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.5230987Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.5231114Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.5231458Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.5231526Z graph_break [] 2025-12-04T11:13:50.5231616Z =================================== FAILURES =================================== 2025-12-04T11:13:50.5231954Z _ TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_8_in_features_128_out_features_65_cuda_bfloat16 _ 2025-12-04T11:13:50.5232068Z Traceback (most recent call last): 2025-12-04T11:13:50.5232374Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.5232453Z method(*args, **kwargs) 2025-12-04T11:13:50.5232750Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.5232818Z method(*args, **kwargs) 2025-12-04T11:13:50.5233120Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T11:13:50.5233182Z with policy(): 2025-12-04T11:13:50.5233653Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T11:13:50.5233778Z raise RuntimeError(msg) 2025-12-04T11:13:50.5234627Z RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_8_in_features_128_out_features_65_cuda_bfloat16! Caching allocator allocated memory was 17408 and is now reported as 26112 on device 0. CUDA driver allocated memory was 276824064 and is now 278921216. 2025-12-04T11:13:50.5234666Z 2025-12-04T11:13:50.5234829Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.5235490Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_8_in_features_128_out_features_65_cuda_bfloat16 2025-12-04T11:13:50.5235494Z 2025-12-04T11:13:50.5235710Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.5235928Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.5236059Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.5236443Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.5236625Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.5236776Z graph_break [] 2025-12-04T11:13:50.5237003Z ----------------------------- Captured stderr call ----------------------------- 2025-12-04T11:13:50.5237724Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/constant_folding.py:256: UserWarning: Unsupported unwinding pattern: Address not in range (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/profiler/unwind/unwind.cpp:219.) 2025-12-04T11:13:50.5237830Z if out == self.unknown_value: 2025-12-04T11:13:50.5243533Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.5243748Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.5243947Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.5244405Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.5244470Z graph_break [] 2025-12-04T11:13:50.5244617Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.5244716Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.5244844Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.5245203Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.5245268Z graph_break [] 2025-12-04T11:13:50.5245774Z - generated xml file: /var/lib/jenkins/workspace/test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-c41d68e69339995e.xml - 2025-12-04T11:13:50.5245958Z =========================== short test summary info ============================ 2025-12-04T11:13:50.5247292Z FAILED [0.4196s] inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_32_mid_dim_8_in_features_128_out_features_65_cuda_bfloat16 - RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_8_in_features_128_out_features_65_cuda_bfloat16! Caching allocator allocated memory was 17408 and is now reported as 26112 on device 0. CUDA driver allocated memory was 276824064 and is now 278921216. 2025-12-04T11:13:50.5247302Z 2025-12-04T11:13:50.5247441Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.5248000Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_8_in_features_128_out_features_65_cuda_bfloat16 2025-12-04T11:13:50.5248008Z 2025-12-04T11:13:50.5248177Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.5248291Z !!!!!!!!!!!!!!!!!!!!!!!!!! stopping after 1 failures !!!!!!!!!!!!!!!!!!!!!!!!!!! 2025-12-04T11:13:50.5248418Z ================== 1 failed, 57 deselected, 2 rerun in 12.12s ================== 2025-12-04T11:13:50.5248490Z Got exit code 1 2025-12-04T11:13:50.5248558Z Retrying single test... 2025-12-04T11:13:50.5248834Z W1204 11:11:07.189000 87399 site-packages/torch/_inductor/utils.py:1703] Not enough SMs to use max_autotune_gemm mode 2025-12-04T11:13:50.5249225Z Test results will be stored in test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-dfa47b8dd194f01b.xml 2025-12-04T11:13:50.5249327Z ============================= test session starts ============================== 2025-12-04T11:13:50.5249556Z platform linux -- Python 3.10.14, pytest-7.3.2, pluggy-1.6.0 -- /opt/conda/envs/py_3.10/bin/python 2025-12-04T11:13:50.5249627Z cachedir: .pytest_cache 2025-12-04T11:13:50.5249941Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow] 2025-12-04T11:13:50.5250021Z rootdir: /var/lib/jenkins/workspace 2025-12-04T11:13:50.5250090Z configfile: pytest.ini 2025-12-04T11:13:50.5250412Z plugins: hypothesis-6.56.4, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-14.0, subtests-0.13.1, xdist-3.3.1, xdoctest-1.3.0, typeguard-4.3.0 2025-12-04T11:13:50.5250547Z collecting ... collected 58 items / 57 deselected / 1 selected 2025-12-04T11:13:50.5251130Z stepcurrent: skipping 54 already run items. Running only test/inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_32_mid_dim_8_in_features_128_out_features_65_cuda_bfloat16 2025-12-04T11:13:50.5251201Z Running 1 items in this shard 2025-12-04T11:13:50.5251247Z 2025-12-04T11:13:50.5251980Z inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_32_mid_dim_8_in_features_128_out_features_65_cuda_bfloat16 [W1204 11:11:08.488356368 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.5251985Z 2025-12-04T11:13:50.5252286Z [W1204 11:11:17.686621099 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.5252291Z 2025-12-04T11:13:50.5252577Z [W1204 11:11:17.686861254 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.5252580Z 2025-12-04T11:13:50.5252872Z [W1204 11:11:17.693271756 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.5252876Z 2025-12-04T11:13:50.5253161Z [W1204 11:11:17.693823418 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.5253201Z 2025-12-04T11:13:50.5253522Z [W1204 11:11:17.693987442 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.5253526Z 2025-12-04T11:13:50.5253813Z [W1204 11:11:17.699316290 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.5253816Z 2025-12-04T11:13:50.5254104Z [W1204 11:11:17.699840501 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.5254107Z 2025-12-04T11:13:50.5254395Z [W1204 11:11:17.699992285 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.5254431Z 2025-12-04T11:13:50.5254515Z ('RERUN', {'yellow': True}) [11.1129s] [100%] 2025-12-04T11:13:50.5255242Z inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_32_mid_dim_8_in_features_128_out_features_65_cuda_bfloat16 [W1204 11:11:18.700990739 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.5255247Z 2025-12-04T11:13:50.5255535Z [W1204 11:11:18.701576392 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.5255539Z 2025-12-04T11:13:50.5255828Z [W1204 11:11:18.701719075 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.5255831Z 2025-12-04T11:13:50.5256115Z [W1204 11:11:18.704721071 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.5256121Z 2025-12-04T11:13:50.5256411Z [W1204 11:11:18.705306994 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.5256417Z 2025-12-04T11:13:50.5256702Z [W1204 11:11:18.705444257 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.5256705Z 2025-12-04T11:13:50.5256992Z [W1204 11:11:18.710051398 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.5256996Z 2025-12-04T11:13:50.5257280Z [W1204 11:11:18.710533579 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.5257284Z 2025-12-04T11:13:50.5257573Z [W1204 11:11:18.710668302 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.5257576Z 2025-12-04T11:13:50.5257657Z ('RERUN', {'yellow': True}) [0.4166s] [100%] 2025-12-04T11:13:50.5258420Z inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_32_mid_dim_8_in_features_128_out_features_65_cuda_bfloat16 [W1204 11:11:19.113528037 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.5258429Z 2025-12-04T11:13:50.5258731Z [W1204 11:11:19.114107060 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.5258735Z 2025-12-04T11:13:50.5259032Z [W1204 11:11:19.114251623 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.5259035Z 2025-12-04T11:13:50.5259325Z [W1204 11:11:19.117233989 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.5259330Z 2025-12-04T11:13:50.5259623Z [W1204 11:11:19.117814051 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.5259664Z 2025-12-04T11:13:50.5260011Z [W1204 11:11:19.117949554 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.5260015Z 2025-12-04T11:13:50.5260298Z [W1204 11:11:19.122563416 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.5260302Z 2025-12-04T11:13:50.5260589Z [W1204 11:11:19.123039567 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.5260593Z 2025-12-04T11:13:50.5260881Z [W1204 11:11:19.123173970 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.5260884Z 2025-12-04T11:13:50.5260981Z FAILED [0.4106s] [100%] 2025-12-04T11:13:50.5260989Z 2025-12-04T11:13:50.5261079Z ==================================== RERUNS ==================================== 2025-12-04T11:13:50.5261376Z _ TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_8_in_features_128_out_features_65_cuda_bfloat16 _ 2025-12-04T11:13:50.5261458Z Traceback (most recent call last): 2025-12-04T11:13:50.5261774Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.5261840Z method(*args, **kwargs) 2025-12-04T11:13:50.5262139Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.5262203Z method(*args, **kwargs) 2025-12-04T11:13:50.5262502Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T11:13:50.5262562Z with policy(): 2025-12-04T11:13:50.5262856Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T11:13:50.5262927Z raise RuntimeError(msg) 2025-12-04T11:13:50.5263834Z RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_8_in_features_128_out_features_65_cuda_bfloat16! Caching allocator allocated memory was 0 and is now reported as 8704 on device 0. CUDA driver allocated memory was 230686720 and is now 274726912. 2025-12-04T11:13:50.5263839Z 2025-12-04T11:13:50.5263977Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.5264497Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_8_in_features_128_out_features_65_cuda_bfloat16 2025-12-04T11:13:50.5264501Z 2025-12-04T11:13:50.5264668Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.5264805Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.5264960Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.5265319Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.5265451Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.5265512Z graph_break [] 2025-12-04T11:13:50.5265645Z ----------------------------- Captured stderr call ----------------------------- 2025-12-04T11:13:50.5266339Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/constant_folding.py:256: UserWarning: Unsupported unwinding pattern: Address not in range (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/profiler/unwind/unwind.cpp:219.) 2025-12-04T11:13:50.5266416Z if out == self.unknown_value: 2025-12-04T11:13:50.5266708Z _ TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_8_in_features_128_out_features_65_cuda_bfloat16 _ 2025-12-04T11:13:50.5266786Z Traceback (most recent call last): 2025-12-04T11:13:50.5267163Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.5267229Z method(*args, **kwargs) 2025-12-04T11:13:50.5267521Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.5267588Z method(*args, **kwargs) 2025-12-04T11:13:50.5267872Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T11:13:50.5267936Z with policy(): 2025-12-04T11:13:50.5268227Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T11:13:50.5268327Z raise RuntimeError(msg) 2025-12-04T11:13:50.5269134Z RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_8_in_features_128_out_features_65_cuda_bfloat16! Caching allocator allocated memory was 8704 and is now reported as 17408 on device 0. CUDA driver allocated memory was 274726912 and is now 276824064. 2025-12-04T11:13:50.5269142Z 2025-12-04T11:13:50.5269269Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.5269781Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_8_in_features_128_out_features_65_cuda_bfloat16 2025-12-04T11:13:50.5269785Z 2025-12-04T11:13:50.5269942Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.5270072Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.5270169Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.5270516Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.5270650Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.5270711Z graph_break [] 2025-12-04T11:13:50.5270836Z ----------------------------- Captured stderr call ----------------------------- 2025-12-04T11:13:50.5271532Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/constant_folding.py:256: UserWarning: Unsupported unwinding pattern: Address not in range (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/profiler/unwind/unwind.cpp:219.) 2025-12-04T11:13:50.5271602Z if out == self.unknown_value: 2025-12-04T11:13:50.5271739Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.5271835Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.5271961Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.5272350Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.5272412Z graph_break [] 2025-12-04T11:13:50.5272499Z =================================== FAILURES =================================== 2025-12-04T11:13:50.5272795Z _ TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_8_in_features_128_out_features_65_cuda_bfloat16 _ 2025-12-04T11:13:50.5272870Z Traceback (most recent call last): 2025-12-04T11:13:50.5273177Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.5273241Z method(*args, **kwargs) 2025-12-04T11:13:50.5273533Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.5273598Z method(*args, **kwargs) 2025-12-04T11:13:50.5273896Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T11:13:50.5273998Z with policy(): 2025-12-04T11:13:50.5274329Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T11:13:50.5274397Z raise RuntimeError(msg) 2025-12-04T11:13:50.5275211Z RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_8_in_features_128_out_features_65_cuda_bfloat16! Caching allocator allocated memory was 17408 and is now reported as 26112 on device 0. CUDA driver allocated memory was 276824064 and is now 278921216. 2025-12-04T11:13:50.5275215Z 2025-12-04T11:13:50.5275341Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.5275898Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_8_in_features_128_out_features_65_cuda_bfloat16 2025-12-04T11:13:50.5275906Z 2025-12-04T11:13:50.5276064Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.5276194Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.5276288Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.5276632Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.5276763Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.5276821Z graph_break [] 2025-12-04T11:13:50.5276944Z ----------------------------- Captured stderr call ----------------------------- 2025-12-04T11:13:50.5277638Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/constant_folding.py:256: UserWarning: Unsupported unwinding pattern: Address not in range (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/profiler/unwind/unwind.cpp:219.) 2025-12-04T11:13:50.5277713Z if out == self.unknown_value: 2025-12-04T11:13:50.5278050Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.5278151Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.5278276Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.5278629Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.5278688Z graph_break [] 2025-12-04T11:13:50.5278813Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.5278906Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.5279034Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.5279459Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.5279526Z graph_break [] 2025-12-04T11:13:50.5280026Z - generated xml file: /var/lib/jenkins/workspace/test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-dfa47b8dd194f01b.xml - 2025-12-04T11:13:50.5280132Z =========================== short test summary info ============================ 2025-12-04T11:13:50.5281417Z FAILED [0.4106s] inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_32_mid_dim_8_in_features_128_out_features_65_cuda_bfloat16 - RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_8_in_features_128_out_features_65_cuda_bfloat16! Caching allocator allocated memory was 17408 and is now reported as 26112 on device 0. CUDA driver allocated memory was 276824064 and is now 278921216. 2025-12-04T11:13:50.5281482Z 2025-12-04T11:13:50.5281618Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.5282183Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_8_in_features_128_out_features_65_cuda_bfloat16 2025-12-04T11:13:50.5282187Z 2025-12-04T11:13:50.5282351Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.5282470Z !!!!!!!!!!!!!!!!!!!!!!!!!! stopping after 1 failures !!!!!!!!!!!!!!!!!!!!!!!!!!! 2025-12-04T11:13:50.5282589Z ================== 1 failed, 57 deselected, 2 rerun in 11.96s ================== 2025-12-04T11:13:50.5282654Z Got exit code 1 2025-12-04T11:13:50.5283187Z FAILED CONSISTENTLY: test/inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_32_mid_dim_8_in_features_128_out_features_65_cuda_bfloat16 2025-12-04T11:13:50.5283438Z Test failed consistently, continuing with the rest of the tests due to continue-through-error being set 2025-12-04T11:13:50.5283707Z W1204 11:11:25.974000 87585 site-packages/torch/_inductor/utils.py:1703] Not enough SMs to use max_autotune_gemm mode 2025-12-04T11:13:50.5284096Z Test results will be stored in test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-6b849be23abc3e95.xml 2025-12-04T11:13:50.5284196Z ============================= test session starts ============================== 2025-12-04T11:13:50.5284404Z platform linux -- Python 3.10.14, pytest-7.3.2, pluggy-1.6.0 -- /opt/conda/envs/py_3.10/bin/python 2025-12-04T11:13:50.5284475Z cachedir: .pytest_cache 2025-12-04T11:13:50.5284779Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow] 2025-12-04T11:13:50.5284858Z rootdir: /var/lib/jenkins/workspace 2025-12-04T11:13:50.5284930Z configfile: pytest.ini 2025-12-04T11:13:50.5285248Z plugins: hypothesis-6.56.4, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-14.0, subtests-0.13.1, xdist-3.3.1, xdoctest-1.3.0, typeguard-4.3.0 2025-12-04T11:13:50.5285380Z collecting ... collected 58 items / 55 deselected / 3 selected 2025-12-04T11:13:50.5285479Z stepcurrent: skipping 55 already run items. 2025-12-04T11:13:50.5285551Z Running 3 items in this shard 2025-12-04T11:13:50.5285555Z 2025-12-04T11:13:50.5286067Z inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_32_mid_dim_8_in_features_144_out_features_1024_cuda_bfloat16 ('RERUN', {'yellow': True}) [1.9127s] [ 33%] 2025-12-04T11:13:50.5286556Z inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_32_mid_dim_8_in_features_144_out_features_1024_cuda_bfloat16 ('RERUN', {'yellow': True}) [0.4955s] [ 33%] 2025-12-04T11:13:50.5287002Z inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_32_mid_dim_8_in_features_144_out_features_1024_cuda_bfloat16 FAILED [0.4950s] [ 33%] 2025-12-04T11:13:50.5287055Z 2025-12-04T11:13:50.5287140Z ==================================== RERUNS ==================================== 2025-12-04T11:13:50.5287439Z _ TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_8_in_features_144_out_features_1024_cuda_bfloat16 _ 2025-12-04T11:13:50.5287523Z Traceback (most recent call last): 2025-12-04T11:13:50.5287832Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.5287898Z method(*args, **kwargs) 2025-12-04T11:13:50.5288196Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.5288268Z method(*args, **kwargs) 2025-12-04T11:13:50.5288569Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T11:13:50.5288630Z with policy(): 2025-12-04T11:13:50.5288981Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T11:13:50.5289086Z raise RuntimeError(msg) 2025-12-04T11:13:50.5289903Z RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_8_in_features_144_out_features_1024_cuda_bfloat16! Caching allocator allocated memory was 0 and is now reported as 147456 on device 0. CUDA driver allocated memory was 230686720 and is now 274726912. 2025-12-04T11:13:50.5289908Z 2025-12-04T11:13:50.5290035Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.5290674Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_8_in_features_144_out_features_1024_cuda_bfloat16 2025-12-04T11:13:50.5290682Z 2025-12-04T11:13:50.5290840Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.5290973Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.5291070Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.5291421Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.5291548Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.5291607Z graph_break [] 2025-12-04T11:13:50.5291900Z _ TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_8_in_features_144_out_features_1024_cuda_bfloat16 _ 2025-12-04T11:13:50.5291974Z Traceback (most recent call last): 2025-12-04T11:13:50.5292284Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.5292350Z method(*args, **kwargs) 2025-12-04T11:13:50.5292639Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.5292710Z method(*args, **kwargs) 2025-12-04T11:13:50.5292996Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T11:13:50.5293056Z with policy(): 2025-12-04T11:13:50.5293351Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T11:13:50.5293417Z raise RuntimeError(msg) 2025-12-04T11:13:50.5294241Z RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_8_in_features_144_out_features_1024_cuda_bfloat16! Caching allocator allocated memory was 147456 and is now reported as 294912 on device 0. CUDA driver allocated memory was 274726912 and is now 276824064. 2025-12-04T11:13:50.5294296Z 2025-12-04T11:13:50.5294425Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.5294962Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_8_in_features_144_out_features_1024_cuda_bfloat16 2025-12-04T11:13:50.5294966Z 2025-12-04T11:13:50.5295145Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.5295282Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.5295383Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.5295748Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.5295883Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.5295946Z graph_break [] 2025-12-04T11:13:50.5296078Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.5296211Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.5296415Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.5296769Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.5296829Z graph_break [] 2025-12-04T11:13:50.5296918Z =================================== FAILURES =================================== 2025-12-04T11:13:50.5297220Z _ TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_8_in_features_144_out_features_1024_cuda_bfloat16 _ 2025-12-04T11:13:50.5297294Z Traceback (most recent call last): 2025-12-04T11:13:50.5297636Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.5297706Z method(*args, **kwargs) 2025-12-04T11:13:50.5298049Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.5298116Z method(*args, **kwargs) 2025-12-04T11:13:50.5298423Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T11:13:50.5298490Z with policy(): 2025-12-04T11:13:50.5298804Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T11:13:50.5298873Z raise RuntimeError(msg) 2025-12-04T11:13:50.5299748Z RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_8_in_features_144_out_features_1024_cuda_bfloat16! Caching allocator allocated memory was 294912 and is now reported as 442368 on device 0. CUDA driver allocated memory was 276824064 and is now 278921216. 2025-12-04T11:13:50.5299754Z 2025-12-04T11:13:50.5299888Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.5300436Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_8_in_features_144_out_features_1024_cuda_bfloat16 2025-12-04T11:13:50.5300440Z 2025-12-04T11:13:50.5300605Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.5300834Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.5301001Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.5301420Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.5301689Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.5301847Z graph_break [] 2025-12-04T11:13:50.5301990Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.5302095Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.5302228Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.5302740Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.5302804Z graph_break [] 2025-12-04T11:13:50.5302936Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.5303036Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.5303162Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.5303602Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.5303667Z graph_break [] 2025-12-04T11:13:50.5304271Z - generated xml file: /var/lib/jenkins/workspace/test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-6b849be23abc3e95.xml - 2025-12-04T11:13:50.5304381Z =========================== short test summary info ============================ 2025-12-04T11:13:50.5305810Z FAILED [0.4950s] inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_32_mid_dim_8_in_features_144_out_features_1024_cuda_bfloat16 - RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_8_in_features_144_out_features_1024_cuda_bfloat16! Caching allocator allocated memory was 294912 and is now reported as 442368 on device 0. CUDA driver allocated memory was 276824064 and is now 278921216. 2025-12-04T11:13:50.5305816Z 2025-12-04T11:13:50.5305957Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.5306500Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_8_in_features_144_out_features_1024_cuda_bfloat16 2025-12-04T11:13:50.5306505Z 2025-12-04T11:13:50.5306677Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.5306786Z !!!!!!!!!!!!!!!!!!!!!!!!!! stopping after 1 failures !!!!!!!!!!!!!!!!!!!!!!!!!!! 2025-12-04T11:13:50.5306905Z ================== 1 failed, 55 deselected, 2 rerun in 2.93s =================== 2025-12-04T11:13:50.5306971Z Got exit code 1 2025-12-04T11:13:50.5307040Z Retrying single test... 2025-12-04T11:13:50.5307341Z W1204 11:11:36.062000 87774 site-packages/torch/_inductor/utils.py:1703] Not enough SMs to use max_autotune_gemm mode 2025-12-04T11:13:50.5307775Z Test results will be stored in test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-1eeada475e93262f.xml 2025-12-04T11:13:50.5307874Z ============================= test session starts ============================== 2025-12-04T11:13:50.5308106Z platform linux -- Python 3.10.14, pytest-7.3.2, pluggy-1.6.0 -- /opt/conda/envs/py_3.10/bin/python 2025-12-04T11:13:50.5308176Z cachedir: .pytest_cache 2025-12-04T11:13:50.5308493Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow] 2025-12-04T11:13:50.5308578Z rootdir: /var/lib/jenkins/workspace 2025-12-04T11:13:50.5308646Z configfile: pytest.ini 2025-12-04T11:13:50.5308972Z plugins: hypothesis-6.56.4, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-14.0, subtests-0.13.1, xdist-3.3.1, xdoctest-1.3.0, typeguard-4.3.0 2025-12-04T11:13:50.5309109Z collecting ... collected 58 items / 57 deselected / 1 selected 2025-12-04T11:13:50.5309697Z stepcurrent: skipping 55 already run items. Running only test/inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_32_mid_dim_8_in_features_144_out_features_1024_cuda_bfloat16 2025-12-04T11:13:50.5309856Z Running 1 items in this shard 2025-12-04T11:13:50.5309860Z 2025-12-04T11:13:50.5310618Z inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_32_mid_dim_8_in_features_144_out_features_1024_cuda_bfloat16 [W1204 11:11:37.195219095 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.5310622Z 2025-12-04T11:13:50.5310930Z [W1204 11:11:46.366044732 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.5310933Z 2025-12-04T11:13:50.5311227Z [W1204 11:11:46.366323778 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.5311232Z 2025-12-04T11:13:50.5311531Z [W1204 11:11:46.372035304 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.5311574Z 2025-12-04T11:13:50.5311903Z [W1204 11:11:46.372604086 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.5311908Z 2025-12-04T11:13:50.5312240Z [W1204 11:11:46.372780190 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.5312244Z 2025-12-04T11:13:50.5312536Z [W1204 11:11:46.378167140 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.5312540Z 2025-12-04T11:13:50.5312860Z [W1204 11:11:46.378673311 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.5312863Z 2025-12-04T11:13:50.5313199Z [W1204 11:11:46.378829834 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.5313206Z 2025-12-04T11:13:50.5313292Z ('RERUN', {'yellow': True}) [11.0982s] [100%] 2025-12-04T11:13:50.5314038Z inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_32_mid_dim_8_in_features_144_out_features_1024_cuda_bfloat16 [W1204 11:11:47.606863622 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.5314043Z 2025-12-04T11:13:50.5314340Z [W1204 11:11:47.607422474 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.5314343Z 2025-12-04T11:13:50.5314652Z [W1204 11:11:47.607567097 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.5314655Z 2025-12-04T11:13:50.5314952Z [W1204 11:11:47.610611644 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.5314958Z 2025-12-04T11:13:50.5315256Z [W1204 11:11:47.611201637 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.5315259Z 2025-12-04T11:13:50.5315553Z [W1204 11:11:47.611342260 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.5315556Z 2025-12-04T11:13:50.5315850Z [W1204 11:11:47.615924011 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.5315859Z 2025-12-04T11:13:50.5316150Z [W1204 11:11:47.616406392 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.5316153Z 2025-12-04T11:13:50.5316449Z [W1204 11:11:47.616541835 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.5316512Z 2025-12-04T11:13:50.5316602Z ('RERUN', {'yellow': True}) [0.4663s] [100%] 2025-12-04T11:13:50.5317340Z inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_32_mid_dim_8_in_features_144_out_features_1024_cuda_bfloat16 [W1204 11:11:48.068168259 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.5317343Z 2025-12-04T11:13:50.5317646Z [W1204 11:11:48.068723591 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.5317650Z 2025-12-04T11:13:50.5317944Z [W1204 11:11:48.068870995 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.5317947Z 2025-12-04T11:13:50.5318243Z [W1204 11:11:48.071849721 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.5318247Z 2025-12-04T11:13:50.5318577Z [W1204 11:11:48.072445994 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.5318615Z 2025-12-04T11:13:50.5318917Z [W1204 11:11:48.072596047 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.5318920Z 2025-12-04T11:13:50.5319211Z [W1204 11:11:48.077178168 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.5319214Z 2025-12-04T11:13:50.5319506Z [W1204 11:11:48.077650239 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.5319513Z 2025-12-04T11:13:50.5319841Z [W1204 11:11:48.077791502 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.5319847Z 2025-12-04T11:13:50.5319912Z FAILED [0.4599s] [100%] 2025-12-04T11:13:50.5319917Z 2025-12-04T11:13:50.5320007Z ==================================== RERUNS ==================================== 2025-12-04T11:13:50.5320312Z _ TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_8_in_features_144_out_features_1024_cuda_bfloat16 _ 2025-12-04T11:13:50.5320390Z Traceback (most recent call last): 2025-12-04T11:13:50.5320720Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.5320788Z method(*args, **kwargs) 2025-12-04T11:13:50.5321095Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.5321160Z method(*args, **kwargs) 2025-12-04T11:13:50.5321455Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T11:13:50.5321521Z with policy(): 2025-12-04T11:13:50.5321823Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T11:13:50.5321895Z raise RuntimeError(msg) 2025-12-04T11:13:50.5322729Z RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_8_in_features_144_out_features_1024_cuda_bfloat16! Caching allocator allocated memory was 0 and is now reported as 147456 on device 0. CUDA driver allocated memory was 230686720 and is now 274726912. 2025-12-04T11:13:50.5322733Z 2025-12-04T11:13:50.5322867Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.5323410Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_8_in_features_144_out_features_1024_cuda_bfloat16 2025-12-04T11:13:50.5323452Z 2025-12-04T11:13:50.5323618Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.5323753Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.5323851Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.5324210Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.5324349Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.5324407Z graph_break [] 2025-12-04T11:13:50.5324538Z ----------------------------- Captured stderr call ----------------------------- 2025-12-04T11:13:50.5325284Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/constant_folding.py:256: UserWarning: Unsupported unwinding pattern: Address not in range (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/profiler/unwind/unwind.cpp:219.) 2025-12-04T11:13:50.5325362Z if out == self.unknown_value: 2025-12-04T11:13:50.5325706Z _ TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_8_in_features_144_out_features_1024_cuda_bfloat16 _ 2025-12-04T11:13:50.5325819Z Traceback (most recent call last): 2025-12-04T11:13:50.5326130Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.5326194Z method(*args, **kwargs) 2025-12-04T11:13:50.5326488Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.5326557Z method(*args, **kwargs) 2025-12-04T11:13:50.5326854Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T11:13:50.5326915Z with policy(): 2025-12-04T11:13:50.5327252Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T11:13:50.5327322Z raise RuntimeError(msg) 2025-12-04T11:13:50.5328167Z RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_8_in_features_144_out_features_1024_cuda_bfloat16! Caching allocator allocated memory was 147456 and is now reported as 294912 on device 0. CUDA driver allocated memory was 274726912 and is now 276824064. 2025-12-04T11:13:50.5328171Z 2025-12-04T11:13:50.5328304Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.5328845Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_8_in_features_144_out_features_1024_cuda_bfloat16 2025-12-04T11:13:50.5328849Z 2025-12-04T11:13:50.5329011Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.5329137Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.5329241Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.5329604Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.5329742Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.5329802Z graph_break [] 2025-12-04T11:13:50.5329929Z ----------------------------- Captured stderr call ----------------------------- 2025-12-04T11:13:50.5330637Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/constant_folding.py:256: UserWarning: Unsupported unwinding pattern: Address not in range (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/profiler/unwind/unwind.cpp:219.) 2025-12-04T11:13:50.5330707Z if out == self.unknown_value: 2025-12-04T11:13:50.5330834Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.5330985Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.5331113Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.5331474Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.5331533Z graph_break [] 2025-12-04T11:13:50.5331618Z =================================== FAILURES =================================== 2025-12-04T11:13:50.5331921Z _ TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_8_in_features_144_out_features_1024_cuda_bfloat16 _ 2025-12-04T11:13:50.5331998Z Traceback (most recent call last): 2025-12-04T11:13:50.5332306Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.5332372Z method(*args, **kwargs) 2025-12-04T11:13:50.5332666Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.5332776Z method(*args, **kwargs) 2025-12-04T11:13:50.5333109Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T11:13:50.5333171Z with policy(): 2025-12-04T11:13:50.5333472Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T11:13:50.5333537Z raise RuntimeError(msg) 2025-12-04T11:13:50.5334425Z RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_8_in_features_144_out_features_1024_cuda_bfloat16! Caching allocator allocated memory was 294912 and is now reported as 442368 on device 0. CUDA driver allocated memory was 276824064 and is now 278921216. 2025-12-04T11:13:50.5334430Z 2025-12-04T11:13:50.5334562Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.5335098Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_8_in_features_144_out_features_1024_cuda_bfloat16 2025-12-04T11:13:50.5335107Z 2025-12-04T11:13:50.5335265Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.5335391Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.5335488Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.5335841Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.5335967Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.5336031Z graph_break [] 2025-12-04T11:13:50.5336155Z ----------------------------- Captured stderr call ----------------------------- 2025-12-04T11:13:50.5336863Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/constant_folding.py:256: UserWarning: Unsupported unwinding pattern: Address not in range (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/profiler/unwind/unwind.cpp:219.) 2025-12-04T11:13:50.5336936Z if out == self.unknown_value: 2025-12-04T11:13:50.5337062Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.5337157Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.5337280Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.5337636Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.5337696Z graph_break [] 2025-12-04T11:13:50.5337821Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.5337966Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.5338095Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.5338443Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.5338505Z graph_break [] 2025-12-04T11:13:50.5339040Z - generated xml file: /var/lib/jenkins/workspace/test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-1eeada475e93262f.xml - 2025-12-04T11:13:50.5339151Z =========================== short test summary info ============================ 2025-12-04T11:13:50.5340483Z FAILED [0.4599s] inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_32_mid_dim_8_in_features_144_out_features_1024_cuda_bfloat16 - RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_8_in_features_144_out_features_1024_cuda_bfloat16! Caching allocator allocated memory was 294912 and is now reported as 442368 on device 0. CUDA driver allocated memory was 276824064 and is now 278921216. 2025-12-04T11:13:50.5340562Z 2025-12-04T11:13:50.5340695Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.5341224Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_8_in_features_144_out_features_1024_cuda_bfloat16 2025-12-04T11:13:50.5341228Z 2025-12-04T11:13:50.5341395Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.5341501Z !!!!!!!!!!!!!!!!!!!!!!!!!! stopping after 1 failures !!!!!!!!!!!!!!!!!!!!!!!!!!! 2025-12-04T11:13:50.5341671Z ================== 1 failed, 57 deselected, 2 rerun in 12.05s ================== 2025-12-04T11:13:50.5341735Z Got exit code 1 2025-12-04T11:13:50.5341802Z Retrying single test... 2025-12-04T11:13:50.5342072Z W1204 11:11:54.947000 87967 site-packages/torch/_inductor/utils.py:1703] Not enough SMs to use max_autotune_gemm mode 2025-12-04T11:13:50.5342474Z Test results will be stored in test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-3a596bc2a015c0a5.xml 2025-12-04T11:13:50.5342574Z ============================= test session starts ============================== 2025-12-04T11:13:50.5342787Z platform linux -- Python 3.10.14, pytest-7.3.2, pluggy-1.6.0 -- /opt/conda/envs/py_3.10/bin/python 2025-12-04T11:13:50.5342855Z cachedir: .pytest_cache 2025-12-04T11:13:50.5343169Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow] 2025-12-04T11:13:50.5343251Z rootdir: /var/lib/jenkins/workspace 2025-12-04T11:13:50.5343319Z configfile: pytest.ini 2025-12-04T11:13:50.5343718Z plugins: hypothesis-6.56.4, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-14.0, subtests-0.13.1, xdist-3.3.1, xdoctest-1.3.0, typeguard-4.3.0 2025-12-04T11:13:50.5343864Z collecting ... collected 58 items / 57 deselected / 1 selected 2025-12-04T11:13:50.5344453Z stepcurrent: skipping 55 already run items. Running only test/inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_32_mid_dim_8_in_features_144_out_features_1024_cuda_bfloat16 2025-12-04T11:13:50.5344530Z Running 1 items in this shard 2025-12-04T11:13:50.5344534Z 2025-12-04T11:13:50.5345286Z inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_32_mid_dim_8_in_features_144_out_features_1024_cuda_bfloat16 [W1204 11:11:56.078897695 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.5345290Z 2025-12-04T11:13:50.5345598Z [W1204 11:12:05.313077521 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.5345645Z 2025-12-04T11:13:50.5345944Z [W1204 11:12:05.313309337 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.5345949Z 2025-12-04T11:13:50.5346249Z [W1204 11:12:05.318965872 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.5346253Z 2025-12-04T11:13:50.5346546Z [W1204 11:12:05.319523944 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.5346549Z 2025-12-04T11:13:50.5346843Z [W1204 11:12:05.319687828 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.5346847Z 2025-12-04T11:13:50.5347144Z [W1204 11:12:05.325174089 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.5347149Z 2025-12-04T11:13:50.5347442Z [W1204 11:12:05.325721951 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.5347481Z 2025-12-04T11:13:50.5347824Z [W1204 11:12:05.325877665 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.5347827Z 2025-12-04T11:13:50.5347911Z ('RERUN', {'yellow': True}) [11.1642s] [100%] 2025-12-04T11:13:50.5348661Z inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_32_mid_dim_8_in_features_144_out_features_1024_cuda_bfloat16 [W1204 11:12:06.555993835 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.5348665Z 2025-12-04T11:13:50.5348994Z [W1204 11:12:06.556565427 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.5349000Z 2025-12-04T11:13:50.5349301Z [W1204 11:12:06.556718581 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.5349306Z 2025-12-04T11:13:50.5349609Z [W1204 11:12:06.559741928 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.5349612Z 2025-12-04T11:13:50.5349905Z [W1204 11:12:06.560348531 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.5349915Z 2025-12-04T11:13:50.5350209Z [W1204 11:12:06.560492494 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.5350213Z 2025-12-04T11:13:50.5350506Z [W1204 11:12:06.565132937 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.5350511Z 2025-12-04T11:13:50.5350807Z [W1204 11:12:06.565619318 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.5350811Z 2025-12-04T11:13:50.5351104Z [W1204 11:12:06.565755651 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.5351108Z 2025-12-04T11:13:50.5351191Z ('RERUN', {'yellow': True}) [0.4642s] [100%] 2025-12-04T11:13:50.5351938Z inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_32_mid_dim_8_in_features_144_out_features_1024_cuda_bfloat16 [W1204 11:12:07.013851193 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.5351942Z 2025-12-04T11:13:50.5352257Z [W1204 11:12:07.014406615 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.5352298Z 2025-12-04T11:13:50.5352598Z [W1204 11:12:07.014551978 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.5352603Z 2025-12-04T11:13:50.5352903Z [W1204 11:12:07.017563565 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.5352906Z 2025-12-04T11:13:50.5353199Z [W1204 11:12:07.018146628 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.5353202Z 2025-12-04T11:13:50.5353494Z [W1204 11:12:07.018287711 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.5353502Z 2025-12-04T11:13:50.5353797Z [W1204 11:12:07.022953524 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.5353802Z 2025-12-04T11:13:50.5354093Z [W1204 11:12:07.023437645 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.5354131Z 2025-12-04T11:13:50.5354477Z [W1204 11:12:07.023571538 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.5354482Z 2025-12-04T11:13:50.5354547Z FAILED [0.4557s] [100%] 2025-12-04T11:13:50.5354551Z 2025-12-04T11:13:50.5354641Z ==================================== RERUNS ==================================== 2025-12-04T11:13:50.5354947Z _ TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_8_in_features_144_out_features_1024_cuda_bfloat16 _ 2025-12-04T11:13:50.5355025Z Traceback (most recent call last): 2025-12-04T11:13:50.5355419Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.5355489Z method(*args, **kwargs) 2025-12-04T11:13:50.5355796Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.5355862Z method(*args, **kwargs) 2025-12-04T11:13:50.5356161Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T11:13:50.5356227Z with policy(): 2025-12-04T11:13:50.5356528Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T11:13:50.5356594Z raise RuntimeError(msg) 2025-12-04T11:13:50.5357431Z RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_8_in_features_144_out_features_1024_cuda_bfloat16! Caching allocator allocated memory was 0 and is now reported as 147456 on device 0. CUDA driver allocated memory was 230686720 and is now 274726912. 2025-12-04T11:13:50.5357436Z 2025-12-04T11:13:50.5357574Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.5358116Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_8_in_features_144_out_features_1024_cuda_bfloat16 2025-12-04T11:13:50.5358120Z 2025-12-04T11:13:50.5358283Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.5358418Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.5358526Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.5358890Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.5359031Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.5359090Z graph_break [] 2025-12-04T11:13:50.5359216Z ----------------------------- Captured stderr call ----------------------------- 2025-12-04T11:13:50.5360016Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/constant_folding.py:256: UserWarning: Unsupported unwinding pattern: Address not in range (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/profiler/unwind/unwind.cpp:219.) 2025-12-04T11:13:50.5360089Z if out == self.unknown_value: 2025-12-04T11:13:50.5360396Z _ TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_8_in_features_144_out_features_1024_cuda_bfloat16 _ 2025-12-04T11:13:50.5360471Z Traceback (most recent call last): 2025-12-04T11:13:50.5360778Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.5360847Z method(*args, **kwargs) 2025-12-04T11:13:50.5361146Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.5361216Z method(*args, **kwargs) 2025-12-04T11:13:50.5361510Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T11:13:50.5361611Z with policy(): 2025-12-04T11:13:50.5361949Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T11:13:50.5362016Z raise RuntimeError(msg) 2025-12-04T11:13:50.5362855Z RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_8_in_features_144_out_features_1024_cuda_bfloat16! Caching allocator allocated memory was 147456 and is now reported as 294912 on device 0. CUDA driver allocated memory was 274726912 and is now 276824064. 2025-12-04T11:13:50.5362859Z 2025-12-04T11:13:50.5362992Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.5363564Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_8_in_features_144_out_features_1024_cuda_bfloat16 2025-12-04T11:13:50.5363576Z 2025-12-04T11:13:50.5363740Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.5363868Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.5363979Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.5364335Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.5364463Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.5364528Z graph_break [] 2025-12-04T11:13:50.5364655Z ----------------------------- Captured stderr call ----------------------------- 2025-12-04T11:13:50.5365363Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/constant_folding.py:256: UserWarning: Unsupported unwinding pattern: Address not in range (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/profiler/unwind/unwind.cpp:219.) 2025-12-04T11:13:50.5365436Z if out == self.unknown_value: 2025-12-04T11:13:50.5365562Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.5365661Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.5365787Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.5366140Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.5366208Z graph_break [] 2025-12-04T11:13:50.5366293Z =================================== FAILURES =================================== 2025-12-04T11:13:50.5366597Z _ TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_8_in_features_144_out_features_1024_cuda_bfloat16 _ 2025-12-04T11:13:50.5366713Z Traceback (most recent call last): 2025-12-04T11:13:50.5367017Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.5367089Z method(*args, **kwargs) 2025-12-04T11:13:50.5367420Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.5367490Z method(*args, **kwargs) 2025-12-04T11:13:50.5367785Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T11:13:50.5367845Z with policy(): 2025-12-04T11:13:50.5368150Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T11:13:50.5368218Z raise RuntimeError(msg) 2025-12-04T11:13:50.5369095Z RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_8_in_features_144_out_features_1024_cuda_bfloat16! Caching allocator allocated memory was 294912 and is now reported as 442368 on device 0. CUDA driver allocated memory was 276824064 and is now 278921216. 2025-12-04T11:13:50.5369157Z 2025-12-04T11:13:50.5369288Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.5369828Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_8_in_features_144_out_features_1024_cuda_bfloat16 2025-12-04T11:13:50.5369832Z 2025-12-04T11:13:50.5369996Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.5370124Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.5370255Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.5370609Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.5370741Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.5370818Z graph_break [] 2025-12-04T11:13:50.5370943Z ----------------------------- Captured stderr call ----------------------------- 2025-12-04T11:13:50.5371646Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/constant_folding.py:256: UserWarning: Unsupported unwinding pattern: Address not in range (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/profiler/unwind/unwind.cpp:219.) 2025-12-04T11:13:50.5371714Z if out == self.unknown_value: 2025-12-04T11:13:50.5371835Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.5371933Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.5372059Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.5372409Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.5372474Z graph_break [] 2025-12-04T11:13:50.5372600Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.5372695Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.5372817Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.5373163Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.5373227Z graph_break [] 2025-12-04T11:13:50.5373726Z - generated xml file: /var/lib/jenkins/workspace/test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-3a596bc2a015c0a5.xml - 2025-12-04T11:13:50.5373834Z =========================== short test summary info ============================ 2025-12-04T11:13:50.5375211Z FAILED [0.4557s] inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_32_mid_dim_8_in_features_144_out_features_1024_cuda_bfloat16 - RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_8_in_features_144_out_features_1024_cuda_bfloat16! Caching allocator allocated memory was 294912 and is now reported as 442368 on device 0. CUDA driver allocated memory was 276824064 and is now 278921216. 2025-12-04T11:13:50.5375217Z 2025-12-04T11:13:50.5375349Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.5375879Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_8_in_features_144_out_features_1024_cuda_bfloat16 2025-12-04T11:13:50.5375883Z 2025-12-04T11:13:50.5376047Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.5376196Z !!!!!!!!!!!!!!!!!!!!!!!!!! stopping after 1 failures !!!!!!!!!!!!!!!!!!!!!!!!!!! 2025-12-04T11:13:50.5376348Z ================== 1 failed, 57 deselected, 2 rerun in 12.11s ================== 2025-12-04T11:13:50.5376415Z Got exit code 1 2025-12-04T11:13:50.5376907Z FAILED CONSISTENTLY: test/inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_32_mid_dim_8_in_features_144_out_features_1024_cuda_bfloat16 2025-12-04T11:13:50.5377157Z Test failed consistently, continuing with the rest of the tests due to continue-through-error being set 2025-12-04T11:13:50.5377428Z W1204 11:12:13.831000 88160 site-packages/torch/_inductor/utils.py:1703] Not enough SMs to use max_autotune_gemm mode 2025-12-04T11:13:50.5378084Z Test results will be stored in test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-ff629078718dd71f.xml 2025-12-04T11:13:50.5378218Z ============================= test session starts ============================== 2025-12-04T11:13:50.5378443Z platform linux -- Python 3.10.14, pytest-7.3.2, pluggy-1.6.0 -- /opt/conda/envs/py_3.10/bin/python 2025-12-04T11:13:50.5378515Z cachedir: .pytest_cache 2025-12-04T11:13:50.5378838Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow] 2025-12-04T11:13:50.5378919Z rootdir: /var/lib/jenkins/workspace 2025-12-04T11:13:50.5378985Z configfile: pytest.ini 2025-12-04T11:13:50.5379312Z plugins: hypothesis-6.56.4, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-14.0, subtests-0.13.1, xdist-3.3.1, xdoctest-1.3.0, typeguard-4.3.0 2025-12-04T11:13:50.5379442Z collecting ... collected 58 items / 56 deselected / 2 selected 2025-12-04T11:13:50.5379536Z stepcurrent: skipping 56 already run items. 2025-12-04T11:13:50.5379611Z Running 2 items in this shard 2025-12-04T11:13:50.5379616Z 2025-12-04T11:13:50.5380136Z inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_32_mid_dim_8_in_features_144_out_features_64_cuda_bfloat16 ('RERUN', {'yellow': True}) [1.8836s] [ 50%] 2025-12-04T11:13:50.5380638Z inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_32_mid_dim_8_in_features_144_out_features_64_cuda_bfloat16 ('RERUN', {'yellow': True}) [0.4623s] [ 50%] 2025-12-04T11:13:50.5381131Z inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_32_mid_dim_8_in_features_144_out_features_64_cuda_bfloat16 FAILED [0.4677s] [ 50%] 2025-12-04T11:13:50.5381136Z 2025-12-04T11:13:50.5381223Z ==================================== RERUNS ==================================== 2025-12-04T11:13:50.5381522Z _ TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_8_in_features_144_out_features_64_cuda_bfloat16 _ 2025-12-04T11:13:50.5381599Z Traceback (most recent call last): 2025-12-04T11:13:50.5381996Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.5382065Z method(*args, **kwargs) 2025-12-04T11:13:50.5382368Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.5382434Z method(*args, **kwargs) 2025-12-04T11:13:50.5382727Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T11:13:50.5382792Z with policy(): 2025-12-04T11:13:50.5383091Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T11:13:50.5383159Z raise RuntimeError(msg) 2025-12-04T11:13:50.5384023Z RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_8_in_features_144_out_features_64_cuda_bfloat16! Caching allocator allocated memory was 0 and is now reported as 9216 on device 0. CUDA driver allocated memory was 230686720 and is now 274726912. 2025-12-04T11:13:50.5384088Z 2025-12-04T11:13:50.5384282Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.5384822Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_8_in_features_144_out_features_64_cuda_bfloat16 2025-12-04T11:13:50.5384826Z 2025-12-04T11:13:50.5384989Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.5385134Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.5385233Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.5385628Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.5385768Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.5385830Z graph_break [] 2025-12-04T11:13:50.5386134Z _ TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_8_in_features_144_out_features_64_cuda_bfloat16 _ 2025-12-04T11:13:50.5386211Z Traceback (most recent call last): 2025-12-04T11:13:50.5386516Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.5386585Z method(*args, **kwargs) 2025-12-04T11:13:50.5386886Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.5386951Z method(*args, **kwargs) 2025-12-04T11:13:50.5387252Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T11:13:50.5387311Z with policy(): 2025-12-04T11:13:50.5387629Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T11:13:50.5387698Z raise RuntimeError(msg) 2025-12-04T11:13:50.5388522Z RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_8_in_features_144_out_features_64_cuda_bfloat16! Caching allocator allocated memory was 9216 and is now reported as 18432 on device 0. CUDA driver allocated memory was 274726912 and is now 276824064. 2025-12-04T11:13:50.5388527Z 2025-12-04T11:13:50.5388662Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.5389192Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_8_in_features_144_out_features_64_cuda_bfloat16 2025-12-04T11:13:50.5389196Z 2025-12-04T11:13:50.5389364Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.5389531Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.5389629Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.5389995Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.5390124Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.5390188Z graph_break [] 2025-12-04T11:13:50.5390315Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.5390408Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.5390540Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.5390891Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.5390956Z graph_break [] 2025-12-04T11:13:50.5391041Z =================================== FAILURES =================================== 2025-12-04T11:13:50.5391408Z _ TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_8_in_features_144_out_features_64_cuda_bfloat16 _ 2025-12-04T11:13:50.5391489Z Traceback (most recent call last): 2025-12-04T11:13:50.5391794Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.5391859Z method(*args, **kwargs) 2025-12-04T11:13:50.5392162Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.5392226Z method(*args, **kwargs) 2025-12-04T11:13:50.5392568Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T11:13:50.5392629Z with policy(): 2025-12-04T11:13:50.5392930Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T11:13:50.5393002Z raise RuntimeError(msg) 2025-12-04T11:13:50.5393826Z RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_8_in_features_144_out_features_64_cuda_bfloat16! Caching allocator allocated memory was 18432 and is now reported as 27648 on device 0. CUDA driver allocated memory was 276824064 and is now 278921216. 2025-12-04T11:13:50.5393830Z 2025-12-04T11:13:50.5393974Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.5394506Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_8_in_features_144_out_features_64_cuda_bfloat16 2025-12-04T11:13:50.5394511Z 2025-12-04T11:13:50.5394671Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.5394803Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.5394897Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.5395254Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.5395382Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.5395441Z graph_break [] 2025-12-04T11:13:50.5395569Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.5395661Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.5395783Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.5396137Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.5396238Z graph_break [] 2025-12-04T11:13:50.5396374Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.5396466Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.5396591Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.5396943Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.5397003Z graph_break [] 2025-12-04T11:13:50.5397504Z - generated xml file: /var/lib/jenkins/workspace/test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-ff629078718dd71f.xml - 2025-12-04T11:13:50.5397613Z =========================== short test summary info ============================ 2025-12-04T11:13:50.5398983Z FAILED [0.4677s] inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_32_mid_dim_8_in_features_144_out_features_64_cuda_bfloat16 - RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_8_in_features_144_out_features_64_cuda_bfloat16! Caching allocator allocated memory was 18432 and is now reported as 27648 on device 0. CUDA driver allocated memory was 276824064 and is now 278921216. 2025-12-04T11:13:50.5399026Z 2025-12-04T11:13:50.5399154Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.5399683Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_8_in_features_144_out_features_64_cuda_bfloat16 2025-12-04T11:13:50.5399686Z 2025-12-04T11:13:50.5399848Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.5399989Z !!!!!!!!!!!!!!!!!!!!!!!!!! stopping after 1 failures !!!!!!!!!!!!!!!!!!!!!!!!!!! 2025-12-04T11:13:50.5400113Z ================== 1 failed, 56 deselected, 2 rerun in 2.84s =================== 2025-12-04T11:13:50.5400176Z Got exit code 1 2025-12-04T11:13:50.5400244Z Retrying single test... 2025-12-04T11:13:50.5400521Z W1204 11:12:23.831000 88341 site-packages/torch/_inductor/utils.py:1703] Not enough SMs to use max_autotune_gemm mode 2025-12-04T11:13:50.5400916Z Test results will be stored in test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-2cc0c50f93d222cd.xml 2025-12-04T11:13:50.5401015Z ============================= test session starts ============================== 2025-12-04T11:13:50.5401231Z platform linux -- Python 3.10.14, pytest-7.3.2, pluggy-1.6.0 -- /opt/conda/envs/py_3.10/bin/python 2025-12-04T11:13:50.5401299Z cachedir: .pytest_cache 2025-12-04T11:13:50.5401618Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow] 2025-12-04T11:13:50.5401697Z rootdir: /var/lib/jenkins/workspace 2025-12-04T11:13:50.5401765Z configfile: pytest.ini 2025-12-04T11:13:50.5402104Z plugins: hypothesis-6.56.4, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-14.0, subtests-0.13.1, xdist-3.3.1, xdoctest-1.3.0, typeguard-4.3.0 2025-12-04T11:13:50.5402239Z collecting ... collected 58 items / 57 deselected / 1 selected 2025-12-04T11:13:50.5402827Z stepcurrent: skipping 56 already run items. Running only test/inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_32_mid_dim_8_in_features_144_out_features_64_cuda_bfloat16 2025-12-04T11:13:50.5402900Z Running 1 items in this shard 2025-12-04T11:13:50.5402904Z 2025-12-04T11:13:50.5403648Z inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_32_mid_dim_8_in_features_144_out_features_64_cuda_bfloat16 [W1204 11:12:24.929142640 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.5403696Z 2025-12-04T11:13:50.5404003Z [W1204 11:12:34.982910304 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.5404008Z 2025-12-04T11:13:50.5404306Z [W1204 11:12:34.983134649 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.5404310Z 2025-12-04T11:13:50.5404610Z [W1204 11:12:34.988837376 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.5404614Z 2025-12-04T11:13:50.5404908Z [W1204 11:12:34.989401108 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.5404911Z 2025-12-04T11:13:50.5405210Z [W1204 11:12:34.989573852 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.5405215Z 2025-12-04T11:13:50.5405512Z [W1204 11:12:34.995088943 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.5405551Z 2025-12-04T11:13:50.5405963Z [W1204 11:12:34.995618835 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.5405967Z 2025-12-04T11:13:50.5406265Z [W1204 11:12:34.995777858 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.5406268Z 2025-12-04T11:13:50.5406352Z ('RERUN', {'yellow': True}) [10.9419s] [100%] 2025-12-04T11:13:50.5407127Z inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_32_mid_dim_8_in_features_144_out_features_64_cuda_bfloat16 [W1204 11:12:35.173745122 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.5407133Z 2025-12-04T11:13:50.5407430Z [W1204 11:12:35.174325955 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.5407435Z 2025-12-04T11:13:50.5407734Z [W1204 11:12:35.174467278 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.5407738Z 2025-12-04T11:13:50.5408032Z [W1204 11:12:35.177411533 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.5408035Z 2025-12-04T11:13:50.5408339Z [W1204 11:12:35.177986305 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.5408342Z 2025-12-04T11:13:50.5408637Z [W1204 11:12:35.178126819 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.5408640Z 2025-12-04T11:13:50.5408941Z [W1204 11:12:35.182715000 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.5408946Z 2025-12-04T11:13:50.5409241Z [W1204 11:12:35.183192171 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.5409245Z 2025-12-04T11:13:50.5409543Z [W1204 11:12:35.183326874 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.5409546Z 2025-12-04T11:13:50.5409626Z ('RERUN', {'yellow': True}) [0.4168s] [100%] 2025-12-04T11:13:50.5410359Z inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_32_mid_dim_8_in_features_144_out_features_64_cuda_bfloat16 [W1204 11:12:35.591073115 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.5410366Z 2025-12-04T11:13:50.5410711Z [W1204 11:12:35.591651608 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.5410716Z 2025-12-04T11:13:50.5411013Z [W1204 11:12:35.591793521 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.5411016Z 2025-12-04T11:13:50.5411316Z [W1204 11:12:35.594733316 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.5411319Z 2025-12-04T11:13:50.5411613Z [W1204 11:12:35.595299879 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.5411616Z 2025-12-04T11:13:50.5411916Z [W1204 11:12:35.595438052 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.5411920Z 2025-12-04T11:13:50.5412211Z [W1204 11:12:35.599992412 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.5412251Z 2025-12-04T11:13:50.5412584Z [W1204 11:12:35.600480683 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.5412588Z 2025-12-04T11:13:50.5412882Z [W1204 11:12:35.600625357 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.5412885Z 2025-12-04T11:13:50.5412952Z FAILED [0.4142s] [100%] 2025-12-04T11:13:50.5412956Z 2025-12-04T11:13:50.5413044Z ==================================== RERUNS ==================================== 2025-12-04T11:13:50.5413343Z _ TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_8_in_features_144_out_features_64_cuda_bfloat16 _ 2025-12-04T11:13:50.5413460Z Traceback (most recent call last): 2025-12-04T11:13:50.5413775Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.5413844Z method(*args, **kwargs) 2025-12-04T11:13:50.5414148Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.5414214Z method(*args, **kwargs) 2025-12-04T11:13:50.5414513Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T11:13:50.5414574Z with policy(): 2025-12-04T11:13:50.5414879Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T11:13:50.5414954Z raise RuntimeError(msg) 2025-12-04T11:13:50.5415772Z RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_8_in_features_144_out_features_64_cuda_bfloat16! Caching allocator allocated memory was 0 and is now reported as 9216 on device 0. CUDA driver allocated memory was 230686720 and is now 274726912. 2025-12-04T11:13:50.5415780Z 2025-12-04T11:13:50.5415925Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.5416462Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_8_in_features_144_out_features_64_cuda_bfloat16 2025-12-04T11:13:50.5416467Z 2025-12-04T11:13:50.5416630Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.5416769Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.5416866Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.5417229Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.5417400Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.5417463Z graph_break [] 2025-12-04T11:13:50.5417595Z ----------------------------- Captured stderr call ----------------------------- 2025-12-04T11:13:50.5418301Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/constant_folding.py:256: UserWarning: Unsupported unwinding pattern: Address not in range (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/profiler/unwind/unwind.cpp:219.) 2025-12-04T11:13:50.5418379Z if out == self.unknown_value: 2025-12-04T11:13:50.5418679Z _ TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_8_in_features_144_out_features_64_cuda_bfloat16 _ 2025-12-04T11:13:50.5418761Z Traceback (most recent call last): 2025-12-04T11:13:50.5419070Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.5419136Z method(*args, **kwargs) 2025-12-04T11:13:50.5419437Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.5419550Z method(*args, **kwargs) 2025-12-04T11:13:50.5419881Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T11:13:50.5419949Z with policy(): 2025-12-04T11:13:50.5420249Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T11:13:50.5420316Z raise RuntimeError(msg) 2025-12-04T11:13:50.5421181Z RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_8_in_features_144_out_features_64_cuda_bfloat16! Caching allocator allocated memory was 9216 and is now reported as 18432 on device 0. CUDA driver allocated memory was 274726912 and is now 276824064. 2025-12-04T11:13:50.5421185Z 2025-12-04T11:13:50.5421317Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.5421857Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_8_in_features_144_out_features_64_cuda_bfloat16 2025-12-04T11:13:50.5421863Z 2025-12-04T11:13:50.5422029Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.5422161Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.5422257Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.5422610Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.5422746Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.5422819Z graph_break [] 2025-12-04T11:13:50.5422946Z ----------------------------- Captured stderr call ----------------------------- 2025-12-04T11:13:50.5423707Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/constant_folding.py:256: UserWarning: Unsupported unwinding pattern: Address not in range (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/profiler/unwind/unwind.cpp:219.) 2025-12-04T11:13:50.5423782Z if out == self.unknown_value: 2025-12-04T11:13:50.5423914Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.5424007Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.5424133Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.5424487Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.5424547Z graph_break [] 2025-12-04T11:13:50.5424637Z =================================== FAILURES =================================== 2025-12-04T11:13:50.5424997Z _ TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_8_in_features_144_out_features_64_cuda_bfloat16 _ 2025-12-04T11:13:50.5425075Z Traceback (most recent call last): 2025-12-04T11:13:50.5425384Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.5425449Z method(*args, **kwargs) 2025-12-04T11:13:50.5425744Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.5425812Z method(*args, **kwargs) 2025-12-04T11:13:50.5426119Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T11:13:50.5426187Z with policy(): 2025-12-04T11:13:50.5426487Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T11:13:50.5426558Z raise RuntimeError(msg) 2025-12-04T11:13:50.5427430Z RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_8_in_features_144_out_features_64_cuda_bfloat16! Caching allocator allocated memory was 18432 and is now reported as 27648 on device 0. CUDA driver allocated memory was 276824064 and is now 278921216. 2025-12-04T11:13:50.5427468Z 2025-12-04T11:13:50.5427603Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.5428142Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_8_in_features_144_out_features_64_cuda_bfloat16 2025-12-04T11:13:50.5428148Z 2025-12-04T11:13:50.5428310Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.5428472Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.5428572Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.5428926Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.5429063Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.5429122Z graph_break [] 2025-12-04T11:13:50.5429247Z ----------------------------- Captured stderr call ----------------------------- 2025-12-04T11:13:50.5429948Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/constant_folding.py:256: UserWarning: Unsupported unwinding pattern: Address not in range (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/profiler/unwind/unwind.cpp:219.) 2025-12-04T11:13:50.5430018Z if out == self.unknown_value: 2025-12-04T11:13:50.5430148Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.5430241Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.5430367Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.5430723Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.5430783Z graph_break [] 2025-12-04T11:13:50.5430907Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.5431003Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.5431126Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.5431478Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.5431536Z graph_break [] 2025-12-04T11:13:50.5432034Z - generated xml file: /var/lib/jenkins/workspace/test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-2cc0c50f93d222cd.xml - 2025-12-04T11:13:50.5432185Z =========================== short test summary info ============================ 2025-12-04T11:13:50.5433493Z FAILED [0.4142s] inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_32_mid_dim_8_in_features_144_out_features_64_cuda_bfloat16 - RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_8_in_features_144_out_features_64_cuda_bfloat16! Caching allocator allocated memory was 18432 and is now reported as 27648 on device 0. CUDA driver allocated memory was 276824064 and is now 278921216. 2025-12-04T11:13:50.5433498Z 2025-12-04T11:13:50.5433632Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.5434159Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_8_in_features_144_out_features_64_cuda_bfloat16 2025-12-04T11:13:50.5434199Z 2025-12-04T11:13:50.5434364Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.5434502Z !!!!!!!!!!!!!!!!!!!!!!!!!! stopping after 1 failures !!!!!!!!!!!!!!!!!!!!!!!!!!! 2025-12-04T11:13:50.5434621Z ================== 1 failed, 57 deselected, 2 rerun in 11.80s ================== 2025-12-04T11:13:50.5434696Z Got exit code 1 2025-12-04T11:13:50.5434763Z Retrying single test... 2025-12-04T11:13:50.5435038Z W1204 11:12:42.448000 88527 site-packages/torch/_inductor/utils.py:1703] Not enough SMs to use max_autotune_gemm mode 2025-12-04T11:13:50.5435434Z Test results will be stored in test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-ca536e35d6aacc32.xml 2025-12-04T11:13:50.5435566Z ============================= test session starts ============================== 2025-12-04T11:13:50.5435786Z platform linux -- Python 3.10.14, pytest-7.3.2, pluggy-1.6.0 -- /opt/conda/envs/py_3.10/bin/python 2025-12-04T11:13:50.5435857Z cachedir: .pytest_cache 2025-12-04T11:13:50.5436170Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow] 2025-12-04T11:13:50.5436255Z rootdir: /var/lib/jenkins/workspace 2025-12-04T11:13:50.5436323Z configfile: pytest.ini 2025-12-04T11:13:50.5436646Z plugins: hypothesis-6.56.4, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-14.0, subtests-0.13.1, xdist-3.3.1, xdoctest-1.3.0, typeguard-4.3.0 2025-12-04T11:13:50.5436777Z collecting ... collected 58 items / 57 deselected / 1 selected 2025-12-04T11:13:50.5437356Z stepcurrent: skipping 56 already run items. Running only test/inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_32_mid_dim_8_in_features_144_out_features_64_cuda_bfloat16 2025-12-04T11:13:50.5437434Z Running 1 items in this shard 2025-12-04T11:13:50.5437438Z 2025-12-04T11:13:50.5438184Z inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_32_mid_dim_8_in_features_144_out_features_64_cuda_bfloat16 [W1204 11:12:43.546058595 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.5438189Z 2025-12-04T11:13:50.5438497Z [W1204 11:12:52.897494018 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.5438501Z 2025-12-04T11:13:50.5438796Z [W1204 11:12:52.897729343 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.5438800Z 2025-12-04T11:13:50.5439098Z [W1204 11:12:52.903278346 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.5439102Z 2025-12-04T11:13:50.5439393Z [W1204 11:12:52.903845629 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.5439436Z 2025-12-04T11:13:50.5439743Z [W1204 11:12:52.904012192 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.5439747Z 2025-12-04T11:13:50.5440040Z [W1204 11:12:52.909418342 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.5440043Z 2025-12-04T11:13:50.5440338Z [W1204 11:12:52.909957714 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.5440346Z 2025-12-04T11:13:50.5440640Z [W1204 11:12:52.910145368 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.5440643Z 2025-12-04T11:13:50.5440729Z ('RERUN', {'yellow': True}) [11.2424s] [100%] 2025-12-04T11:13:50.5441502Z inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_32_mid_dim_8_in_features_144_out_features_64_cuda_bfloat16 [W1204 11:12:54.091264650 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.5441541Z 2025-12-04T11:13:50.5441839Z [W1204 11:12:54.091845183 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.5441842Z 2025-12-04T11:13:50.5442139Z [W1204 11:12:54.091984016 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.5442142Z 2025-12-04T11:13:50.5442435Z [W1204 11:12:54.094907811 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.5442439Z 2025-12-04T11:13:50.5442769Z [W1204 11:12:54.095477313 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.5442775Z 2025-12-04T11:13:50.5443071Z [W1204 11:12:54.095616556 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.5443076Z 2025-12-04T11:13:50.5443374Z [W1204 11:12:54.100129806 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.5443377Z 2025-12-04T11:13:50.5443669Z [W1204 11:12:54.100608087 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.5443673Z 2025-12-04T11:13:50.5443966Z [W1204 11:12:54.100744580 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.5443975Z 2025-12-04T11:13:50.5444055Z ('RERUN', {'yellow': True}) [0.4140s] [100%] 2025-12-04T11:13:50.5444788Z inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_32_mid_dim_8_in_features_144_out_features_64_cuda_bfloat16 [W1204 11:12:54.505336972 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.5444797Z 2025-12-04T11:13:50.5445095Z [W1204 11:12:54.505908084 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.5445099Z 2025-12-04T11:13:50.5445394Z [W1204 11:12:54.506050338 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.5445398Z 2025-12-04T11:13:50.5445694Z [W1204 11:12:54.508995753 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.5445697Z 2025-12-04T11:13:50.5445992Z [W1204 11:12:54.509562935 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.5446032Z 2025-12-04T11:13:50.5446334Z [W1204 11:12:54.509702068 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.5446339Z 2025-12-04T11:13:50.5446635Z [W1204 11:12:54.514214688 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.5446638Z 2025-12-04T11:13:50.5446929Z [W1204 11:12:54.514685009 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.5446936Z 2025-12-04T11:13:50.5447234Z [W1204 11:12:54.514820512 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.5447237Z 2025-12-04T11:13:50.5447299Z FAILED [0.4121s] [100%] 2025-12-04T11:13:50.5447302Z 2025-12-04T11:13:50.5447394Z ==================================== RERUNS ==================================== 2025-12-04T11:13:50.5447693Z _ TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_8_in_features_144_out_features_64_cuda_bfloat16 _ 2025-12-04T11:13:50.5447814Z Traceback (most recent call last): 2025-12-04T11:13:50.5448162Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.5448230Z method(*args, **kwargs) 2025-12-04T11:13:50.5448532Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.5448594Z method(*args, **kwargs) 2025-12-04T11:13:50.5448889Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T11:13:50.5448953Z with policy(): 2025-12-04T11:13:50.5449305Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T11:13:50.5449381Z raise RuntimeError(msg) 2025-12-04T11:13:50.5450200Z RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_8_in_features_144_out_features_64_cuda_bfloat16! Caching allocator allocated memory was 0 and is now reported as 9216 on device 0. CUDA driver allocated memory was 230686720 and is now 274726912. 2025-12-04T11:13:50.5450206Z 2025-12-04T11:13:50.5450341Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.5450878Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_8_in_features_144_out_features_64_cuda_bfloat16 2025-12-04T11:13:50.5450882Z 2025-12-04T11:13:50.5451044Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.5451176Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.5451277Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.5451632Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.5451779Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.5451842Z graph_break [] 2025-12-04T11:13:50.5451978Z ----------------------------- Captured stderr call ----------------------------- 2025-12-04T11:13:50.5452682Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/constant_folding.py:256: UserWarning: Unsupported unwinding pattern: Address not in range (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/profiler/unwind/unwind.cpp:219.) 2025-12-04T11:13:50.5452754Z if out == self.unknown_value: 2025-12-04T11:13:50.5453057Z _ TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_8_in_features_144_out_features_64_cuda_bfloat16 _ 2025-12-04T11:13:50.5453173Z Traceback (most recent call last): 2025-12-04T11:13:50.5453479Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.5453546Z method(*args, **kwargs) 2025-12-04T11:13:50.5453842Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.5453912Z method(*args, **kwargs) 2025-12-04T11:13:50.5454207Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T11:13:50.5454268Z with policy(): 2025-12-04T11:13:50.5454570Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T11:13:50.5454636Z raise RuntimeError(msg) 2025-12-04T11:13:50.5455471Z RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_8_in_features_144_out_features_64_cuda_bfloat16! Caching allocator allocated memory was 9216 and is now reported as 18432 on device 0. CUDA driver allocated memory was 274726912 and is now 276824064. 2025-12-04T11:13:50.5455514Z 2025-12-04T11:13:50.5455677Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.5456211Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_8_in_features_144_out_features_64_cuda_bfloat16 2025-12-04T11:13:50.5456215Z 2025-12-04T11:13:50.5456378Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.5456505Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.5456606Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.5456994Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.5457131Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.5457192Z graph_break [] 2025-12-04T11:13:50.5457319Z ----------------------------- Captured stderr call ----------------------------- 2025-12-04T11:13:50.5458026Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/constant_folding.py:256: UserWarning: Unsupported unwinding pattern: Address not in range (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/profiler/unwind/unwind.cpp:219.) 2025-12-04T11:13:50.5458098Z if out == self.unknown_value: 2025-12-04T11:13:50.5458222Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.5458321Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.5458449Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.5458805Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.5458869Z graph_break [] 2025-12-04T11:13:50.5458954Z =================================== FAILURES =================================== 2025-12-04T11:13:50.5459258Z _ TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_8_in_features_144_out_features_64_cuda_bfloat16 _ 2025-12-04T11:13:50.5459337Z Traceback (most recent call last): 2025-12-04T11:13:50.5459661Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.5459733Z method(*args, **kwargs) 2025-12-04T11:13:50.5460033Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.5460104Z method(*args, **kwargs) 2025-12-04T11:13:50.5460398Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T11:13:50.5460500Z with policy(): 2025-12-04T11:13:50.5460802Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T11:13:50.5460870Z raise RuntimeError(msg) 2025-12-04T11:13:50.5461702Z RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_8_in_features_144_out_features_64_cuda_bfloat16! Caching allocator allocated memory was 18432 and is now reported as 27648 on device 0. CUDA driver allocated memory was 276824064 and is now 278921216. 2025-12-04T11:13:50.5461706Z 2025-12-04T11:13:50.5461835Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.5462367Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_8_in_features_144_out_features_64_cuda_bfloat16 2025-12-04T11:13:50.5462373Z 2025-12-04T11:13:50.5462582Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.5462741Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.5462838Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.5463190Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.5463321Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.5463383Z graph_break [] 2025-12-04T11:13:50.5463546Z ----------------------------- Captured stderr call ----------------------------- 2025-12-04T11:13:50.5464291Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/constant_folding.py:256: UserWarning: Unsupported unwinding pattern: Address not in range (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/profiler/unwind/unwind.cpp:219.) 2025-12-04T11:13:50.5464366Z if out == self.unknown_value: 2025-12-04T11:13:50.5464489Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.5464586Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.5464712Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.5465068Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.5465128Z graph_break [] 2025-12-04T11:13:50.5465253Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.5465354Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.5465481Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.5465829Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.5465908Z graph_break [] 2025-12-04T11:13:50.5466414Z - generated xml file: /var/lib/jenkins/workspace/test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-ca536e35d6aacc32.xml - 2025-12-04T11:13:50.5466521Z =========================== short test summary info ============================ 2025-12-04T11:13:50.5467833Z FAILED [0.4121s] inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_32_mid_dim_8_in_features_144_out_features_64_cuda_bfloat16 - RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_8_in_features_144_out_features_64_cuda_bfloat16! Caching allocator allocated memory was 18432 and is now reported as 27648 on device 0. CUDA driver allocated memory was 276824064 and is now 278921216. 2025-12-04T11:13:50.5467879Z 2025-12-04T11:13:50.5468014Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.5468544Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_8_in_features_144_out_features_64_cuda_bfloat16 2025-12-04T11:13:50.5468548Z 2025-12-04T11:13:50.5468720Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.5468827Z !!!!!!!!!!!!!!!!!!!!!!!!!! stopping after 1 failures !!!!!!!!!!!!!!!!!!!!!!!!!!! 2025-12-04T11:13:50.5468947Z ================== 1 failed, 57 deselected, 2 rerun in 12.09s ================== 2025-12-04T11:13:50.5469009Z Got exit code 1 2025-12-04T11:13:50.5469500Z FAILED CONSISTENTLY: test/inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_32_mid_dim_8_in_features_144_out_features_64_cuda_bfloat16 2025-12-04T11:13:50.5469749Z Test failed consistently, continuing with the rest of the tests due to continue-through-error being set 2025-12-04T11:13:50.5470059Z W1204 11:13:01.405000 88713 site-packages/torch/_inductor/utils.py:1703] Not enough SMs to use max_autotune_gemm mode 2025-12-04T11:13:50.5470488Z Test results will be stored in test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-732261681906a561.xml 2025-12-04T11:13:50.5470589Z ============================= test session starts ============================== 2025-12-04T11:13:50.5470800Z platform linux -- Python 3.10.14, pytest-7.3.2, pluggy-1.6.0 -- /opt/conda/envs/py_3.10/bin/python 2025-12-04T11:13:50.5470869Z cachedir: .pytest_cache 2025-12-04T11:13:50.5471200Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow] 2025-12-04T11:13:50.5471278Z rootdir: /var/lib/jenkins/workspace 2025-12-04T11:13:50.5471384Z configfile: pytest.ini 2025-12-04T11:13:50.5471702Z plugins: hypothesis-6.56.4, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-14.0, subtests-0.13.1, xdist-3.3.1, xdoctest-1.3.0, typeguard-4.3.0 2025-12-04T11:13:50.5471837Z collecting ... collected 58 items / 57 deselected / 1 selected 2025-12-04T11:13:50.5471932Z stepcurrent: skipping 57 already run items. 2025-12-04T11:13:50.5472003Z Running 1 items in this shard 2025-12-04T11:13:50.5472007Z 2025-12-04T11:13:50.5472513Z inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_32_mid_dim_8_in_features_144_out_features_65_cuda_bfloat16 ('RERUN', {'yellow': True}) [1.9032s] [100%] 2025-12-04T11:13:50.5473012Z inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_32_mid_dim_8_in_features_144_out_features_65_cuda_bfloat16 ('RERUN', {'yellow': True}) [0.4552s] [100%] 2025-12-04T11:13:50.5473463Z inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_32_mid_dim_8_in_features_144_out_features_65_cuda_bfloat16 FAILED [0.4631s] [100%] 2025-12-04T11:13:50.5473469Z 2025-12-04T11:13:50.5473558Z ==================================== RERUNS ==================================== 2025-12-04T11:13:50.5473855Z _ TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_8_in_features_144_out_features_65_cuda_bfloat16 _ 2025-12-04T11:13:50.5473936Z Traceback (most recent call last): 2025-12-04T11:13:50.5474246Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.5474312Z method(*args, **kwargs) 2025-12-04T11:13:50.5474614Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.5474679Z method(*args, **kwargs) 2025-12-04T11:13:50.5474974Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T11:13:50.5475039Z with policy(): 2025-12-04T11:13:50.5475394Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T11:13:50.5475471Z raise RuntimeError(msg) 2025-12-04T11:13:50.5476285Z RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_8_in_features_144_out_features_65_cuda_bfloat16! Caching allocator allocated memory was 0 and is now reported as 9728 on device 0. CUDA driver allocated memory was 230686720 and is now 274726912. 2025-12-04T11:13:50.5476289Z 2025-12-04T11:13:50.5476425Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.5476957Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_8_in_features_144_out_features_65_cuda_bfloat16 2025-12-04T11:13:50.5476962Z 2025-12-04T11:13:50.5477124Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.5477259Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.5477395Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.5477982Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.5478133Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.5478196Z graph_break [] 2025-12-04T11:13:50.5478502Z _ TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_8_in_features_144_out_features_65_cuda_bfloat16 _ 2025-12-04T11:13:50.5478580Z Traceback (most recent call last): 2025-12-04T11:13:50.5478975Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.5479050Z method(*args, **kwargs) 2025-12-04T11:13:50.5479355Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.5479425Z method(*args, **kwargs) 2025-12-04T11:13:50.5479723Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T11:13:50.5479785Z with policy(): 2025-12-04T11:13:50.5480095Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T11:13:50.5480162Z raise RuntimeError(msg) 2025-12-04T11:13:50.5481003Z RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_8_in_features_144_out_features_65_cuda_bfloat16! Caching allocator allocated memory was 9728 and is now reported as 19456 on device 0. CUDA driver allocated memory was 274726912 and is now 276824064. 2025-12-04T11:13:50.5481007Z 2025-12-04T11:13:50.5481142Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.5481683Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_8_in_features_144_out_features_65_cuda_bfloat16 2025-12-04T11:13:50.5481688Z 2025-12-04T11:13:50.5481857Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.5481987Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.5482092Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.5482453Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.5482581Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.5482647Z graph_break [] 2025-12-04T11:13:50.5482774Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.5482932Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.5483058Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.5483411Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.5483475Z graph_break [] 2025-12-04T11:13:50.5483562Z =================================== FAILURES =================================== 2025-12-04T11:13:50.5483861Z _ TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_8_in_features_144_out_features_65_cuda_bfloat16 _ 2025-12-04T11:13:50.5483956Z Traceback (most recent call last): 2025-12-04T11:13:50.5484276Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.5484346Z method(*args, **kwargs) 2025-12-04T11:13:50.5484650Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.5484769Z method(*args, **kwargs) 2025-12-04T11:13:50.5485115Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T11:13:50.5485179Z with policy(): 2025-12-04T11:13:50.5485481Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T11:13:50.5485551Z raise RuntimeError(msg) 2025-12-04T11:13:50.5486418Z RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_8_in_features_144_out_features_65_cuda_bfloat16! Caching allocator allocated memory was 19456 and is now reported as 29184 on device 0. CUDA driver allocated memory was 276824064 and is now 278921216. 2025-12-04T11:13:50.5486422Z 2025-12-04T11:13:50.5486561Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.5487097Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_8_in_features_144_out_features_65_cuda_bfloat16 2025-12-04T11:13:50.5487101Z 2025-12-04T11:13:50.5487268Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.5487396Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.5487491Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.5487848Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.5487975Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.5488042Z graph_break [] 2025-12-04T11:13:50.5488166Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.5488262Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.5488392Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.5488742Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.5488801Z graph_break [] 2025-12-04T11:13:50.5488931Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.5489021Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.5489150Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.5489502Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.5489561Z graph_break [] 2025-12-04T11:13:50.5490116Z - generated xml file: /var/lib/jenkins/workspace/test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-732261681906a561.xml - 2025-12-04T11:13:50.5490224Z =========================== short test summary info ============================ 2025-12-04T11:13:50.5491540Z FAILED [0.4631s] inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_32_mid_dim_8_in_features_144_out_features_65_cuda_bfloat16 - RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_8_in_features_144_out_features_65_cuda_bfloat16! Caching allocator allocated memory was 19456 and is now reported as 29184 on device 0. CUDA driver allocated memory was 276824064 and is now 278921216. 2025-12-04T11:13:50.5491545Z 2025-12-04T11:13:50.5491675Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.5492208Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_8_in_features_144_out_features_65_cuda_bfloat16 2025-12-04T11:13:50.5492250Z 2025-12-04T11:13:50.5492445Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.5492554Z !!!!!!!!!!!!!!!!!!!!!!!!!! stopping after 1 failures !!!!!!!!!!!!!!!!!!!!!!!!!!! 2025-12-04T11:13:50.5492676Z ================== 1 failed, 57 deselected, 2 rerun in 2.85s =================== 2025-12-04T11:13:50.5492735Z Got exit code 1 2025-12-04T11:13:50.5492802Z Retrying single test... 2025-12-04T11:13:50.5493074Z W1204 11:13:11.519000 88894 site-packages/torch/_inductor/utils.py:1703] Not enough SMs to use max_autotune_gemm mode 2025-12-04T11:13:50.5493503Z Test results will be stored in test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-dd0f5dd918ba20c2.xml 2025-12-04T11:13:50.5493607Z ============================= test session starts ============================== 2025-12-04T11:13:50.5493823Z platform linux -- Python 3.10.14, pytest-7.3.2, pluggy-1.6.0 -- /opt/conda/envs/py_3.10/bin/python 2025-12-04T11:13:50.5493892Z cachedir: .pytest_cache 2025-12-04T11:13:50.5494211Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow] 2025-12-04T11:13:50.5494289Z rootdir: /var/lib/jenkins/workspace 2025-12-04T11:13:50.5494357Z configfile: pytest.ini 2025-12-04T11:13:50.5494678Z plugins: hypothesis-6.56.4, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-14.0, subtests-0.13.1, xdist-3.3.1, xdoctest-1.3.0, typeguard-4.3.0 2025-12-04T11:13:50.5494812Z collecting ... collected 58 items / 57 deselected / 1 selected 2025-12-04T11:13:50.5495397Z stepcurrent: skipping 57 already run items. Running only test/inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_32_mid_dim_8_in_features_144_out_features_65_cuda_bfloat16 2025-12-04T11:13:50.5495472Z Running 1 items in this shard 2025-12-04T11:13:50.5495475Z 2025-12-04T11:13:50.5496229Z inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_32_mid_dim_8_in_features_144_out_features_65_cuda_bfloat16 [W1204 11:13:12.830996342 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.5496233Z 2025-12-04T11:13:50.5496542Z [W1204 11:13:22.954453649 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.5496546Z 2025-12-04T11:13:50.5496844Z [W1204 11:13:22.954683634 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.5496850Z 2025-12-04T11:13:50.5497147Z [W1204 11:13:22.960366972 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.5497188Z 2025-12-04T11:13:50.5497483Z [W1204 11:13:22.960936465 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.5497488Z 2025-12-04T11:13:50.5497785Z [W1204 11:13:22.961117909 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.5497788Z 2025-12-04T11:13:50.5498081Z [W1204 11:13:22.966636433 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.5498085Z 2025-12-04T11:13:50.5498382Z [W1204 11:13:22.967159545 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.5498386Z 2025-12-04T11:13:50.5498680Z [W1204 11:13:22.967315938 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.5498683Z 2025-12-04T11:13:50.5498773Z ('RERUN', {'yellow': True}) [11.0467s] [100%] 2025-12-04T11:13:50.5499594Z inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_32_mid_dim_8_in_features_144_out_features_65_cuda_bfloat16 [W1204 11:13:23.964379082 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.5499599Z 2025-12-04T11:13:50.5499899Z [W1204 11:13:23.964987206 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.5499903Z 2025-12-04T11:13:50.5500195Z [W1204 11:13:23.965126619 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.5500199Z 2025-12-04T11:13:50.5500524Z [W1204 11:13:23.968091635 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.5500534Z 2025-12-04T11:13:50.5500828Z [W1204 11:13:23.968673548 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.5500833Z 2025-12-04T11:13:50.5501129Z [W1204 11:13:23.968812992 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.5501133Z 2025-12-04T11:13:50.5501430Z [W1204 11:13:23.973477666 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.5501433Z 2025-12-04T11:13:50.5501727Z [W1204 11:13:23.973950667 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.5501730Z 2025-12-04T11:13:50.5502030Z [W1204 11:13:23.974085760 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.5502034Z 2025-12-04T11:13:50.5502118Z ('RERUN', {'yellow': True}) [0.4170s] [100%] 2025-12-04T11:13:50.5502865Z inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_32_mid_dim_8_in_features_144_out_features_65_cuda_bfloat16 [W1204 11:13:23.376335461 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.5502871Z 2025-12-04T11:13:50.5503166Z [W1204 11:13:23.376908794 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.5503169Z 2025-12-04T11:13:50.5503468Z [W1204 11:13:23.377047457 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.5503472Z 2025-12-04T11:13:50.5503807Z [W1204 11:13:23.380000263 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.5503811Z 2025-12-04T11:13:50.5504143Z [W1204 11:13:23.380594647 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.5504149Z 2025-12-04T11:13:50.5504462Z [W1204 11:13:23.380733030 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.5504466Z 2025-12-04T11:13:50.5504770Z [W1204 11:13:23.385340813 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.5504774Z 2025-12-04T11:13:50.5505074Z [W1204 11:13:23.385805074 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.5505077Z 2025-12-04T11:13:50.5505370Z [W1204 11:13:23.385938887 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.5505374Z 2025-12-04T11:13:50.5505440Z FAILED [0.4087s] [100%] 2025-12-04T11:13:50.5505445Z 2025-12-04T11:13:50.5505529Z ==================================== RERUNS ==================================== 2025-12-04T11:13:50.5505916Z _ TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_8_in_features_144_out_features_65_cuda_bfloat16 _ 2025-12-04T11:13:50.5505998Z Traceback (most recent call last): 2025-12-04T11:13:50.5506310Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.5506382Z method(*args, **kwargs) 2025-12-04T11:13:50.5506682Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.5506747Z method(*args, **kwargs) 2025-12-04T11:13:50.5507056Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T11:13:50.5507151Z with policy(): 2025-12-04T11:13:50.5507454Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T11:13:50.5507528Z raise RuntimeError(msg) 2025-12-04T11:13:50.5508340Z RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_8_in_features_144_out_features_65_cuda_bfloat16! Caching allocator allocated memory was 0 and is now reported as 9728 on device 0. CUDA driver allocated memory was 230686720 and is now 274726912. 2025-12-04T11:13:50.5508345Z 2025-12-04T11:13:50.5508480Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.5509009Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_8_in_features_144_out_features_65_cuda_bfloat16 2025-12-04T11:13:50.5509014Z 2025-12-04T11:13:50.5509184Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.5509315Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.5509414Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.5509792Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.5509924Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.5509994Z graph_break [] 2025-12-04T11:13:50.5510121Z ----------------------------- Captured stderr call ----------------------------- 2025-12-04T11:13:50.5510825Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/constant_folding.py:256: UserWarning: Unsupported unwinding pattern: Address not in range (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/profiler/unwind/unwind.cpp:219.) 2025-12-04T11:13:50.5510915Z if out == self.unknown_value: 2025-12-04T11:13:50.5511217Z _ TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_8_in_features_144_out_features_65_cuda_bfloat16 _ 2025-12-04T11:13:50.5511337Z Traceback (most recent call last): 2025-12-04T11:13:50.5511650Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.5511714Z method(*args, **kwargs) 2025-12-04T11:13:50.5512020Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.5512084Z method(*args, **kwargs) 2025-12-04T11:13:50.5512377Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T11:13:50.5512441Z with policy(): 2025-12-04T11:13:50.5512745Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T11:13:50.5512818Z raise RuntimeError(msg) 2025-12-04T11:13:50.5513684Z RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_8_in_features_144_out_features_65_cuda_bfloat16! Caching allocator allocated memory was 9728 and is now reported as 19456 on device 0. CUDA driver allocated memory was 274726912 and is now 276824064. 2025-12-04T11:13:50.5513721Z 2025-12-04T11:13:50.5513854Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.5514391Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_8_in_features_144_out_features_65_cuda_bfloat16 2025-12-04T11:13:50.5514395Z 2025-12-04T11:13:50.5514556Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.5514728Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.5514828Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.5515185Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.5515322Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.5515382Z graph_break [] 2025-12-04T11:13:50.5515513Z ----------------------------- Captured stderr call ----------------------------- 2025-12-04T11:13:50.5516215Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/constant_folding.py:256: UserWarning: Unsupported unwinding pattern: Address not in range (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/profiler/unwind/unwind.cpp:219.) 2025-12-04T11:13:50.5516287Z if out == self.unknown_value: 2025-12-04T11:13:50.5516415Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.5516509Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.5516640Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.5516995Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.5517053Z graph_break [] 2025-12-04T11:13:50.5517141Z =================================== FAILURES =================================== 2025-12-04T11:13:50.5517437Z _ TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_8_in_features_144_out_features_65_cuda_bfloat16 _ 2025-12-04T11:13:50.5517512Z Traceback (most recent call last): 2025-12-04T11:13:50.5517817Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.5517881Z method(*args, **kwargs) 2025-12-04T11:13:50.5518184Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.5518289Z method(*args, **kwargs) 2025-12-04T11:13:50.5518584Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T11:13:50.5518651Z with policy(): 2025-12-04T11:13:50.5518951Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T11:13:50.5519019Z raise RuntimeError(msg) 2025-12-04T11:13:50.5519864Z RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_8_in_features_144_out_features_65_cuda_bfloat16! Caching allocator allocated memory was 19456 and is now reported as 29184 on device 0. CUDA driver allocated memory was 276824064 and is now 278921216. 2025-12-04T11:13:50.5519868Z 2025-12-04T11:13:50.5519996Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.5520533Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_8_in_features_144_out_features_65_cuda_bfloat16 2025-12-04T11:13:50.5520656Z 2025-12-04T11:13:50.5521242Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.5521381Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.5521476Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.5521828Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.5521961Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.5522022Z graph_break [] 2025-12-04T11:13:50.5522161Z ----------------------------- Captured stderr call ----------------------------- 2025-12-04T11:13:50.5522905Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/constant_folding.py:256: UserWarning: Unsupported unwinding pattern: Address not in range (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/profiler/unwind/unwind.cpp:219.) 2025-12-04T11:13:50.5522981Z if out == self.unknown_value: 2025-12-04T11:13:50.5523111Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.5523208Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.5523332Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.5523690Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.5523749Z graph_break [] 2025-12-04T11:13:50.5523876Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.5523968Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.5524093Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.5524451Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.5524514Z graph_break [] 2025-12-04T11:13:50.5525020Z - generated xml file: /var/lib/jenkins/workspace/test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-dd0f5dd918ba20c2.xml - 2025-12-04T11:13:50.5525123Z =========================== short test summary info ============================ 2025-12-04T11:13:50.5526438Z FAILED [0.4087s] inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_32_mid_dim_8_in_features_144_out_features_65_cuda_bfloat16 - RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_8_in_features_144_out_features_65_cuda_bfloat16! Caching allocator allocated memory was 19456 and is now reported as 29184 on device 0. CUDA driver allocated memory was 276824064 and is now 278921216. 2025-12-04T11:13:50.5526512Z 2025-12-04T11:13:50.5526643Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.5527176Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_8_in_features_144_out_features_65_cuda_bfloat16 2025-12-04T11:13:50.5527180Z 2025-12-04T11:13:50.5527346Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.5527452Z !!!!!!!!!!!!!!!!!!!!!!!!!! stopping after 1 failures !!!!!!!!!!!!!!!!!!!!!!!!!!! 2025-12-04T11:13:50.5527577Z ================== 1 failed, 57 deselected, 2 rerun in 11.90s ================== 2025-12-04T11:13:50.5527636Z Got exit code 1 2025-12-04T11:13:50.5527701Z Retrying single test... 2025-12-04T11:13:50.5527987Z W1204 11:13:30.208000 89080 site-packages/torch/_inductor/utils.py:1703] Not enough SMs to use max_autotune_gemm mode 2025-12-04T11:13:50.5528425Z Test results will be stored in test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-2967af23244bf7ea.xml 2025-12-04T11:13:50.5528560Z ============================= test session starts ============================== 2025-12-04T11:13:50.5528781Z platform linux -- Python 3.10.14, pytest-7.3.2, pluggy-1.6.0 -- /opt/conda/envs/py_3.10/bin/python 2025-12-04T11:13:50.5528849Z cachedir: .pytest_cache 2025-12-04T11:13:50.5529167Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow] 2025-12-04T11:13:50.5529248Z rootdir: /var/lib/jenkins/workspace 2025-12-04T11:13:50.5529316Z configfile: pytest.ini 2025-12-04T11:13:50.5529685Z plugins: hypothesis-6.56.4, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-14.0, subtests-0.13.1, xdist-3.3.1, xdoctest-1.3.0, typeguard-4.3.0 2025-12-04T11:13:50.5529824Z collecting ... collected 58 items / 57 deselected / 1 selected 2025-12-04T11:13:50.5530416Z stepcurrent: skipping 57 already run items. Running only test/inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_32_mid_dim_8_in_features_144_out_features_65_cuda_bfloat16 2025-12-04T11:13:50.5530491Z Running 1 items in this shard 2025-12-04T11:13:50.5530494Z 2025-12-04T11:13:50.5531244Z inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_32_mid_dim_8_in_features_144_out_features_65_cuda_bfloat16 [W1204 11:13:31.507621028 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.5531258Z 2025-12-04T11:13:50.5531572Z [W1204 11:13:40.581989301 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.5531577Z 2025-12-04T11:13:50.5531874Z [W1204 11:13:40.582226096 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.5531880Z 2025-12-04T11:13:50.5532183Z [W1204 11:13:40.587867921 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.5532186Z 2025-12-04T11:13:50.5532479Z [W1204 11:13:40.588411283 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.5532483Z 2025-12-04T11:13:50.5532780Z [W1204 11:13:40.588582547 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.5532783Z 2025-12-04T11:13:50.5533077Z [W1204 11:13:40.594081118 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.5533080Z 2025-12-04T11:13:50.5533380Z [W1204 11:13:40.594619950 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.5533439Z 2025-12-04T11:13:50.5533735Z [W1204 11:13:40.594778224 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.5533740Z 2025-12-04T11:13:50.5533825Z ('RERUN', {'yellow': True}) [10.9824s] [100%] 2025-12-04T11:13:50.5534570Z inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_32_mid_dim_8_in_features_144_out_features_65_cuda_bfloat16 [W1204 11:13:41.588488499 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.5534574Z 2025-12-04T11:13:50.5534871Z [W1204 11:13:41.589080352 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.5534875Z 2025-12-04T11:13:50.5535172Z [W1204 11:13:41.589228325 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.5535213Z 2025-12-04T11:13:50.5535540Z [W1204 11:13:41.592242592 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.5535544Z 2025-12-04T11:13:50.5535845Z [W1204 11:13:41.592843035 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.5535848Z 2025-12-04T11:13:50.5536141Z [W1204 11:13:41.592982558 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.5536144Z 2025-12-04T11:13:50.5536444Z [W1204 11:13:41.597646432 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.5536447Z 2025-12-04T11:13:50.5536774Z [W1204 11:13:41.598128382 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.5536779Z 2025-12-04T11:13:50.5537077Z [W1204 11:13:41.598263405 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.5537082Z 2025-12-04T11:13:50.5537165Z ('RERUN', {'yellow': True}) [0.4154s] [100%] 2025-12-04T11:13:50.5537905Z inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_32_mid_dim_8_in_features_144_out_features_65_cuda_bfloat16 [W1204 11:13:42.002245365 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.5537913Z 2025-12-04T11:13:50.5538208Z [W1204 11:13:42.002838188 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.5538212Z 2025-12-04T11:13:50.5538506Z [W1204 11:13:42.002980731 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.5538511Z 2025-12-04T11:13:50.5538810Z [W1204 11:13:42.005992728 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.5538816Z 2025-12-04T11:13:50.5539110Z [W1204 11:13:42.006566591 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.5539113Z 2025-12-04T11:13:50.5539412Z [W1204 11:13:42.006702383 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.5539415Z 2025-12-04T11:13:50.5539709Z [W1204 11:13:42.011329576 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.5539712Z 2025-12-04T11:13:50.5540009Z [W1204 11:13:42.011804846 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.5540050Z 2025-12-04T11:13:50.5540348Z [W1204 11:13:42.011941359 Module.cpp:201] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1... 2025-12-04T11:13:50.5540353Z 2025-12-04T11:13:50.5540416Z FAILED [0.4110s] [100%] 2025-12-04T11:13:50.5540424Z 2025-12-04T11:13:50.5540508Z ==================================== RERUNS ==================================== 2025-12-04T11:13:50.5540807Z _ TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_8_in_features_144_out_features_65_cuda_bfloat16 _ 2025-12-04T11:13:50.5540887Z Traceback (most recent call last): 2025-12-04T11:13:50.5541201Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.5541268Z method(*args, **kwargs) 2025-12-04T11:13:50.5541573Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.5541640Z method(*args, **kwargs) 2025-12-04T11:13:50.5541939Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T11:13:50.5542049Z with policy(): 2025-12-04T11:13:50.5542387Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T11:13:50.5542460Z raise RuntimeError(msg) 2025-12-04T11:13:50.5543270Z RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_8_in_features_144_out_features_65_cuda_bfloat16! Caching allocator allocated memory was 0 and is now reported as 9728 on device 0. CUDA driver allocated memory was 230686720 and is now 274726912. 2025-12-04T11:13:50.5543274Z 2025-12-04T11:13:50.5543446Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.5544026Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_8_in_features_144_out_features_65_cuda_bfloat16 2025-12-04T11:13:50.5544034Z 2025-12-04T11:13:50.5544200Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.5544335Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.5544434Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.5544809Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.5544941Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.5545001Z graph_break [] 2025-12-04T11:13:50.5545131Z ----------------------------- Captured stderr call ----------------------------- 2025-12-04T11:13:50.5545838Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/constant_folding.py:256: UserWarning: Unsupported unwinding pattern: Address not in range (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/profiler/unwind/unwind.cpp:219.) 2025-12-04T11:13:50.5545917Z if out == self.unknown_value: 2025-12-04T11:13:50.5546219Z _ TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_8_in_features_144_out_features_65_cuda_bfloat16 _ 2025-12-04T11:13:50.5546297Z Traceback (most recent call last): 2025-12-04T11:13:50.5546604Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.5546671Z method(*args, **kwargs) 2025-12-04T11:13:50.5546966Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.5547037Z method(*args, **kwargs) 2025-12-04T11:13:50.5547333Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T11:13:50.5547442Z with policy(): 2025-12-04T11:13:50.5547747Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T11:13:50.5547816Z raise RuntimeError(msg) 2025-12-04T11:13:50.5548641Z RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_8_in_features_144_out_features_65_cuda_bfloat16! Caching allocator allocated memory was 9728 and is now reported as 19456 on device 0. CUDA driver allocated memory was 274726912 and is now 276824064. 2025-12-04T11:13:50.5548645Z 2025-12-04T11:13:50.5548779Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.5549314Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_8_in_features_144_out_features_65_cuda_bfloat16 2025-12-04T11:13:50.5549319Z 2025-12-04T11:13:50.5549491Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.5549661Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.5549792Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.5550146Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.5550281Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.5550343Z graph_break [] 2025-12-04T11:13:50.5550469Z ----------------------------- Captured stderr call ----------------------------- 2025-12-04T11:13:50.5551214Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/constant_folding.py:256: UserWarning: Unsupported unwinding pattern: Address not in range (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/profiler/unwind/unwind.cpp:219.) 2025-12-04T11:13:50.5551290Z if out == self.unknown_value: 2025-12-04T11:13:50.5551418Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.5551511Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.5551639Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.5551994Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.5552053Z graph_break [] 2025-12-04T11:13:50.5552142Z =================================== FAILURES =================================== 2025-12-04T11:13:50.5552439Z _ TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_8_in_features_144_out_features_65_cuda_bfloat16 _ 2025-12-04T11:13:50.5552516Z Traceback (most recent call last): 2025-12-04T11:13:50.5552827Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.5552900Z method(*args, **kwargs) 2025-12-04T11:13:50.5553200Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T11:13:50.5553270Z method(*args, **kwargs) 2025-12-04T11:13:50.5553571Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T11:13:50.5553638Z with policy(): 2025-12-04T11:13:50.5553960Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T11:13:50.5554029Z raise RuntimeError(msg) 2025-12-04T11:13:50.5554868Z RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_8_in_features_144_out_features_65_cuda_bfloat16! Caching allocator allocated memory was 19456 and is now reported as 29184 on device 0. CUDA driver allocated memory was 276824064 and is now 278921216. 2025-12-04T11:13:50.5554913Z 2025-12-04T11:13:50.5555047Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.5555588Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_8_in_features_144_out_features_65_cuda_bfloat16 2025-12-04T11:13:50.5555592Z 2025-12-04T11:13:50.5555757Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.5555883Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.5555982Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.5556341Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.5556478Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.5556539Z graph_break [] 2025-12-04T11:13:50.5556702Z ----------------------------- Captured stderr call ----------------------------- 2025-12-04T11:13:50.5557461Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/constant_folding.py:256: UserWarning: Unsupported unwinding pattern: Address not in range (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/profiler/unwind/unwind.cpp:219.) 2025-12-04T11:13:50.5557534Z if out == self.unknown_value: 2025-12-04T11:13:50.5557662Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.5557755Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.5557880Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.5558288Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.5558352Z graph_break [] 2025-12-04T11:13:50.5558477Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:13:50.5558577Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:13:50.5558702Z aot_autograd [('total', 1), ('autograd_cache_bypass', 1), ('ok', 1)] 2025-12-04T11:13:50.5559062Z inductor [('pattern_matcher_nodes', 8), ('woq_matcher_nodes', 6), ('pattern_matcher_count', 3), ('fxgraph_cache_miss', 1), ('woq_matcher_count', 1), ('extern_calls', 1)] 2025-12-04T11:13:50.5559127Z graph_break [] 2025-12-04T11:13:50.5559636Z - generated xml file: /var/lib/jenkins/workspace/test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-2967af23244bf7ea.xml - 2025-12-04T11:13:50.5559745Z =========================== short test summary info ============================ 2025-12-04T11:13:50.5561060Z FAILED [0.4110s] inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_32_mid_dim_8_in_features_144_out_features_65_cuda_bfloat16 - RuntimeError: CUDA driver API confirmed a leak in __main__.TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_8_in_features_144_out_features_65_cuda_bfloat16! Caching allocator allocated memory was 19456 and is now reported as 29184 on device 0. CUDA driver allocated memory was 276824064 and is now 278921216. 2025-12-04T11:13:50.5561068Z 2025-12-04T11:13:50.5561206Z To execute this test, run the following from the base repo dir: 2025-12-04T11:13:50.5561732Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cuda_select_algorithm.py TestSelectAlgorithmCudaCUDA.test_int8_woq_mm_cuda_batch_size_32_mid_dim_8_in_features_144_out_features_65_cuda_bfloat16 2025-12-04T11:13:50.5561736Z 2025-12-04T11:13:50.5561907Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:13:50.5562016Z !!!!!!!!!!!!!!!!!!!!!!!!!! stopping after 1 failures !!!!!!!!!!!!!!!!!!!!!!!!!!! 2025-12-04T11:13:50.5562176Z ================== 1 failed, 57 deselected, 2 rerun in 11.83s ================== 2025-12-04T11:13:50.5562242Z Got exit code 1 2025-12-04T11:13:50.5562729Z FAILED CONSISTENTLY: test/inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_32_mid_dim_8_in_features_144_out_features_65_cuda_bfloat16 2025-12-04T11:13:50.5562985Z Test failed consistently, continuing with the rest of the tests due to continue-through-error being set 2025-12-04T11:13:50.5563255Z W1204 11:13:48.844000 89266 site-packages/torch/_inductor/utils.py:1703] Not enough SMs to use max_autotune_gemm mode 2025-12-04T11:13:50.5563650Z Test results will be stored in test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-defe8bc47342e70f.xml 2025-12-04T11:13:50.5563754Z ============================= test session starts ============================== 2025-12-04T11:13:50.5563966Z platform linux -- Python 3.10.14, pytest-7.3.2, pluggy-1.6.0 -- /opt/conda/envs/py_3.10/bin/python 2025-12-04T11:13:50.5564040Z cachedir: .pytest_cache 2025-12-04T11:13:50.5564393Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow] 2025-12-04T11:13:50.5564507Z rootdir: /var/lib/jenkins/workspace 2025-12-04T11:13:50.5564580Z configfile: pytest.ini 2025-12-04T11:13:50.5564902Z plugins: hypothesis-6.56.4, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-14.0, subtests-0.13.1, xdist-3.3.1, xdoctest-1.3.0, typeguard-4.3.0 2025-12-04T11:13:50.5565034Z collecting ... collected 58 items / 58 deselected / 0 selected 2025-12-04T11:13:50.5565131Z stepcurrent: skipping 58 already run items. 2025-12-04T11:13:50.5565202Z Running 0 items in this shard 2025-12-04T11:13:50.5565206Z 2025-12-04T11:13:50.5565750Z - generated xml file: /var/lib/jenkins/workspace/test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-defe8bc47342e70f.xml - 2025-12-04T11:13:50.5565854Z ============================ 58 deselected in 0.01s ============================ 2025-12-04T11:13:50.5592549Z The following tests failed consistently: ['test/inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_concat_cuda_batch_size_1_mid_dim_1_in_features_128_out_features_64_cuda_bfloat16', 'test/inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_concat_cuda_batch_size_1_mid_dim_8_in_features_128_out_features_64_cuda_bfloat16', 'test/inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_concat_cuda_batch_size_32_mid_dim_1_in_features_128_out_features_64_cuda_bfloat16', 'test/inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_concat_cuda_batch_size_32_mid_dim_8_in_features_128_out_features_64_cuda_bfloat16', 'test/inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_17_mid_dim_1_in_features_1024_out_features_1024_cuda_bfloat16', 'test/inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_17_mid_dim_1_in_features_1024_out_features_64_cuda_bfloat16', 'test/inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_17_mid_dim_1_in_features_1024_out_features_65_cuda_bfloat16', 'test/inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_17_mid_dim_1_in_features_128_out_features_1024_cuda_bfloat16', 'test/inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_17_mid_dim_1_in_features_128_out_features_64_cuda_bfloat16', 'test/inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_17_mid_dim_1_in_features_128_out_features_65_cuda_bfloat16', 'test/inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_17_mid_dim_1_in_features_144_out_features_1024_cuda_bfloat16', 'test/inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_17_mid_dim_1_in_features_144_out_features_64_cuda_bfloat16', 'test/inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_17_mid_dim_1_in_features_144_out_features_65_cuda_bfloat16', 'test/inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_17_mid_dim_8_in_features_1024_out_features_1024_cuda_bfloat16', 'test/inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_17_mid_dim_8_in_features_1024_out_features_64_cuda_bfloat16', 'test/inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_17_mid_dim_8_in_features_1024_out_features_65_cuda_bfloat16', 'test/inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_17_mid_dim_8_in_features_128_out_features_1024_cuda_bfloat16', 'test/inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_17_mid_dim_8_in_features_128_out_features_64_cuda_bfloat16', 'test/inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_17_mid_dim_8_in_features_128_out_features_65_cuda_bfloat16', 'test/inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_17_mid_dim_8_in_features_144_out_features_1024_cuda_bfloat16', 'test/inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_17_mid_dim_8_in_features_144_out_features_64_cuda_bfloat16', 'test/inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_17_mid_dim_8_in_features_144_out_features_65_cuda_bfloat16', 'test/inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_1_mid_dim_1_in_features_1024_out_features_1024_cuda_bfloat16', 'test/inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_1_mid_dim_1_in_features_1024_out_features_64_cuda_bfloat16', 'test/inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_1_mid_dim_1_in_features_1024_out_features_65_cuda_bfloat16', 'test/inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_1_mid_dim_1_in_features_128_out_features_1024_cuda_bfloat16', 'test/inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_1_mid_dim_1_in_features_128_out_features_64_cuda_bfloat16', 'test/inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_1_mid_dim_1_in_features_128_out_features_65_cuda_bfloat16', 'test/inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_1_mid_dim_1_in_features_144_out_features_1024_cuda_bfloat16', 'test/inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_1_mid_dim_1_in_features_144_out_features_64_cuda_bfloat16', 'test/inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_1_mid_dim_1_in_features_144_out_features_65_cuda_bfloat16', 'test/inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_1_mid_dim_8_in_features_1024_out_features_1024_cuda_bfloat16', 'test/inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_1_mid_dim_8_in_features_1024_out_features_64_cuda_bfloat16', 'test/inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_1_mid_dim_8_in_features_1024_out_features_65_cuda_bfloat16', 'test/inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_1_mid_dim_8_in_features_128_out_features_1024_cuda_bfloat16', 'test/inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_1_mid_dim_8_in_features_128_out_features_64_cuda_bfloat16', 'test/inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_1_mid_dim_8_in_features_128_out_features_65_cuda_bfloat16', 'test/inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_1_mid_dim_8_in_features_144_out_features_1024_cuda_bfloat16', 'test/inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_1_mid_dim_8_in_features_144_out_features_64_cuda_bfloat16', 'test/inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_1_mid_dim_8_in_features_144_out_features_65_cuda_bfloat16', 'test/inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_32_mid_dim_1_in_features_1024_out_features_1024_cuda_bfloat16', 'test/inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_32_mid_dim_1_in_features_1024_out_features_64_cuda_bfloat16', 'test/inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_32_mid_dim_1_in_features_1024_out_features_65_cuda_bfloat16', 'test/inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_32_mid_dim_1_in_features_128_out_features_1024_cuda_bfloat16', 'test/inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_32_mid_dim_1_in_features_128_out_features_64_cuda_bfloat16', 'test/inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_32_mid_dim_1_in_features_128_out_features_65_cuda_bfloat16', 'test/inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_32_mid_dim_1_in_features_144_out_features_1024_cuda_bfloat16', 'test/inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_32_mid_dim_1_in_features_144_out_features_64_cuda_bfloat16', 'test/inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_32_mid_dim_1_in_features_144_out_features_65_cuda_bfloat16', 'test/inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_32_mid_dim_8_in_features_1024_out_features_1024_cuda_bfloat16', 'test/inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_32_mid_dim_8_in_features_1024_out_features_64_cuda_bfloat16', 'test/inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_32_mid_dim_8_in_features_1024_out_features_65_cuda_bfloat16', 'test/inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_32_mid_dim_8_in_features_128_out_features_1024_cuda_bfloat16', 'test/inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_32_mid_dim_8_in_features_128_out_features_64_cuda_bfloat16', 'test/inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_32_mid_dim_8_in_features_128_out_features_65_cuda_bfloat16', 'test/inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_32_mid_dim_8_in_features_144_out_features_1024_cuda_bfloat16', 'test/inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_32_mid_dim_8_in_features_144_out_features_64_cuda_bfloat16', 'test/inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_32_mid_dim_8_in_features_144_out_features_65_cuda_bfloat16'] 2025-12-04T11:13:50.5592744Z 2025-12-04T11:13:50.5593174Z FINISHED PRINTING LOG FILE of inductor/test_cuda_select_algorithm 1/1 (test/test-reports/inductor.test_cuda_select_algorithm_1.1_4871a8a3689f9e63_.log) 2025-12-04T11:13:50.5593215Z 2025-12-04T11:13:50.5593457Z Finished inductor/test_cuda_select_algorithm 1/1 ... [2025-12-04 11:13:49.934063][5677.876271403], took 46.43min 2025-12-04T11:13:50.5594066Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-9c66153466707d74.xml 2025-12-04T11:13:50.5594674Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-11449d70dc195ff4.xml 2025-12-04T11:13:50.5595329Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-6c093515a384d9fc.xml 2025-12-04T11:13:50.5595969Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-7ea688002f0554bd.xml 2025-12-04T11:13:50.5596494Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-759714b1d86e3c4c.xml 2025-12-04T11:13:50.5597133Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-7a190e1b598a9023.xml 2025-12-04T11:13:50.5597650Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-c45e67249599e6c2.xml 2025-12-04T11:13:50.5598166Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-8112676a6b085940.xml 2025-12-04T11:13:50.5598739Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-2280a7982827e03d.xml 2025-12-04T11:13:50.5599272Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-ecd8523a161788bb.xml 2025-12-04T11:13:50.5599786Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-d90aab22855519e4.xml 2025-12-04T11:13:50.5600304Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-5cb074eac86f05e5.xml 2025-12-04T11:13:50.5600823Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-7681ec3fadfa4a43.xml 2025-12-04T11:13:50.5601336Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-49c0a334019b663e.xml 2025-12-04T11:13:50.5601953Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-a4c3d2e69c8841e3.xml 2025-12-04T11:13:50.5602633Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-0f60f7321324a86a.xml 2025-12-04T11:13:50.5603162Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-294e2abb077cceb2.xml 2025-12-04T11:13:50.5603689Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-b24025703d1a5b1f.xml 2025-12-04T11:13:50.5856566Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-531381cfa4745e03.xml 2025-12-04T11:13:50.6167386Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-bb14ed3958363693.xml 2025-12-04T11:13:50.6447574Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-ec0356b9a2627168.xml 2025-12-04T11:13:50.6755305Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-c81b585b68f6cf60.xml 2025-12-04T11:13:50.7012975Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-647efcde8452d0c7.xml 2025-12-04T11:13:50.7296047Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-abfd2b283c5f3df6.xml 2025-12-04T11:13:50.7594894Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-e0f3393878857c02.xml 2025-12-04T11:13:50.7919712Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-d5da2e41e9fcecb9.xml 2025-12-04T11:13:50.8203222Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-5fb75fde833c163c.xml 2025-12-04T11:13:50.8445056Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-2f9606f02b59f87b.xml 2025-12-04T11:13:50.8716123Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-94458b88fceefe4e.xml 2025-12-04T11:13:50.9005228Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-6787c22d74999180.xml 2025-12-04T11:13:50.9295069Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-cdf948359e80bb41.xml 2025-12-04T11:13:50.9575590Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-bd192fd6fb52c26e.xml 2025-12-04T11:13:50.9918306Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-7d48593db7bb6f60.xml 2025-12-04T11:13:51.0231083Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-91ed99958e7f9cc9.xml 2025-12-04T11:13:51.0504558Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-f4094aba19e671d4.xml 2025-12-04T11:13:51.0827884Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-204edfd0c56893dd.xml 2025-12-04T11:13:51.1233250Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-5d3545fec05b298e.xml 2025-12-04T11:13:51.1506844Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-c2388e46be16ba24.xml 2025-12-04T11:13:51.1794802Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-e9cf01ceaaab6f89.xml 2025-12-04T11:13:51.2106409Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-ff104bb89c063280.xml 2025-12-04T11:13:51.2405187Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-61993e92defaa624.xml 2025-12-04T11:13:51.2724849Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-16a80675dc6db2f9.xml 2025-12-04T11:13:51.3016965Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-601480a9a7ef82d7.xml 2025-12-04T11:13:51.3347525Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-fda38496367dc0a9.xml 2025-12-04T11:13:51.3687886Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-13c7a98252f889c4.xml 2025-12-04T11:13:51.4034719Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-d1b821ab0fc5c966.xml 2025-12-04T11:13:51.4323647Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-d70416164f95a17a.xml 2025-12-04T11:13:51.4606810Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-a1832326f6932688.xml 2025-12-04T11:13:51.4888105Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-1965a836c7535ef2.xml 2025-12-04T11:13:51.5186192Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-12f95c060627756e.xml 2025-12-04T11:13:51.5493614Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-c82b9fa27197518a.xml 2025-12-04T11:13:51.5793167Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-36ce77ac1c8c9763.xml 2025-12-04T11:13:51.6137193Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-e4f38ecbc39decbb.xml 2025-12-04T11:13:51.6485326Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-e20c92a1eaab2de9.xml 2025-12-04T11:13:51.6744982Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-2048f1d3bc205458.xml 2025-12-04T11:13:51.7017406Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-d670b011b4f3da72.xml 2025-12-04T11:13:51.7286663Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-56e0b2ecffd4aa2f.xml 2025-12-04T11:13:51.7550527Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-1e9198c3a728666b.xml 2025-12-04T11:13:51.8216107Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-c1c9525c22bede8d.xml 2025-12-04T11:13:51.8485976Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-cc4ec46989be7746.xml 2025-12-04T11:13:51.8955045Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-3f42debbb8d01db8.xml 2025-12-04T11:13:51.9245758Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-92851bf92613c0d7.xml 2025-12-04T11:13:51.9528233Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-3610b3094a676974.xml 2025-12-04T11:13:51.9853618Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-dba8bfd8733d5e09.xml 2025-12-04T11:13:52.0146277Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-6c64eb8af833b17a.xml 2025-12-04T11:13:52.0426377Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-f878b685b7c3b00a.xml 2025-12-04T11:13:52.0744428Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-232061e660a2ff26.xml 2025-12-04T11:13:52.1039806Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-fe4ea69fd2c99307.xml 2025-12-04T11:13:52.1295405Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-4bc55824cd7c5544.xml 2025-12-04T11:13:52.1605579Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-c2fc9a8f37b6f624.xml 2025-12-04T11:13:52.1894582Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-df0458d9683e0406.xml 2025-12-04T11:13:52.2197632Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-1f5454d7efc5fad5.xml 2025-12-04T11:13:52.2454988Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-0e640f0818a9f631.xml 2025-12-04T11:13:52.2742340Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-e912b3c1622dc5a6.xml 2025-12-04T11:13:52.3225690Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-9bf082709afd4107.xml 2025-12-04T11:13:52.3495783Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-417661a6a446d83a.xml 2025-12-04T11:13:52.3786437Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-2d84d0ab3f5cbf16.xml 2025-12-04T11:13:52.4079220Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-3c429c2af5dedece.xml 2025-12-04T11:13:52.4355191Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-b9ae48951e1b6488.xml 2025-12-04T11:13:52.4665955Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-a9948af371c98fbd.xml 2025-12-04T11:13:52.4935081Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-f058047ef4d14b46.xml 2025-12-04T11:13:52.5219602Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-5d247f52fbae6573.xml 2025-12-04T11:13:52.5555463Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-b685fc5b32b50b90.xml 2025-12-04T11:13:52.5898264Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-bc2410b75aa40646.xml 2025-12-04T11:13:52.6198044Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-71cfec7d40715d91.xml 2025-12-04T11:13:52.6486757Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-c9dbc1d234ce26dc.xml 2025-12-04T11:13:52.6778813Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-8a076bf12e47858e.xml 2025-12-04T11:13:52.7036880Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-bb0bac5f3b803919.xml 2025-12-04T11:13:52.7372571Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-0645969e8b49cd4b.xml 2025-12-04T11:13:52.7667799Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-20accf8a0f4e3024.xml 2025-12-04T11:13:52.8094104Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-55dc1e1f737da98a.xml 2025-12-04T11:13:52.8478694Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-458d9d61d3eb81db.xml 2025-12-04T11:13:52.8759321Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-0cf2b0bead25849b.xml 2025-12-04T11:13:52.9033988Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-e47d4d68d26f0ad1.xml 2025-12-04T11:13:52.9298011Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-044dfd62cda981dc.xml 2025-12-04T11:13:52.9612014Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-f4c669524343c1b1.xml 2025-12-04T11:13:52.9900474Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-9f9264fbba4173a1.xml 2025-12-04T11:13:53.0187722Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-c85f34f0c90d933d.xml 2025-12-04T11:13:53.0649478Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-4b2e7fd1ed886dfe.xml 2025-12-04T11:13:53.0935387Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-f54f533501201f35.xml 2025-12-04T11:13:53.1220393Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-cbbd83fa84eb990f.xml 2025-12-04T11:13:53.2235403Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-67e17348ec4a01d8.xml 2025-12-04T11:13:53.2630649Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-f72cf4cc8675ff8a.xml 2025-12-04T11:13:53.2947609Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-bd700a0c553d5cfe.xml 2025-12-04T11:13:53.3267552Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-28b04335f093e360.xml 2025-12-04T11:13:53.3583718Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-857ce5cc87a4a050.xml 2025-12-04T11:13:53.3866214Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-4e0b90b7cea75841.xml 2025-12-04T11:13:53.4147930Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-14983efcd69bf6ac.xml 2025-12-04T11:13:53.4445902Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-2104559418af7772.xml 2025-12-04T11:13:53.4746709Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-49eccf27aa184bc6.xml 2025-12-04T11:13:53.5117880Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-71736827b1650ca5.xml 2025-12-04T11:13:53.5446186Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-b5c5c102e82be0c3.xml 2025-12-04T11:13:53.5738001Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-6b2f788f9a7d50ec.xml 2025-12-04T11:13:53.6035967Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-61d7ccaaf8cfd370.xml 2025-12-04T11:13:53.6356192Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-cc96100da976ce4e.xml 2025-12-04T11:13:53.6715178Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-e2ca8845caeab461.xml 2025-12-04T11:13:53.6999225Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-ea9e1f7f9950a9ad.xml 2025-12-04T11:13:53.7314029Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-778cc31dcd227b51.xml 2025-12-04T11:13:53.7606955Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-d8517590777959a7.xml 2025-12-04T11:13:53.7916806Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-7c1369cc4e5815f4.xml 2025-12-04T11:13:53.8203266Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-f43c4b9f36164ff6.xml 2025-12-04T11:13:53.8509881Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-d56aa1824afdfad3.xml 2025-12-04T11:13:53.8826168Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-6a9d289c68c250dc.xml 2025-12-04T11:13:53.9127538Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-04e5068c64f117f5.xml 2025-12-04T11:13:53.9434100Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-85ab96cf3cf02016.xml 2025-12-04T11:13:53.9800893Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-888ba6ece96c2997.xml 2025-12-04T11:13:54.0137277Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-58cf8d0fed87176f.xml 2025-12-04T11:13:54.0458735Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-0022ebcfefe802db.xml 2025-12-04T11:13:54.0785750Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-dac6db8ed6c97e37.xml 2025-12-04T11:13:54.1065058Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-040b4092def69522.xml 2025-12-04T11:13:54.1363836Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-084397a772e6c7f7.xml 2025-12-04T11:13:54.1660369Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-2942f663738ee80d.xml 2025-12-04T11:13:54.1956127Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-ac0d8e76747a1a1b.xml 2025-12-04T11:13:54.2258375Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-7f452630b39063ce.xml 2025-12-04T11:13:54.2540682Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-bb668a52e682b2dd.xml 2025-12-04T11:13:54.2886133Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-60466f6816d813eb.xml 2025-12-04T11:13:54.3188020Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-3b52d4f5cebad92d.xml 2025-12-04T11:13:54.3480580Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-2ff4d68a3a0ad43c.xml 2025-12-04T11:13:54.3780869Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-7240a6bd78b3c1ba.xml 2025-12-04T11:13:54.4367265Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-7cb1dae7cd5b9c6d.xml 2025-12-04T11:13:54.4660718Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-1238d86169b8b87a.xml 2025-12-04T11:13:54.4958628Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-86e4cbaa0e57366b.xml 2025-12-04T11:13:54.5263426Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-1c1918cf02456db6.xml 2025-12-04T11:13:54.5548727Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-4d7a6e50e4cb57cd.xml 2025-12-04T11:13:54.5845005Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-cd0db30d38ee28ec.xml 2025-12-04T11:13:54.6134228Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-97331d68f5e395ac.xml 2025-12-04T11:13:54.6507442Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-266b0df1c83533e9.xml 2025-12-04T11:13:54.6787858Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-0728186d53d0914c.xml 2025-12-04T11:13:54.7070750Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-ace5a67d4c3c2a01.xml 2025-12-04T11:13:54.7358280Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-eeeef84645c1bb52.xml 2025-12-04T11:13:54.7658134Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-ffbbb699e0b6f991.xml 2025-12-04T11:13:54.7926109Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-90e4a91c9bbb810f.xml 2025-12-04T11:13:54.8215955Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-2ee8d3487d6282f0.xml 2025-12-04T11:13:54.8536418Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-2ab4541f9392b10c.xml 2025-12-04T11:13:54.8846661Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-ce43d3c4fee34ad0.xml 2025-12-04T11:13:54.9189135Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-d04b19868e6e9971.xml 2025-12-04T11:13:54.9477562Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-cf6ace4ea5db8dfb.xml 2025-12-04T11:13:54.9780299Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-8b65a3dabcd39e70.xml 2025-12-04T11:13:55.0118529Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-c85748e91a458eac.xml 2025-12-04T11:13:55.0427776Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-85bca917f8457708.xml 2025-12-04T11:13:55.0814277Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-6a70392d2e5080cc.xml 2025-12-04T11:13:55.1116150Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-1b506e26b1410ad4.xml 2025-12-04T11:13:55.1434825Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-f7f9c3afca4dce49.xml 2025-12-04T11:13:55.1865203Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-c41d68e69339995e.xml 2025-12-04T11:13:55.2189792Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-dfa47b8dd194f01b.xml 2025-12-04T11:13:55.2730873Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-6b849be23abc3e95.xml 2025-12-04T11:13:55.3566031Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-1eeada475e93262f.xml 2025-12-04T11:13:55.3868952Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-3a596bc2a015c0a5.xml 2025-12-04T11:13:55.4177753Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-ff629078718dd71f.xml 2025-12-04T11:13:55.4489666Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-2cc0c50f93d222cd.xml 2025-12-04T11:13:55.4809268Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-ca536e35d6aacc32.xml 2025-12-04T11:13:55.5285012Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-732261681906a561.xml 2025-12-04T11:13:55.5558538Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-dd0f5dd918ba20c2.xml 2025-12-04T11:13:55.5843343Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-2967af23244bf7ea.xml 2025-12-04T11:13:55.6166093Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-defe8bc47342e70f.xml 2025-12-04T11:13:56.0686603Z Uploading logs for 57120265687 to S3 2025-12-04T11:13:56.1403482Z Uploading artifacts took 0.50 seconds 2025-12-04T11:13:56.1404097Z inductor/test_cuda_select_algorithm 1/1 failed! 2025-12-04T11:13:56.1407453Z Running inductor/test_compile_subprocess 1/2 ... [2025-12-04 11:13:56.140542][5684.082758042] 2025-12-04T11:13:56.1408097Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2025-12-04T11:13:56.1411397Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'inductor/test_compile_subprocess.py', '--shard-id=1', '--num-shards=2', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-12-04 11:13:56.140882] 2025-12-04T11:21:33.4238562Z 2025-12-04T11:21:33.4239853Z PRINTING LOG FILE of inductor/test_compile_subprocess 1/2 (test/test-reports/inductor.test_compile_subprocess_1.2_abe6ea6a5cdf0a6c_.log) 2025-12-04T11:21:33.4241109Z W1204 11:14:01.189000 89367 site-packages/torch/_inductor/utils.py:1703] Not enough SMs to use max_autotune_gemm mode 2025-12-04T11:21:33.4242341Z Test results will be stored in test-reports/python-pytest/inductor.test_compile_subprocess/inductor.test_compile_subprocess-576cab637f4fea56.xml 2025-12-04T11:21:33.4243292Z ============================= test session starts ============================== 2025-12-04T11:21:33.4243962Z platform linux -- Python 3.10.14, pytest-7.3.2, pluggy-1.6.0 -- /opt/conda/envs/py_3.10/bin/python 2025-12-04T11:21:33.4244556Z cachedir: .pytest_cache 2025-12-04T11:21:33.4245277Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow] 2025-12-04T11:21:33.4246049Z rootdir: /var/lib/jenkins/workspace 2025-12-04T11:21:33.4246410Z configfile: pytest.ini 2025-12-04T11:21:33.4247122Z plugins: hypothesis-6.56.4, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-14.0, subtests-0.13.1, xdist-3.3.1, xdoctest-1.3.0, typeguard-4.3.0 2025-12-04T11:21:33.4247911Z collecting ... collected 897 items 2025-12-04T11:21:33.4248331Z stepcurrent: Cannot find last run test, not skipping 2025-12-04T11:21:33.4425399Z Running 433 items in this shard: test/inductor/test_compile_subprocess.py::TestSubprocess::test_progressive, test/inductor/test_compile_subprocess.py::GPUTests::test_AllenaiLongformerBase_repro_cuda, test/inductor/test_compile_subprocess.py::GPUTests::test__dyn_quant_matmul_4bit_bf16_input_cuda, test/inductor/test_compile_subprocess.py::GPUTests::test__dyn_quant_matmul_4bit_fp32_input_cuda, test/inductor/test_compile_subprocess.py::GPUTests::test__dyn_quant_pack_4bit_weight_fp32_cuda, test/inductor/test_compile_subprocess.py::GPUTests::test__unsafe_masked_index_cuda, test/inductor/test_compile_subprocess.py::GPUTests::test_abs_cuda, test/inductor/test_compile_subprocess.py::GPUTests::test_adaptive_avg_pool1d_argmax_cuda, test/inductor/test_compile_subprocess.py::GPUTests::test_adaptive_avg_pool2d2_cuda, test/inductor/test_compile_subprocess.py::GPUTests::test_adaptive_avg_pool_errors_with_long_cuda, test/inductor/test_compile_subprocess.py::GPUTests::test_adaptive_avg_pool_with_output_size_0_cuda, test/inductor/test_compile_subprocess.py::GPUTests::test_adaptive_max_pool2d1_cuda, test/inductor/test_compile_subprocess.py::GPUTests::test_adaptive_max_pool2d3_cuda, test/inductor/test_compile_subprocess.py::GPUTests::test_adaptive_pool_errors_with_long_cuda, test/inductor/test_compile_subprocess.py::GPUTests::test_add_complex10_cuda, test/inductor/test_compile_subprocess.py::GPUTests::test_add_complex3_cuda, test/inductor/test_compile_subprocess.py::GPUTests::test_add_complex8_cuda, test/inductor/test_compile_subprocess.py::GPUTests::test_add_inplace_permuted_cuda, test/inductor/test_compile_subprocess.py::GPUTests::test_adding_tensor_offsets_cuda, test/inductor/test_compile_subprocess.py::GPUTests::test_addmm_cuda, test/inductor/test_compile_subprocess.py::GPUTests::test_addmv_cuda, test/inductor/test_compile_subprocess.py::GPUTests::test_alexnet_prefix_cuda, test/inductor/test_compile_subprocess.py::GPUTests::test_allow_reuse_disable_if_exceed_peak_cuda, test/inductor/test_compile_subprocess.py::GPUTests::test_angle_cuda, test/inductor/test_compile_subprocess.py::GPUTests::test_aoti_eager_override_registration_cuda, test/inductor/test_compile_subprocess.py::GPUTests::test_aoti_eager_support_out_cuda, test/inductor/test_compile_subprocess.py::GPUTests::test_aoti_eager_with_scalar_cuda, test/inductor/test_compile_subprocess.py::GPUTests::test_arange4_cuda, test/inductor/test_compile_subprocess.py::GPUTests::test_argmax_argmin1_cuda, test/inductor/test_compile_subprocess.py::GPUTests::test_argmax_argmin2_cuda, test/inductor/test_compile_subprocess.py::GPUTests::test_argmax_argmin3_cuda, test/inductor/test_compile_subprocess.py::GPUTests::test_argmax_to_float_cuda, test/inductor/test_compile_subprocess.py::GPUTests::test_as_strided_scatter_cuda, test/inductor/test_compile_subprocess.py::GPUTests::test_assert_alignment_op_name_fail_cuda, test/inductor/test_compile_subprocess.py::GPUTests::test_assert_size_stride_op_name_pass_cuda, test/inductor/test_compile_subprocess.py::GPUTests::test_avg_pool2d1_cuda, test/inductor/test_compile_subprocess.py::GPUTests::test_avg_pool2d4_cuda, test/inductor/test_compile_subprocess.py::GPUTests::test_avg_pool2d5_cuda, test/inductor/test_compile_subprocess.py::GPUTests::test_avg_pool2d6_cuda, test/inductor/test_compile_subprocess.py::GPUTests::test_avg_pool2d8_cuda, test/inductor/test_compile_subprocess.py::GPUTests::test_avg_pool2d_backward3_cuda, test/inductor/test_compile_subprocess.py::GPUTests::test_avg_pool2d_backward4_cuda, test/inductor/test_compile_subprocess.py::GPUTests::test_avg_pool3d_backward2_cuda, test/inductor/test_compile_subprocess.py::GPUTests::test_avg_pool3d_backward3_cuda, test/inductor/test_compile_subprocess.py::GPUTests::test_avg_pool3d_backward4_cuda, test/inductor/test_compile_subprocess.py::GPUTests::test_baddbmm_cuda, test/inductor/test_compile_subprocess.py::GPUTests::test_bernoulli1_cuda, test/inductor/test_compile_subprocess.py::GPUTests::test_bernoulli2_cuda, test/inductor/test_compile_subprocess.py::GPUTests::test_bitwise2_cuda, test/inductor/test_compile_subprocess.py::GPUTests::test_bitwise_cuda, test/inductor/test_compile_subprocess.py::GPUTests::test_bmm1_cuda, test/inductor/test_compile_subprocess.py::GPUTests::test_bool_cuda, test/inductor/test_compile_subprocess.py::GPUTests::test_both_scalars_cuda, test/inductor/test_compile_subprocess.py::GPUTests::test_bucketize_broadcast_cuda, test/inductor/test_compile_subprocess.py::GPUTests::test_bucketize_computed_offsets_cuda, test/inductor/test_compile_subprocess.py::GPUTests::test_bucketize_int_int16_uint8_cuda, test/inductor/test_compile_subprocess.py::GPUTests::test_bucketize_int_int32_int16_cuda, test/inductor/test_compile_subprocess.py::GPUTests::test_bucketize_int_int32_int32_cuda, test/inductor/test_compile_subprocess.py::GPUTests::test_bucketize_int_int32_int8_cuda, test/inductor/test_compile_subprocess.py::GPUTests::test_bucketize_int_int64_int64_cuda, test/inductor/test_compile_subprocess.py::GPUTests::test_bucketize_int_int64_int8_cuda, test/inductor/test_compile_subprocess.py::GPUTests::test_bucketize_int_uint8_int16_cuda, test/inductor/test_compile_subprocess.py::GPUTests::test_bucketize_int_uint8_int8_cuda, test/inductor/test_compile_subprocess.py::GPUTests::test_bucketize_nd_tiling_True_cuda, test/inductor/test_compile_subprocess.py::GPUTests::test_buffer_copied_in_graph_with_different_shapes_cuda, test/inductor/test_compile_subprocess.py::GPUTests::test_buffer_use_after_remove_cuda, test/inductor/test_compile_subprocess.py::GPUTests::test_builtins_round_float_ndigits_neg_cuda, test/inductor/test_compile_subprocess.py::GPUTests::test_builtins_round_float_ndigits_zero_cuda, test/inductor/test_compile_subprocess.py::GPUTests::test_builtins_round_int_ndigits_pos_cuda, test/inductor/test_compile_subprocess.py::GPUTests::test_builtins_round_int_ndigits_zero_cuda, test/inductor/test_compile_subprocess.py::GPUTests::test_cat_inplace_cuda, test/inductor/test_compile_subprocess.py::GPUTests::test_cat_negative_dim_cuda, test/inductor/test_compile_subprocess.py::GPUTests::test_cat_single_empty_cuda, test/inductor/test_compile_subprocess.py::GPUTests::test_cat_unbacked_2d_cuda, test/inductor/test_compile_subprocess.py::GPUTests::test_cat_unbacked_legacy_empty_cuda, test/inductor/test_compile_subprocess.py::GPUTests::test_chunk_recompiles_cuda, test/inductor/test_compile_subprocess.py::GPUTests::test_clamp_type_promotion_non_tensor_cuda, test/inductor/test_compile_subprocess.py::GPUTests::test_compar_cuda, test/inductor/test_compile_subprocess.py::GPUTests::test_complex_fallback_cuda, test/inductor/test_compile_subprocess.py::GPUTests::test_complex_from_real_imag_cuda, test/inductor/test_compile_subprocess.py::GPUTests::test_complex_memory_overlap_cuda, test/inductor/test_compile_subprocess.py::GPUTests::test_config_option_dont_assume_alignment_cuda, test/inductor/test_compile_subprocess.py::GPUTests::test_const_int32_to_float_cuda, test/inductor/test_compile_subprocess.py::GPUTests::test_constant_pad_1d_cuda, test/inductor/test_compile_subprocess.py::GPUTests::test_constant_pad_2d_strides_nonpositive_cuda, test/inductor/test_compile_subprocess.py::GPUTests::test_constant_pad_fill_dtype_cuda, test/inductor/test_compile_subprocess.py::GPUTests::test_constant_pad_float64_cuda, test/inductor/test_compile_subprocess.py::GPUTests::test_conv1d_with_permute_cuda, test/inductor/test_compile_subprocess.py::GPUTests::test_conv3d_channels_last_use_block_ptr_False_cuda, test/inductor/test_compile_subprocess.py::GPUTests::test_conv3d_cuda, test/inductor/test_compile_subprocess.py::GPUTests::test_conv_backward_cuda, test/inductor/test_compile_subprocess.py::GPUTests::test_conv_bn_fuse_cuda, test/inductor/test_compile_subprocess.py::GPUTests::test_conv_inference_heuristics_cuda, test/inductor/test_compile_subprocess.py::GPUTests::test_conv_shape_check_cuda, test/inductor/test_compile_subprocess.py::GPUTests::test_conv_with_as_strided_cuda, test/inductor/test_compile_subprocess.py::GPUTests::test_convolution1_cuda, test/inductor/test_compile_subprocess.py::GPUTests::test_convolution3_cuda, test/inductor/test_compile_subprocess.py::GPUTests::test_convolution4_cuda, test/inductor/test_compile_subprocess.py::GPUTests::test_copy_with_scalar_src_cuda, test/inductor/test_compile_subprocess.py::GPUTests::test_cpu_scalar_with_gpu_tensor_dynamic_cuda, test/inductor/test_compile_subprocess.py::GPUTests::test_cpu_tensor_with_gpu_tensor_cuda, test/inductor/test_compile_subprocess.py::GPUTests::test_cumsum_inf_cuda, test/inductor/test_compile_subprocess.py::GPUTests::test_cumsum_pattern_matcher_issue_cuda, test/inductor/test_compile_subprocess.py::GPUTests::test_custom_op_1_cuda, test/inductor/test_compile_subprocess.py::GPUTests::test_custom_op_2_cuda, test/inductor/test_compile_subprocess.py::GPUTests::test_custom_op_default_layout_constraint_cuda, test/inductor/test_compile_subprocess.py::GPUTests::test_custom_op_fixed_layout_channels_last_cuda, test/inductor/test_compile_subprocess.py::GPUTests::test_custom_op_fixed_layout_sequential_cuda, test/inductor/test_compile_subprocess.py::GPUTests::test_custom_op_unbacked_symints_cuda, test/inductor/test_compile_subprocess.py::GPUTests::test_custom_scan_op_cuda, test/inductor/test_compile_subprocess.py::GPUTests::test_custom_scan_op_multi_input_cuda, test/inductor/test_compile_subprocess.py::GPUTests::test_data_type_propogation_cuda, test/inductor/test_compile_subprocess.py::GPUTests::test_dense_mask_index_cuda, test/inductor/test_compile_subprocess.py::GPUTests::test_deterministic_codegen_with_suffix_cuda, test/inductor/test_compile_subprocess.py::GPUTests::test_diagonal_copy_cuda, test/inductor/test_compile_subprocess.py::GPUTests::test_div2_cuda, test/inductor/test_compile_subprocess.py::GPUTests::test_div3_cuda, test/inductor/test_compile_subprocess.py::GPUTests::test_div4_cuda, test/inductor/test_compile_subprocess.py::GPUTests::test_div8_cuda, test/inductor/test_compile_subprocess.py::GPUTests::test_div9_cuda, test/inductor/test_compile_subprocess.py::GPUTests::test_div_by_zero_cuda, test/inductor/test_compile_subprocess.py::GPUTests::test_div_precision_cuda, test/inductor/test_compile_subprocess.py::GPUTests::test_div_presicion_accuracy_cuda, test/inductor/test_compile_subprocess.py::GPUTests::test_div_prim_cuda, test/inductor/test_compile_subprocess.py::GPUTests::test_div_softmax_symfloat_cuda, test/inductor/test_compile_subprocess.py::GPUTests::test_dont_constant_fold_cuda, test/inductor/test_compile_subprocess.py::GPUTests::test_dropout_trivial_1_cuda, test/inductor/test_compile_subprocess.py::GPUTests::test_dtype_mismatch_issue_cuda, test/inductor/test_compile_subprocess.py::GPUTests::test_dtype_sympy_expr_cuda, test/inductor/test_compile_subprocess.py::GPUTests::test_dtypeview_bfloat16_bfloat16_cuda, test/inductor/test_compile_subprocess.py::GPUTests::test_dtypeview_bfloat16_int32_cuda, test/inductor/test_compile_subprocess.py::GPUTests::test_dtypeview_bfloat16_int64_cuda, test/inductor/test_compile_subprocess.py::GPUTests::test_dtypeview_float16_bfloat16_cuda, test/inductor/test_compile_subprocess.py::GPUTests::test_dtypeview_float32_float16_cuda, test/inductor/test_compile_subprocess.py::GPUTests::test_dtypeview_float32_float64_cuda, test/inductor/test_compile_subprocess.py::GPUTests::test_dtypeview_float32_int64_cuda, test/inductor/test_compile_subprocess.py::GPUTests::test_dtypeview_float32_int8_cuda, test/inductor/test_compile_subprocess.py::GPUTests::test_dtypeview_float64_bfloat16_cuda, test/inductor/test_compile_subprocess.py::GPUTests::test_dtypeview_float64_float32_cuda, test/inductor/test_compile_subprocess.py::GPUTests::test_dtypeview_float64_float64_cuda, test/inductor/test_compile_subprocess.py::GPUTests::test_dtypeview_float64_int16_cuda, test/inductor/test_compile_subprocess.py::GPUTests::test_dtypeview_float64_uint8_cuda, test/inductor/test_compile_subprocess.py::GPUTests::test_dtypeview_fusion_cuda, test/inductor/test_compile_subprocess.py::GPUTests::test_dtypeview_int16_bfloat16_cuda, test/inductor/test_compile_subprocess.py::GPUTests::test_dtypeview_int16_float16_cuda, test/inductor/test_compile_subprocess.py::GPUTests::test_dtypeview_int16_float64_cuda, test/inductor/test_compile_subprocess.py::GPUTests::test_dtypeview_int16_int16_cuda, test/inductor/test_compile_subprocess.py::GPUTests::test_dtypeview_int16_int64_cuda, test/inductor/test_compile_subprocess.py::GPUTests::test_dtypeview_int32_float16_cuda, test/inductor/test_compile_subprocess.py::GPUTests::test_dtypeview_int32_float64_cuda, test/inductor/test_compile_subprocess.py::GPUTests::test_dtypeview_int32_int64_cuda, test/inductor/test_compile_subprocess.py::GPUTests::test_dtypeview_int64_float32_cuda, test/inductor/test_compile_subprocess.py::GPUTests::test_dtypeview_int64_float64_cuda, test/inductor/test_compile_subprocess.py::GPUTests::test_dtypeview_int64_int64_cuda, test/inductor/test_compile_subprocess.py::GPUTests::test_dtypeview_int64_int8_cuda, test/inductor/test_compile_subprocess.py::GPUTests::test_dtypeview_int64_uint8_cuda, test/inductor/test_compile_subprocess.py::GPUTests::test_dtypeview_int8_bfloat16_cuda, test/inductor/test_compile_subprocess.py::GPUTests::test_dtypeview_int8_float16_cuda, test/inductor/test_compile_subprocess.py::GPUTests::test_dtypeview_int8_float64_cuda, test/inductor/test_compile_subprocess.py::GPUTests::test_dtypeview_int8_int32_cuda, test/inductor/test_compile_subprocess.py::GPUTests::test_dtypeview_int8_int64_cuda, test/inductor/test_compile_subprocess.py::GPUTests::test_dtypeview_int8_uint8_cuda, test/inductor/test_compile_subprocess.py::GPUTests::test_dtypeview_uint8_bfloat16_cuda, test/inductor/test_compile_subprocess.py::GPUTests::test_dtypeview_uint8_int16_cuda, test/inductor/test_compile_subprocess.py::GPUTests::test_dtypeview_uint8_int32_cuda, test/inductor/test_compile_subprocess.py::GPUTests::test_dtypeview_uint8_int8_cuda, test/inductor/test_compile_subprocess.py::GPUTests::test_dtypeview_uint8_uint8_cuda, test/inductor/test_compile_subprocess.py::GPUTests::test_elu_cuda, test/inductor/test_compile_subprocess.py::GPUTests::test_embedding_bag_byte_unpack_cuda, test/inductor/test_compile_subprocess.py::GPUTests::test_embedding_cuda, test/inductor/test_compile_subprocess.py::GPUTests::test_empty_strided_cuda, test/inductor/test_compile_subprocess.py::GPUTests::test_exact_stride_cuda, test/inductor/test_compile_subprocess.py::GPUTests::test_exp2_cuda, test/inductor/test_compile_subprocess.py::GPUTests::test_exp_cuda, test/inductor/test_compile_subprocess.py::GPUTests::test_expand_as_cuda, test/inductor/test_compile_subprocess.py::GPUTests::test_expand_cuda, test/inductor/test_compile_subprocess.py::GPUTests::test_fft_real_input_real_output_cuda, test/inductor/test_compile_subprocess.py::GPUTests::test_fill2_cuda, test/inductor/test_compile_subprocess.py::GPUTests::test_flexible_layout_immutable_free_symbols_cuda, test/inductor/test_compile_subprocess.py::GPUTests::test_flip_cat_cuda, test/inductor/test_compile_subprocess.py::GPUTests::test_flip_cuda, test/inductor/test_compile_subprocess.py::GPUTests::test_float_repr_dynamic_shapes_cuda, test/inductor/test_compile_subprocess.py::GPUTests::test_floordiv_cuda, test/inductor/test_compile_subprocess.py::GPUTests::test_forced_buffer_realize_cuda, test/inductor/test_compile_subprocess.py::GPUTests::test_fractional_max_pool2d3_cuda, test/inductor/test_compile_subprocess.py::GPUTests::test_fractional_max_pool2d5_cuda, test/inductor/test_compile_subprocess.py::GPUTests::test_full_boolean_cuda, test/inductor/test_compile_subprocess.py::GPUTests::test_full_like_cuda, test/inductor/test_compile_subprocess.py::GPUTests::test_functionalize_rng_wrappers_cuda, test/inductor/test_compile_subprocess.py::GPUTests::test_gather1_cuda, test/inductor/test_compile_subprocess.py::GPUTests::test_gather_scatter_cuda, test/inductor/test_compile_subprocess.py::GPUTests::test_gelu_cuda, test/inductor/test_compile_subprocess.py::GPUTests::test_getitem_cuda, test/inductor/test_compile_subprocess.py::GPUTests::test_graph_partition_arange2_cuda, test/inductor/test_compile_subprocess.py::GPUTests::test_graph_partition_misaligned_input_cuda, test/inductor/test_compile_subprocess.py::GPUTests::test_graph_partition_mutation_real_name_cuda, test/inductor/test_compile_subprocess.py::GPUTests::test_graph_partition_scalar_inputs_cuda, test/inductor/test_compile_subprocess.py::GPUTests::test_hardtanh_cuda, test/inductor/test_compile_subprocess.py::GPUTests::test_horizonal_fusion1_cuda, test/inductor/test_compile_subprocess.py::GPUTests::test_index2_cuda, test/inductor/test_compile_subprocess.py::GPUTests::test_index_put1_cuda, test/inductor/test_compile_subprocess.py::GPUTests::test_index_put3_cuda, test/inductor/test_compile_subprocess.py::GPUTests::test_index_put4_cuda, test/inductor/test_compile_subprocess.py::GPUTests::test_index_put_as_masked_fill_cuda, test/inductor/test_compile_subprocess.py::GPUTests::test_index_put_deterministic_fallback_cuda, test/inductor/test_compile_subprocess.py::GPUTests::test_index_put_failed_reinplace_cuda, test/inductor/test_compile_subprocess.py::GPUTests::test_index_put_index_cuda, test/inductor/test_compile_subprocess.py::GPUTests::test_index_put_reinplace_cuda, test/inductor/test_compile_subprocess.py::GPUTests::test_index_remainder_cuda, test/inductor/test_compile_subprocess.py::GPUTests::test_indirect_load_broadcast_cuda, test/inductor/test_compile_subprocess.py::GPUTests::test_inf_cuda, test/inductor/test_compile_subprocess.py::GPUTests::test_inner_reduction_detection_cuda, test/inductor/test_compile_subprocess.py::GPUTests::test_inplace_add_cuda, test/inductor/test_compile_subprocess.py::GPUTests::test_inplace_flip_cuda, test/inductor/test_compile_subprocess.py::GPUTests::test_inplace_resize_as_cuda, test/inductor/test_compile_subprocess.py::GPUTests::test_input_mutation2_cuda, test/inductor/test_compile_subprocess.py::GPUTests::test_input_mutation5_cuda, test/inductor/test_compile_subprocess.py::GPUTests::test_insignificant_strides_cuda, test/inductor/test_compile_subprocess.py::GPUTests::test_int8_weight_only_quant_cuda, test/inductor/test_compile_subprocess.py::GPUTests::test_isin_tensor_scalar_cuda, test/inductor/test_compile_subprocess.py::GPUTests::test_issue102546_cuda, test/inductor/test_compile_subprocess.py::GPUTests::test_kernel_names_cuda, test/inductor/test_compile_subprocess.py::GPUTests::test_kwargs_cuda, test/inductor/test_compile_subprocess.py::GPUTests::test_l1_loss_cuda, test/inductor/test_compile_subprocess.py::GPUTests::test_large_block_sizes_cuda, test/inductor/test_compile_subprocess.py::GPUTests::test_large_broadcast_reduction_cuda, test/inductor/test_compile_subprocess.py::GPUTests::test_large_offset_pointwise_cuda, test/inductor/test_compile_subprocess.py::GPUTests::test_large_tensor_reduction_cuda, test/inductor/test_compile_subprocess.py::GPUTests::test_linalg_eig_stride_consistency_cuda, test/inductor/test_compile_subprocess.py::GPUTests::test_linspace2_cuda, test/inductor/test_compile_subprocess.py::GPUTests::test_linspace4_cuda, test/inductor/test_compile_subprocess.py::GPUTests::test_list_clearing_cuda, test/inductor/test_compile_subprocess.py::GPUTests::test_lite_mode_not_decompose_cuda, test/inductor/test_compile_subprocess.py::GPUTests::test_lite_triton_kernel_wrapper_functional_cuda, test/inductor/test_compile_subprocess.py::GPUTests::test_log2_cuda, test/inductor/test_compile_subprocess.py::GPUTests::test_log_fp64_cuda, test/inductor/test_compile_subprocess.py::GPUTests::test_logaddexp_cuda, test/inductor/test_compile_subprocess.py::GPUTests::test_logcumsumexp_cuda, test/inductor/test_compile_subprocess.py::GPUTests::test_logcumsumexp_zero_dim_cuda, test/inductor/test_compile_subprocess.py::GPUTests::test_logsumexp_cuda, test/inductor/test_compile_subprocess.py::GPUTests::test_low_memory_max_pool_dilation_1_dim_3_cuda, test/inductor/test_compile_subprocess.py::GPUTests::test_low_memory_max_pool_dilation_2_dim_2_cuda, test/inductor/test_compile_subprocess.py::GPUTests::test_low_memory_max_pool_dilation_2_dim_3_cuda, test/inductor/test_compile_subprocess.py::GPUTests::test_masked_fill_cuda, test/inductor/test_compile_subprocess.py::GPUTests::test_masked_scatter_cuda, test/inductor/test_compile_subprocess.py::GPUTests::test_max_pool2d1_cuda, test/inductor/test_compile_subprocess.py::GPUTests::test_max_pool2d3_cuda, test/inductor/test_compile_subprocess.py::GPUTests::test_max_pool2d4_cuda, test/inductor/test_compile_subprocess.py::GPUTests::test_max_pool2d5_cuda, test/inductor/test_compile_subprocess.py::GPUTests::test_max_pool2d6_dilation_1_cuda, test/inductor/test_compile_subprocess.py::GPUTests::test_max_pool2d_with_indices_backward2_cuda, test/inductor/test_compile_subprocess.py::GPUTests::test_max_pool2d_with_indices_backward3_cuda, test/inductor/test_compile_subprocess.py::GPUTests::test_max_pool2d_with_indices_backward4_cuda, test/inductor/test_compile_subprocess.py::GPUTests::test_max_pool2d_with_indices_backward6_cuda, test/inductor/test_compile_subprocess.py::GPUTests::test_max_pool2d_with_indices_backward_cuda, test/inductor/test_compile_subprocess.py::GPUTests::test_mean_cuda, test/inductor/test_compile_subprocess.py::GPUTests::test_min_max_reduction_cuda, test/inductor/test_compile_subprocess.py::GPUTests::test_misaligned_address_issue1_cuda, test/inductor/test_compile_subprocess.py::GPUTests::test_mix_device_index_cuda, test/inductor/test_compile_subprocess.py::GPUTests::test_mixed_mm3_cuda, test/inductor/test_compile_subprocess.py::GPUTests::test_mixed_mm_cuda, test/inductor/test_compile_subprocess.py::GPUTests::test_move_arange_cuda, test/inductor/test_compile_subprocess.py::GPUTests::test_mul_index_expr_cuda, test/inductor/test_compile_subprocess.py::GPUTests::test_mul_softmax_symfloat_cuda, test/inductor/test_compile_subprocess.py::GPUTests::test_multi_gpu_recompile_on_index_cuda, test/inductor/test_compile_subprocess.py::GPUTests::test_multi_threading_cuda, test/inductor/test_compile_subprocess.py::GPUTests::test_multilayer_prime_size_cuda, test/inductor/test_compile_subprocess.py::GPUTests::test_multilayer_var_cuda, test/inductor/test_compile_subprocess.py::GPUTests::test_nan_sort_stable_True_descending_False_cuda, test/inductor/test_compile_subprocess.py::GPUTests::test_nan_sort_stable_True_descending_True_cuda, test/inductor/test_compile_subprocess.py::GPUTests::test_needs_contiguous_strides_cuda, test/inductor/test_compile_subprocess.py::GPUTests::test_neg_index_cuda, test/inductor/test_compile_subprocess.py::GPUTests::test_new_empty_cuda, test/inductor/test_compile_subprocess.py::GPUTests::test_new_empty_strided_cuda, test/inductor/test_compile_subprocess.py::GPUTests::test_new_ones_cuda, test/inductor/test_compile_subprocess.py::GPUTests::test_no_mega_fusion_during_lowering_cuda, test/inductor/test_compile_subprocess.py::GPUTests::test_no_op_reduction_cuda, test/inductor/test_compile_subprocess.py::GPUTests::test_norm_constant_overflow_cuda, test/inductor/test_compile_subprocess.py::GPUTests::test_one_hot_cuda, test/inductor/test_compile_subprocess.py::GPUTests::test_pad_single_cuda, test/inductor/test_compile_subprocess.py::GPUTests::test_pattern_matcher_unbacked_cuda, test/inductor/test_compile_subprocess.py::GPUTests::test_permute1_cuda, test/inductor/test_compile_subprocess.py::GPUTests::test_philox_rand_cuda, test/inductor/test_compile_subprocess.py::GPUTests::test_pointwise_bessel_j0_cuda, test/inductor/test_compile_subprocess.py::GPUTests::test_pointwise_bessel_j1_cuda, test/inductor/test_compile_subprocess.py::GPUTests::test_pointwise_bessel_y0_cuda, test/inductor/test_compile_subprocess.py::GPUTests::test_pointwise_bessel_y1_cuda, test/inductor/test_compile_subprocess.py::GPUTests::test_pointwise_chebyshev_polynomial_t_cuda, test/inductor/test_compile_subprocess.py::GPUTests::test_pointwise_chebyshev_polynomial_u_cuda, test/inductor/test_compile_subprocess.py::GPUTests::test_pointwise_digamma_cuda, test/inductor/test_compile_subprocess.py::GPUTests::test_pointwise_entr_cuda, test/inductor/test_compile_subprocess.py::GPUTests::test_pointwise_erfc_cuda, test/inductor/test_compile_subprocess.py::GPUTests::test_pointwise_erfinv_cuda, test/inductor/test_compile_subprocess.py::GPUTests::test_pointwise_exp2_cuda, test/inductor/test_compile_subprocess.py::GPUTests::test_pointwise_expm1_cuda, test/inductor/test_compile_subprocess.py::GPUTests::test_pointwise_gammainc_cuda, test/inductor/test_compile_subprocess.py::GPUTests::test_pointwise_gammaln_cuda, test/inductor/test_compile_subprocess.py::GPUTests::test_pointwise_hermite_polynomial_h_cuda, test/inductor/test_compile_subprocess.py::GPUTests::test_pointwise_laguerre_polynomial_l_cuda, test/inductor/test_compile_subprocess.py::GPUTests::test_pointwise_legendre_polynomial_p_cuda, test/inductor/test_compile_subprocess.py::GPUTests::test_pointwise_log1p_cuda, test/inductor/test_compile_subprocess.py::GPUTests::test_pointwise_modified_bessel_i0_cuda, test/inductor/test_compile_subprocess.py::GPUTests::test_pointwise_modified_bessel_i1_cuda, test/inductor/test_compile_subprocess.py::GPUTests::test_pointwise_modified_bessel_k0_cuda, test/inductor/test_compile_subprocess.py::GPUTests::test_pointwise_modified_bessel_k1_cuda, test/inductor/test_compile_subprocess.py::GPUTests::test_pointwise_ndtri_cuda, test/inductor/test_compile_subprocess.py::GPUTests::test_pointwise_polygamma_cuda, test/inductor/test_compile_subprocess.py::GPUTests::test_pointwise_round_cuda, test/inductor/test_compile_subprocess.py::GPUTests::test_pointwise_scaled_modified_bessel_k0_cuda, test/inductor/test_compile_subprocess.py::GPUTests::test_pointwise_scaled_modified_bessel_k1_cuda, test/inductor/test_compile_subprocess.py::GPUTests::test_pointwise_sinc_cuda, test/inductor/test_compile_subprocess.py::GPUTests::test_pointwise_spherical_bessel_j0_cuda, test/inductor/test_compile_subprocess.py::GPUTests::test_pointwise_xlog1py_cuda, test/inductor/test_compile_subprocess.py::GPUTests::test_pointwise_xlogy_cuda, test/inductor/test_compile_subprocess.py::GPUTests::test_pointwise_zeta_cuda, test/inductor/test_compile_subprocess.py::GPUTests::test_pow3_cuda, test/inductor/test_compile_subprocess.py::GPUTests::test_pow_int_cuda, test/inductor/test_compile_subprocess.py::GPUTests::test_pow_symfloat_cuda, test/inductor/test_compile_subprocess.py::GPUTests::test_prod_cuda, test/inductor/test_compile_subprocess.py::GPUTests::test_profiler_mark_wrapper_call_cuda, test/inductor/test_compile_subprocess.py::GPUTests::test_rand_like_deterministic_cuda, test/inductor/test_compile_subprocess.py::GPUTests::test_randint_cuda, test/inductor/test_compile_subprocess.py::GPUTests::test_randn_generator_cuda, test/inductor/test_compile_subprocess.py::GPUTests::test_reduction1_cuda, test/inductor/test_compile_subprocess.py::GPUTests::test_reduction3_cuda, test/inductor/test_compile_subprocess.py::GPUTests::test_reduction_config_limit_cuda, test/inductor/test_compile_subprocess.py::GPUTests::test_reflection_pad2d_backward_cuda, test/inductor/test_compile_subprocess.py::GPUTests::test_reflection_pad2d_cuda, test/inductor/test_compile_subprocess.py::GPUTests::test_reinterpret_dtypeview_cuda, test/inductor/test_compile_subprocess.py::GPUTests::test_relu_cuda, test/inductor/test_compile_subprocess.py::GPUTests::test_remove_no_ops_cuda, test/inductor/test_compile_subprocess.py::GPUTests::test_remove_noop_slice_cuda, test/inductor/test_compile_subprocess.py::GPUTests::test_remove_noop_view_default_cuda, test/inductor/test_compile_subprocess.py::GPUTests::test_remove_noop_view_dtype_cuda, test/inductor/test_compile_subprocess.py::GPUTests::test_repeat_as_strided_cuda, test/inductor/test_compile_subprocess.py::GPUTests::test_repeat_interleave_2_cuda, test/inductor/test_compile_subprocess.py::GPUTests::test_repeat_interleave_Tensor_decomp_int64_nd_1_cuda, test/inductor/test_compile_subprocess.py::GPUTests::test_repeat_interleave_Tensor_decomp_int64_nd_2_cuda, test/inductor/test_compile_subprocess.py::GPUTests::test_repeat_interleave_cuda, test/inductor/test_compile_subprocess.py::GPUTests::test_reuse_buffers_with_aliasing_cuda, test/inductor/test_compile_subprocess.py::GPUTests::test_rsqrt_cuda, test/inductor/test_compile_subprocess.py::GPUTests::test_rsqrt_dynamic_shapes_cuda, test/inductor/test_compile_subprocess.py::GPUTests::test_scatter2_cuda, test/inductor/test_compile_subprocess.py::GPUTests::test_scatter3_cuda, test/inductor/test_compile_subprocess.py::GPUTests::test_scatter6_cuda, test/inductor/test_compile_subprocess.py::GPUTests::test_scatter_add1_cuda, test/inductor/test_compile_subprocess.py::GPUTests::test_scatter_bf16_cuda, test/inductor/test_compile_subprocess.py::GPUTests::test_scatter_reduce1_cuda, test/inductor/test_compile_subprocess.py::GPUTests::test_scatter_reduce3_cuda, test/inductor/test_compile_subprocess.py::GPUTests::test_sdpa_prefer_nd_tiling_True_use_block_ptr_True_cuda, test/inductor/test_compile_subprocess.py::GPUTests::test_searchsorted_cuda, test/inductor/test_compile_subprocess.py::GPUTests::test_select_scatter_cuda, test/inductor/test_compile_subprocess.py::GPUTests::test_setitem_with_int_parameter_cuda, test/inductor/test_compile_subprocess.py::GPUTests::test_sgn_cuda, test/inductor/test_compile_subprocess.py::GPUTests::test_sgn_extremal_cuda, test/inductor/test_compile_subprocess.py::GPUTests::test_shape_padding_cuda, test/inductor/test_compile_subprocess.py::GPUTests::test_silu_cuda, test/inductor/test_compile_subprocess.py::GPUTests::test_size_asserts_for_multi_output_fallback_cuda, test/inductor/test_compile_subprocess.py::GPUTests::test_slice2_cuda, test/inductor/test_compile_subprocess.py::GPUTests::test_slice4_cuda, test/inductor/test_compile_subprocess.py::GPUTests::test_slice_mutation3_cuda, test/inductor/test_compile_subprocess.py::GPUTests::test_slice_scatter3_cuda, test/inductor/test_compile_subprocess.py::GPUTests::test_slice_scatter4_cuda, test/inductor/test_compile_subprocess.py::GPUTests::test_slice_scatter_cuda, test/inductor/test_compile_subprocess.py::GPUTests::test_slice_scatter_dtype_consistency_cuda, test/inductor/test_compile_subprocess.py::GPUTests::test_slice_view_with_graph_break_cuda, test/inductor/test_compile_subprocess.py::GPUTests::test_softmax_backward_data_cuda, test/inductor/test_compile_subprocess.py::GPUTests::test_softmax_one_kernel_loop_cuda, test/inductor/test_compile_subprocess.py::GPUTests::test_sort_bool_cuda, test/inductor/test_compile_subprocess.py::GPUTests::test_sort_cuda, test/inductor/test_compile_subprocess.py::GPUTests::test_sort_stable_cuda, test/inductor/test_compile_subprocess.py::GPUTests::test_sort_transpose_cuda, test/inductor/test_compile_subprocess.py::GPUTests::test_special_polygamma_cuda, test/inductor/test_compile_subprocess.py::GPUTests::test_split_cumprod_cuda, test/inductor/test_compile_subprocess.py::GPUTests::test_split_cumsum_low_prec_cuda, test/inductor/test_compile_subprocess.py::GPUTests::test_split_failed_cuda, test/inductor/test_compile_subprocess.py::GPUTests::test_split_with_list_cuda, test/inductor/test_compile_subprocess.py::GPUTests::test_split_with_sizes_with_unbacked_symints_cuda, test/inductor/test_compile_subprocess.py::GPUTests::test_std_cuda, test/inductor/test_compile_subprocess.py::GPUTests::test_stride_preservation_with_stride_modifying_fx_pass_cuda, test/inductor/test_compile_subprocess.py::GPUTests::test_strided_inputs_cuda, test/inductor/test_compile_subprocess.py::GPUTests::test_sum1_cuda, test/inductor/test_compile_subprocess.py::GPUTests::test_sum3_cuda, test/inductor/test_compile_subprocess.py::GPUTests::test_sum_dtype_cuda, test/inductor/test_compile_subprocess.py::GPUTests::test_sum_int_cuda, test/inductor/test_compile_subprocess.py::GPUTests::test_sum_keepdims_cuda, test/inductor/test_compile_subprocess.py::GPUTests::test_tanh_cuda, test/inductor/test_compile_subprocess.py::GPUTests::test_tensor1_cuda, test/inductor/test_compile_subprocess.py::GPUTests::test_tensor2_cuda, test/inductor/test_compile_subprocess.py::GPUTests::test_tensor3_cuda, test/inductor/test_compile_subprocess.py::GPUTests::test_tensor_index_put_slice_cuda, test/inductor/test_compile_subprocess.py::GPUTests::test_tmp_not_defined_issue1_use_block_ptr_True_cuda, test/inductor/test_compile_subprocess.py::GPUTests::test_tmp_not_defined_issue2_cuda, test/inductor/test_compile_subprocess.py::GPUTests::test_tmp_not_defined_issue3_cuda, test/inductor/test_compile_subprocess.py::GPUTests::test_to_device_constant_cuda, test/inductor/test_compile_subprocess.py::GPUTests::test_to_memory_format_cuda, test/inductor/test_compile_subprocess.py::GPUTests::test_topk_cuda, test/inductor/test_compile_subprocess.py::GPUTests::test_transpose_add_cuda, test/inductor/test_compile_subprocess.py::GPUTests::test_transposed_propagates_cuda, test/inductor/test_compile_subprocess.py::GPUTests::test_triton_argmin_argmax_transpose_logical_index_cuda, test/inductor/test_compile_subprocess.py::GPUTests::test_triton_kernel_bool_param_cuda, test/inductor/test_compile_subprocess.py::GPUTests::test_triu_cuda, test/inductor/test_compile_subprocess.py::GPUTests::test_uint4x2_mixed_mm_cuda, test/inductor/test_compile_subprocess.py::GPUTests::test_unbacked_float_item_cuda, test/inductor/test_compile_subprocess.py::GPUTests::test_unbacked_floordiv_simplify_cuda, test/inductor/test_compile_subprocess.py::GPUTests::test_unbacked_floordiv_simplify_errors_cuda, test/inductor/test_compile_subprocess.py::GPUTests::test_unroll_small_reduction_cuda, test/inductor/test_compile_subprocess.py::GPUTests::test_unspec_inputs_float32_cuda, test/inductor/test_compile_subprocess.py::GPUTests::test_unspec_inputs_float64_cuda, test/inductor/test_compile_subprocess.py::GPUTests::test_unspec_inputs_int16_cuda, test/inductor/test_compile_subprocess.py::GPUTests::test_unspec_inputs_int32_cuda, test/inductor/test_compile_subprocess.py::GPUTests::test_unspec_inputs_int64_cuda, test/inductor/test_compile_subprocess.py::GPUTests::test_unspec_inputs_int8_cuda, test/inductor/test_compile_subprocess.py::GPUTests::test_unsqueeze_inplace_cuda, test/inductor/test_compile_subprocess.py::GPUTests::test_upsample_bilinear2d_a_cuda, test/inductor/test_compile_subprocess.py::GPUTests::test_upsample_cat_conv_cuda, test/inductor/test_compile_subprocess.py::GPUTests::test_upsample_nearest1d_cuda, test/inductor/test_compile_subprocess.py::GPUTests::test_var_correction_cuda, test/inductor/test_compile_subprocess.py::GPUTests::test_var_mean_tile_reduction_False_cuda, test/inductor/test_compile_subprocess.py::GPUTests::test_vdd_clamp_cuda, test/inductor/test_compile_subprocess.py::GPUTests::test_view_as_complex_cuda, test/inductor/test_compile_subprocess.py::GPUTests::test_view_as_real_cuda, test/inductor/test_compile_subprocess.py::GPUTests::test_view_detach_cuda, test/inductor/test_compile_subprocess.py::GPUTests::test_views2_cuda, test/inductor/test_compile_subprocess.py::GPUTests::test_views4_cuda, test/inductor/test_compile_subprocess.py::GPUTests::test_views5_cuda, test/inductor/test_compile_subprocess.py::GPUTests::test_views6_cuda, test/inductor/test_compile_subprocess.py::GPUTests::test_weight_norm_bwd_cuda, test/inductor/test_compile_subprocess.py::GPUTests::test_weight_norm_conv2d_cuda, test/inductor/test_compile_subprocess.py::GPUTests::test_where_broadcast_cuda, test/inductor/test_compile_subprocess.py::GPUTests::test_where_with_logical_op_cuda, test/inductor/test_compile_subprocess.py::GPUTests::test_zero_dim_reductions_cuda, test/inductor/test_compile_subprocess.py::GPUTests::test_zero_element_mutation_cuda 2025-12-04T11:21:33.4603076Z 2025-12-04T11:21:33.4603870Z inductor/test_compile_subprocess.py::TestSubprocess::test_progressive SKIPPED [0.0002s] (Skipping triton backend only since not big GPU (not enough SM)) [ 0%] 2025-12-04T11:21:33.4605397Z inductor/test_compile_subprocess.py::GPUTests::test_AllenaiLongformerBase_repro_cuda <- test/inductor/test_torchinductor.py PASSED [9.6225s] [ 0%] 2025-12-04T11:21:33.4607117Z inductor/test_compile_subprocess.py::GPUTests::test__dyn_quant_matmul_4bit_bf16_input_cuda <- test/inductor/test_torchinductor.py SKIPPED [0.0025s] (No _dyn_quant_matmul_4bit implementation on CUDA) [ 0%] 2025-12-04T11:21:33.4609030Z inductor/test_compile_subprocess.py::GPUTests::test__dyn_quant_matmul_4bit_fp32_input_cuda <- test/inductor/test_torchinductor.py SKIPPED [0.0023s] (No _dyn_quant_matmul_4bit implementation on CUDA) [ 0%] 2025-12-04T11:21:33.4610990Z inductor/test_compile_subprocess.py::GPUTests::test__dyn_quant_pack_4bit_weight_fp32_cuda <- test/inductor/test_torchinductor.py SKIPPED [0.0022s] (No _dyn_quant_pack_4bit_weight implementation on CUDA) [ 1%] 2025-12-04T11:21:33.4612628Z inductor/test_compile_subprocess.py::GPUTests::test__unsafe_masked_index_cuda <- test/inductor/test_torchinductor.py PASSED [0.6870s] [ 1%] 2025-12-04T11:21:33.4613851Z inductor/test_compile_subprocess.py::GPUTests::test_abs_cuda <- test/inductor/test_torchinductor.py PASSED [0.5889s] [ 1%] 2025-12-04T11:21:33.4615117Z inductor/test_compile_subprocess.py::GPUTests::test_adaptive_avg_pool1d_argmax_cuda <- test/inductor/test_torchinductor.py PASSED [0.5892s] [ 1%] 2025-12-04T11:21:33.4616464Z inductor/test_compile_subprocess.py::GPUTests::test_adaptive_avg_pool2d2_cuda <- test/inductor/test_torchinductor.py PASSED [0.1006s] [ 2%] 2025-12-04T11:21:33.4617838Z inductor/test_compile_subprocess.py::GPUTests::test_adaptive_avg_pool_errors_with_long_cuda <- test/inductor/test_torchinductor.py PASSED [0.5711s] [ 2%] 2025-12-04T11:21:33.4619246Z inductor/test_compile_subprocess.py::GPUTests::test_adaptive_avg_pool_with_output_size_0_cuda <- test/inductor/test_torchinductor.py PASSED [0.3941s] [ 2%] 2025-12-04T11:21:33.4621355Z inductor/test_compile_subprocess.py::GPUTests::test_adaptive_max_pool2d1_cuda <- test/inductor/test_torchinductor.py W1204 11:14:17.363000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] Unable to pickle input graph or example inputs 2025-12-04T11:21:33.4623096Z W1204 11:14:17.363000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] Traceback (most recent call last): 2025-12-04T11:21:33.4624725Z W1204 11:14:17.363000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/compile_fx_ext.py", line 486, in serialize_compile 2025-12-04T11:21:33.4626155Z W1204 11:14:17.363000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] ).serialize() 2025-12-04T11:21:33.4627543Z W1204 11:14:17.363000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/compile_fx_ext.py", line 210, in serialize 2025-12-04T11:21:33.4629308Z W1204 11:14:17.363000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] return _WireProtocolPickledInput(GraphPickler.dumps(self)) 2025-12-04T11:21:33.4630870Z W1204 11:14:17.363000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 127, in dumps 2025-12-04T11:21:33.4632237Z W1204 11:14:17.363000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] pickler.dump(obj) 2025-12-04T11:21:33.4633641Z W1204 11:14:17.363000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 94, in reducer_override 2025-12-04T11:21:33.4635260Z W1204 11:14:17.363000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] return _GraphModulePickleData.reduce_helper(self, obj) 2025-12-04T11:21:33.4636907Z W1204 11:14:17.363000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 352, in reduce_helper 2025-12-04T11:21:33.4638405Z W1204 11:14:17.363000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] cls(obj, pickler.options), 2025-12-04T11:21:33.4639901Z W1204 11:14:17.363000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 364, in __init__ 2025-12-04T11:21:33.4641461Z W1204 11:14:17.363000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] self.graph = _GraphPickleData(gm._graph, options) 2025-12-04T11:21:33.4643045Z W1204 11:14:17.363000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 571, in __init__ 2025-12-04T11:21:33.4644601Z W1204 11:14:17.363000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] nodes[node] = _NodePickleData(node, nodes, options) 2025-12-04T11:21:33.4646155Z W1204 11:14:17.363000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 387, in __init__ 2025-12-04T11:21:33.4647739Z W1204 11:14:17.363000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] self.target = _OpPickleData.pickle(node.target, options) 2025-12-04T11:21:33.4649315Z W1204 11:14:17.363000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 435, in pickle 2025-12-04T11:21:33.4650917Z W1204 11:14:17.363000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] return cls._pickle_op(name, _OpOverloadPickleData, options) 2025-12-04T11:21:33.4652621Z W1204 11:14:17.363000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 456, in _pickle_op 2025-12-04T11:21:33.4654315Z W1204 11:14:17.363000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] raise BypassFxGraphCache(f"Unable to pickle non-standard op: {name}") 2025-12-04T11:21:33.4656161Z W1204 11:14:17.363000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] torch._inductor.codecache.BypassFxGraphCache: Unable to pickle non-standard op: torch.ops.prims._low_memory_max_pool_with_offsets.default 2025-12-04T11:21:33.4657876Z W1204 11:14:17.896000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] Unable to pickle input graph or example inputs 2025-12-04T11:21:33.4659070Z W1204 11:14:17.896000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] Traceback (most recent call last): 2025-12-04T11:21:33.4660657Z W1204 11:14:17.896000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/compile_fx_ext.py", line 486, in serialize_compile 2025-12-04T11:21:33.4662283Z W1204 11:14:17.896000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] ).serialize() 2025-12-04T11:21:33.4663793Z W1204 11:14:17.896000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/compile_fx_ext.py", line 210, in serialize 2025-12-04T11:21:33.4665453Z W1204 11:14:17.896000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] return _WireProtocolPickledInput(GraphPickler.dumps(self)) 2025-12-04T11:21:33.4667160Z W1204 11:14:17.896000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 127, in dumps 2025-12-04T11:21:33.4668586Z W1204 11:14:17.896000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] pickler.dump(obj) 2025-12-04T11:21:33.4670076Z W1204 11:14:17.896000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 94, in reducer_override 2025-12-04T11:21:33.4671700Z W1204 11:14:17.896000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] return _GraphModulePickleData.reduce_helper(self, obj) 2025-12-04T11:21:33.4673324Z W1204 11:14:17.896000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 352, in reduce_helper 2025-12-04T11:21:33.4674835Z W1204 11:14:17.896000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] cls(obj, pickler.options), 2025-12-04T11:21:33.4676317Z W1204 11:14:17.896000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 364, in __init__ 2025-12-04T11:21:33.4678112Z W1204 11:14:17.896000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] self.graph = _GraphPickleData(gm._graph, options) 2025-12-04T11:21:33.4679713Z W1204 11:14:17.896000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 571, in __init__ 2025-12-04T11:21:33.4681271Z W1204 11:14:17.896000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] nodes[node] = _NodePickleData(node, nodes, options) 2025-12-04T11:21:33.4682792Z W1204 11:14:17.896000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 387, in __init__ 2025-12-04T11:21:33.4684314Z W1204 11:14:17.896000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] self.target = _OpPickleData.pickle(node.target, options) 2025-12-04T11:21:33.4685992Z W1204 11:14:17.896000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 435, in pickle 2025-12-04T11:21:33.4687557Z W1204 11:14:17.896000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] return cls._pickle_op(name, _OpOverloadPickleData, options) 2025-12-04T11:21:33.4689170Z W1204 11:14:17.896000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 456, in _pickle_op 2025-12-04T11:21:33.4690812Z W1204 11:14:17.896000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] raise BypassFxGraphCache(f"Unable to pickle non-standard op: {name}") 2025-12-04T11:21:33.4692617Z W1204 11:14:17.896000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] torch._inductor.codecache.BypassFxGraphCache: Unable to pickle non-standard op: torch.ops.prims._low_memory_max_pool_with_offsets.default 2025-12-04T11:21:33.4693994Z PASSED [4.1186s] [ 2%] 2025-12-04T11:21:33.4695417Z inductor/test_compile_subprocess.py::GPUTests::test_adaptive_max_pool2d3_cuda <- test/inductor/test_torchinductor.py W1204 11:14:21.013000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] Unable to pickle input graph or example inputs 2025-12-04T11:21:33.4697185Z W1204 11:14:21.013000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] Traceback (most recent call last): 2025-12-04T11:21:33.4698735Z W1204 11:14:21.013000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/compile_fx_ext.py", line 486, in serialize_compile 2025-12-04T11:21:33.4700285Z W1204 11:14:21.013000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] ).serialize() 2025-12-04T11:21:33.4701714Z W1204 11:14:21.013000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/compile_fx_ext.py", line 210, in serialize 2025-12-04T11:21:33.4703342Z W1204 11:14:21.013000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] return _WireProtocolPickledInput(GraphPickler.dumps(self)) 2025-12-04T11:21:33.4704991Z W1204 11:14:21.013000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 127, in dumps 2025-12-04T11:21:33.4706367Z W1204 11:14:21.013000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] pickler.dump(obj) 2025-12-04T11:21:33.4707805Z W1204 11:14:21.013000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 94, in reducer_override 2025-12-04T11:21:33.4709392Z W1204 11:14:21.013000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] return _GraphModulePickleData.reduce_helper(self, obj) 2025-12-04T11:21:33.4710981Z W1204 11:14:21.013000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 352, in reduce_helper 2025-12-04T11:21:33.4712437Z W1204 11:14:21.013000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] cls(obj, pickler.options), 2025-12-04T11:21:33.4713873Z W1204 11:14:21.013000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 364, in __init__ 2025-12-04T11:21:33.4715387Z W1204 11:14:21.013000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] self.graph = _GraphPickleData(gm._graph, options) 2025-12-04T11:21:33.4716903Z W1204 11:14:21.013000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 571, in __init__ 2025-12-04T11:21:33.4718503Z W1204 11:14:21.013000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] nodes[node] = _NodePickleData(node, nodes, options) 2025-12-04T11:21:33.4720049Z W1204 11:14:21.013000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 387, in __init__ 2025-12-04T11:21:33.4721624Z W1204 11:14:21.013000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] self.target = _OpPickleData.pickle(node.target, options) 2025-12-04T11:21:33.4723182Z W1204 11:14:21.013000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 435, in pickle 2025-12-04T11:21:33.4724763Z W1204 11:14:21.013000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] return cls._pickle_op(name, _OpOverloadPickleData, options) 2025-12-04T11:21:33.4726432Z W1204 11:14:21.013000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 456, in _pickle_op 2025-12-04T11:21:33.4728127Z W1204 11:14:21.013000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] raise BypassFxGraphCache(f"Unable to pickle non-standard op: {name}") 2025-12-04T11:21:33.4729891Z W1204 11:14:21.013000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] torch._inductor.codecache.BypassFxGraphCache: Unable to pickle non-standard op: torch.ops.prims._low_memory_max_pool_with_offsets.default 2025-12-04T11:21:33.4731488Z W1204 11:14:21.346000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] Unable to pickle input graph or example inputs 2025-12-04T11:21:33.4732687Z W1204 11:14:21.346000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] Traceback (most recent call last): 2025-12-04T11:21:33.4734169Z W1204 11:14:21.346000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/compile_fx_ext.py", line 486, in serialize_compile 2025-12-04T11:21:33.4735593Z W1204 11:14:21.346000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] ).serialize() 2025-12-04T11:21:33.4736991Z W1204 11:14:21.346000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/compile_fx_ext.py", line 210, in serialize 2025-12-04T11:21:33.4738600Z W1204 11:14:21.346000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] return _WireProtocolPickledInput(GraphPickler.dumps(self)) 2025-12-04T11:21:33.4740126Z W1204 11:14:21.346000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 127, in dumps 2025-12-04T11:21:33.4741472Z W1204 11:14:21.346000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] pickler.dump(obj) 2025-12-04T11:21:33.4742837Z W1204 11:14:21.346000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 94, in reducer_override 2025-12-04T11:21:33.4744483Z W1204 11:14:21.346000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] return _GraphModulePickleData.reduce_helper(self, obj) 2025-12-04T11:21:33.4745973Z W1204 11:14:21.346000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 352, in reduce_helper 2025-12-04T11:21:33.4747386Z W1204 11:14:21.346000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] cls(obj, pickler.options), 2025-12-04T11:21:33.4748739Z W1204 11:14:21.346000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 364, in __init__ 2025-12-04T11:21:33.4750313Z W1204 11:14:21.346000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] self.graph = _GraphPickleData(gm._graph, options) 2025-12-04T11:21:33.4751762Z W1204 11:14:21.346000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 571, in __init__ 2025-12-04T11:21:33.4753174Z W1204 11:14:21.346000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] nodes[node] = _NodePickleData(node, nodes, options) 2025-12-04T11:21:33.4754639Z W1204 11:14:21.346000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 387, in __init__ 2025-12-04T11:21:33.4756173Z W1204 11:14:21.346000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] self.target = _OpPickleData.pickle(node.target, options) 2025-12-04T11:21:33.4757844Z W1204 11:14:21.346000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 435, in pickle 2025-12-04T11:21:33.4759341Z W1204 11:14:21.346000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] return cls._pickle_op(name, _OpOverloadPickleData, options) 2025-12-04T11:21:33.4760824Z W1204 11:14:21.346000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 456, in _pickle_op 2025-12-04T11:21:33.4762384Z W1204 11:14:21.346000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] raise BypassFxGraphCache(f"Unable to pickle non-standard op: {name}") 2025-12-04T11:21:33.4764230Z W1204 11:14:21.346000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] torch._inductor.codecache.BypassFxGraphCache: Unable to pickle non-standard op: torch.ops.prims._low_memory_max_pool_with_offsets.default 2025-12-04T11:21:33.4765486Z PASSED [3.3575s] [ 3%] 2025-12-04T11:21:33.4766344Z inductor/test_compile_subprocess.py::GPUTests::test_adaptive_pool_errors_with_long_cuda <- test/inductor/test_torchinductor.py PASSED [0.5325s] [ 3%] 2025-12-04T11:21:33.4767682Z inductor/test_compile_subprocess.py::GPUTests::test_add_complex10_cuda <- test/inductor/test_torchinductor.py PASSED [0.7188s] [ 3%] 2025-12-04T11:21:33.4768959Z inductor/test_compile_subprocess.py::GPUTests::test_add_complex3_cuda <- test/inductor/test_torchinductor.py PASSED [0.5032s] [ 3%] 2025-12-04T11:21:33.4770192Z inductor/test_compile_subprocess.py::GPUTests::test_add_complex8_cuda <- test/inductor/test_torchinductor.py PASSED [0.6350s] [ 3%] 2025-12-04T11:21:33.4771504Z inductor/test_compile_subprocess.py::GPUTests::test_add_inplace_permuted_cuda <- test/inductor/test_torchinductor.py PASSED [1.0237s] [ 4%] 2025-12-04T11:21:33.4772893Z inductor/test_compile_subprocess.py::GPUTests::test_adding_tensor_offsets_cuda <- test/inductor/test_torchinductor.py PASSED [0.2496s] [ 4%] 2025-12-04T11:21:33.4774579Z inductor/test_compile_subprocess.py::GPUTests::test_addmm_cuda <- test/inductor/test_torchinductor.py W1204 11:14:25.435000 89536 site-packages/torch/_inductor/utils.py:1703] [0/0] Not enough SMs to use max_autotune_gemm mode 2025-12-04T11:21:33.4775782Z PASSED [1.0257s] [ 4%] 2025-12-04T11:21:33.4776932Z inductor/test_compile_subprocess.py::GPUTests::test_addmv_cuda <- test/inductor/test_torchinductor.py W1204 11:14:26.366000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] Unable to pickle input graph or example inputs 2025-12-04T11:21:33.4778869Z W1204 11:14:26.366000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] Traceback (most recent call last): 2025-12-04T11:21:33.4780065Z W1204 11:14:26.366000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/compile_fx_ext.py", line 486, in serialize_compile 2025-12-04T11:21:33.4781320Z W1204 11:14:26.366000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] ).serialize() 2025-12-04T11:21:33.4782363Z W1204 11:14:26.366000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/compile_fx_ext.py", line 210, in serialize 2025-12-04T11:21:33.4783615Z W1204 11:14:26.366000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] return _WireProtocolPickledInput(GraphPickler.dumps(self)) 2025-12-04T11:21:33.4784749Z W1204 11:14:26.366000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 127, in dumps 2025-12-04T11:21:33.4785726Z W1204 11:14:26.366000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] pickler.dump(obj) 2025-12-04T11:21:33.4786910Z W1204 11:14:26.366000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 94, in reducer_override 2025-12-04T11:21:33.4788204Z W1204 11:14:26.366000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] return _GraphModulePickleData.reduce_helper(self, obj) 2025-12-04T11:21:33.4789104Z W1204 11:14:26.366000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 352, in reduce_helper 2025-12-04T11:21:33.4789926Z W1204 11:14:26.366000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] cls(obj, pickler.options), 2025-12-04T11:21:33.4790791Z W1204 11:14:26.366000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 364, in __init__ 2025-12-04T11:21:33.4791643Z W1204 11:14:26.366000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] self.graph = _GraphPickleData(gm._graph, options) 2025-12-04T11:21:33.4792510Z W1204 11:14:26.366000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 571, in __init__ 2025-12-04T11:21:33.4793379Z W1204 11:14:26.366000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] nodes[node] = _NodePickleData(node, nodes, options) 2025-12-04T11:21:33.4794230Z W1204 11:14:26.366000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 387, in __init__ 2025-12-04T11:21:33.4795116Z W1204 11:14:26.366000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] self.target = _OpPickleData.pickle(node.target, options) 2025-12-04T11:21:33.4795980Z W1204 11:14:26.366000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 435, in pickle 2025-12-04T11:21:33.4796846Z W1204 11:14:26.366000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] return cls._pickle_op(name, _OpOverloadPickleData, options) 2025-12-04T11:21:33.4797724Z W1204 11:14:26.366000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 456, in _pickle_op 2025-12-04T11:21:33.4798633Z W1204 11:14:26.366000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] raise BypassFxGraphCache(f"Unable to pickle non-standard op: {name}") 2025-12-04T11:21:33.4799601Z W1204 11:14:26.366000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] torch._inductor.codecache.BypassFxGraphCache: Unable to pickle non-standard op: torch.ops.prims.convert_element_type.default 2025-12-04T11:21:33.4800330Z PASSED [0.7252s] [ 4%] 2025-12-04T11:21:33.4801072Z inductor/test_compile_subprocess.py::GPUTests::test_alexnet_prefix_cuda <- test/inductor/test_torchinductor.py W1204 11:14:27.213000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] Unable to pickle input graph or example inputs 2025-12-04T11:21:33.4802037Z W1204 11:14:27.213000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] Traceback (most recent call last): 2025-12-04T11:21:33.4802903Z W1204 11:14:27.213000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/compile_fx_ext.py", line 486, in serialize_compile 2025-12-04T11:21:33.4803714Z W1204 11:14:27.213000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] ).serialize() 2025-12-04T11:21:33.4804496Z W1204 11:14:27.213000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/compile_fx_ext.py", line 210, in serialize 2025-12-04T11:21:33.4805442Z W1204 11:14:27.213000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] return _WireProtocolPickledInput(GraphPickler.dumps(self)) 2025-12-04T11:21:33.4806370Z W1204 11:14:27.213000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 127, in dumps 2025-12-04T11:21:33.4807146Z W1204 11:14:27.213000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] pickler.dump(obj) 2025-12-04T11:21:33.4807940Z W1204 11:14:27.213000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 94, in reducer_override 2025-12-04T11:21:33.4808865Z W1204 11:14:27.213000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] return _GraphModulePickleData.reduce_helper(self, obj) 2025-12-04T11:21:33.4810338Z W1204 11:14:27.213000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 352, in reduce_helper 2025-12-04T11:21:33.4811216Z W1204 11:14:27.213000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] cls(obj, pickler.options), 2025-12-04T11:21:33.4812074Z W1204 11:14:27.213000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 364, in __init__ 2025-12-04T11:21:33.4812928Z W1204 11:14:27.213000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] self.graph = _GraphPickleData(gm._graph, options) 2025-12-04T11:21:33.4813789Z W1204 11:14:27.213000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 571, in __init__ 2025-12-04T11:21:33.4814639Z W1204 11:14:27.213000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] nodes[node] = _NodePickleData(node, nodes, options) 2025-12-04T11:21:33.4815489Z W1204 11:14:27.213000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 387, in __init__ 2025-12-04T11:21:33.4816349Z W1204 11:14:27.213000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] self.target = _OpPickleData.pickle(node.target, options) 2025-12-04T11:21:33.4817204Z W1204 11:14:27.213000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 435, in pickle 2025-12-04T11:21:33.4818068Z W1204 11:14:27.213000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] return cls._pickle_op(name, _OpOverloadPickleData, options) 2025-12-04T11:21:33.4818943Z W1204 11:14:27.213000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 456, in _pickle_op 2025-12-04T11:21:33.4819920Z W1204 11:14:27.213000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] raise BypassFxGraphCache(f"Unable to pickle non-standard op: {name}") 2025-12-04T11:21:33.4820927Z W1204 11:14:27.213000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] torch._inductor.codecache.BypassFxGraphCache: Unable to pickle non-standard op: torch.ops.prims._low_memory_max_pool_with_offsets.default 2025-12-04T11:21:33.4821849Z W1204 11:14:27.225000 89367 site-packages/torch/_inductor/utils.py:1703] [0/0] Not enough SMs to use max_autotune_gemm mode 2025-12-04T11:21:33.4822497Z W1204 11:14:27.861000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] Unable to pickle input graph or example inputs 2025-12-04T11:21:33.4823144Z W1204 11:14:27.861000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] Traceback (most recent call last): 2025-12-04T11:21:33.4824144Z W1204 11:14:27.861000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/compile_fx_ext.py", line 486, in serialize_compile 2025-12-04T11:21:33.4824989Z W1204 11:14:27.861000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] ).serialize() 2025-12-04T11:21:33.4825784Z W1204 11:14:27.861000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/compile_fx_ext.py", line 210, in serialize 2025-12-04T11:21:33.4826683Z W1204 11:14:27.861000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] return _WireProtocolPickledInput(GraphPickler.dumps(self)) 2025-12-04T11:21:33.4827582Z W1204 11:14:27.861000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 127, in dumps 2025-12-04T11:21:33.4828370Z W1204 11:14:27.861000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] pickler.dump(obj) 2025-12-04T11:21:33.4829157Z W1204 11:14:27.861000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 94, in reducer_override 2025-12-04T11:21:33.4830041Z W1204 11:14:27.861000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] return _GraphModulePickleData.reduce_helper(self, obj) 2025-12-04T11:21:33.4830921Z W1204 11:14:27.861000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 352, in reduce_helper 2025-12-04T11:21:33.4831757Z W1204 11:14:27.861000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] cls(obj, pickler.options), 2025-12-04T11:21:33.4832567Z W1204 11:14:27.861000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 364, in __init__ 2025-12-04T11:21:33.4833410Z W1204 11:14:27.861000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] self.graph = _GraphPickleData(gm._graph, options) 2025-12-04T11:21:33.4834244Z W1204 11:14:27.861000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 571, in __init__ 2025-12-04T11:21:33.4835083Z W1204 11:14:27.861000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] nodes[node] = _NodePickleData(node, nodes, options) 2025-12-04T11:21:33.4835922Z W1204 11:14:27.861000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 387, in __init__ 2025-12-04T11:21:33.4836771Z W1204 11:14:27.861000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] self.target = _OpPickleData.pickle(node.target, options) 2025-12-04T11:21:33.4837705Z W1204 11:14:27.861000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 435, in pickle 2025-12-04T11:21:33.4838566Z W1204 11:14:27.861000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] return cls._pickle_op(name, _OpOverloadPickleData, options) 2025-12-04T11:21:33.4839439Z W1204 11:14:27.861000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 456, in _pickle_op 2025-12-04T11:21:33.4840339Z W1204 11:14:27.861000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] raise BypassFxGraphCache(f"Unable to pickle non-standard op: {name}") 2025-12-04T11:21:33.4841329Z W1204 11:14:27.861000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] torch._inductor.codecache.BypassFxGraphCache: Unable to pickle non-standard op: torch.ops.prims._low_memory_max_pool_with_offsets.default 2025-12-04T11:21:33.4842343Z W1204 11:14:28.448000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] Unable to pickle input graph or example inputs 2025-12-04T11:21:33.4842978Z W1204 11:14:28.448000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] Traceback (most recent call last): 2025-12-04T11:21:33.4843826Z W1204 11:14:28.448000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/compile_fx_ext.py", line 486, in serialize_compile 2025-12-04T11:21:33.4844627Z W1204 11:14:28.448000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] ).serialize() 2025-12-04T11:21:33.4845451Z W1204 11:14:28.448000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/compile_fx_ext.py", line 210, in serialize 2025-12-04T11:21:33.4846349Z W1204 11:14:28.448000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] return _WireProtocolPickledInput(GraphPickler.dumps(self)) 2025-12-04T11:21:33.4847212Z W1204 11:14:28.448000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 127, in dumps 2025-12-04T11:21:33.4847980Z W1204 11:14:28.448000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] pickler.dump(obj) 2025-12-04T11:21:33.4848765Z W1204 11:14:28.448000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 94, in reducer_override 2025-12-04T11:21:33.4849660Z W1204 11:14:28.448000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] return _GraphModulePickleData.reduce_helper(self, obj) 2025-12-04T11:21:33.4850531Z W1204 11:14:28.448000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 352, in reduce_helper 2025-12-04T11:21:33.4851348Z W1204 11:14:28.448000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] cls(obj, pickler.options), 2025-12-04T11:21:33.4852149Z W1204 11:14:28.448000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 364, in __init__ 2025-12-04T11:21:33.4852981Z W1204 11:14:28.448000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] self.graph = _GraphPickleData(gm._graph, options) 2025-12-04T11:21:33.4853822Z W1204 11:14:28.448000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 571, in __init__ 2025-12-04T11:21:33.4854698Z W1204 11:14:28.448000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] nodes[node] = _NodePickleData(node, nodes, options) 2025-12-04T11:21:33.4855546Z W1204 11:14:28.448000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 387, in __init__ 2025-12-04T11:21:33.4856395Z W1204 11:14:28.448000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] self.target = _OpPickleData.pickle(node.target, options) 2025-12-04T11:21:33.4857254Z W1204 11:14:28.448000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 435, in pickle 2025-12-04T11:21:33.4858113Z W1204 11:14:28.448000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] return cls._pickle_op(name, _OpOverloadPickleData, options) 2025-12-04T11:21:33.4858984Z W1204 11:14:28.448000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 456, in _pickle_op 2025-12-04T11:21:33.4859964Z W1204 11:14:28.448000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] raise BypassFxGraphCache(f"Unable to pickle non-standard op: {name}") 2025-12-04T11:21:33.4860971Z W1204 11:14:28.448000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] torch._inductor.codecache.BypassFxGraphCache: Unable to pickle non-standard op: torch.ops.prims._low_memory_max_pool_with_offsets.default 2025-12-04T11:21:33.4861900Z W1204 11:14:28.715000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] Unable to pickle input graph or example inputs 2025-12-04T11:21:33.4862574Z W1204 11:14:28.715000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] Traceback (most recent call last): 2025-12-04T11:21:33.4863433Z W1204 11:14:28.715000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/compile_fx_ext.py", line 486, in serialize_compile 2025-12-04T11:21:33.4864301Z W1204 11:14:28.715000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] ).serialize() 2025-12-04T11:21:33.4865074Z W1204 11:14:28.715000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/compile_fx_ext.py", line 210, in serialize 2025-12-04T11:21:33.4865965Z W1204 11:14:28.715000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] return _WireProtocolPickledInput(GraphPickler.dumps(self)) 2025-12-04T11:21:33.4866832Z W1204 11:14:28.715000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 127, in dumps 2025-12-04T11:21:33.4867612Z W1204 11:14:28.715000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] pickler.dump(obj) 2025-12-04T11:21:33.4868411Z W1204 11:14:28.715000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 94, in reducer_override 2025-12-04T11:21:33.4869291Z W1204 11:14:28.715000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] return _GraphModulePickleData.reduce_helper(self, obj) 2025-12-04T11:21:33.4870167Z W1204 11:14:28.715000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 352, in reduce_helper 2025-12-04T11:21:33.4870976Z W1204 11:14:28.715000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] cls(obj, pickler.options), 2025-12-04T11:21:33.4871770Z W1204 11:14:28.715000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 364, in __init__ 2025-12-04T11:21:33.4872658Z W1204 11:14:28.715000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] self.graph = _GraphPickleData(gm._graph, options) 2025-12-04T11:21:33.4873504Z W1204 11:14:28.715000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 571, in __init__ 2025-12-04T11:21:33.4874337Z W1204 11:14:28.715000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] nodes[node] = _NodePickleData(node, nodes, options) 2025-12-04T11:21:33.4875172Z W1204 11:14:28.715000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 387, in __init__ 2025-12-04T11:21:33.4876028Z W1204 11:14:28.715000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] self.target = _OpPickleData.pickle(node.target, options) 2025-12-04T11:21:33.4876876Z W1204 11:14:28.715000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 435, in pickle 2025-12-04T11:21:33.4878084Z W1204 11:14:28.715000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] return cls._pickle_op(name, _OpOverloadPickleData, options) 2025-12-04T11:21:33.4879048Z W1204 11:14:28.715000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 456, in _pickle_op 2025-12-04T11:21:33.4879966Z W1204 11:14:28.715000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] raise BypassFxGraphCache(f"Unable to pickle non-standard op: {name}") 2025-12-04T11:21:33.4881028Z W1204 11:14:28.715000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] torch._inductor.codecache.BypassFxGraphCache: Unable to pickle non-standard op: torch.ops.prims._low_memory_max_pool_with_offsets.default 2025-12-04T11:21:33.4881739Z PASSED [1.8953s] [ 5%] 2025-12-04T11:21:33.4882255Z inductor/test_compile_subprocess.py::GPUTests::test_allow_reuse_disable_if_exceed_peak_cuda <- test/inductor/test_torchinductor.py PASSED [0.8975s] [ 5%] 2025-12-04T11:21:33.4883021Z inductor/test_compile_subprocess.py::GPUTests::test_angle_cuda <- test/inductor/test_torchinductor.py PASSED [0.5794s] [ 5%] 2025-12-04T11:21:33.4884235Z inductor/test_compile_subprocess.py::GPUTests::test_aoti_eager_override_registration_cuda <- test/inductor/test_torchinductor.py W1204 11:14:30.740000 89367 site-packages/torch/_export/__init__.py:71] +============================+ 2025-12-04T11:21:33.4885281Z W1204 11:14:30.741000 89367 site-packages/torch/_export/__init__.py:72] | !!! WARNING !!! | 2025-12-04T11:21:33.4885786Z W1204 11:14:30.741000 89367 site-packages/torch/_export/__init__.py:73] +============================+ 2025-12-04T11:21:33.4887692Z W1204 11:14:30.741000 89367 site-packages/torch/_export/__init__.py:74] torch._export.aot_compile()/torch._export.aot_load() is being deprecated, please switch to directly calling torch._inductor.aoti_compile_and_package(torch.export.export())/torch._inductor.aoti_load_package() instead. 2025-12-04T11:21:33.4888605Z PASSED [27.8668s] [ 5%] 2025-12-04T11:21:33.4889097Z inductor/test_compile_subprocess.py::GPUTests::test_aoti_eager_support_out_cuda <- test/inductor/test_torchinductor.py PASSED [5.7099s] [ 6%] 2025-12-04T11:21:33.4889872Z inductor/test_compile_subprocess.py::GPUTests::test_aoti_eager_with_scalar_cuda <- test/inductor/test_torchinductor.py PASSED [24.0502s] [ 6%] 2025-12-04T11:21:33.4890933Z inductor/test_compile_subprocess.py::GPUTests::test_arange4_cuda <- test/inductor/test_torchinductor.py W1204 11:15:28.099000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] Unable to pickle input graph or example inputs 2025-12-04T11:21:33.4891924Z W1204 11:15:28.099000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] Traceback (most recent call last): 2025-12-04T11:21:33.4892930Z W1204 11:15:28.099000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/compile_fx_ext.py", line 486, in serialize_compile 2025-12-04T11:21:33.4894047Z W1204 11:15:28.099000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] ).serialize() 2025-12-04T11:21:33.4895133Z W1204 11:15:28.099000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/compile_fx_ext.py", line 210, in serialize 2025-12-04T11:21:33.4896039Z W1204 11:15:28.099000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] return _WireProtocolPickledInput(GraphPickler.dumps(self)) 2025-12-04T11:21:33.4896924Z W1204 11:15:28.099000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 127, in dumps 2025-12-04T11:21:33.4897696Z W1204 11:15:28.099000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] pickler.dump(obj) 2025-12-04T11:21:33.4898640Z W1204 11:15:28.099000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 94, in reducer_override 2025-12-04T11:21:33.4899523Z W1204 11:15:28.099000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] return _GraphModulePickleData.reduce_helper(self, obj) 2025-12-04T11:21:33.4900407Z W1204 11:15:28.099000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 352, in reduce_helper 2025-12-04T11:21:33.4901290Z W1204 11:15:28.099000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] cls(obj, pickler.options), 2025-12-04T11:21:33.4902091Z W1204 11:15:28.099000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 364, in __init__ 2025-12-04T11:21:33.4902941Z W1204 11:15:28.099000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] self.graph = _GraphPickleData(gm._graph, options) 2025-12-04T11:21:33.4903868Z W1204 11:15:28.099000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 571, in __init__ 2025-12-04T11:21:33.4904917Z W1204 11:15:28.099000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] nodes[node] = _NodePickleData(node, nodes, options) 2025-12-04T11:21:33.4905779Z W1204 11:15:28.099000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 387, in __init__ 2025-12-04T11:21:33.4906648Z W1204 11:15:28.099000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] self.target = _OpPickleData.pickle(node.target, options) 2025-12-04T11:21:33.4907683Z W1204 11:15:28.099000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 435, in pickle 2025-12-04T11:21:33.4908723Z W1204 11:15:28.099000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] return cls._pickle_op(name, _OpOverloadPickleData, options) 2025-12-04T11:21:33.4909613Z W1204 11:15:28.099000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 456, in _pickle_op 2025-12-04T11:21:33.4910523Z W1204 11:15:28.099000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] raise BypassFxGraphCache(f"Unable to pickle non-standard op: {name}") 2025-12-04T11:21:33.4911460Z W1204 11:15:28.099000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] torch._inductor.codecache.BypassFxGraphCache: Unable to pickle non-standard op: torch.ops.prims.iota.default 2025-12-04T11:21:33.4912391Z W1204 11:15:28.359000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] Unable to pickle input graph or example inputs 2025-12-04T11:21:33.4913033Z W1204 11:15:28.359000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] Traceback (most recent call last): 2025-12-04T11:21:33.4913908Z W1204 11:15:28.359000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/compile_fx_ext.py", line 486, in serialize_compile 2025-12-04T11:21:33.4914713Z W1204 11:15:28.359000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] ).serialize() 2025-12-04T11:21:33.4915513Z W1204 11:15:28.359000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/compile_fx_ext.py", line 210, in serialize 2025-12-04T11:21:33.4916465Z W1204 11:15:28.359000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] return _WireProtocolPickledInput(GraphPickler.dumps(self)) 2025-12-04T11:21:33.4917410Z W1204 11:15:28.359000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 127, in dumps 2025-12-04T11:21:33.4918199Z W1204 11:15:28.359000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] pickler.dump(obj) 2025-12-04T11:21:33.4919002Z W1204 11:15:28.359000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 94, in reducer_override 2025-12-04T11:21:33.4919923Z W1204 11:15:28.359000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] return _GraphModulePickleData.reduce_helper(self, obj) 2025-12-04T11:21:33.4920808Z W1204 11:15:28.359000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 352, in reduce_helper 2025-12-04T11:21:33.4921649Z W1204 11:15:28.359000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] cls(obj, pickler.options), 2025-12-04T11:21:33.4922444Z W1204 11:15:28.359000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 364, in __init__ 2025-12-04T11:21:33.4923297Z W1204 11:15:28.359000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] self.graph = _GraphPickleData(gm._graph, options) 2025-12-04T11:21:33.4924143Z W1204 11:15:28.359000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 571, in __init__ 2025-12-04T11:21:33.4924994Z W1204 11:15:28.359000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] nodes[node] = _NodePickleData(node, nodes, options) 2025-12-04T11:21:33.4925843Z W1204 11:15:28.359000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 387, in __init__ 2025-12-04T11:21:33.4926706Z W1204 11:15:28.359000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] self.target = _OpPickleData.pickle(node.target, options) 2025-12-04T11:21:33.4927576Z W1204 11:15:28.359000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 435, in pickle 2025-12-04T11:21:33.4928452Z W1204 11:15:28.359000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] return cls._pickle_op(name, _OpOverloadPickleData, options) 2025-12-04T11:21:33.4929338Z W1204 11:15:28.359000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 456, in _pickle_op 2025-12-04T11:21:33.4930296Z W1204 11:15:28.359000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] raise BypassFxGraphCache(f"Unable to pickle non-standard op: {name}") 2025-12-04T11:21:33.4931214Z W1204 11:15:28.359000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] torch._inductor.codecache.BypassFxGraphCache: Unable to pickle non-standard op: torch.ops.prims.iota.default 2025-12-04T11:21:33.4931856Z PASSED [0.5276s] [ 6%] 2025-12-04T11:21:33.4932309Z inductor/test_compile_subprocess.py::GPUTests::test_argmax_argmin1_cuda <- test/inductor/test_torchinductor.py PASSED [0.6002s] [ 6%] 2025-12-04T11:21:33.4933040Z inductor/test_compile_subprocess.py::GPUTests::test_argmax_argmin2_cuda <- test/inductor/test_torchinductor.py PASSED [0.6629s] [ 6%] 2025-12-04T11:21:33.4933768Z inductor/test_compile_subprocess.py::GPUTests::test_argmax_argmin3_cuda <- test/inductor/test_torchinductor.py PASSED [1.2852s] [ 7%] 2025-12-04T11:21:33.4934859Z inductor/test_compile_subprocess.py::GPUTests::test_argmax_to_float_cuda <- test/inductor/test_torchinductor.py W1204 11:15:31.179000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] Unable to pickle input graph or example inputs 2025-12-04T11:21:33.4935818Z W1204 11:15:31.179000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] Traceback (most recent call last): 2025-12-04T11:21:33.4936690Z W1204 11:15:31.179000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/compile_fx_ext.py", line 486, in serialize_compile 2025-12-04T11:21:33.4937500Z W1204 11:15:31.179000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] ).serialize() 2025-12-04T11:21:33.4938338Z W1204 11:15:31.179000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/compile_fx_ext.py", line 210, in serialize 2025-12-04T11:21:33.4939247Z W1204 11:15:31.179000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] return _WireProtocolPickledInput(GraphPickler.dumps(self)) 2025-12-04T11:21:33.4940121Z W1204 11:15:31.179000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 127, in dumps 2025-12-04T11:21:33.4940897Z W1204 11:15:31.179000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] pickler.dump(obj) 2025-12-04T11:21:33.4941694Z W1204 11:15:31.179000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 94, in reducer_override 2025-12-04T11:21:33.4942587Z W1204 11:15:31.179000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] return _GraphModulePickleData.reduce_helper(self, obj) 2025-12-04T11:21:33.4943474Z W1204 11:15:31.179000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 352, in reduce_helper 2025-12-04T11:21:33.4944379Z W1204 11:15:31.179000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] cls(obj, pickler.options), 2025-12-04T11:21:33.4945183Z W1204 11:15:31.179000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 364, in __init__ 2025-12-04T11:21:33.4946032Z W1204 11:15:31.179000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] self.graph = _GraphPickleData(gm._graph, options) 2025-12-04T11:21:33.4946881Z W1204 11:15:31.179000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 571, in __init__ 2025-12-04T11:21:33.4947826Z W1204 11:15:31.179000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] nodes[node] = _NodePickleData(node, nodes, options) 2025-12-04T11:21:33.4948871Z W1204 11:15:31.179000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 387, in __init__ 2025-12-04T11:21:33.4949886Z W1204 11:15:31.179000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] self.target = _OpPickleData.pickle(node.target, options) 2025-12-04T11:21:33.4950854Z W1204 11:15:31.179000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 435, in pickle 2025-12-04T11:21:33.4951894Z W1204 11:15:31.179000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] return cls._pickle_op(name, _OpOverloadPickleData, options) 2025-12-04T11:21:33.4952920Z W1204 11:15:31.179000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 456, in _pickle_op 2025-12-04T11:21:33.4954042Z W1204 11:15:31.179000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] raise BypassFxGraphCache(f"Unable to pickle non-standard op: {name}") 2025-12-04T11:21:33.4955179Z W1204 11:15:31.179000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] torch._inductor.codecache.BypassFxGraphCache: Unable to pickle non-standard op: torch.ops.prims.convert_element_type.default 2025-12-04T11:21:33.4955944Z PASSED [0.6010s] [ 7%] 2025-12-04T11:21:33.4956491Z inductor/test_compile_subprocess.py::GPUTests::test_as_strided_scatter_cuda <- test/inductor/test_torchinductor.py PASSED [0.8050s] [ 7%] 2025-12-04T11:21:33.4957443Z inductor/test_compile_subprocess.py::GPUTests::test_assert_alignment_op_name_fail_cuda <- test/inductor/test_torchinductor.py PASSED [0.0026s] [ 7%] 2025-12-04T11:21:33.4972067Z inductor/test_compile_subprocess.py::GPUTests::test_assert_size_stride_op_name_pass_cuda <- test/inductor/test_torchinductor.py PASSED [0.0023s] [ 8%] 2025-12-04T11:21:33.4973128Z inductor/test_compile_subprocess.py::GPUTests::test_avg_pool2d1_cuda <- test/inductor/test_torchinductor.py PASSED [1.0144s] [ 8%] 2025-12-04T11:21:33.4973869Z inductor/test_compile_subprocess.py::GPUTests::test_avg_pool2d4_cuda <- test/inductor/test_torchinductor.py PASSED [0.8664s] [ 8%] 2025-12-04T11:21:33.4974564Z inductor/test_compile_subprocess.py::GPUTests::test_avg_pool2d5_cuda <- test/inductor/test_torchinductor.py PASSED [0.9780s] [ 8%] 2025-12-04T11:21:33.4975253Z inductor/test_compile_subprocess.py::GPUTests::test_avg_pool2d6_cuda <- test/inductor/test_torchinductor.py PASSED [0.5802s] [ 9%] 2025-12-04T11:21:33.4975952Z inductor/test_compile_subprocess.py::GPUTests::test_avg_pool2d8_cuda <- test/inductor/test_torchinductor.py PASSED [1.1400s] [ 9%] 2025-12-04T11:21:33.4976685Z inductor/test_compile_subprocess.py::GPUTests::test_avg_pool2d_backward3_cuda <- test/inductor/test_torchinductor.py PASSED [1.2254s] [ 9%] 2025-12-04T11:21:33.4977460Z inductor/test_compile_subprocess.py::GPUTests::test_avg_pool2d_backward4_cuda <- test/inductor/test_torchinductor.py PASSED [0.1101s] [ 9%] 2025-12-04T11:21:33.4978685Z inductor/test_compile_subprocess.py::GPUTests::test_avg_pool3d_backward2_cuda <- test/inductor/test_torchinductor.py SKIPPED [0.0003s] (test is slow; run with PYTORCH_TEST_WITH_SLOW to enable test) [ 9%] 2025-12-04T11:21:33.4979594Z inductor/test_compile_subprocess.py::GPUTests::test_avg_pool3d_backward3_cuda <- test/inductor/test_torchinductor.py PASSED [2.0980s] [ 10%] 2025-12-04T11:21:33.4980338Z inductor/test_compile_subprocess.py::GPUTests::test_avg_pool3d_backward4_cuda <- test/inductor/test_torchinductor.py PASSED [0.1106s] [ 10%] 2025-12-04T11:21:33.4981050Z inductor/test_compile_subprocess.py::GPUTests::test_baddbmm_cuda <- test/inductor/test_torchinductor.py PASSED [0.6164s] [ 10%] 2025-12-04T11:21:33.4981865Z inductor/test_compile_subprocess.py::GPUTests::test_bernoulli1_cuda <- test/inductor/test_torchinductor.py PASSED [1.3197s] [ 10%] 2025-12-04T11:21:33.4982856Z inductor/test_compile_subprocess.py::GPUTests::test_bernoulli2_cuda <- test/inductor/test_torchinductor.py W1204 11:15:42.703000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] Unable to pickle input graph or example inputs 2025-12-04T11:21:33.4983899Z W1204 11:15:42.703000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] Traceback (most recent call last): 2025-12-04T11:21:33.4984765Z W1204 11:15:42.703000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/compile_fx_ext.py", line 486, in serialize_compile 2025-12-04T11:21:33.4985570Z W1204 11:15:42.703000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] ).serialize() 2025-12-04T11:21:33.4986367Z W1204 11:15:42.703000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/compile_fx_ext.py", line 210, in serialize 2025-12-04T11:21:33.4987393Z W1204 11:15:42.703000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] return _WireProtocolPickledInput(GraphPickler.dumps(self)) 2025-12-04T11:21:33.4988271Z W1204 11:15:42.703000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 127, in dumps 2025-12-04T11:21:33.4989038Z W1204 11:15:42.703000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] pickler.dump(obj) 2025-12-04T11:21:33.4989893Z W1204 11:15:42.703000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 94, in reducer_override 2025-12-04T11:21:33.4990783Z W1204 11:15:42.703000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] return _GraphModulePickleData.reduce_helper(self, obj) 2025-12-04T11:21:33.4991651Z W1204 11:15:42.703000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 352, in reduce_helper 2025-12-04T11:21:33.4992461Z W1204 11:15:42.703000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] cls(obj, pickler.options), 2025-12-04T11:21:33.4993256Z W1204 11:15:42.703000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 364, in __init__ 2025-12-04T11:21:33.4994113Z W1204 11:15:42.703000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] self.graph = _GraphPickleData(gm._graph, options) 2025-12-04T11:21:33.4994960Z W1204 11:15:42.703000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 571, in __init__ 2025-12-04T11:21:33.4995801Z W1204 11:15:42.703000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] nodes[node] = _NodePickleData(node, nodes, options) 2025-12-04T11:21:33.4996649Z W1204 11:15:42.703000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 387, in __init__ 2025-12-04T11:21:33.4997505Z W1204 11:15:42.703000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] self.target = _OpPickleData.pickle(node.target, options) 2025-12-04T11:21:33.4998365Z W1204 11:15:42.703000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 435, in pickle 2025-12-04T11:21:33.4999218Z W1204 11:15:42.703000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] return cls._pickle_op(name, _OpOverloadPickleData, options) 2025-12-04T11:21:33.5000147Z W1204 11:15:42.703000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 456, in _pickle_op 2025-12-04T11:21:33.5001046Z W1204 11:15:42.703000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] raise BypassFxGraphCache(f"Unable to pickle non-standard op: {name}") 2025-12-04T11:21:33.5001989Z W1204 11:15:42.703000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] torch._inductor.codecache.BypassFxGraphCache: Unable to pickle non-standard op: torch.ops.prims.inductor_seeds.default 2025-12-04T11:21:33.5002877Z W1204 11:15:43.601000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] Unable to pickle input graph or example inputs 2025-12-04T11:21:33.5003519Z W1204 11:15:43.601000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] Traceback (most recent call last): 2025-12-04T11:21:33.5004372Z W1204 11:15:43.601000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/compile_fx_ext.py", line 486, in serialize_compile 2025-12-04T11:21:33.5005269Z W1204 11:15:43.601000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] ).serialize() 2025-12-04T11:21:33.5006051Z W1204 11:15:43.601000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/compile_fx_ext.py", line 210, in serialize 2025-12-04T11:21:33.5006937Z W1204 11:15:43.601000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] return _WireProtocolPickledInput(GraphPickler.dumps(self)) 2025-12-04T11:21:33.5007839Z W1204 11:15:43.601000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 127, in dumps 2025-12-04T11:21:33.5008610Z W1204 11:15:43.601000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] pickler.dump(obj) 2025-12-04T11:21:33.5009409Z W1204 11:15:43.601000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 94, in reducer_override 2025-12-04T11:21:33.5010287Z W1204 11:15:43.601000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] return _GraphModulePickleData.reduce_helper(self, obj) 2025-12-04T11:21:33.5011149Z W1204 11:15:43.601000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 352, in reduce_helper 2025-12-04T11:21:33.5011955Z W1204 11:15:43.601000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] cls(obj, pickler.options), 2025-12-04T11:21:33.5012749Z W1204 11:15:43.601000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 364, in __init__ 2025-12-04T11:21:33.5013581Z W1204 11:15:43.601000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] self.graph = _GraphPickleData(gm._graph, options) 2025-12-04T11:21:33.5014404Z W1204 11:15:43.601000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 571, in __init__ 2025-12-04T11:21:33.5015228Z W1204 11:15:43.601000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] nodes[node] = _NodePickleData(node, nodes, options) 2025-12-04T11:21:33.5016053Z W1204 11:15:43.601000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 387, in __init__ 2025-12-04T11:21:33.5016916Z W1204 11:15:43.601000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] self.target = _OpPickleData.pickle(node.target, options) 2025-12-04T11:21:33.5017821Z W1204 11:15:43.601000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 435, in pickle 2025-12-04T11:21:33.5018681Z W1204 11:15:43.601000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] return cls._pickle_op(name, _OpOverloadPickleData, options) 2025-12-04T11:21:33.5019547Z W1204 11:15:43.601000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 456, in _pickle_op 2025-12-04T11:21:33.5020441Z W1204 11:15:43.601000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] raise BypassFxGraphCache(f"Unable to pickle non-standard op: {name}") 2025-12-04T11:21:33.5021379Z W1204 11:15:43.601000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] torch._inductor.codecache.BypassFxGraphCache: Unable to pickle non-standard op: torch.ops.prims.inductor_seeds.default 2025-12-04T11:21:33.5022087Z PASSED [1.1314s] [ 11%] 2025-12-04T11:21:33.5022563Z inductor/test_compile_subprocess.py::GPUTests::test_bitwise2_cuda <- test/inductor/test_torchinductor.py PASSED [0.2776s] [ 11%] 2025-12-04T11:21:33.5023261Z inductor/test_compile_subprocess.py::GPUTests::test_bitwise_cuda <- test/inductor/test_torchinductor.py PASSED [0.2445s] [ 11%] 2025-12-04T11:21:33.5024008Z inductor/test_compile_subprocess.py::GPUTests::test_bmm1_cuda <- test/inductor/test_torchinductor.py PASSED [0.5369s] [ 11%] 2025-12-04T11:21:33.5024670Z inductor/test_compile_subprocess.py::GPUTests::test_bool_cuda <- test/inductor/test_torchinductor.py PASSED [0.3676s] [ 12%] 2025-12-04T11:21:33.5025397Z inductor/test_compile_subprocess.py::GPUTests::test_both_scalars_cuda <- test/inductor/test_torchinductor.py PASSED [0.6070s] [ 12%] 2025-12-04T11:21:33.5026136Z inductor/test_compile_subprocess.py::GPUTests::test_bucketize_broadcast_cuda <- test/inductor/test_torchinductor.py PASSED [0.5030s] [ 12%] 2025-12-04T11:21:33.5026913Z inductor/test_compile_subprocess.py::GPUTests::test_bucketize_computed_offsets_cuda <- test/inductor/test_torchinductor.py PASSED [0.1839s] [ 12%] 2025-12-04T11:21:33.5027697Z inductor/test_compile_subprocess.py::GPUTests::test_bucketize_int_int16_uint8_cuda <- test/inductor/test_torchinductor.py PASSED [1.6260s] [ 12%] 2025-12-04T11:21:33.5028465Z inductor/test_compile_subprocess.py::GPUTests::test_bucketize_int_int32_int16_cuda <- test/inductor/test_torchinductor.py PASSED [1.6090s] [ 13%] 2025-12-04T11:21:33.5029225Z inductor/test_compile_subprocess.py::GPUTests::test_bucketize_int_int32_int32_cuda <- test/inductor/test_torchinductor.py PASSED [1.5953s] [ 13%] 2025-12-04T11:21:33.5029986Z inductor/test_compile_subprocess.py::GPUTests::test_bucketize_int_int32_int8_cuda <- test/inductor/test_torchinductor.py PASSED [1.6022s] [ 13%] 2025-12-04T11:21:33.5030742Z inductor/test_compile_subprocess.py::GPUTests::test_bucketize_int_int64_int64_cuda <- test/inductor/test_torchinductor.py PASSED [1.5967s] [ 13%] 2025-12-04T11:21:33.5031510Z inductor/test_compile_subprocess.py::GPUTests::test_bucketize_int_int64_int8_cuda <- test/inductor/test_torchinductor.py PASSED [1.6112s] [ 14%] 2025-12-04T11:21:33.5032274Z inductor/test_compile_subprocess.py::GPUTests::test_bucketize_int_uint8_int16_cuda <- test/inductor/test_torchinductor.py PASSED [1.6049s] [ 14%] 2025-12-04T11:21:33.5033029Z inductor/test_compile_subprocess.py::GPUTests::test_bucketize_int_uint8_int8_cuda <- test/inductor/test_torchinductor.py PASSED [1.6251s] [ 14%] 2025-12-04T11:21:33.5033786Z inductor/test_compile_subprocess.py::GPUTests::test_bucketize_nd_tiling_True_cuda <- test/inductor/test_torchinductor.py PASSED [0.7796s] [ 14%] 2025-12-04T11:21:33.5034609Z inductor/test_compile_subprocess.py::GPUTests::test_buffer_copied_in_graph_with_different_shapes_cuda <- test/inductor/test_torchinductor.py PASSED [0.2469s] [ 15%] 2025-12-04T11:21:33.5035479Z inductor/test_compile_subprocess.py::GPUTests::test_buffer_use_after_remove_cuda <- test/inductor/test_torchinductor.py PASSED [1.9532s] [ 15%] 2025-12-04T11:21:33.5036267Z inductor/test_compile_subprocess.py::GPUTests::test_builtins_round_float_ndigits_neg_cuda <- test/inductor/test_torchinductor.py PASSED [0.2400s] [ 15%] 2025-12-04T11:21:33.5037085Z inductor/test_compile_subprocess.py::GPUTests::test_builtins_round_float_ndigits_zero_cuda <- test/inductor/test_torchinductor.py PASSED [0.1978s] [ 15%] 2025-12-04T11:21:33.5037899Z inductor/test_compile_subprocess.py::GPUTests::test_builtins_round_int_ndigits_pos_cuda <- test/inductor/test_torchinductor.py PASSED [0.1573s] [ 15%] 2025-12-04T11:21:33.5038692Z inductor/test_compile_subprocess.py::GPUTests::test_builtins_round_int_ndigits_zero_cuda <- test/inductor/test_torchinductor.py PASSED [0.1574s] [ 16%] 2025-12-04T11:21:33.5039445Z inductor/test_compile_subprocess.py::GPUTests::test_cat_inplace_cuda <- test/inductor/test_torchinductor.py PASSED [0.6584s] [ 16%] 2025-12-04T11:21:33.5040158Z inductor/test_compile_subprocess.py::GPUTests::test_cat_negative_dim_cuda <- test/inductor/test_torchinductor.py PASSED [0.8894s] [ 16%] 2025-12-04T11:21:33.5041573Z inductor/test_compile_subprocess.py::GPUTests::test_cat_single_empty_cuda <- test/inductor/test_torchinductor.py PASSED [0.1806s] [ 16%] 2025-12-04T11:21:33.5042287Z inductor/test_compile_subprocess.py::GPUTests::test_cat_unbacked_2d_cuda <- test/inductor/test_torchinductor.py PASSED [0.8494s] [ 17%] 2025-12-04T11:21:33.5043032Z inductor/test_compile_subprocess.py::GPUTests::test_cat_unbacked_legacy_empty_cuda <- test/inductor/test_torchinductor.py PASSED [0.0208s] [ 17%] 2025-12-04T11:21:33.5043791Z inductor/test_compile_subprocess.py::GPUTests::test_chunk_recompiles_cuda <- test/inductor/test_torchinductor.py PASSED [0.8031s] [ 17%] 2025-12-04T11:21:33.5044887Z inductor/test_compile_subprocess.py::GPUTests::test_clamp_type_promotion_non_tensor_cuda <- test/inductor/test_torchinductor.py W1204 11:16:06.614000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] Unable to pickle input graph or example inputs 2025-12-04T11:21:33.5045895Z W1204 11:16:06.614000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] Traceback (most recent call last): 2025-12-04T11:21:33.5046757Z W1204 11:16:06.614000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/compile_fx_ext.py", line 486, in serialize_compile 2025-12-04T11:21:33.5047746Z W1204 11:16:06.614000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] ).serialize() 2025-12-04T11:21:33.5048534Z W1204 11:16:06.614000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/compile_fx_ext.py", line 210, in serialize 2025-12-04T11:21:33.5049419Z W1204 11:16:06.614000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] return _WireProtocolPickledInput(GraphPickler.dumps(self)) 2025-12-04T11:21:33.5050287Z W1204 11:16:06.614000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 127, in dumps 2025-12-04T11:21:33.5051043Z W1204 11:16:06.614000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] pickler.dump(obj) 2025-12-04T11:21:33.5051834Z W1204 11:16:06.614000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 94, in reducer_override 2025-12-04T11:21:33.5052699Z W1204 11:16:06.614000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] return _GraphModulePickleData.reduce_helper(self, obj) 2025-12-04T11:21:33.5053570Z W1204 11:16:06.614000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 352, in reduce_helper 2025-12-04T11:21:33.5054430Z W1204 11:16:06.614000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] cls(obj, pickler.options), 2025-12-04T11:21:33.5055233Z W1204 11:16:06.614000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 364, in __init__ 2025-12-04T11:21:33.5056076Z W1204 11:16:06.614000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] self.graph = _GraphPickleData(gm._graph, options) 2025-12-04T11:21:33.5056908Z W1204 11:16:06.614000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 571, in __init__ 2025-12-04T11:21:33.5057742Z W1204 11:16:06.614000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] nodes[node] = _NodePickleData(node, nodes, options) 2025-12-04T11:21:33.5058879Z W1204 11:16:06.614000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 387, in __init__ 2025-12-04T11:21:33.5059863Z W1204 11:16:06.614000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] self.target = _OpPickleData.pickle(node.target, options) 2025-12-04T11:21:33.5060721Z W1204 11:16:06.614000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 435, in pickle 2025-12-04T11:21:33.5061573Z W1204 11:16:06.614000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] return cls._pickle_op(name, _OpOverloadPickleData, options) 2025-12-04T11:21:33.5062550Z W1204 11:16:06.614000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 456, in _pickle_op 2025-12-04T11:21:33.5063451Z W1204 11:16:06.614000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] raise BypassFxGraphCache(f"Unable to pickle non-standard op: {name}") 2025-12-04T11:21:33.5064486Z W1204 11:16:06.614000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] torch._inductor.codecache.BypassFxGraphCache: Unable to pickle non-standard op: torch.ops.prims.convert_element_type.default 2025-12-04T11:21:33.5065165Z PASSED [0.2068s] [ 17%] 2025-12-04T11:21:33.5065595Z inductor/test_compile_subprocess.py::GPUTests::test_compar_cuda <- test/inductor/test_torchinductor.py PASSED [0.2535s] [ 18%] 2025-12-04T11:21:33.5066584Z inductor/test_compile_subprocess.py::GPUTests::test_complex_fallback_cuda <- test/inductor/test_torchinductor.py W1204 11:16:07.415000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] Unable to pickle input graph or example inputs 2025-12-04T11:21:33.5067540Z W1204 11:16:07.415000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] Traceback (most recent call last): 2025-12-04T11:21:33.5068395Z W1204 11:16:07.415000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/compile_fx_ext.py", line 486, in serialize_compile 2025-12-04T11:21:33.5069210Z W1204 11:16:07.415000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] ).serialize() 2025-12-04T11:21:33.5069989Z W1204 11:16:07.415000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/compile_fx_ext.py", line 210, in serialize 2025-12-04T11:21:33.5070870Z W1204 11:16:07.415000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] return _WireProtocolPickledInput(GraphPickler.dumps(self)) 2025-12-04T11:21:33.5071731Z W1204 11:16:07.415000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 127, in dumps 2025-12-04T11:21:33.5072554Z W1204 11:16:07.415000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] pickler.dump(obj) 2025-12-04T11:21:33.5073345Z W1204 11:16:07.415000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 94, in reducer_override 2025-12-04T11:21:33.5074205Z W1204 11:16:07.415000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] return _GraphModulePickleData.reduce_helper(self, obj) 2025-12-04T11:21:33.5075088Z W1204 11:16:07.415000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 352, in reduce_helper 2025-12-04T11:21:33.5075908Z W1204 11:16:07.415000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] cls(obj, pickler.options), 2025-12-04T11:21:33.5076712Z W1204 11:16:07.415000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 364, in __init__ 2025-12-04T11:21:33.5077630Z W1204 11:16:07.415000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] self.graph = _GraphPickleData(gm._graph, options) 2025-12-04T11:21:33.5078736Z W1204 11:16:07.415000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 571, in __init__ 2025-12-04T11:21:33.5079580Z W1204 11:16:07.415000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] nodes[node] = _NodePickleData(node, nodes, options) 2025-12-04T11:21:33.5080501Z W1204 11:16:07.415000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 387, in __init__ 2025-12-04T11:21:33.5081381Z W1204 11:16:07.415000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] self.target = _OpPickleData.pickle(node.target, options) 2025-12-04T11:21:33.5082244Z W1204 11:16:07.415000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 435, in pickle 2025-12-04T11:21:33.5083121Z W1204 11:16:07.415000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] return cls._pickle_op(name, _OpOverloadPickleData, options) 2025-12-04T11:21:33.5083993Z W1204 11:16:07.415000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 456, in _pickle_op 2025-12-04T11:21:33.5084887Z W1204 11:16:07.415000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] raise BypassFxGraphCache(f"Unable to pickle non-standard op: {name}") 2025-12-04T11:21:33.5085851Z W1204 11:16:07.415000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] torch._inductor.codecache.BypassFxGraphCache: Unable to pickle non-standard op: torch.ops.prims.convert_element_type.default 2025-12-04T11:21:33.5086518Z PASSED [0.4927s] [ 18%] 2025-12-04T11:21:33.5087340Z inductor/test_compile_subprocess.py::GPUTests::test_complex_from_real_imag_cuda <- test/inductor/test_torchinductor.py [W1204 11:16:07.568859412 EmptyTensor.cpp:57] Warning: ComplexHalf support is experimental and many operators don't support it yet. (function operator()) 2025-12-04T11:21:33.5088173Z PASSED [0.2022s] [ 18%] 2025-12-04T11:21:33.5088637Z inductor/test_compile_subprocess.py::GPUTests::test_complex_memory_overlap_cuda <- test/inductor/test_torchinductor.py PASSED [0.0026s] [ 18%] 2025-12-04T11:21:33.5089437Z inductor/test_compile_subprocess.py::GPUTests::test_config_option_dont_assume_alignment_cuda <- test/inductor/test_torchinductor.py PASSED [0.5931s] [ 18%] 2025-12-04T11:21:33.5090524Z inductor/test_compile_subprocess.py::GPUTests::test_const_int32_to_float_cuda <- test/inductor/test_torchinductor.py W1204 11:16:08.370000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] Unable to pickle input graph or example inputs 2025-12-04T11:21:33.5091563Z W1204 11:16:08.370000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] Traceback (most recent call last): 2025-12-04T11:21:33.5092419Z W1204 11:16:08.370000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/compile_fx_ext.py", line 486, in serialize_compile 2025-12-04T11:21:33.5093225Z W1204 11:16:08.370000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] ).serialize() 2025-12-04T11:21:33.5094005Z W1204 11:16:08.370000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/compile_fx_ext.py", line 210, in serialize 2025-12-04T11:21:33.5094896Z W1204 11:16:08.370000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] return _WireProtocolPickledInput(GraphPickler.dumps(self)) 2025-12-04T11:21:33.5095869Z W1204 11:16:08.370000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 127, in dumps 2025-12-04T11:21:33.5096630Z W1204 11:16:08.370000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] pickler.dump(obj) 2025-12-04T11:21:33.5097435Z W1204 11:16:08.370000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 94, in reducer_override 2025-12-04T11:21:33.5098309Z W1204 11:16:08.370000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] return _GraphModulePickleData.reduce_helper(self, obj) 2025-12-04T11:21:33.5099211Z W1204 11:16:08.370000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 352, in reduce_helper 2025-12-04T11:21:33.5100036Z W1204 11:16:08.370000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] cls(obj, pickler.options), 2025-12-04T11:21:33.5100830Z W1204 11:16:08.370000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 364, in __init__ 2025-12-04T11:21:33.5101661Z W1204 11:16:08.370000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] self.graph = _GraphPickleData(gm._graph, options) 2025-12-04T11:21:33.5102486Z W1204 11:16:08.370000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 571, in __init__ 2025-12-04T11:21:33.5103311Z W1204 11:16:08.370000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] nodes[node] = _NodePickleData(node, nodes, options) 2025-12-04T11:21:33.5104225Z W1204 11:16:08.370000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 387, in __init__ 2025-12-04T11:21:33.5105079Z W1204 11:16:08.370000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] self.target = _OpPickleData.pickle(node.target, options) 2025-12-04T11:21:33.5105921Z W1204 11:16:08.370000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 435, in pickle 2025-12-04T11:21:33.5106777Z W1204 11:16:08.370000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] return cls._pickle_op(name, _OpOverloadPickleData, options) 2025-12-04T11:21:33.5107647Z W1204 11:16:08.370000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 456, in _pickle_op 2025-12-04T11:21:33.5108589Z W1204 11:16:08.370000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] raise BypassFxGraphCache(f"Unable to pickle non-standard op: {name}") 2025-12-04T11:21:33.5109554Z W1204 11:16:08.370000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] torch._inductor.codecache.BypassFxGraphCache: Unable to pickle non-standard op: torch.ops.prims.convert_element_type.default 2025-12-04T11:21:33.5110228Z PASSED [0.5647s] [ 19%] 2025-12-04T11:21:33.5110706Z inductor/test_compile_subprocess.py::GPUTests::test_constant_pad_1d_cuda <- test/inductor/test_torchinductor.py PASSED [0.7908s] [ 19%] 2025-12-04T11:21:33.5111501Z inductor/test_compile_subprocess.py::GPUTests::test_constant_pad_2d_strides_nonpositive_cuda <- test/inductor/test_torchinductor.py PASSED [0.3193s] [ 19%] 2025-12-04T11:21:33.5112303Z inductor/test_compile_subprocess.py::GPUTests::test_constant_pad_fill_dtype_cuda <- test/inductor/test_torchinductor.py PASSED [0.2438s] [ 19%] 2025-12-04T11:21:33.5113058Z inductor/test_compile_subprocess.py::GPUTests::test_constant_pad_float64_cuda <- test/inductor/test_torchinductor.py PASSED [0.2376s] [ 20%] 2025-12-04T11:21:33.5114196Z inductor/test_compile_subprocess.py::GPUTests::test_conv1d_with_permute_cuda <- test/inductor/test_torchinductor.py W1204 11:16:10.564000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] Unable to pickle input graph or example inputs 2025-12-04T11:21:33.5115163Z W1204 11:16:10.564000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] Traceback (most recent call last): 2025-12-04T11:21:33.5116022Z W1204 11:16:10.564000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/compile_fx_ext.py", line 486, in serialize_compile 2025-12-04T11:21:33.5116877Z W1204 11:16:10.564000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] ).serialize() 2025-12-04T11:21:33.5117660Z W1204 11:16:10.564000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/compile_fx_ext.py", line 210, in serialize 2025-12-04T11:21:33.5118559Z W1204 11:16:10.564000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] return _WireProtocolPickledInput(GraphPickler.dumps(self)) 2025-12-04T11:21:33.5119435Z W1204 11:16:10.564000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 127, in dumps 2025-12-04T11:21:33.5120201Z W1204 11:16:10.564000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] pickler.dump(obj) 2025-12-04T11:21:33.5120954Z W1204 11:16:10.564000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] AttributeError: Can't pickle local object 'CommonTemplate.test_conv1d_with_permute..ConvModel' 2025-12-04T11:21:33.5121540Z PASSED [0.3172s] [ 20%] 2025-12-04T11:21:33.5122167Z inductor/test_compile_subprocess.py::GPUTests::test_conv3d_channels_last_use_block_ptr_False_cuda <- test/inductor/test_torchinductor.py SKIPPED [0.0025s] (only support cpu conv3d channels_last) [ 20%] 2025-12-04T11:21:33.5123058Z inductor/test_compile_subprocess.py::GPUTests::test_conv3d_cuda <- test/inductor/test_torchinductor.py PASSED [0.6454s] [ 20%] 2025-12-04T11:21:33.5123759Z inductor/test_compile_subprocess.py::GPUTests::test_conv_backward_cuda <- test/inductor/test_torchinductor.py PASSED [0.2786s] [ 21%] 2025-12-04T11:21:33.5124621Z inductor/test_compile_subprocess.py::GPUTests::test_conv_bn_fuse_cuda <- test/inductor/test_torchinductor.py SKIPPED [0.0025s] (test is slow; run with PYTORCH_TEST_WITH_SLOW to enable test) [ 21%] 2025-12-04T11:21:33.5125516Z inductor/test_compile_subprocess.py::GPUTests::test_conv_inference_heuristics_cuda <- test/inductor/test_torchinductor.py PASSED [0.6855s] [ 21%] 2025-12-04T11:21:33.5126268Z inductor/test_compile_subprocess.py::GPUTests::test_conv_shape_check_cuda <- test/inductor/test_torchinductor.py PASSED [0.0588s] [ 21%] 2025-12-04T11:21:33.5127347Z inductor/test_compile_subprocess.py::GPUTests::test_conv_with_as_strided_cuda <- test/inductor/test_torchinductor.py W1204 11:16:12.619000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] Unable to pickle input graph or example inputs 2025-12-04T11:21:33.5128311Z W1204 11:16:12.619000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] Traceback (most recent call last): 2025-12-04T11:21:33.5129155Z W1204 11:16:12.619000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/compile_fx_ext.py", line 486, in serialize_compile 2025-12-04T11:21:33.5129961Z W1204 11:16:12.619000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] ).serialize() 2025-12-04T11:21:33.5130739Z W1204 11:16:12.619000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/compile_fx_ext.py", line 210, in serialize 2025-12-04T11:21:33.5131700Z W1204 11:16:12.619000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] return _WireProtocolPickledInput(GraphPickler.dumps(self)) 2025-12-04T11:21:33.5132566Z W1204 11:16:12.619000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 127, in dumps 2025-12-04T11:21:33.5133315Z W1204 11:16:12.619000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] pickler.dump(obj) 2025-12-04T11:21:33.5134065Z W1204 11:16:12.619000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] AttributeError: Can't pickle local object 'CommonTemplate.test_conv_with_as_strided..Model' 2025-12-04T11:21:33.5134907Z W1204 11:16:13.147000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] Unable to pickle input graph or example inputs 2025-12-04T11:21:33.5135541Z W1204 11:16:13.147000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] Traceback (most recent call last): 2025-12-04T11:21:33.5136380Z W1204 11:16:13.147000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/compile_fx_ext.py", line 486, in serialize_compile 2025-12-04T11:21:33.5137172Z W1204 11:16:13.147000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] ).serialize() 2025-12-04T11:21:33.5137937Z W1204 11:16:13.147000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/compile_fx_ext.py", line 210, in serialize 2025-12-04T11:21:33.5138812Z W1204 11:16:13.147000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] return _WireProtocolPickledInput(GraphPickler.dumps(self)) 2025-12-04T11:21:33.5139671Z W1204 11:16:13.147000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 127, in dumps 2025-12-04T11:21:33.5140425Z W1204 11:16:13.147000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] pickler.dump(obj) 2025-12-04T11:21:33.5141165Z W1204 11:16:13.147000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] AttributeError: Can't pickle local object 'CommonTemplate.test_conv_with_as_strided..Model' 2025-12-04T11:21:33.5141738Z PASSED [0.9813s] [ 21%] 2025-12-04T11:21:33.5142169Z inductor/test_compile_subprocess.py::GPUTests::test_convolution1_cuda <- test/inductor/test_torchinductor.py PASSED [0.7670s] [ 22%] 2025-12-04T11:21:33.5142878Z inductor/test_compile_subprocess.py::GPUTests::test_convolution3_cuda <- test/inductor/test_torchinductor.py PASSED [0.7134s] [ 22%] 2025-12-04T11:21:33.5143640Z inductor/test_compile_subprocess.py::GPUTests::test_convolution4_cuda <- test/inductor/test_torchinductor.py PASSED [0.9853s] [ 22%] 2025-12-04T11:21:33.5144414Z inductor/test_compile_subprocess.py::GPUTests::test_copy_with_scalar_src_cuda <- test/inductor/test_torchinductor.py PASSED [0.5556s] [ 22%] 2025-12-04T11:21:33.5145423Z inductor/test_compile_subprocess.py::GPUTests::test_cpu_scalar_with_gpu_tensor_dynamic_cuda <- test/inductor/test_torchinductor.py W1204 11:16:16.593000 89536 site-packages/torch/_inductor/utils.py:2565] [0/0] DeviceCopy in input program 2025-12-04T11:21:33.5146154Z PASSED [0.1980s] [ 23%] 2025-12-04T11:21:33.5146605Z inductor/test_compile_subprocess.py::GPUTests::test_cpu_tensor_with_gpu_tensor_cuda <- test/inductor/test_torchinductor.py PASSED [0.0126s] [ 23%] 2025-12-04T11:21:33.5147337Z inductor/test_compile_subprocess.py::GPUTests::test_cumsum_inf_cuda <- test/inductor/test_torchinductor.py PASSED [0.5260s] [ 23%] 2025-12-04T11:21:33.5148072Z inductor/test_compile_subprocess.py::GPUTests::test_cumsum_pattern_matcher_issue_cuda <- test/inductor/test_torchinductor.py PASSED [0.3263s] [ 23%] 2025-12-04T11:21:33.5149098Z inductor/test_compile_subprocess.py::GPUTests::test_custom_op_1_cuda <- test/inductor/test_torchinductor.py W1204 11:16:17.621000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] Unable to pickle input graph or example inputs 2025-12-04T11:21:33.5150096Z W1204 11:16:17.621000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] Traceback (most recent call last): 2025-12-04T11:21:33.5150960Z W1204 11:16:17.621000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/compile_fx_ext.py", line 486, in serialize_compile 2025-12-04T11:21:33.5151759Z W1204 11:16:17.621000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] ).serialize() 2025-12-04T11:21:33.5152587Z W1204 11:16:17.621000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/compile_fx_ext.py", line 210, in serialize 2025-12-04T11:21:33.5153470Z W1204 11:16:17.621000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] return _WireProtocolPickledInput(GraphPickler.dumps(self)) 2025-12-04T11:21:33.5154328Z W1204 11:16:17.621000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 127, in dumps 2025-12-04T11:21:33.5155092Z W1204 11:16:17.621000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] pickler.dump(obj) 2025-12-04T11:21:33.5155877Z W1204 11:16:17.621000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 94, in reducer_override 2025-12-04T11:21:33.5156736Z W1204 11:16:17.621000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] return _GraphModulePickleData.reduce_helper(self, obj) 2025-12-04T11:21:33.5157596Z W1204 11:16:17.621000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 352, in reduce_helper 2025-12-04T11:21:33.5158403Z W1204 11:16:17.621000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] cls(obj, pickler.options), 2025-12-04T11:21:33.5159195Z W1204 11:16:17.621000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 364, in __init__ 2025-12-04T11:21:33.5160026Z W1204 11:16:17.621000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] self.graph = _GraphPickleData(gm._graph, options) 2025-12-04T11:21:33.5160852Z W1204 11:16:17.621000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 571, in __init__ 2025-12-04T11:21:33.5161684Z W1204 11:16:17.621000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] nodes[node] = _NodePickleData(node, nodes, options) 2025-12-04T11:21:33.5162565Z W1204 11:16:17.621000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 387, in __init__ 2025-12-04T11:21:33.5163416Z W1204 11:16:17.621000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] self.target = _OpPickleData.pickle(node.target, options) 2025-12-04T11:21:33.5164263Z W1204 11:16:17.621000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 435, in pickle 2025-12-04T11:21:33.5165108Z W1204 11:16:17.621000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] return cls._pickle_op(name, _OpOverloadPickleData, options) 2025-12-04T11:21:33.5165979Z W1204 11:16:17.621000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 456, in _pickle_op 2025-12-04T11:21:33.5166966Z W1204 11:16:17.621000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] raise BypassFxGraphCache(f"Unable to pickle non-standard op: {name}") 2025-12-04T11:21:33.5167876Z W1204 11:16:17.621000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] torch._inductor.codecache.BypassFxGraphCache: Unable to pickle non-standard op: torch.ops.test.foo.default 2025-12-04T11:21:33.5168487Z PASSED [0.4481s] [ 24%] 2025-12-04T11:21:33.5169185Z inductor/test_compile_subprocess.py::GPUTests::test_custom_op_2_cuda <- test/inductor/test_torchinductor.py W1204 11:16:18.069000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] Unable to pickle input graph or example inputs 2025-12-04T11:21:33.5170154Z W1204 11:16:18.069000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] Traceback (most recent call last): 2025-12-04T11:21:33.5171006Z W1204 11:16:18.069000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/compile_fx_ext.py", line 486, in serialize_compile 2025-12-04T11:21:33.5171814Z W1204 11:16:18.069000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] ).serialize() 2025-12-04T11:21:33.5172582Z W1204 11:16:18.069000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/compile_fx_ext.py", line 210, in serialize 2025-12-04T11:21:33.5173472Z W1204 11:16:18.069000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] return _WireProtocolPickledInput(GraphPickler.dumps(self)) 2025-12-04T11:21:33.5174323Z W1204 11:16:18.069000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 127, in dumps 2025-12-04T11:21:33.5175071Z W1204 11:16:18.069000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] pickler.dump(obj) 2025-12-04T11:21:33.5175858Z W1204 11:16:18.069000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 94, in reducer_override 2025-12-04T11:21:33.5176716Z W1204 11:16:18.069000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] return _GraphModulePickleData.reduce_helper(self, obj) 2025-12-04T11:21:33.5177573Z W1204 11:16:18.069000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 352, in reduce_helper 2025-12-04T11:21:33.5178540Z W1204 11:16:18.069000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] cls(obj, pickler.options), 2025-12-04T11:21:33.5179327Z W1204 11:16:18.069000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 364, in __init__ 2025-12-04T11:21:33.5180252Z W1204 11:16:18.069000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] self.graph = _GraphPickleData(gm._graph, options) 2025-12-04T11:21:33.5181083Z W1204 11:16:18.069000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 571, in __init__ 2025-12-04T11:21:33.5181909Z W1204 11:16:18.069000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] nodes[node] = _NodePickleData(node, nodes, options) 2025-12-04T11:21:33.5182738Z W1204 11:16:18.069000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 387, in __init__ 2025-12-04T11:21:33.5183639Z W1204 11:16:18.069000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] self.target = _OpPickleData.pickle(node.target, options) 2025-12-04T11:21:33.5184482Z W1204 11:16:18.069000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 435, in pickle 2025-12-04T11:21:33.5185453Z W1204 11:16:18.069000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] return cls._pickle_op(name, _OpOverloadPickleData, options) 2025-12-04T11:21:33.5186323Z W1204 11:16:18.069000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 456, in _pickle_op 2025-12-04T11:21:33.5187210Z W1204 11:16:18.069000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] raise BypassFxGraphCache(f"Unable to pickle non-standard op: {name}") 2025-12-04T11:21:33.5188174Z W1204 11:16:18.069000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] torch._inductor.codecache.BypassFxGraphCache: Unable to pickle non-standard op: torch.ops.test.foo2.default 2025-12-04T11:21:33.5188801Z PASSED [0.2714s] [ 24%] 2025-12-04T11:21:33.5189576Z inductor/test_compile_subprocess.py::GPUTests::test_custom_op_default_layout_constraint_cuda <- test/inductor/test_torchinductor.py W1204 11:16:18.348000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] Unable to pickle input graph or example inputs 2025-12-04T11:21:33.5190580Z W1204 11:16:18.348000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] Traceback (most recent call last): 2025-12-04T11:21:33.5191433Z W1204 11:16:18.348000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/compile_fx_ext.py", line 486, in serialize_compile 2025-12-04T11:21:33.5192239Z W1204 11:16:18.348000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] ).serialize() 2025-12-04T11:21:33.5193018Z W1204 11:16:18.348000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/compile_fx_ext.py", line 210, in serialize 2025-12-04T11:21:33.5193907Z W1204 11:16:18.348000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] return _WireProtocolPickledInput(GraphPickler.dumps(self)) 2025-12-04T11:21:33.5194767Z W1204 11:16:18.348000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 127, in dumps 2025-12-04T11:21:33.5195524Z W1204 11:16:18.348000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] pickler.dump(obj) 2025-12-04T11:21:33.5196303Z W1204 11:16:18.348000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 96, in reducer_override 2025-12-04T11:21:33.5197149Z W1204 11:16:18.348000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] return _OpPickleData.reduce_helper(self, obj) 2025-12-04T11:21:33.5198052Z W1204 11:16:18.348000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 418, in reduce_helper 2025-12-04T11:21:33.5198881Z W1204 11:16:18.348000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] result = cls.pickle(op, pickler.options) 2025-12-04T11:21:33.5199685Z W1204 11:16:18.348000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 435, in pickle 2025-12-04T11:21:33.5200536Z W1204 11:16:18.348000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] return cls._pickle_op(name, _OpOverloadPickleData, options) 2025-12-04T11:21:33.5201402Z W1204 11:16:18.348000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 456, in _pickle_op 2025-12-04T11:21:33.5202350Z W1204 11:16:18.348000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] raise BypassFxGraphCache(f"Unable to pickle non-standard op: {name}") 2025-12-04T11:21:33.5203305Z W1204 11:16:18.348000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] torch._inductor.codecache.BypassFxGraphCache: Unable to pickle non-standard op: torch.ops.mylib.copy_.default 2025-12-04T11:21:33.5203927Z PASSED [0.1668s] [ 24%] 2025-12-04T11:21:33.5204709Z inductor/test_compile_subprocess.py::GPUTests::test_custom_op_fixed_layout_channels_last_cuda <- test/inductor/test_torchinductor.py W1204 11:16:18.547000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] Unable to pickle input graph or example inputs 2025-12-04T11:21:33.5205761Z W1204 11:16:18.547000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] Traceback (most recent call last): 2025-12-04T11:21:33.5206622Z W1204 11:16:18.547000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/compile_fx_ext.py", line 486, in serialize_compile 2025-12-04T11:21:33.5207427Z W1204 11:16:18.547000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] ).serialize() 2025-12-04T11:21:33.5208198Z W1204 11:16:18.547000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/compile_fx_ext.py", line 210, in serialize 2025-12-04T11:21:33.5209091Z W1204 11:16:18.547000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] return _WireProtocolPickledInput(GraphPickler.dumps(self)) 2025-12-04T11:21:33.5209949Z W1204 11:16:18.547000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 127, in dumps 2025-12-04T11:21:33.5210699Z W1204 11:16:18.547000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] pickler.dump(obj) 2025-12-04T11:21:33.5211483Z W1204 11:16:18.547000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 94, in reducer_override 2025-12-04T11:21:33.5212363Z W1204 11:16:18.547000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] return _GraphModulePickleData.reduce_helper(self, obj) 2025-12-04T11:21:33.5213224Z W1204 11:16:18.547000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 352, in reduce_helper 2025-12-04T11:21:33.5214024Z W1204 11:16:18.547000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] cls(obj, pickler.options), 2025-12-04T11:21:33.5214813Z W1204 11:16:18.547000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 364, in __init__ 2025-12-04T11:21:33.5215701Z W1204 11:16:18.547000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] self.graph = _GraphPickleData(gm._graph, options) 2025-12-04T11:21:33.5216534Z W1204 11:16:18.547000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 571, in __init__ 2025-12-04T11:21:33.5217363Z W1204 11:16:18.547000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] nodes[node] = _NodePickleData(node, nodes, options) 2025-12-04T11:21:33.5218196Z W1204 11:16:18.547000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 387, in __init__ 2025-12-04T11:21:33.5219042Z W1204 11:16:18.547000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] self.target = _OpPickleData.pickle(node.target, options) 2025-12-04T11:21:33.5219925Z W1204 11:16:18.547000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 435, in pickle 2025-12-04T11:21:33.5220841Z W1204 11:16:18.547000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] return cls._pickle_op(name, _OpOverloadPickleData, options) 2025-12-04T11:21:33.5221705Z W1204 11:16:18.547000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 456, in _pickle_op 2025-12-04T11:21:33.5222599Z W1204 11:16:18.547000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] raise BypassFxGraphCache(f"Unable to pickle non-standard op: {name}") 2025-12-04T11:21:33.5223621Z W1204 11:16:18.547000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] torch._inductor.codecache.BypassFxGraphCache: Unable to pickle non-standard op: torch.ops.prims.inductor_seeds.default 2025-12-04T11:21:33.5224285Z PASSED [0.6318s] [ 24%] 2025-12-04T11:21:33.5225054Z inductor/test_compile_subprocess.py::GPUTests::test_custom_op_fixed_layout_sequential_cuda <- test/inductor/test_torchinductor.py W1204 11:16:19.152000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] Unable to pickle input graph or example inputs 2025-12-04T11:21:33.5226048Z W1204 11:16:19.152000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] Traceback (most recent call last): 2025-12-04T11:21:33.5226901Z W1204 11:16:19.152000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/compile_fx_ext.py", line 486, in serialize_compile 2025-12-04T11:21:33.5227698Z W1204 11:16:19.152000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] ).serialize() 2025-12-04T11:21:33.5228482Z W1204 11:16:19.152000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/compile_fx_ext.py", line 210, in serialize 2025-12-04T11:21:33.5229366Z W1204 11:16:19.152000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] return _WireProtocolPickledInput(GraphPickler.dumps(self)) 2025-12-04T11:21:33.5230224Z W1204 11:16:19.152000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 127, in dumps 2025-12-04T11:21:33.5230983Z W1204 11:16:19.152000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] pickler.dump(obj) 2025-12-04T11:21:33.5232083Z W1204 11:16:19.152000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 94, in reducer_override 2025-12-04T11:21:33.5233026Z W1204 11:16:19.152000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] return _GraphModulePickleData.reduce_helper(self, obj) 2025-12-04T11:21:33.5233966Z W1204 11:16:19.152000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 352, in reduce_helper 2025-12-04T11:21:33.5234813Z W1204 11:16:19.152000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] cls(obj, pickler.options), 2025-12-04T11:21:33.5235620Z W1204 11:16:19.152000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 364, in __init__ 2025-12-04T11:21:33.5236462Z W1204 11:16:19.152000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] self.graph = _GraphPickleData(gm._graph, options) 2025-12-04T11:21:33.5237305Z W1204 11:16:19.152000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 571, in __init__ 2025-12-04T11:21:33.5238203Z W1204 11:16:19.152000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] nodes[node] = _NodePickleData(node, nodes, options) 2025-12-04T11:21:33.5239074Z W1204 11:16:19.152000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 387, in __init__ 2025-12-04T11:21:33.5239925Z W1204 11:16:19.152000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] self.target = _OpPickleData.pickle(node.target, options) 2025-12-04T11:21:33.5240782Z W1204 11:16:19.152000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 435, in pickle 2025-12-04T11:21:33.5241719Z W1204 11:16:19.152000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] return cls._pickle_op(name, _OpOverloadPickleData, options) 2025-12-04T11:21:33.5242595Z W1204 11:16:19.152000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 456, in _pickle_op 2025-12-04T11:21:33.5243912Z W1204 11:16:19.152000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] raise BypassFxGraphCache(f"Unable to pickle non-standard op: {name}") 2025-12-04T11:21:33.5244842Z W1204 11:16:19.152000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] torch._inductor.codecache.BypassFxGraphCache: Unable to pickle non-standard op: torch.ops.test.bar.default 2025-12-04T11:21:33.5245694Z W1204 11:16:19.381000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] Unable to pickle input graph or example inputs 2025-12-04T11:21:33.5246355Z W1204 11:16:19.381000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] Traceback (most recent call last): 2025-12-04T11:21:33.5247223Z W1204 11:16:19.381000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/compile_fx_ext.py", line 486, in serialize_compile 2025-12-04T11:21:33.5248019Z W1204 11:16:19.381000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] ).serialize() 2025-12-04T11:21:33.5248793Z W1204 11:16:19.381000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/compile_fx_ext.py", line 210, in serialize 2025-12-04T11:21:33.5249688Z W1204 11:16:19.381000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] return _WireProtocolPickledInput(GraphPickler.dumps(self)) 2025-12-04T11:21:33.5250546Z W1204 11:16:19.381000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 127, in dumps 2025-12-04T11:21:33.5251303Z W1204 11:16:19.381000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] pickler.dump(obj) 2025-12-04T11:21:33.5252150Z W1204 11:16:19.381000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 94, in reducer_override 2025-12-04T11:21:33.5253211Z W1204 11:16:19.381000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] return _GraphModulePickleData.reduce_helper(self, obj) 2025-12-04T11:21:33.5254071Z W1204 11:16:19.381000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 352, in reduce_helper 2025-12-04T11:21:33.5254872Z W1204 11:16:19.381000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] cls(obj, pickler.options), 2025-12-04T11:21:33.5255652Z W1204 11:16:19.381000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 364, in __init__ 2025-12-04T11:21:33.5256551Z W1204 11:16:19.381000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] self.graph = _GraphPickleData(gm._graph, options) 2025-12-04T11:21:33.5257420Z W1204 11:16:19.381000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 571, in __init__ 2025-12-04T11:21:33.5258247Z W1204 11:16:19.381000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] nodes[node] = _NodePickleData(node, nodes, options) 2025-12-04T11:21:33.5259068Z W1204 11:16:19.381000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 387, in __init__ 2025-12-04T11:21:33.5259959Z W1204 11:16:19.381000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] self.target = _OpPickleData.pickle(node.target, options) 2025-12-04T11:21:33.5260821Z W1204 11:16:19.381000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 435, in pickle 2025-12-04T11:21:33.5261673Z W1204 11:16:19.381000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] return cls._pickle_op(name, _OpOverloadPickleData, options) 2025-12-04T11:21:33.5262539Z W1204 11:16:19.381000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 456, in _pickle_op 2025-12-04T11:21:33.5263427Z W1204 11:16:19.381000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] raise BypassFxGraphCache(f"Unable to pickle non-standard op: {name}") 2025-12-04T11:21:33.5264382Z W1204 11:16:19.381000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] torch._inductor.codecache.BypassFxGraphCache: Unable to pickle non-standard op: torch.ops.test.bar.default 2025-12-04T11:21:33.5264996Z PASSED [0.3722s] [ 24%] 2025-12-04T11:21:33.5265768Z inductor/test_compile_subprocess.py::GPUTests::test_custom_op_unbacked_symints_cuda <- test/inductor/test_torchinductor.py W1204 11:16:19.625000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] Unable to pickle input graph or example inputs 2025-12-04T11:21:33.5266741Z W1204 11:16:19.625000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] Traceback (most recent call last): 2025-12-04T11:21:33.5267577Z W1204 11:16:19.625000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/compile_fx_ext.py", line 486, in serialize_compile 2025-12-04T11:21:33.5268377Z W1204 11:16:19.625000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] ).serialize() 2025-12-04T11:21:33.5269156Z W1204 11:16:19.625000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/compile_fx_ext.py", line 210, in serialize 2025-12-04T11:21:33.5270095Z W1204 11:16:19.625000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] return _WireProtocolPickledInput(GraphPickler.dumps(self)) 2025-12-04T11:21:33.5270947Z W1204 11:16:19.625000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 127, in dumps 2025-12-04T11:21:33.5271695Z W1204 11:16:19.625000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] pickler.dump(obj) 2025-12-04T11:21:33.5272473Z W1204 11:16:19.625000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 94, in reducer_override 2025-12-04T11:21:33.5273337Z W1204 11:16:19.625000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] return _GraphModulePickleData.reduce_helper(self, obj) 2025-12-04T11:21:33.5274289Z W1204 11:16:19.625000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 352, in reduce_helper 2025-12-04T11:21:33.5275087Z W1204 11:16:19.625000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] cls(obj, pickler.options), 2025-12-04T11:21:33.5275863Z W1204 11:16:19.625000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 364, in __init__ 2025-12-04T11:21:33.5276685Z W1204 11:16:19.625000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] self.graph = _GraphPickleData(gm._graph, options) 2025-12-04T11:21:33.5277553Z W1204 11:16:19.625000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 571, in __init__ 2025-12-04T11:21:33.5278565Z W1204 11:16:19.625000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] nodes[node] = _NodePickleData(node, nodes, options) 2025-12-04T11:21:33.5279398Z W1204 11:16:19.625000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 387, in __init__ 2025-12-04T11:21:33.5280248Z W1204 11:16:19.625000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] self.target = _OpPickleData.pickle(node.target, options) 2025-12-04T11:21:33.5281095Z W1204 11:16:19.625000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 435, in pickle 2025-12-04T11:21:33.5281943Z W1204 11:16:19.625000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] return cls._pickle_op(name, _OpOverloadPickleData, options) 2025-12-04T11:21:33.5282803Z W1204 11:16:19.625000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 456, in _pickle_op 2025-12-04T11:21:33.5283688Z W1204 11:16:19.625000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] raise BypassFxGraphCache(f"Unable to pickle non-standard op: {name}") 2025-12-04T11:21:33.5284630Z W1204 11:16:19.625000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] torch._inductor.codecache.BypassFxGraphCache: Unable to pickle non-standard op: torch.ops.test_unbacked_symints.foo.default 2025-12-04T11:21:33.5285283Z PASSED [0.2585s] [ 25%] 2025-12-04T11:21:33.5285715Z inductor/test_compile_subprocess.py::GPUTests::test_custom_scan_op_cuda <- test/inductor/test_torchinductor.py PASSED [0.1746s] [ 25%] 2025-12-04T11:21:33.5286458Z inductor/test_compile_subprocess.py::GPUTests::test_custom_scan_op_multi_input_cuda <- test/inductor/test_torchinductor.py PASSED [0.1172s] [ 25%] 2025-12-04T11:21:33.5287372Z inductor/test_compile_subprocess.py::GPUTests::test_data_type_propogation_cuda <- test/inductor/test_torchinductor.py SKIPPED [0.0024s] (triton not supported) [ 25%] 2025-12-04T11:21:33.5288182Z inductor/test_compile_subprocess.py::GPUTests::test_dense_mask_index_cuda <- test/inductor/test_torchinductor.py PASSED [0.4506s] [ 26%] 2025-12-04T11:21:33.5288945Z inductor/test_compile_subprocess.py::GPUTests::test_deterministic_codegen_with_suffix_cuda <- test/inductor/test_torchinductor.py PASSED [0.6673s] [ 26%] 2025-12-04T11:21:33.5289711Z inductor/test_compile_subprocess.py::GPUTests::test_diagonal_copy_cuda <- test/inductor/test_torchinductor.py PASSED [0.8168s] [ 26%] 2025-12-04T11:21:33.5290386Z inductor/test_compile_subprocess.py::GPUTests::test_div2_cuda <- test/inductor/test_torchinductor.py PASSED [0.5435s] [ 26%] 2025-12-04T11:21:33.5291041Z inductor/test_compile_subprocess.py::GPUTests::test_div3_cuda <- test/inductor/test_torchinductor.py PASSED [0.2927s] [ 27%] 2025-12-04T11:21:33.5291697Z inductor/test_compile_subprocess.py::GPUTests::test_div4_cuda <- test/inductor/test_torchinductor.py PASSED [0.2786s] [ 27%] 2025-12-04T11:21:33.5292460Z inductor/test_compile_subprocess.py::GPUTests::test_div8_cuda <- test/inductor/test_torchinductor.py PASSED [0.6037s] [ 27%] 2025-12-04T11:21:33.5293106Z inductor/test_compile_subprocess.py::GPUTests::test_div9_cuda <- test/inductor/test_torchinductor.py PASSED [0.3365s] [ 27%] 2025-12-04T11:21:33.5293771Z inductor/test_compile_subprocess.py::GPUTests::test_div_by_zero_cuda <- test/inductor/test_torchinductor.py PASSED [0.4928s] [ 27%] 2025-12-04T11:21:33.5294751Z inductor/test_compile_subprocess.py::GPUTests::test_div_precision_cuda <- test/inductor/test_torchinductor.py W1204 11:16:24.844000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] Unable to pickle input graph or example inputs 2025-12-04T11:21:33.5295744Z W1204 11:16:24.844000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] Traceback (most recent call last): 2025-12-04T11:21:33.5296613Z W1204 11:16:24.844000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/compile_fx_ext.py", line 486, in serialize_compile 2025-12-04T11:21:33.5297422Z W1204 11:16:24.844000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] ).serialize() 2025-12-04T11:21:33.5298196Z W1204 11:16:24.844000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/compile_fx_ext.py", line 210, in serialize 2025-12-04T11:21:33.5299077Z W1204 11:16:24.844000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] return _WireProtocolPickledInput(GraphPickler.dumps(self)) 2025-12-04T11:21:33.5299927Z W1204 11:16:24.844000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 127, in dumps 2025-12-04T11:21:33.5300681Z W1204 11:16:24.844000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] pickler.dump(obj) 2025-12-04T11:21:33.5301459Z W1204 11:16:24.844000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 94, in reducer_override 2025-12-04T11:21:33.5302327Z W1204 11:16:24.844000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] return _GraphModulePickleData.reduce_helper(self, obj) 2025-12-04T11:21:33.5303185Z W1204 11:16:24.844000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 352, in reduce_helper 2025-12-04T11:21:33.5304068Z W1204 11:16:24.844000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] cls(obj, pickler.options), 2025-12-04T11:21:33.5304849Z W1204 11:16:24.844000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 364, in __init__ 2025-12-04T11:21:33.5305722Z W1204 11:16:24.844000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] self.graph = _GraphPickleData(gm._graph, options) 2025-12-04T11:21:33.5306546Z W1204 11:16:24.844000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 571, in __init__ 2025-12-04T11:21:33.5307372Z W1204 11:16:24.844000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] nodes[node] = _NodePickleData(node, nodes, options) 2025-12-04T11:21:33.5308193Z W1204 11:16:24.844000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 387, in __init__ 2025-12-04T11:21:33.5309341Z W1204 11:16:24.844000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] self.target = _OpPickleData.pickle(node.target, options) 2025-12-04T11:21:33.5310286Z W1204 11:16:24.844000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 435, in pickle 2025-12-04T11:21:33.5311134Z W1204 11:16:24.844000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] return cls._pickle_op(name, _OpOverloadPickleData, options) 2025-12-04T11:21:33.5311996Z W1204 11:16:24.844000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 456, in _pickle_op 2025-12-04T11:21:33.5312925Z W1204 11:16:24.844000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] raise BypassFxGraphCache(f"Unable to pickle non-standard op: {name}") 2025-12-04T11:21:33.5313888Z W1204 11:16:24.844000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] torch._inductor.codecache.BypassFxGraphCache: Unable to pickle non-standard op: torch.ops.prims.convert_element_type.default 2025-12-04T11:21:33.5314792Z W1204 11:16:25.462000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] Unable to pickle input graph or example inputs 2025-12-04T11:21:33.5315421Z W1204 11:16:25.462000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] Traceback (most recent call last): 2025-12-04T11:21:33.5316263Z W1204 11:16:25.462000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/compile_fx_ext.py", line 486, in serialize_compile 2025-12-04T11:21:33.5317052Z W1204 11:16:25.462000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] ).serialize() 2025-12-04T11:21:33.5317822Z W1204 11:16:25.462000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/compile_fx_ext.py", line 210, in serialize 2025-12-04T11:21:33.5318699Z W1204 11:16:25.462000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] return _WireProtocolPickledInput(GraphPickler.dumps(self)) 2025-12-04T11:21:33.5319554Z W1204 11:16:25.462000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 127, in dumps 2025-12-04T11:21:33.5320309Z W1204 11:16:25.462000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] pickler.dump(obj) 2025-12-04T11:21:33.5321085Z W1204 11:16:25.462000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 94, in reducer_override 2025-12-04T11:21:33.5321956Z W1204 11:16:25.462000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] return _GraphModulePickleData.reduce_helper(self, obj) 2025-12-04T11:21:33.5322867Z W1204 11:16:25.462000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 352, in reduce_helper 2025-12-04T11:21:33.5323671Z W1204 11:16:25.462000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] cls(obj, pickler.options), 2025-12-04T11:21:33.5324450Z W1204 11:16:25.462000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 364, in __init__ 2025-12-04T11:21:33.5325277Z W1204 11:16:25.462000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] self.graph = _GraphPickleData(gm._graph, options) 2025-12-04T11:21:33.5326111Z W1204 11:16:25.462000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 571, in __init__ 2025-12-04T11:21:33.5326944Z W1204 11:16:25.462000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] nodes[node] = _NodePickleData(node, nodes, options) 2025-12-04T11:21:33.5327850Z W1204 11:16:25.462000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 387, in __init__ 2025-12-04T11:21:33.5328696Z W1204 11:16:25.462000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] self.target = _OpPickleData.pickle(node.target, options) 2025-12-04T11:21:33.5329537Z W1204 11:16:25.462000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 435, in pickle 2025-12-04T11:21:33.5330435Z W1204 11:16:25.462000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] return cls._pickle_op(name, _OpOverloadPickleData, options) 2025-12-04T11:21:33.5331307Z W1204 11:16:25.462000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 456, in _pickle_op 2025-12-04T11:21:33.5332207Z W1204 11:16:25.462000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] raise BypassFxGraphCache(f"Unable to pickle non-standard op: {name}") 2025-12-04T11:21:33.5333159Z W1204 11:16:25.462000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] torch._inductor.codecache.BypassFxGraphCache: Unable to pickle non-standard op: torch.ops.prims.convert_element_type.default 2025-12-04T11:21:33.5333828Z PASSED [1.2344s] [ 28%] 2025-12-04T11:21:33.5334289Z inductor/test_compile_subprocess.py::GPUTests::test_div_presicion_accuracy_cuda <- test/inductor/test_torchinductor.py PASSED [0.3301s] [ 28%] 2025-12-04T11:21:33.5335299Z inductor/test_compile_subprocess.py::GPUTests::test_div_prim_cuda <- test/inductor/test_torchinductor.py W1204 11:16:26.144000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] Unable to pickle input graph or example inputs 2025-12-04T11:21:33.5336235Z W1204 11:16:26.144000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] Traceback (most recent call last): 2025-12-04T11:21:33.5337085Z W1204 11:16:26.144000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/compile_fx_ext.py", line 486, in serialize_compile 2025-12-04T11:21:33.5337883Z W1204 11:16:26.144000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] ).serialize() 2025-12-04T11:21:33.5338662Z W1204 11:16:26.144000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/compile_fx_ext.py", line 210, in serialize 2025-12-04T11:21:33.5339545Z W1204 11:16:26.144000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] return _WireProtocolPickledInput(GraphPickler.dumps(self)) 2025-12-04T11:21:33.5340443Z W1204 11:16:26.144000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 127, in dumps 2025-12-04T11:21:33.5341205Z W1204 11:16:26.144000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] pickler.dump(obj) 2025-12-04T11:21:33.5341991Z W1204 11:16:26.144000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 94, in reducer_override 2025-12-04T11:21:33.5342858Z W1204 11:16:26.144000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] return _GraphModulePickleData.reduce_helper(self, obj) 2025-12-04T11:21:33.5343759Z W1204 11:16:26.144000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 352, in reduce_helper 2025-12-04T11:21:33.5344572Z W1204 11:16:26.144000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] cls(obj, pickler.options), 2025-12-04T11:21:33.5345531Z W1204 11:16:26.144000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 364, in __init__ 2025-12-04T11:21:33.5346369Z W1204 11:16:26.144000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] self.graph = _GraphPickleData(gm._graph, options) 2025-12-04T11:21:33.5347200Z W1204 11:16:26.144000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 571, in __init__ 2025-12-04T11:21:33.5348016Z W1204 11:16:26.144000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] nodes[node] = _NodePickleData(node, nodes, options) 2025-12-04T11:21:33.5348886Z W1204 11:16:26.144000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 387, in __init__ 2025-12-04T11:21:33.5349742Z W1204 11:16:26.144000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] self.target = _OpPickleData.pickle(node.target, options) 2025-12-04T11:21:33.5350589Z W1204 11:16:26.144000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 435, in pickle 2025-12-04T11:21:33.5351434Z W1204 11:16:26.144000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] return cls._pickle_op(name, _OpOverloadPickleData, options) 2025-12-04T11:21:33.5352310Z W1204 11:16:26.144000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 456, in _pickle_op 2025-12-04T11:21:33.5353207Z W1204 11:16:26.144000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] raise BypassFxGraphCache(f"Unable to pickle non-standard op: {name}") 2025-12-04T11:21:33.5354121Z W1204 11:16:26.144000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] torch._inductor.codecache.BypassFxGraphCache: Unable to pickle non-standard op: torch.ops.prims.div.default 2025-12-04T11:21:33.5354972Z W1204 11:16:26.281000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] Unable to pickle input graph or example inputs 2025-12-04T11:21:33.5355597Z W1204 11:16:26.281000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] Traceback (most recent call last): 2025-12-04T11:21:33.5356464Z W1204 11:16:26.281000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/compile_fx_ext.py", line 486, in serialize_compile 2025-12-04T11:21:33.5357273Z W1204 11:16:26.281000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] ).serialize() 2025-12-04T11:21:33.5358105Z W1204 11:16:26.281000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/compile_fx_ext.py", line 210, in serialize 2025-12-04T11:21:33.5358999Z W1204 11:16:26.281000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] return _WireProtocolPickledInput(GraphPickler.dumps(self)) 2025-12-04T11:21:33.5359860Z W1204 11:16:26.281000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 127, in dumps 2025-12-04T11:21:33.5360623Z W1204 11:16:26.281000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] pickler.dump(obj) 2025-12-04T11:21:33.5361414Z W1204 11:16:26.281000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 94, in reducer_override 2025-12-04T11:21:33.5362281Z W1204 11:16:26.281000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] return _GraphModulePickleData.reduce_helper(self, obj) 2025-12-04T11:21:33.5363218Z W1204 11:16:26.281000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 352, in reduce_helper 2025-12-04T11:21:33.5364021Z W1204 11:16:26.281000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] cls(obj, pickler.options), 2025-12-04T11:21:33.5364804Z W1204 11:16:26.281000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 364, in __init__ 2025-12-04T11:21:33.5365677Z W1204 11:16:26.281000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] self.graph = _GraphPickleData(gm._graph, options) 2025-12-04T11:21:33.5366524Z W1204 11:16:26.281000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 571, in __init__ 2025-12-04T11:21:33.5367353Z W1204 11:16:26.281000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] nodes[node] = _NodePickleData(node, nodes, options) 2025-12-04T11:21:33.5368188Z W1204 11:16:26.281000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 387, in __init__ 2025-12-04T11:21:33.5369036Z W1204 11:16:26.281000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] self.target = _OpPickleData.pickle(node.target, options) 2025-12-04T11:21:33.5369883Z W1204 11:16:26.281000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 435, in pickle 2025-12-04T11:21:33.5370744Z W1204 11:16:26.281000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] return cls._pickle_op(name, _OpOverloadPickleData, options) 2025-12-04T11:21:33.5371606Z W1204 11:16:26.281000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 456, in _pickle_op 2025-12-04T11:21:33.5372497Z W1204 11:16:26.281000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] raise BypassFxGraphCache(f"Unable to pickle non-standard op: {name}") 2025-12-04T11:21:33.5373409Z W1204 11:16:26.281000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] torch._inductor.codecache.BypassFxGraphCache: Unable to pickle non-standard op: torch.ops.prims.div.default 2025-12-04T11:21:33.5374253Z W1204 11:16:26.416000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] Unable to pickle input graph or example inputs 2025-12-04T11:21:33.5374893Z W1204 11:16:26.416000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] Traceback (most recent call last): 2025-12-04T11:21:33.5375816Z W1204 11:16:26.416000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/compile_fx_ext.py", line 486, in serialize_compile 2025-12-04T11:21:33.5376618Z W1204 11:16:26.416000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] ).serialize() 2025-12-04T11:21:33.5377392Z W1204 11:16:26.416000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/compile_fx_ext.py", line 210, in serialize 2025-12-04T11:21:33.5378487Z W1204 11:16:26.416000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] return _WireProtocolPickledInput(GraphPickler.dumps(self)) 2025-12-04T11:21:33.5379353Z W1204 11:16:26.416000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 127, in dumps 2025-12-04T11:21:33.5380116Z W1204 11:16:26.416000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] pickler.dump(obj) 2025-12-04T11:21:33.5381037Z W1204 11:16:26.416000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 94, in reducer_override 2025-12-04T11:21:33.5381908Z W1204 11:16:26.416000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] return _GraphModulePickleData.reduce_helper(self, obj) 2025-12-04T11:21:33.5382771Z W1204 11:16:26.416000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 352, in reduce_helper 2025-12-04T11:21:33.5383670Z W1204 11:16:26.416000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] cls(obj, pickler.options), 2025-12-04T11:21:33.5384477Z W1204 11:16:26.416000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 364, in __init__ 2025-12-04T11:21:33.5385308Z W1204 11:16:26.416000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] self.graph = _GraphPickleData(gm._graph, options) 2025-12-04T11:21:33.5386141Z W1204 11:16:26.416000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 571, in __init__ 2025-12-04T11:21:33.5386968Z W1204 11:16:26.416000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] nodes[node] = _NodePickleData(node, nodes, options) 2025-12-04T11:21:33.5387802Z W1204 11:16:26.416000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 387, in __init__ 2025-12-04T11:21:33.5388660Z W1204 11:16:26.416000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] self.target = _OpPickleData.pickle(node.target, options) 2025-12-04T11:21:33.5389515Z W1204 11:16:26.416000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 435, in pickle 2025-12-04T11:21:33.5390370Z W1204 11:16:26.416000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] return cls._pickle_op(name, _OpOverloadPickleData, options) 2025-12-04T11:21:33.5391229Z W1204 11:16:26.416000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 456, in _pickle_op 2025-12-04T11:21:33.5392126Z W1204 11:16:26.416000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] raise BypassFxGraphCache(f"Unable to pickle non-standard op: {name}") 2025-12-04T11:21:33.5393040Z W1204 11:16:26.416000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] torch._inductor.codecache.BypassFxGraphCache: Unable to pickle non-standard op: torch.ops.prims.div.default 2025-12-04T11:21:33.5393722Z PASSED [0.4644s] [ 28%] 2025-12-04T11:21:33.5394473Z inductor/test_compile_subprocess.py::GPUTests::test_div_softmax_symfloat_cuda <- test/inductor/test_torchinductor.py W1204 11:16:26.742000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0_1] Unable to pickle input graph or example inputs 2025-12-04T11:21:33.5395440Z W1204 11:16:26.742000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0_1] Traceback (most recent call last): 2025-12-04T11:21:33.5396302Z W1204 11:16:26.742000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0_1] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/compile_fx_ext.py", line 486, in serialize_compile 2025-12-04T11:21:33.5397127Z W1204 11:16:26.742000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0_1] ).serialize() 2025-12-04T11:21:33.5397917Z W1204 11:16:26.742000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0_1] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/compile_fx_ext.py", line 210, in serialize 2025-12-04T11:21:33.5398889Z W1204 11:16:26.742000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0_1] return _WireProtocolPickledInput(GraphPickler.dumps(self)) 2025-12-04T11:21:33.5399778Z W1204 11:16:26.742000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0_1] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 127, in dumps 2025-12-04T11:21:33.5400548Z W1204 11:16:26.742000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0_1] pickler.dump(obj) 2025-12-04T11:21:33.5401390Z W1204 11:16:26.742000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0_1] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 94, in reducer_override 2025-12-04T11:21:33.5402279Z W1204 11:16:26.742000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0_1] return _GraphModulePickleData.reduce_helper(self, obj) 2025-12-04T11:21:33.5403145Z W1204 11:16:26.742000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0_1] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 352, in reduce_helper 2025-12-04T11:21:33.5403957Z W1204 11:16:26.742000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0_1] cls(obj, pickler.options), 2025-12-04T11:21:33.5404760Z W1204 11:16:26.742000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0_1] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 364, in __init__ 2025-12-04T11:21:33.5405604Z W1204 11:16:26.742000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0_1] self.graph = _GraphPickleData(gm._graph, options) 2025-12-04T11:21:33.5406447Z W1204 11:16:26.742000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0_1] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 571, in __init__ 2025-12-04T11:21:33.5407285Z W1204 11:16:26.742000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0_1] nodes[node] = _NodePickleData(node, nodes, options) 2025-12-04T11:21:33.5408129Z W1204 11:16:26.742000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0_1] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 387, in __init__ 2025-12-04T11:21:33.5408983Z W1204 11:16:26.742000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0_1] self.target = _OpPickleData.pickle(node.target, options) 2025-12-04T11:21:33.5409835Z W1204 11:16:26.742000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0_1] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 435, in pickle 2025-12-04T11:21:33.5410687Z W1204 11:16:26.742000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0_1] return cls._pickle_op(name, _OpOverloadPickleData, options) 2025-12-04T11:21:33.5411608Z W1204 11:16:26.742000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0_1] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 456, in _pickle_op 2025-12-04T11:21:33.5412506Z W1204 11:16:26.742000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0_1] raise BypassFxGraphCache(f"Unable to pickle non-standard op: {name}") 2025-12-04T11:21:33.5413466Z W1204 11:16:26.742000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0_1] torch._inductor.codecache.BypassFxGraphCache: Unable to pickle non-standard op: torch.ops.prims.convert_element_type.default 2025-12-04T11:21:33.5414360Z W1204 11:16:27.652000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/1] Unable to pickle input graph or example inputs 2025-12-04T11:21:33.5414992Z W1204 11:16:27.652000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/1] Traceback (most recent call last): 2025-12-04T11:21:33.5415915Z W1204 11:16:27.652000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/1] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/compile_fx_ext.py", line 486, in serialize_compile 2025-12-04T11:21:33.5416719Z W1204 11:16:27.652000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/1] ).serialize() 2025-12-04T11:21:33.5417490Z W1204 11:16:27.652000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/1] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/compile_fx_ext.py", line 210, in serialize 2025-12-04T11:21:33.5418380Z W1204 11:16:27.652000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/1] return _WireProtocolPickledInput(GraphPickler.dumps(self)) 2025-12-04T11:21:33.5419278Z W1204 11:16:27.652000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/1] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 127, in dumps 2025-12-04T11:21:33.5420045Z W1204 11:16:27.652000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/1] pickler.dump(obj) 2025-12-04T11:21:33.5420842Z W1204 11:16:27.652000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/1] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 94, in reducer_override 2025-12-04T11:21:33.5421703Z W1204 11:16:27.652000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/1] return _GraphModulePickleData.reduce_helper(self, obj) 2025-12-04T11:21:33.5422571Z W1204 11:16:27.652000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/1] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 352, in reduce_helper 2025-12-04T11:21:33.5423380Z W1204 11:16:27.652000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/1] cls(obj, pickler.options), 2025-12-04T11:21:33.5424208Z W1204 11:16:27.652000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/1] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 364, in __init__ 2025-12-04T11:21:33.5425056Z W1204 11:16:27.652000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/1] self.graph = _GraphPickleData(gm._graph, options) 2025-12-04T11:21:33.5425890Z W1204 11:16:27.652000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/1] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 571, in __init__ 2025-12-04T11:21:33.5426713Z W1204 11:16:27.652000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/1] nodes[node] = _NodePickleData(node, nodes, options) 2025-12-04T11:21:33.5427545Z W1204 11:16:27.652000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/1] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 387, in __init__ 2025-12-04T11:21:33.5428443Z W1204 11:16:27.652000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/1] self.target = _OpPickleData.pickle(node.target, options) 2025-12-04T11:21:33.5429291Z W1204 11:16:27.652000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/1] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 435, in pickle 2025-12-04T11:21:33.5430142Z W1204 11:16:27.652000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/1] return cls._pickle_op(name, _OpOverloadPickleData, options) 2025-12-04T11:21:33.5431010Z W1204 11:16:27.652000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/1] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 456, in _pickle_op 2025-12-04T11:21:33.5431909Z W1204 11:16:27.652000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/1] raise BypassFxGraphCache(f"Unable to pickle non-standard op: {name}") 2025-12-04T11:21:33.5432884Z W1204 11:16:27.652000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/1] torch._inductor.codecache.BypassFxGraphCache: Unable to pickle non-standard op: torch.ops.prims.convert_element_type.default 2025-12-04T11:21:33.5433596Z PASSED [1.8644s] [ 28%] 2025-12-04T11:21:33.5434213Z inductor/test_compile_subprocess.py::GPUTests::test_dont_constant_fold_cuda <- test/inductor/test_torchinductor.py SKIPPED [0.0026s] (triton not supported) [ 29%] 2025-12-04T11:21:33.5435055Z inductor/test_compile_subprocess.py::GPUTests::test_dropout_trivial_1_cuda <- test/inductor/test_torchinductor.py PASSED [0.1921s] [ 29%] 2025-12-04T11:21:33.5436091Z inductor/test_compile_subprocess.py::GPUTests::test_dtype_mismatch_issue_cuda <- test/inductor/test_torchinductor.py W1204 11:16:29.301000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] Unable to pickle input graph or example inputs 2025-12-04T11:21:33.5437105Z W1204 11:16:29.301000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] Traceback (most recent call last): 2025-12-04T11:21:33.5437970Z W1204 11:16:29.301000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/compile_fx_ext.py", line 486, in serialize_compile 2025-12-04T11:21:33.5438772Z W1204 11:16:29.301000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] ).serialize() 2025-12-04T11:21:33.5439556Z W1204 11:16:29.301000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/compile_fx_ext.py", line 210, in serialize 2025-12-04T11:21:33.5440450Z W1204 11:16:29.301000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] return _WireProtocolPickledInput(GraphPickler.dumps(self)) 2025-12-04T11:21:33.5441308Z W1204 11:16:29.301000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 127, in dumps 2025-12-04T11:21:33.5442071Z W1204 11:16:29.301000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] pickler.dump(obj) 2025-12-04T11:21:33.5442854Z W1204 11:16:29.301000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 94, in reducer_override 2025-12-04T11:21:33.5443737Z W1204 11:16:29.301000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] return _GraphModulePickleData.reduce_helper(self, obj) 2025-12-04T11:21:33.5444605Z W1204 11:16:29.301000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 352, in reduce_helper 2025-12-04T11:21:33.5445415Z W1204 11:16:29.301000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] cls(obj, pickler.options), 2025-12-04T11:21:33.5446205Z W1204 11:16:29.301000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 364, in __init__ 2025-12-04T11:21:33.5447107Z W1204 11:16:29.301000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] self.graph = _GraphPickleData(gm._graph, options) 2025-12-04T11:21:33.5447956Z W1204 11:16:29.301000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 571, in __init__ 2025-12-04T11:21:33.5448787Z W1204 11:16:29.301000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] nodes[node] = _NodePickleData(node, nodes, options) 2025-12-04T11:21:33.5449621Z W1204 11:16:29.301000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 387, in __init__ 2025-12-04T11:21:33.5450467Z W1204 11:16:29.301000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] self.target = _OpPickleData.pickle(node.target, options) 2025-12-04T11:21:33.5451412Z W1204 11:16:29.301000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 435, in pickle 2025-12-04T11:21:33.5452267Z W1204 11:16:29.301000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] return cls._pickle_op(name, _OpOverloadPickleData, options) 2025-12-04T11:21:33.5453127Z W1204 11:16:29.301000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 456, in _pickle_op 2025-12-04T11:21:33.5454022Z W1204 11:16:29.301000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] raise BypassFxGraphCache(f"Unable to pickle non-standard op: {name}") 2025-12-04T11:21:33.5455023Z W1204 11:16:29.301000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] torch._inductor.codecache.BypassFxGraphCache: Unable to pickle non-standard op: torch.ops.prims.convert_element_type.default 2025-12-04T11:21:33.5455709Z PASSED [1.2180s] [ 29%] 2025-12-04T11:21:33.5456159Z inductor/test_compile_subprocess.py::GPUTests::test_dtype_sympy_expr_cuda <- test/inductor/test_torchinductor.py PASSED [1.7119s] [ 29%] 2025-12-04T11:21:33.5456931Z inductor/test_compile_subprocess.py::GPUTests::test_dtypeview_bfloat16_bfloat16_cuda <- test/inductor/test_torchinductor.py PASSED [0.4961s] [ 30%] 2025-12-04T11:21:33.5457705Z inductor/test_compile_subprocess.py::GPUTests::test_dtypeview_bfloat16_int32_cuda <- test/inductor/test_torchinductor.py PASSED [0.4041s] [ 30%] 2025-12-04T11:21:33.5458478Z inductor/test_compile_subprocess.py::GPUTests::test_dtypeview_bfloat16_int64_cuda <- test/inductor/test_torchinductor.py PASSED [0.7011s] [ 30%] 2025-12-04T11:21:33.5459253Z inductor/test_compile_subprocess.py::GPUTests::test_dtypeview_float16_bfloat16_cuda <- test/inductor/test_torchinductor.py PASSED [0.3808s] [ 30%] 2025-12-04T11:21:33.5460034Z inductor/test_compile_subprocess.py::GPUTests::test_dtypeview_float32_float16_cuda <- test/inductor/test_torchinductor.py PASSED [0.2017s] [ 30%] 2025-12-04T11:21:33.5460806Z inductor/test_compile_subprocess.py::GPUTests::test_dtypeview_float32_float64_cuda <- test/inductor/test_torchinductor.py PASSED [0.1959s] [ 31%] 2025-12-04T11:21:33.5461572Z inductor/test_compile_subprocess.py::GPUTests::test_dtypeview_float32_int64_cuda <- test/inductor/test_torchinductor.py PASSED [0.1936s] [ 31%] 2025-12-04T11:21:33.5462329Z inductor/test_compile_subprocess.py::GPUTests::test_dtypeview_float32_int8_cuda <- test/inductor/test_torchinductor.py PASSED [0.0043s] [ 31%] 2025-12-04T11:21:33.5463100Z inductor/test_compile_subprocess.py::GPUTests::test_dtypeview_float64_bfloat16_cuda <- test/inductor/test_torchinductor.py PASSED [0.0038s] [ 31%] 2025-12-04T11:21:33.5463922Z inductor/test_compile_subprocess.py::GPUTests::test_dtypeview_float64_float32_cuda <- test/inductor/test_torchinductor.py PASSED [0.2193s] [ 32%] 2025-12-04T11:21:33.5464753Z inductor/test_compile_subprocess.py::GPUTests::test_dtypeview_float64_float64_cuda <- test/inductor/test_torchinductor.py PASSED [0.1851s] [ 32%] 2025-12-04T11:21:33.5465518Z inductor/test_compile_subprocess.py::GPUTests::test_dtypeview_float64_int16_cuda <- test/inductor/test_torchinductor.py PASSED [0.0041s] [ 32%] 2025-12-04T11:21:33.5466272Z inductor/test_compile_subprocess.py::GPUTests::test_dtypeview_float64_uint8_cuda <- test/inductor/test_torchinductor.py PASSED [0.0037s] [ 32%] 2025-12-04T11:21:33.5467020Z inductor/test_compile_subprocess.py::GPUTests::test_dtypeview_fusion_cuda <- test/inductor/test_torchinductor.py PASSED [0.4501s] [ 33%] 2025-12-04T11:21:33.5467763Z inductor/test_compile_subprocess.py::GPUTests::test_dtypeview_int16_bfloat16_cuda <- test/inductor/test_torchinductor.py PASSED [0.3886s] [ 33%] 2025-12-04T11:21:33.5468534Z inductor/test_compile_subprocess.py::GPUTests::test_dtypeview_int16_float16_cuda <- test/inductor/test_torchinductor.py PASSED [0.3880s] [ 33%] 2025-12-04T11:21:33.5469350Z inductor/test_compile_subprocess.py::GPUTests::test_dtypeview_int16_float64_cuda <- test/inductor/test_torchinductor.py PASSED [0.3965s] [ 33%] 2025-12-04T11:21:33.5470142Z inductor/test_compile_subprocess.py::GPUTests::test_dtypeview_int16_int16_cuda <- test/inductor/test_torchinductor.py PASSED [0.3870s] [ 33%] 2025-12-04T11:21:33.5470888Z inductor/test_compile_subprocess.py::GPUTests::test_dtypeview_int16_int64_cuda <- test/inductor/test_torchinductor.py PASSED [0.3942s] [ 34%] 2025-12-04T11:21:33.5471640Z inductor/test_compile_subprocess.py::GPUTests::test_dtypeview_int32_float16_cuda <- test/inductor/test_torchinductor.py PASSED [0.1979s] [ 34%] 2025-12-04T11:21:33.5472398Z inductor/test_compile_subprocess.py::GPUTests::test_dtypeview_int32_float64_cuda <- test/inductor/test_torchinductor.py PASSED [0.1965s] [ 34%] 2025-12-04T11:21:33.5473191Z inductor/test_compile_subprocess.py::GPUTests::test_dtypeview_int32_int64_cuda <- test/inductor/test_torchinductor.py PASSED [0.1979s] [ 34%] 2025-12-04T11:21:33.5473946Z inductor/test_compile_subprocess.py::GPUTests::test_dtypeview_int64_float32_cuda <- test/inductor/test_torchinductor.py PASSED [0.1999s] [ 35%] 2025-12-04T11:21:33.5474709Z inductor/test_compile_subprocess.py::GPUTests::test_dtypeview_int64_float64_cuda <- test/inductor/test_torchinductor.py PASSED [0.1978s] [ 35%] 2025-12-04T11:21:33.5475474Z inductor/test_compile_subprocess.py::GPUTests::test_dtypeview_int64_int64_cuda <- test/inductor/test_torchinductor.py PASSED [0.3984s] [ 35%] 2025-12-04T11:21:33.5476222Z inductor/test_compile_subprocess.py::GPUTests::test_dtypeview_int64_int8_cuda <- test/inductor/test_torchinductor.py PASSED [0.0039s] [ 35%] 2025-12-04T11:21:33.5476979Z inductor/test_compile_subprocess.py::GPUTests::test_dtypeview_int64_uint8_cuda <- test/inductor/test_torchinductor.py PASSED [0.0037s] [ 36%] 2025-12-04T11:21:33.5477728Z inductor/test_compile_subprocess.py::GPUTests::test_dtypeview_int8_bfloat16_cuda <- test/inductor/test_torchinductor.py PASSED [0.0037s] [ 36%] 2025-12-04T11:21:33.5478671Z inductor/test_compile_subprocess.py::GPUTests::test_dtypeview_int8_float16_cuda <- test/inductor/test_torchinductor.py PASSED [0.0037s] [ 36%] 2025-12-04T11:21:33.5479424Z inductor/test_compile_subprocess.py::GPUTests::test_dtypeview_int8_float64_cuda <- test/inductor/test_torchinductor.py PASSED [0.0036s] [ 36%] 2025-12-04T11:21:33.5480173Z inductor/test_compile_subprocess.py::GPUTests::test_dtypeview_int8_int32_cuda <- test/inductor/test_torchinductor.py PASSED [0.0035s] [ 36%] 2025-12-04T11:21:33.5480903Z inductor/test_compile_subprocess.py::GPUTests::test_dtypeview_int8_int64_cuda <- test/inductor/test_torchinductor.py PASSED [0.0035s] [ 37%] 2025-12-04T11:21:33.5481641Z inductor/test_compile_subprocess.py::GPUTests::test_dtypeview_int8_uint8_cuda <- test/inductor/test_torchinductor.py PASSED [0.0035s] [ 37%] 2025-12-04T11:21:33.5482394Z inductor/test_compile_subprocess.py::GPUTests::test_dtypeview_uint8_bfloat16_cuda <- test/inductor/test_torchinductor.py PASSED [0.0036s] [ 37%] 2025-12-04T11:21:33.5483236Z inductor/test_compile_subprocess.py::GPUTests::test_dtypeview_uint8_int16_cuda <- test/inductor/test_torchinductor.py PASSED [0.0035s] [ 37%] 2025-12-04T11:21:33.5483975Z inductor/test_compile_subprocess.py::GPUTests::test_dtypeview_uint8_int32_cuda <- test/inductor/test_torchinductor.py PASSED [0.0035s] [ 38%] 2025-12-04T11:21:33.5484717Z inductor/test_compile_subprocess.py::GPUTests::test_dtypeview_uint8_int8_cuda <- test/inductor/test_torchinductor.py PASSED [0.0039s] [ 38%] 2025-12-04T11:21:33.5485463Z inductor/test_compile_subprocess.py::GPUTests::test_dtypeview_uint8_uint8_cuda <- test/inductor/test_torchinductor.py PASSED [0.0035s] [ 38%] 2025-12-04T11:21:33.5486448Z inductor/test_compile_subprocess.py::GPUTests::test_elu_cuda <- test/inductor/test_torchinductor.py W1204 11:16:38.935000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] Unable to pickle input graph or example inputs 2025-12-04T11:21:33.5487364Z W1204 11:16:38.935000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] Traceback (most recent call last): 2025-12-04T11:21:33.5488332Z W1204 11:16:38.935000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/compile_fx_ext.py", line 486, in serialize_compile 2025-12-04T11:21:33.5489155Z W1204 11:16:38.935000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] ).serialize() 2025-12-04T11:21:33.5490134Z W1204 11:16:38.935000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/compile_fx_ext.py", line 210, in serialize 2025-12-04T11:21:33.5491099Z W1204 11:16:38.935000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] return _WireProtocolPickledInput(GraphPickler.dumps(self)) 2025-12-04T11:21:33.5491972Z W1204 11:16:38.935000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 127, in dumps 2025-12-04T11:21:33.5492736Z W1204 11:16:38.935000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] pickler.dump(obj) 2025-12-04T11:21:33.5493519Z W1204 11:16:38.935000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 94, in reducer_override 2025-12-04T11:21:33.5494419Z W1204 11:16:38.935000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] return _GraphModulePickleData.reduce_helper(self, obj) 2025-12-04T11:21:33.5495292Z W1204 11:16:38.935000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 352, in reduce_helper 2025-12-04T11:21:33.5496096Z W1204 11:16:38.935000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] cls(obj, pickler.options), 2025-12-04T11:21:33.5496898Z W1204 11:16:38.935000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 364, in __init__ 2025-12-04T11:21:33.5497737Z W1204 11:16:38.935000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] self.graph = _GraphPickleData(gm._graph, options) 2025-12-04T11:21:33.5498569Z W1204 11:16:38.935000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 571, in __init__ 2025-12-04T11:21:33.5499398Z W1204 11:16:38.935000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] nodes[node] = _NodePickleData(node, nodes, options) 2025-12-04T11:21:33.5500429Z W1204 11:16:38.935000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 387, in __init__ 2025-12-04T11:21:33.5501441Z W1204 11:16:38.935000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] self.target = _OpPickleData.pickle(node.target, options) 2025-12-04T11:21:33.5502300Z W1204 11:16:38.935000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 435, in pickle 2025-12-04T11:21:33.5503163Z W1204 11:16:38.935000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] return cls._pickle_op(name, _OpOverloadPickleData, options) 2025-12-04T11:21:33.5504103Z W1204 11:16:38.935000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 456, in _pickle_op 2025-12-04T11:21:33.5505010Z W1204 11:16:38.935000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] raise BypassFxGraphCache(f"Unable to pickle non-standard op: {name}") 2025-12-04T11:21:33.5506024Z W1204 11:16:38.935000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] torch._inductor.codecache.BypassFxGraphCache: Unable to pickle non-standard op: torch.ops.prims.convert_element_type.default 2025-12-04T11:21:33.5506743Z PASSED [0.8265s] [ 38%] 2025-12-04T11:21:33.5507340Z inductor/test_compile_subprocess.py::GPUTests::test_embedding_bag_byte_unpack_cuda <- test/inductor/test_torchinductor.py SKIPPED [0.0025s] (No cuda implementation (it returns empty)) [ 39%] 2025-12-04T11:21:33.5508199Z inductor/test_compile_subprocess.py::GPUTests::test_embedding_cuda <- test/inductor/test_torchinductor.py PASSED [0.5240s] [ 39%] 2025-12-04T11:21:33.5508912Z inductor/test_compile_subprocess.py::GPUTests::test_empty_strided_cuda <- test/inductor/test_torchinductor.py PASSED [0.1277s] [ 39%] 2025-12-04T11:21:33.5509686Z inductor/test_compile_subprocess.py::GPUTests::test_exact_stride_cuda <- test/inductor/test_torchinductor.py PASSED [0.4001s] [ 39%] 2025-12-04T11:21:33.5510383Z inductor/test_compile_subprocess.py::GPUTests::test_exp2_cuda <- test/inductor/test_torchinductor.py PASSED [0.4727s] [ 39%] 2025-12-04T11:21:33.5511042Z inductor/test_compile_subprocess.py::GPUTests::test_exp_cuda <- test/inductor/test_torchinductor.py PASSED [0.4076s] [ 40%] 2025-12-04T11:21:33.5511721Z inductor/test_compile_subprocess.py::GPUTests::test_expand_as_cuda <- test/inductor/test_torchinductor.py PASSED [0.6255s] [ 40%] 2025-12-04T11:21:33.5512413Z inductor/test_compile_subprocess.py::GPUTests::test_expand_cuda <- test/inductor/test_torchinductor.py PASSED [0.6645s] [ 40%] 2025-12-04T11:21:33.5513432Z inductor/test_compile_subprocess.py::GPUTests::test_fft_real_input_real_output_cuda <- test/inductor/test_torchinductor.py W1204 11:16:42.572000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] Unable to pickle input graph or example inputs 2025-12-04T11:21:33.5514402Z W1204 11:16:42.572000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] Traceback (most recent call last): 2025-12-04T11:21:33.5515260Z W1204 11:16:42.572000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/compile_fx_ext.py", line 486, in serialize_compile 2025-12-04T11:21:33.5516067Z W1204 11:16:42.572000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] ).serialize() 2025-12-04T11:21:33.5516840Z W1204 11:16:42.572000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/compile_fx_ext.py", line 210, in serialize 2025-12-04T11:21:33.5517725Z W1204 11:16:42.572000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] return _WireProtocolPickledInput(GraphPickler.dumps(self)) 2025-12-04T11:21:33.5518584Z W1204 11:16:42.572000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 127, in dumps 2025-12-04T11:21:33.5519397Z W1204 11:16:42.572000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] pickler.dump(obj) 2025-12-04T11:21:33.5520198Z W1204 11:16:42.572000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 94, in reducer_override 2025-12-04T11:21:33.5521064Z W1204 11:16:42.572000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] return _GraphModulePickleData.reduce_helper(self, obj) 2025-12-04T11:21:33.5521926Z W1204 11:16:42.572000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 352, in reduce_helper 2025-12-04T11:21:33.5522725Z W1204 11:16:42.572000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] cls(obj, pickler.options), 2025-12-04T11:21:33.5523525Z W1204 11:16:42.572000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 364, in __init__ 2025-12-04T11:21:33.5524440Z W1204 11:16:42.572000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] self.graph = _GraphPickleData(gm._graph, options) 2025-12-04T11:21:33.5525272Z W1204 11:16:42.572000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 571, in __init__ 2025-12-04T11:21:33.5526105Z W1204 11:16:42.572000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] nodes[node] = _NodePickleData(node, nodes, options) 2025-12-04T11:21:33.5526990Z W1204 11:16:42.572000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 387, in __init__ 2025-12-04T11:21:33.5527850Z W1204 11:16:42.572000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] self.target = _OpPickleData.pickle(node.target, options) 2025-12-04T11:21:33.5528699Z W1204 11:16:42.572000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 435, in pickle 2025-12-04T11:21:33.5529551Z W1204 11:16:42.572000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] return cls._pickle_op(name, _OpOverloadPickleData, options) 2025-12-04T11:21:33.5530417Z W1204 11:16:42.572000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 456, in _pickle_op 2025-12-04T11:21:33.5531314Z W1204 11:16:42.572000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] raise BypassFxGraphCache(f"Unable to pickle non-standard op: {name}") 2025-12-04T11:21:33.5532270Z W1204 11:16:42.572000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] torch._inductor.codecache.BypassFxGraphCache: Unable to pickle non-standard op: torch.ops.prims.convert_element_type.default 2025-12-04T11:21:33.5532940Z PASSED [0.1085s] [ 40%] 2025-12-04T11:21:33.5533353Z inductor/test_compile_subprocess.py::GPUTests::test_fill2_cuda <- test/inductor/test_torchinductor.py PASSED [0.7255s] [ 41%] 2025-12-04T11:21:33.5534112Z inductor/test_compile_subprocess.py::GPUTests::test_flexible_layout_immutable_free_symbols_cuda <- test/inductor/test_torchinductor.py PASSED [0.0029s] [ 41%] 2025-12-04T11:21:33.5535160Z inductor/test_compile_subprocess.py::GPUTests::test_flip_cat_cuda <- test/inductor/test_torchinductor.py W1204 11:16:43.439000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] Unable to pickle input graph or example inputs 2025-12-04T11:21:33.5536084Z W1204 11:16:43.439000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] Traceback (most recent call last): 2025-12-04T11:21:33.5536995Z W1204 11:16:43.439000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/compile_fx_ext.py", line 486, in serialize_compile 2025-12-04T11:21:33.5537792Z W1204 11:16:43.439000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] ).serialize() 2025-12-04T11:21:33.5538574Z W1204 11:16:43.439000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/compile_fx_ext.py", line 210, in serialize 2025-12-04T11:21:33.5539462Z W1204 11:16:43.439000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] return _WireProtocolPickledInput(GraphPickler.dumps(self)) 2025-12-04T11:21:33.5540324Z W1204 11:16:43.439000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 127, in dumps 2025-12-04T11:21:33.5541088Z W1204 11:16:43.439000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] pickler.dump(obj) 2025-12-04T11:21:33.5541964Z W1204 11:16:43.439000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 94, in reducer_override 2025-12-04T11:21:33.5542845Z W1204 11:16:43.439000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] return _GraphModulePickleData.reduce_helper(self, obj) 2025-12-04T11:21:33.5543754Z W1204 11:16:43.439000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 352, in reduce_helper 2025-12-04T11:21:33.5544567Z W1204 11:16:43.439000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] cls(obj, pickler.options), 2025-12-04T11:21:33.5545400Z W1204 11:16:43.439000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 364, in __init__ 2025-12-04T11:21:33.5546243Z W1204 11:16:43.439000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] self.graph = _GraphPickleData(gm._graph, options) 2025-12-04T11:21:33.5547078Z W1204 11:16:43.439000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 571, in __init__ 2025-12-04T11:21:33.5547914Z W1204 11:16:43.439000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] nodes[node] = _NodePickleData(node, nodes, options) 2025-12-04T11:21:33.5548752Z W1204 11:16:43.439000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 387, in __init__ 2025-12-04T11:21:33.5549594Z W1204 11:16:43.439000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] self.target = _OpPickleData.pickle(node.target, options) 2025-12-04T11:21:33.5550438Z W1204 11:16:43.439000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 435, in pickle 2025-12-04T11:21:33.5551282Z W1204 11:16:43.439000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] return cls._pickle_op(name, _OpOverloadPickleData, options) 2025-12-04T11:21:33.5552133Z W1204 11:16:43.439000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 456, in _pickle_op 2025-12-04T11:21:33.5553032Z W1204 11:16:43.439000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] raise BypassFxGraphCache(f"Unable to pickle non-standard op: {name}") 2025-12-04T11:21:33.5553934Z W1204 11:16:43.439000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] torch._inductor.codecache.BypassFxGraphCache: Unable to pickle non-standard op: torch.ops.prims.rev.default 2025-12-04T11:21:33.5554820Z W1204 11:16:43.657000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] Unable to pickle input graph or example inputs 2025-12-04T11:21:33.5555444Z W1204 11:16:43.657000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] Traceback (most recent call last): 2025-12-04T11:21:33.5556595Z W1204 11:16:43.657000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/compile_fx_ext.py", line 486, in serialize_compile 2025-12-04T11:21:33.5557425Z W1204 11:16:43.657000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] ).serialize() 2025-12-04T11:21:33.5558202Z W1204 11:16:43.657000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/compile_fx_ext.py", line 210, in serialize 2025-12-04T11:21:33.5559088Z W1204 11:16:43.657000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] return _WireProtocolPickledInput(GraphPickler.dumps(self)) 2025-12-04T11:21:33.5560043Z W1204 11:16:43.657000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 127, in dumps 2025-12-04T11:21:33.5560809Z W1204 11:16:43.657000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] pickler.dump(obj) 2025-12-04T11:21:33.5561605Z W1204 11:16:43.657000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 94, in reducer_override 2025-12-04T11:21:33.5562678Z W1204 11:16:43.657000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] return _GraphModulePickleData.reduce_helper(self, obj) 2025-12-04T11:21:33.5563606Z W1204 11:16:43.657000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 352, in reduce_helper 2025-12-04T11:21:33.5564426Z W1204 11:16:43.657000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] cls(obj, pickler.options), 2025-12-04T11:21:33.5565235Z W1204 11:16:43.657000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 364, in __init__ 2025-12-04T11:21:33.5566070Z W1204 11:16:43.657000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] self.graph = _GraphPickleData(gm._graph, options) 2025-12-04T11:21:33.5566908Z W1204 11:16:43.657000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 571, in __init__ 2025-12-04T11:21:33.5567747Z W1204 11:16:43.657000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] nodes[node] = _NodePickleData(node, nodes, options) 2025-12-04T11:21:33.5568597Z W1204 11:16:43.657000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 387, in __init__ 2025-12-04T11:21:33.5569449Z W1204 11:16:43.657000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] self.target = _OpPickleData.pickle(node.target, options) 2025-12-04T11:21:33.5570300Z W1204 11:16:43.657000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 435, in pickle 2025-12-04T11:21:33.5571157Z W1204 11:16:43.657000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] return cls._pickle_op(name, _OpOverloadPickleData, options) 2025-12-04T11:21:33.5572031Z W1204 11:16:43.657000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 456, in _pickle_op 2025-12-04T11:21:33.5572990Z W1204 11:16:43.657000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] raise BypassFxGraphCache(f"Unable to pickle non-standard op: {name}") 2025-12-04T11:21:33.5573901Z W1204 11:16:43.657000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] torch._inductor.codecache.BypassFxGraphCache: Unable to pickle non-standard op: torch.ops.prims.rev.default 2025-12-04T11:21:33.5574514Z PASSED [0.4216s] [ 41%] 2025-12-04T11:21:33.5575215Z inductor/test_compile_subprocess.py::GPUTests::test_flip_cuda <- test/inductor/test_torchinductor.py W1204 11:16:43.833000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] Unable to pickle input graph or example inputs 2025-12-04T11:21:33.5576141Z W1204 11:16:43.833000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] Traceback (most recent call last): 2025-12-04T11:21:33.5576988Z W1204 11:16:43.833000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/compile_fx_ext.py", line 486, in serialize_compile 2025-12-04T11:21:33.5578088Z W1204 11:16:43.833000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] ).serialize() 2025-12-04T11:21:33.5578893Z W1204 11:16:43.833000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/compile_fx_ext.py", line 210, in serialize 2025-12-04T11:21:33.5579779Z W1204 11:16:43.833000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] return _WireProtocolPickledInput(GraphPickler.dumps(self)) 2025-12-04T11:21:33.5580703Z W1204 11:16:43.833000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 127, in dumps 2025-12-04T11:21:33.5581461Z W1204 11:16:43.833000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] pickler.dump(obj) 2025-12-04T11:21:33.5582259Z W1204 11:16:43.833000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 94, in reducer_override 2025-12-04T11:21:33.5603256Z W1204 11:16:43.833000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] return _GraphModulePickleData.reduce_helper(self, obj) 2025-12-04T11:21:33.5604273Z W1204 11:16:43.833000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 352, in reduce_helper 2025-12-04T11:21:33.5605108Z W1204 11:16:43.833000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] cls(obj, pickler.options), 2025-12-04T11:21:33.5605918Z W1204 11:16:43.833000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 364, in __init__ 2025-12-04T11:21:33.5606760Z W1204 11:16:43.833000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] self.graph = _GraphPickleData(gm._graph, options) 2025-12-04T11:21:33.5607640Z W1204 11:16:43.833000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 571, in __init__ 2025-12-04T11:21:33.5608485Z W1204 11:16:43.833000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] nodes[node] = _NodePickleData(node, nodes, options) 2025-12-04T11:21:33.5609342Z W1204 11:16:43.833000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 387, in __init__ 2025-12-04T11:21:33.5610192Z W1204 11:16:43.833000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] self.target = _OpPickleData.pickle(node.target, options) 2025-12-04T11:21:33.5611145Z W1204 11:16:43.833000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 435, in pickle 2025-12-04T11:21:33.5611999Z W1204 11:16:43.833000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] return cls._pickle_op(name, _OpOverloadPickleData, options) 2025-12-04T11:21:33.5612891Z W1204 11:16:43.833000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 456, in _pickle_op 2025-12-04T11:21:33.5613805Z W1204 11:16:43.833000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] raise BypassFxGraphCache(f"Unable to pickle non-standard op: {name}") 2025-12-04T11:21:33.5614729Z W1204 11:16:43.833000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] torch._inductor.codecache.BypassFxGraphCache: Unable to pickle non-standard op: torch.ops.prims.rev.default 2025-12-04T11:21:33.5615572Z W1204 11:16:44.016000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] Unable to pickle input graph or example inputs 2025-12-04T11:21:33.5616322Z W1204 11:16:44.016000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] Traceback (most recent call last): 2025-12-04T11:21:33.5617178Z W1204 11:16:44.016000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/compile_fx_ext.py", line 486, in serialize_compile 2025-12-04T11:21:33.5618002Z W1204 11:16:44.016000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] ).serialize() 2025-12-04T11:21:33.5618831Z W1204 11:16:44.016000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/compile_fx_ext.py", line 210, in serialize 2025-12-04T11:21:33.5619727Z W1204 11:16:44.016000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] return _WireProtocolPickledInput(GraphPickler.dumps(self)) 2025-12-04T11:21:33.5620596Z W1204 11:16:44.016000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 127, in dumps 2025-12-04T11:21:33.5621378Z W1204 11:16:44.016000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] pickler.dump(obj) 2025-12-04T11:21:33.5622190Z W1204 11:16:44.016000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 94, in reducer_override 2025-12-04T11:21:33.5623061Z W1204 11:16:44.016000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] return _GraphModulePickleData.reduce_helper(self, obj) 2025-12-04T11:21:33.5623994Z W1204 11:16:44.016000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 352, in reduce_helper 2025-12-04T11:21:33.5624800Z W1204 11:16:44.016000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] cls(obj, pickler.options), 2025-12-04T11:21:33.5625591Z W1204 11:16:44.016000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 364, in __init__ 2025-12-04T11:21:33.5626422Z W1204 11:16:44.016000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] self.graph = _GraphPickleData(gm._graph, options) 2025-12-04T11:21:33.5627256Z W1204 11:16:44.016000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 571, in __init__ 2025-12-04T11:21:33.5628096Z W1204 11:16:44.016000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] nodes[node] = _NodePickleData(node, nodes, options) 2025-12-04T11:21:33.5628966Z W1204 11:16:44.016000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 387, in __init__ 2025-12-04T11:21:33.5629818Z W1204 11:16:44.016000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] self.target = _OpPickleData.pickle(node.target, options) 2025-12-04T11:21:33.5630669Z W1204 11:16:44.016000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 435, in pickle 2025-12-04T11:21:33.5631522Z W1204 11:16:44.016000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] return cls._pickle_op(name, _OpOverloadPickleData, options) 2025-12-04T11:21:33.5632401Z W1204 11:16:44.016000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 456, in _pickle_op 2025-12-04T11:21:33.5633303Z W1204 11:16:44.016000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] raise BypassFxGraphCache(f"Unable to pickle non-standard op: {name}") 2025-12-04T11:21:33.5634291Z W1204 11:16:44.016000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] torch._inductor.codecache.BypassFxGraphCache: Unable to pickle non-standard op: torch.ops.prims.rev.default 2025-12-04T11:21:33.5634362Z PASSED [0.3429s] [ 41%] 2025-12-04T11:21:33.5635015Z inductor/test_compile_subprocess.py::GPUTests::test_float_repr_dynamic_shapes_cuda <- test/inductor/test_torchinductor.py W1204 11:16:44.370000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] Unable to pickle input graph or example inputs 2025-12-04T11:21:33.5635283Z W1204 11:16:44.370000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] Traceback (most recent call last): 2025-12-04T11:21:33.5635837Z W1204 11:16:44.370000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/compile_fx_ext.py", line 486, in serialize_compile 2025-12-04T11:21:33.5636067Z W1204 11:16:44.370000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] ).serialize() 2025-12-04T11:21:33.5636546Z W1204 11:16:44.370000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/compile_fx_ext.py", line 210, in serialize 2025-12-04T11:21:33.5636876Z W1204 11:16:44.370000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] return _WireProtocolPickledInput(GraphPickler.dumps(self)) 2025-12-04T11:21:33.5637323Z W1204 11:16:44.370000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 127, in dumps 2025-12-04T11:21:33.5637564Z W1204 11:16:44.370000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] pickler.dump(obj) 2025-12-04T11:21:33.5638041Z W1204 11:16:44.370000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 94, in reducer_override 2025-12-04T11:21:33.5638362Z W1204 11:16:44.370000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] return _GraphModulePickleData.reduce_helper(self, obj) 2025-12-04T11:21:33.5638833Z W1204 11:16:44.370000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 352, in reduce_helper 2025-12-04T11:21:33.5639099Z W1204 11:16:44.370000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] cls(obj, pickler.options), 2025-12-04T11:21:33.5639550Z W1204 11:16:44.370000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 364, in __init__ 2025-12-04T11:21:33.5639887Z W1204 11:16:44.370000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] self.graph = _GraphPickleData(gm._graph, options) 2025-12-04T11:21:33.5640345Z W1204 11:16:44.370000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 571, in __init__ 2025-12-04T11:21:33.5640643Z W1204 11:16:44.370000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] nodes[node] = _NodePickleData(node, nodes, options) 2025-12-04T11:21:33.5641096Z W1204 11:16:44.370000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 387, in __init__ 2025-12-04T11:21:33.5641415Z W1204 11:16:44.370000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] self.target = _OpPickleData.pickle(node.target, options) 2025-12-04T11:21:33.5641869Z W1204 11:16:44.370000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 435, in pickle 2025-12-04T11:21:33.5642267Z W1204 11:16:44.370000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] return cls._pickle_op(name, _OpOverloadPickleData, options) 2025-12-04T11:21:33.5642725Z W1204 11:16:44.370000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 456, in _pickle_op 2025-12-04T11:21:33.5643261Z W1204 11:16:44.370000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] raise BypassFxGraphCache(f"Unable to pickle non-standard op: {name}") 2025-12-04T11:21:33.5643836Z W1204 11:16:44.370000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] torch._inductor.codecache.BypassFxGraphCache: Unable to pickle non-standard op: torch.ops.prims.iota.default 2025-12-04T11:21:33.5644138Z W1204 11:16:45.023000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [1/0] Unable to pickle input graph or example inputs 2025-12-04T11:21:33.5644404Z W1204 11:16:45.023000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [1/0] Traceback (most recent call last): 2025-12-04T11:21:33.5644912Z W1204 11:16:45.023000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [1/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/compile_fx_ext.py", line 486, in serialize_compile 2025-12-04T11:21:33.5645136Z W1204 11:16:45.023000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [1/0] ).serialize() 2025-12-04T11:21:33.5645608Z W1204 11:16:45.023000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [1/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/compile_fx_ext.py", line 210, in serialize 2025-12-04T11:21:33.5645938Z W1204 11:16:45.023000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [1/0] return _WireProtocolPickledInput(GraphPickler.dumps(self)) 2025-12-04T11:21:33.5646385Z W1204 11:16:45.023000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [1/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 127, in dumps 2025-12-04T11:21:33.5646627Z W1204 11:16:45.023000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [1/0] pickler.dump(obj) 2025-12-04T11:21:33.5647106Z W1204 11:16:45.023000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [1/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 94, in reducer_override 2025-12-04T11:21:33.5647418Z W1204 11:16:45.023000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [1/0] return _GraphModulePickleData.reduce_helper(self, obj) 2025-12-04T11:21:33.5647891Z W1204 11:16:45.023000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [1/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 352, in reduce_helper 2025-12-04T11:21:33.5648189Z W1204 11:16:45.023000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [1/0] cls(obj, pickler.options), 2025-12-04T11:21:33.5648648Z W1204 11:16:45.023000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [1/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 364, in __init__ 2025-12-04T11:21:33.5648945Z W1204 11:16:45.023000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [1/0] self.graph = _GraphPickleData(gm._graph, options) 2025-12-04T11:21:33.5649401Z W1204 11:16:45.023000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [1/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 571, in __init__ 2025-12-04T11:21:33.5649699Z W1204 11:16:45.023000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [1/0] nodes[node] = _NodePickleData(node, nodes, options) 2025-12-04T11:21:33.5650148Z W1204 11:16:45.023000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [1/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 387, in __init__ 2025-12-04T11:21:33.5650544Z W1204 11:16:45.023000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [1/0] self.target = _OpPickleData.pickle(node.target, options) 2025-12-04T11:21:33.5650990Z W1204 11:16:45.023000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [1/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 435, in pickle 2025-12-04T11:21:33.5651317Z W1204 11:16:45.023000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [1/0] return cls._pickle_op(name, _OpOverloadPickleData, options) 2025-12-04T11:21:33.5651820Z W1204 11:16:45.023000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [1/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 456, in _pickle_op 2025-12-04T11:21:33.5652184Z W1204 11:16:45.023000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [1/0] raise BypassFxGraphCache(f"Unable to pickle non-standard op: {name}") 2025-12-04T11:21:33.5652662Z W1204 11:16:45.023000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [1/0] torch._inductor.codecache.BypassFxGraphCache: Unable to pickle non-standard op: torch.ops.prims.iota.default 2025-12-04T11:21:33.5652950Z W1204 11:16:45.558000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] Unable to pickle input graph or example inputs 2025-12-04T11:21:33.5653215Z W1204 11:16:45.558000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] Traceback (most recent call last): 2025-12-04T11:21:33.5653717Z W1204 11:16:45.558000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/compile_fx_ext.py", line 486, in serialize_compile 2025-12-04T11:21:33.5653940Z W1204 11:16:45.558000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] ).serialize() 2025-12-04T11:21:33.5654587Z W1204 11:16:45.558000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/compile_fx_ext.py", line 210, in serialize 2025-12-04T11:21:33.5654927Z W1204 11:16:45.558000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] return _WireProtocolPickledInput(GraphPickler.dumps(self)) 2025-12-04T11:21:33.5655383Z W1204 11:16:45.558000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 127, in dumps 2025-12-04T11:21:33.5655613Z W1204 11:16:45.558000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] pickler.dump(obj) 2025-12-04T11:21:33.5656098Z W1204 11:16:45.558000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 94, in reducer_override 2025-12-04T11:21:33.5656473Z W1204 11:16:45.558000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] return _GraphModulePickleData.reduce_helper(self, obj) 2025-12-04T11:21:33.5656940Z W1204 11:16:45.558000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 352, in reduce_helper 2025-12-04T11:21:33.5657196Z W1204 11:16:45.558000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] cls(obj, pickler.options), 2025-12-04T11:21:33.5657645Z W1204 11:16:45.558000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 364, in __init__ 2025-12-04T11:21:33.5657948Z W1204 11:16:45.558000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] self.graph = _GraphPickleData(gm._graph, options) 2025-12-04T11:21:33.5658399Z W1204 11:16:45.558000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 571, in __init__ 2025-12-04T11:21:33.5658794Z W1204 11:16:45.558000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] nodes[node] = _NodePickleData(node, nodes, options) 2025-12-04T11:21:33.5659243Z W1204 11:16:45.558000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 387, in __init__ 2025-12-04T11:21:33.5659568Z W1204 11:16:45.558000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] self.target = _OpPickleData.pickle(node.target, options) 2025-12-04T11:21:33.5660058Z W1204 11:16:45.558000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 435, in pickle 2025-12-04T11:21:33.5660389Z W1204 11:16:45.558000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] return cls._pickle_op(name, _OpOverloadPickleData, options) 2025-12-04T11:21:33.5660853Z W1204 11:16:45.558000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 456, in _pickle_op 2025-12-04T11:21:33.5661208Z W1204 11:16:45.558000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] raise BypassFxGraphCache(f"Unable to pickle non-standard op: {name}") 2025-12-04T11:21:33.5661742Z W1204 11:16:45.558000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] torch._inductor.codecache.BypassFxGraphCache: Unable to pickle non-standard op: torch.ops.prims.convert_element_type.default 2025-12-04T11:21:33.5662036Z W1204 11:16:46.232000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [1/0] Unable to pickle input graph or example inputs 2025-12-04T11:21:33.5662306Z W1204 11:16:46.232000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [1/0] Traceback (most recent call last): 2025-12-04T11:21:33.5663090Z W1204 11:16:46.232000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [1/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/compile_fx_ext.py", line 486, in serialize_compile 2025-12-04T11:21:33.5663340Z W1204 11:16:46.232000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [1/0] ).serialize() 2025-12-04T11:21:33.5663925Z W1204 11:16:46.232000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [1/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/compile_fx_ext.py", line 210, in serialize 2025-12-04T11:21:33.5664263Z W1204 11:16:46.232000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [1/0] return _WireProtocolPickledInput(GraphPickler.dumps(self)) 2025-12-04T11:21:33.5664728Z W1204 11:16:46.232000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [1/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 127, in dumps 2025-12-04T11:21:33.5665028Z W1204 11:16:46.232000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [1/0] pickler.dump(obj) 2025-12-04T11:21:33.5665505Z W1204 11:16:46.232000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [1/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 94, in reducer_override 2025-12-04T11:21:33.5665830Z W1204 11:16:46.232000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [1/0] return _GraphModulePickleData.reduce_helper(self, obj) 2025-12-04T11:21:33.5666296Z W1204 11:16:46.232000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [1/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 352, in reduce_helper 2025-12-04T11:21:33.5666570Z W1204 11:16:46.232000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [1/0] cls(obj, pickler.options), 2025-12-04T11:21:33.5667061Z W1204 11:16:46.232000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [1/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 364, in __init__ 2025-12-04T11:21:33.5667406Z W1204 11:16:46.232000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [1/0] self.graph = _GraphPickleData(gm._graph, options) 2025-12-04T11:21:33.5667855Z W1204 11:16:46.232000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [1/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 571, in __init__ 2025-12-04T11:21:33.5668152Z W1204 11:16:46.232000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [1/0] nodes[node] = _NodePickleData(node, nodes, options) 2025-12-04T11:21:33.5668648Z W1204 11:16:46.232000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [1/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 387, in __init__ 2025-12-04T11:21:33.5668974Z W1204 11:16:46.232000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [1/0] self.target = _OpPickleData.pickle(node.target, options) 2025-12-04T11:21:33.5669432Z W1204 11:16:46.232000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [1/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 435, in pickle 2025-12-04T11:21:33.5669761Z W1204 11:16:46.232000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [1/0] return cls._pickle_op(name, _OpOverloadPickleData, options) 2025-12-04T11:21:33.5670226Z W1204 11:16:46.232000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [1/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 456, in _pickle_op 2025-12-04T11:21:33.5670584Z W1204 11:16:46.232000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [1/0] raise BypassFxGraphCache(f"Unable to pickle non-standard op: {name}") 2025-12-04T11:21:33.5671109Z W1204 11:16:46.232000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [1/0] torch._inductor.codecache.BypassFxGraphCache: Unable to pickle non-standard op: torch.ops.prims.convert_element_type.default 2025-12-04T11:21:33.5671189Z PASSED [2.4477s] [ 42%] 2025-12-04T11:21:33.5671504Z inductor/test_compile_subprocess.py::GPUTests::test_floordiv_cuda <- test/inductor/test_torchinductor.py PASSED [0.5036s] [ 42%] 2025-12-04T11:21:33.5672140Z inductor/test_compile_subprocess.py::GPUTests::test_forced_buffer_realize_cuda <- test/inductor/test_torchinductor.py W1204 11:16:47.119000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] Unable to pickle input graph or example inputs 2025-12-04T11:21:33.5672404Z W1204 11:16:47.119000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] Traceback (most recent call last): 2025-12-04T11:21:33.5672912Z W1204 11:16:47.119000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/compile_fx_ext.py", line 486, in serialize_compile 2025-12-04T11:21:33.5673174Z W1204 11:16:47.119000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] ).serialize() 2025-12-04T11:21:33.5673647Z W1204 11:16:47.119000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/compile_fx_ext.py", line 210, in serialize 2025-12-04T11:21:33.5673983Z W1204 11:16:47.119000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] return _WireProtocolPickledInput(GraphPickler.dumps(self)) 2025-12-04T11:21:33.5674432Z W1204 11:16:47.119000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 127, in dumps 2025-12-04T11:21:33.5674681Z W1204 11:16:47.119000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] pickler.dump(obj) 2025-12-04T11:21:33.5675195Z W1204 11:16:47.119000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 94, in reducer_override 2025-12-04T11:21:33.5675554Z W1204 11:16:47.119000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] return _GraphModulePickleData.reduce_helper(self, obj) 2025-12-04T11:21:33.5676022Z W1204 11:16:47.119000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 352, in reduce_helper 2025-12-04T11:21:33.5676277Z W1204 11:16:47.119000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] cls(obj, pickler.options), 2025-12-04T11:21:33.5676762Z W1204 11:16:47.119000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 364, in __init__ 2025-12-04T11:21:33.5677063Z W1204 11:16:47.119000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] self.graph = _GraphPickleData(gm._graph, options) 2025-12-04T11:21:33.5677520Z W1204 11:16:47.119000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 571, in __init__ 2025-12-04T11:21:33.5678000Z W1204 11:16:47.119000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] nodes[node] = _NodePickleData(node, nodes, options) 2025-12-04T11:21:33.5678460Z W1204 11:16:47.119000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 387, in __init__ 2025-12-04T11:21:33.5678787Z W1204 11:16:47.119000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] self.target = _OpPickleData.pickle(node.target, options) 2025-12-04T11:21:33.5679236Z W1204 11:16:47.119000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 435, in pickle 2025-12-04T11:21:33.5679566Z W1204 11:16:47.119000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] return cls._pickle_op(name, _OpOverloadPickleData, options) 2025-12-04T11:21:33.5680023Z W1204 11:16:47.119000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 456, in _pickle_op 2025-12-04T11:21:33.5680383Z W1204 11:16:47.119000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] raise BypassFxGraphCache(f"Unable to pickle non-standard op: {name}") 2025-12-04T11:21:33.5680888Z W1204 11:16:47.119000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] torch._inductor.codecache.BypassFxGraphCache: Unable to pickle non-standard op: torch.ops._inductor_test.realize.default 2025-12-04T11:21:33.5681265Z W1204 11:16:47.244000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] Unable to pickle input graph or example inputs 2025-12-04T11:21:33.5681534Z W1204 11:16:47.244000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] Traceback (most recent call last): 2025-12-04T11:21:33.5682036Z W1204 11:16:47.244000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/compile_fx_ext.py", line 486, in serialize_compile 2025-12-04T11:21:33.5682260Z W1204 11:16:47.244000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] ).serialize() 2025-12-04T11:21:33.5682731Z W1204 11:16:47.244000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/compile_fx_ext.py", line 210, in serialize 2025-12-04T11:21:33.5683065Z W1204 11:16:47.244000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] return _WireProtocolPickledInput(GraphPickler.dumps(self)) 2025-12-04T11:21:33.5683622Z W1204 11:16:47.244000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 127, in dumps 2025-12-04T11:21:33.5683855Z W1204 11:16:47.244000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] pickler.dump(obj) 2025-12-04T11:21:33.5684334Z W1204 11:16:47.244000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 94, in reducer_override 2025-12-04T11:21:33.5684645Z W1204 11:16:47.244000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] return _GraphModulePickleData.reduce_helper(self, obj) 2025-12-04T11:21:33.5685163Z W1204 11:16:47.244000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 352, in reduce_helper 2025-12-04T11:21:33.5685422Z W1204 11:16:47.244000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] cls(obj, pickler.options), 2025-12-04T11:21:33.5685874Z W1204 11:16:47.244000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 364, in __init__ 2025-12-04T11:21:33.5686175Z W1204 11:16:47.244000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] self.graph = _GraphPickleData(gm._graph, options) 2025-12-04T11:21:33.5686622Z W1204 11:16:47.244000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 571, in __init__ 2025-12-04T11:21:33.5686927Z W1204 11:16:47.244000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] nodes[node] = _NodePickleData(node, nodes, options) 2025-12-04T11:21:33.5687375Z W1204 11:16:47.244000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 387, in __init__ 2025-12-04T11:21:33.5687699Z W1204 11:16:47.244000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] self.target = _OpPickleData.pickle(node.target, options) 2025-12-04T11:21:33.5688153Z W1204 11:16:47.244000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 435, in pickle 2025-12-04T11:21:33.5688478Z W1204 11:16:47.244000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] return cls._pickle_op(name, _OpOverloadPickleData, options) 2025-12-04T11:21:33.5688945Z W1204 11:16:47.244000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 456, in _pickle_op 2025-12-04T11:21:33.5689340Z W1204 11:16:47.244000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] raise BypassFxGraphCache(f"Unable to pickle non-standard op: {name}") 2025-12-04T11:21:33.5689850Z W1204 11:16:47.244000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] torch._inductor.codecache.BypassFxGraphCache: Unable to pickle non-standard op: torch.ops._inductor_test.realize.default 2025-12-04T11:21:33.5689918Z PASSED [0.2530s] [ 42%] 2025-12-04T11:21:33.5690274Z inductor/test_compile_subprocess.py::GPUTests::test_fractional_max_pool2d3_cuda <- test/inductor/test_torchinductor.py PASSED [0.4864s] [ 42%] 2025-12-04T11:21:33.5690626Z inductor/test_compile_subprocess.py::GPUTests::test_fractional_max_pool2d5_cuda <- test/inductor/test_torchinductor.py PASSED [0.7017s] [ 42%] 2025-12-04T11:21:33.5690951Z inductor/test_compile_subprocess.py::GPUTests::test_full_boolean_cuda <- test/inductor/test_torchinductor.py PASSED [0.3662s] [ 43%] 2025-12-04T11:21:33.5691261Z inductor/test_compile_subprocess.py::GPUTests::test_full_like_cuda <- test/inductor/test_torchinductor.py PASSED [0.2912s] [ 43%] 2025-12-04T11:21:33.5691714Z inductor/test_compile_subprocess.py::GPUTests::test_functionalize_rng_wrappers_cuda <- test/inductor/test_torchinductor.py PASSED [0.0423s] [ 43%] 2025-12-04T11:21:33.5692025Z inductor/test_compile_subprocess.py::GPUTests::test_gather1_cuda <- test/inductor/test_torchinductor.py PASSED [0.7991s] [ 43%] 2025-12-04T11:21:33.5692341Z inductor/test_compile_subprocess.py::GPUTests::test_gather_scatter_cuda <- test/inductor/test_torchinductor.py PASSED [0.5409s] [ 44%] 2025-12-04T11:21:33.5692914Z inductor/test_compile_subprocess.py::GPUTests::test_gelu_cuda <- test/inductor/test_torchinductor.py W1204 11:16:51.029000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] Unable to pickle input graph or example inputs 2025-12-04T11:21:33.5693220Z W1204 11:16:51.029000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] Traceback (most recent call last): 2025-12-04T11:21:33.5693729Z W1204 11:16:51.029000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/compile_fx_ext.py", line 486, in serialize_compile 2025-12-04T11:21:33.5693951Z W1204 11:16:51.029000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] ).serialize() 2025-12-04T11:21:33.5694428Z W1204 11:16:51.029000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/compile_fx_ext.py", line 210, in serialize 2025-12-04T11:21:33.5694766Z W1204 11:16:51.029000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] return _WireProtocolPickledInput(GraphPickler.dumps(self)) 2025-12-04T11:21:33.5695212Z W1204 11:16:51.029000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 127, in dumps 2025-12-04T11:21:33.5695445Z W1204 11:16:51.029000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] pickler.dump(obj) 2025-12-04T11:21:33.5695924Z W1204 11:16:51.029000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 94, in reducer_override 2025-12-04T11:21:33.5696240Z W1204 11:16:51.029000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] return _GraphModulePickleData.reduce_helper(self, obj) 2025-12-04T11:21:33.5696709Z W1204 11:16:51.029000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 352, in reduce_helper 2025-12-04T11:21:33.5696967Z W1204 11:16:51.029000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] cls(obj, pickler.options), 2025-12-04T11:21:33.5697418Z W1204 11:16:51.029000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 364, in __init__ 2025-12-04T11:21:33.5697778Z W1204 11:16:51.029000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] self.graph = _GraphPickleData(gm._graph, options) 2025-12-04T11:21:33.5698227Z W1204 11:16:51.029000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 571, in __init__ 2025-12-04T11:21:33.5698529Z W1204 11:16:51.029000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] nodes[node] = _NodePickleData(node, nodes, options) 2025-12-04T11:21:33.5698982Z W1204 11:16:51.029000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 387, in __init__ 2025-12-04T11:21:33.5699304Z W1204 11:16:51.029000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] self.target = _OpPickleData.pickle(node.target, options) 2025-12-04T11:21:33.5699820Z W1204 11:16:51.029000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 435, in pickle 2025-12-04T11:21:33.5700144Z W1204 11:16:51.029000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] return cls._pickle_op(name, _OpOverloadPickleData, options) 2025-12-04T11:21:33.5700605Z W1204 11:16:51.029000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 456, in _pickle_op 2025-12-04T11:21:33.5700961Z W1204 11:16:51.029000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] raise BypassFxGraphCache(f"Unable to pickle non-standard op: {name}") 2025-12-04T11:21:33.5701530Z W1204 11:16:51.029000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] torch._inductor.codecache.BypassFxGraphCache: Unable to pickle non-standard op: torch.ops.prims.convert_element_type.default 2025-12-04T11:21:33.5701600Z PASSED [1.1049s] [ 44%] 2025-12-04T11:21:33.5701917Z inductor/test_compile_subprocess.py::GPUTests::test_getitem_cuda <- test/inductor/test_torchinductor.py PASSED [0.0177s] [ 44%] 2025-12-04T11:21:33.5702547Z inductor/test_compile_subprocess.py::GPUTests::test_graph_partition_arange2_cuda <- test/inductor/test_torchinductor.py W1204 11:16:51.741000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] Unable to pickle input graph or example inputs 2025-12-04T11:21:33.5702811Z W1204 11:16:51.741000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] Traceback (most recent call last): 2025-12-04T11:21:33.5703318Z W1204 11:16:51.741000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/compile_fx_ext.py", line 486, in serialize_compile 2025-12-04T11:21:33.5703575Z W1204 11:16:51.741000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] ).serialize() 2025-12-04T11:21:33.5704057Z W1204 11:16:51.741000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/compile_fx_ext.py", line 210, in serialize 2025-12-04T11:21:33.5704389Z W1204 11:16:51.741000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] return _WireProtocolPickledInput(GraphPickler.dumps(self)) 2025-12-04T11:21:33.5704834Z W1204 11:16:51.741000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 127, in dumps 2025-12-04T11:21:33.5705063Z W1204 11:16:51.741000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] pickler.dump(obj) 2025-12-04T11:21:33.5705534Z W1204 11:16:51.741000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 94, in reducer_override 2025-12-04T11:21:33.5705908Z W1204 11:16:51.741000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] return _GraphModulePickleData.reduce_helper(self, obj) 2025-12-04T11:21:33.5706372Z W1204 11:16:51.741000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 352, in reduce_helper 2025-12-04T11:21:33.5706632Z W1204 11:16:51.741000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] cls(obj, pickler.options), 2025-12-04T11:21:33.5707083Z W1204 11:16:51.741000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 364, in __init__ 2025-12-04T11:21:33.5707381Z W1204 11:16:51.741000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] self.graph = _GraphPickleData(gm._graph, options) 2025-12-04T11:21:33.5707915Z W1204 11:16:51.741000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 571, in __init__ 2025-12-04T11:21:33.5708219Z W1204 11:16:51.741000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] nodes[node] = _NodePickleData(node, nodes, options) 2025-12-04T11:21:33.5708673Z W1204 11:16:51.741000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 387, in __init__ 2025-12-04T11:21:33.5708987Z W1204 11:16:51.741000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] self.target = _OpPickleData.pickle(node.target, options) 2025-12-04T11:21:33.5709470Z W1204 11:16:51.741000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 435, in pickle 2025-12-04T11:21:33.5709800Z W1204 11:16:51.741000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] return cls._pickle_op(name, _OpOverloadPickleData, options) 2025-12-04T11:21:33.5710254Z W1204 11:16:51.741000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 456, in _pickle_op 2025-12-04T11:21:33.5710614Z W1204 11:16:51.741000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] raise BypassFxGraphCache(f"Unable to pickle non-standard op: {name}") 2025-12-04T11:21:33.5711086Z W1204 11:16:51.741000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] torch._inductor.codecache.BypassFxGraphCache: Unable to pickle non-standard op: torch.ops.prims.iota.default 2025-12-04T11:21:33.5711393Z W1204 11:16:51.863000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/1] Unable to pickle input graph or example inputs 2025-12-04T11:21:33.5711656Z W1204 11:16:51.863000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/1] Traceback (most recent call last): 2025-12-04T11:21:33.5712163Z W1204 11:16:51.863000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/1] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/compile_fx_ext.py", line 486, in serialize_compile 2025-12-04T11:21:33.5712378Z W1204 11:16:51.863000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/1] ).serialize() 2025-12-04T11:21:33.5712849Z W1204 11:16:51.863000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/1] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/compile_fx_ext.py", line 210, in serialize 2025-12-04T11:21:33.5713184Z W1204 11:16:51.863000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/1] return _WireProtocolPickledInput(GraphPickler.dumps(self)) 2025-12-04T11:21:33.5713625Z W1204 11:16:51.863000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/1] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 127, in dumps 2025-12-04T11:21:33.5713906Z W1204 11:16:51.863000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/1] pickler.dump(obj) 2025-12-04T11:21:33.5714378Z W1204 11:16:51.863000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/1] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 94, in reducer_override 2025-12-04T11:21:33.5714697Z W1204 11:16:51.863000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/1] return _GraphModulePickleData.reduce_helper(self, obj) 2025-12-04T11:21:33.5715161Z W1204 11:16:51.863000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/1] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 352, in reduce_helper 2025-12-04T11:21:33.5715417Z W1204 11:16:51.863000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/1] cls(obj, pickler.options), 2025-12-04T11:21:33.5715945Z W1204 11:16:51.863000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/1] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 364, in __init__ 2025-12-04T11:21:33.5716242Z W1204 11:16:51.863000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/1] self.graph = _GraphPickleData(gm._graph, options) 2025-12-04T11:21:33.5716696Z W1204 11:16:51.863000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/1] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 571, in __init__ 2025-12-04T11:21:33.5717003Z W1204 11:16:51.863000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/1] nodes[node] = _NodePickleData(node, nodes, options) 2025-12-04T11:21:33.5717487Z W1204 11:16:51.863000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/1] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 387, in __init__ 2025-12-04T11:21:33.5717812Z W1204 11:16:51.863000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/1] self.target = _OpPickleData.pickle(node.target, options) 2025-12-04T11:21:33.5718259Z W1204 11:16:51.863000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/1] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 435, in pickle 2025-12-04T11:21:33.5718589Z W1204 11:16:51.863000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/1] return cls._pickle_op(name, _OpOverloadPickleData, options) 2025-12-04T11:21:33.5719045Z W1204 11:16:51.863000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/1] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 456, in _pickle_op 2025-12-04T11:21:33.5719408Z W1204 11:16:51.863000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/1] raise BypassFxGraphCache(f"Unable to pickle non-standard op: {name}") 2025-12-04T11:21:33.5719889Z W1204 11:16:51.863000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/1] torch._inductor.codecache.BypassFxGraphCache: Unable to pickle non-standard op: torch.ops.prims.iota.default 2025-12-04T11:21:33.5719954Z PASSED [0.2443s] [ 44%] 2025-12-04T11:21:33.5720337Z inductor/test_compile_subprocess.py::GPUTests::test_graph_partition_misaligned_input_cuda <- test/inductor/test_torchinductor.py PASSED [1.0976s] [ 45%] 2025-12-04T11:21:33.5720996Z inductor/test_compile_subprocess.py::GPUTests::test_graph_partition_mutation_real_name_cuda <- test/inductor/test_torchinductor.py W1204 11:16:53.125000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] Unable to pickle input graph or example inputs 2025-12-04T11:21:33.5721265Z W1204 11:16:53.125000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] Traceback (most recent call last): 2025-12-04T11:21:33.5721769Z W1204 11:16:53.125000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/compile_fx_ext.py", line 486, in serialize_compile 2025-12-04T11:21:33.5722055Z W1204 11:16:53.125000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] ).serialize() 2025-12-04T11:21:33.5722535Z W1204 11:16:53.125000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/compile_fx_ext.py", line 210, in serialize 2025-12-04T11:21:33.5722862Z W1204 11:16:53.125000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] return _WireProtocolPickledInput(GraphPickler.dumps(self)) 2025-12-04T11:21:33.5723311Z W1204 11:16:53.125000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 127, in dumps 2025-12-04T11:21:33.5723540Z W1204 11:16:53.125000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] pickler.dump(obj) 2025-12-04T11:21:33.5724168Z W1204 11:16:53.125000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 94, in reducer_override 2025-12-04T11:21:33.5724486Z W1204 11:16:53.125000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] return _GraphModulePickleData.reduce_helper(self, obj) 2025-12-04T11:21:33.5724954Z W1204 11:16:53.125000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 352, in reduce_helper 2025-12-04T11:21:33.5725206Z W1204 11:16:53.125000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] cls(obj, pickler.options), 2025-12-04T11:21:33.5725688Z W1204 11:16:53.125000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 364, in __init__ 2025-12-04T11:21:33.5725994Z W1204 11:16:53.125000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] self.graph = _GraphPickleData(gm._graph, options) 2025-12-04T11:21:33.5726442Z W1204 11:16:53.125000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 571, in __init__ 2025-12-04T11:21:33.5726753Z W1204 11:16:53.125000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] nodes[node] = _NodePickleData(node, nodes, options) 2025-12-04T11:21:33.5727206Z W1204 11:16:53.125000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 387, in __init__ 2025-12-04T11:21:33.5727523Z W1204 11:16:53.125000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] self.target = _OpPickleData.pickle(node.target, options) 2025-12-04T11:21:33.5727976Z W1204 11:16:53.125000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 435, in pickle 2025-12-04T11:21:33.5728303Z W1204 11:16:53.125000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] return cls._pickle_op(name, _OpOverloadPickleData, options) 2025-12-04T11:21:33.5728766Z W1204 11:16:53.125000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 456, in _pickle_op 2025-12-04T11:21:33.5729124Z W1204 11:16:53.125000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] raise BypassFxGraphCache(f"Unable to pickle non-standard op: {name}") 2025-12-04T11:21:33.5729648Z W1204 11:16:53.125000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] torch._inductor.codecache.BypassFxGraphCache: Unable to pickle non-standard op: torch.ops.prims.convert_element_type.default 2025-12-04T11:21:33.5729917Z W1204 11:16:53.145000 89367 site-packages/torch/_inductor/utils.py:2565] [0/0] DeviceCopy in input program 2025-12-04T11:21:33.5730143Z W1204 11:16:53.146000 89367 site-packages/torch/_inductor/utils.py:2565] [0/0] DeviceCopy in input program 2025-12-04T11:21:33.5730211Z PASSED [0.2272s] [ 45%] 2025-12-04T11:21:33.5730577Z inductor/test_compile_subprocess.py::GPUTests::test_graph_partition_scalar_inputs_cuda <- test/inductor/test_torchinductor.py PASSED [0.5480s] [ 45%] 2025-12-04T11:21:33.5731168Z inductor/test_compile_subprocess.py::GPUTests::test_hardtanh_cuda <- test/inductor/test_torchinductor.py W1204 11:16:54.094000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] Unable to pickle input graph or example inputs 2025-12-04T11:21:33.5731432Z W1204 11:16:54.094000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] Traceback (most recent call last): 2025-12-04T11:21:33.5731945Z W1204 11:16:54.094000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/compile_fx_ext.py", line 486, in serialize_compile 2025-12-04T11:21:33.5732240Z W1204 11:16:54.094000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] ).serialize() 2025-12-04T11:21:33.5732713Z W1204 11:16:54.094000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/compile_fx_ext.py", line 210, in serialize 2025-12-04T11:21:33.5733056Z W1204 11:16:54.094000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] return _WireProtocolPickledInput(GraphPickler.dumps(self)) 2025-12-04T11:21:33.5733534Z W1204 11:16:54.094000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 127, in dumps 2025-12-04T11:21:33.5733769Z W1204 11:16:54.094000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] pickler.dump(obj) 2025-12-04T11:21:33.5734246Z W1204 11:16:54.094000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 94, in reducer_override 2025-12-04T11:21:33.5734560Z W1204 11:16:54.094000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] return _GraphModulePickleData.reduce_helper(self, obj) 2025-12-04T11:21:33.5735040Z W1204 11:16:54.094000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 352, in reduce_helper 2025-12-04T11:21:33.5735299Z W1204 11:16:54.094000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] cls(obj, pickler.options), 2025-12-04T11:21:33.5735758Z W1204 11:16:54.094000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 364, in __init__ 2025-12-04T11:21:33.5736057Z W1204 11:16:54.094000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] self.graph = _GraphPickleData(gm._graph, options) 2025-12-04T11:21:33.5736517Z W1204 11:16:54.094000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 571, in __init__ 2025-12-04T11:21:33.5736812Z W1204 11:16:54.094000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] nodes[node] = _NodePickleData(node, nodes, options) 2025-12-04T11:21:33.5737260Z W1204 11:16:54.094000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 387, in __init__ 2025-12-04T11:21:33.5737580Z W1204 11:16:54.094000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] self.target = _OpPickleData.pickle(node.target, options) 2025-12-04T11:21:33.5738072Z W1204 11:16:54.094000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 435, in pickle 2025-12-04T11:21:33.5738403Z W1204 11:16:54.094000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] return cls._pickle_op(name, _OpOverloadPickleData, options) 2025-12-04T11:21:33.5738862Z W1204 11:16:54.094000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 456, in _pickle_op 2025-12-04T11:21:33.5739221Z W1204 11:16:54.094000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] raise BypassFxGraphCache(f"Unable to pickle non-standard op: {name}") 2025-12-04T11:21:33.5739741Z W1204 11:16:54.094000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] torch._inductor.codecache.BypassFxGraphCache: Unable to pickle non-standard op: torch.ops.prims.convert_element_type.default 2025-12-04T11:21:33.5739842Z PASSED [0.4096s] [ 45%] 2025-12-04T11:21:33.5740223Z inductor/test_compile_subprocess.py::GPUTests::test_horizonal_fusion1_cuda <- test/inductor/test_torchinductor.py PASSED [0.6710s] [ 45%] 2025-12-04T11:21:33.5740528Z inductor/test_compile_subprocess.py::GPUTests::test_index2_cuda <- test/inductor/test_torchinductor.py PASSED [0.9556s] [ 46%] 2025-12-04T11:21:33.5740845Z inductor/test_compile_subprocess.py::GPUTests::test_index_put1_cuda <- test/inductor/test_torchinductor.py PASSED [2.4684s] [ 46%] 2025-12-04T11:21:33.5741148Z inductor/test_compile_subprocess.py::GPUTests::test_index_put3_cuda <- test/inductor/test_torchinductor.py PASSED [1.1726s] [ 46%] 2025-12-04T11:21:33.5741452Z inductor/test_compile_subprocess.py::GPUTests::test_index_put4_cuda <- test/inductor/test_torchinductor.py PASSED [0.3435s] [ 46%] 2025-12-04T11:21:33.5741840Z inductor/test_compile_subprocess.py::GPUTests::test_index_put_as_masked_fill_cuda <- test/inductor/test_torchinductor.py PASSED [1.1856s] [ 47%] 2025-12-04T11:21:33.5742218Z inductor/test_compile_subprocess.py::GPUTests::test_index_put_deterministic_fallback_cuda <- test/inductor/test_torchinductor.py PASSED [0.2883s] [ 47%] 2025-12-04T11:21:33.5742579Z inductor/test_compile_subprocess.py::GPUTests::test_index_put_failed_reinplace_cuda <- test/inductor/test_torchinductor.py PASSED [0.5663s] [ 47%] 2025-12-04T11:21:33.5742899Z inductor/test_compile_subprocess.py::GPUTests::test_index_put_index_cuda <- test/inductor/test_torchinductor.py PASSED [0.4366s] [ 47%] 2025-12-04T11:21:33.5743233Z inductor/test_compile_subprocess.py::GPUTests::test_index_put_reinplace_cuda <- test/inductor/test_torchinductor.py PASSED [0.3920s] [ 48%] 2025-12-04T11:21:33.5743596Z inductor/test_compile_subprocess.py::GPUTests::test_index_remainder_cuda <- test/inductor/test_torchinductor.py PASSED [0.4003s] [ 48%] 2025-12-04T11:21:33.5743940Z inductor/test_compile_subprocess.py::GPUTests::test_indirect_load_broadcast_cuda <- test/inductor/test_torchinductor.py PASSED [1.7297s] [ 48%] 2025-12-04T11:21:33.5744240Z inductor/test_compile_subprocess.py::GPUTests::test_inf_cuda <- test/inductor/test_torchinductor.py PASSED [0.3317s] [ 48%] 2025-12-04T11:21:33.5744593Z inductor/test_compile_subprocess.py::GPUTests::test_inner_reduction_detection_cuda <- test/inductor/test_torchinductor.py PASSED [0.2296s] [ 48%] 2025-12-04T11:21:33.5744916Z inductor/test_compile_subprocess.py::GPUTests::test_inplace_add_cuda <- test/inductor/test_torchinductor.py PASSED [0.1655s] [ 49%] 2025-12-04T11:21:33.5745516Z inductor/test_compile_subprocess.py::GPUTests::test_inplace_flip_cuda <- test/inductor/test_torchinductor.py W1204 11:17:05.869000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] Unable to pickle input graph or example inputs 2025-12-04T11:21:33.5745786Z W1204 11:17:05.869000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] Traceback (most recent call last): 2025-12-04T11:21:33.5746344Z W1204 11:17:05.869000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/compile_fx_ext.py", line 486, in serialize_compile 2025-12-04T11:21:33.5746568Z W1204 11:17:05.869000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] ).serialize() 2025-12-04T11:21:33.5747045Z W1204 11:17:05.869000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/compile_fx_ext.py", line 210, in serialize 2025-12-04T11:21:33.5747369Z W1204 11:17:05.869000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] return _WireProtocolPickledInput(GraphPickler.dumps(self)) 2025-12-04T11:21:33.5747820Z W1204 11:17:05.869000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 127, in dumps 2025-12-04T11:21:33.5748063Z W1204 11:17:05.869000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] pickler.dump(obj) 2025-12-04T11:21:33.5748606Z W1204 11:17:05.869000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 94, in reducer_override 2025-12-04T11:21:33.5748926Z W1204 11:17:05.869000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] return _GraphModulePickleData.reduce_helper(self, obj) 2025-12-04T11:21:33.5749392Z W1204 11:17:05.869000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 352, in reduce_helper 2025-12-04T11:21:33.5749657Z W1204 11:17:05.869000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] cls(obj, pickler.options), 2025-12-04T11:21:33.5750165Z W1204 11:17:05.869000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 364, in __init__ 2025-12-04T11:21:33.5750473Z W1204 11:17:05.869000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] self.graph = _GraphPickleData(gm._graph, options) 2025-12-04T11:21:33.5750922Z W1204 11:17:05.869000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 571, in __init__ 2025-12-04T11:21:33.5751218Z W1204 11:17:05.869000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] nodes[node] = _NodePickleData(node, nodes, options) 2025-12-04T11:21:33.5751670Z W1204 11:17:05.869000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 387, in __init__ 2025-12-04T11:21:33.5751991Z W1204 11:17:05.869000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] self.target = _OpPickleData.pickle(node.target, options) 2025-12-04T11:21:33.5752443Z W1204 11:17:05.869000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 435, in pickle 2025-12-04T11:21:33.5752767Z W1204 11:17:05.869000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] return cls._pickle_op(name, _OpOverloadPickleData, options) 2025-12-04T11:21:33.5753230Z W1204 11:17:05.869000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 456, in _pickle_op 2025-12-04T11:21:33.5753582Z W1204 11:17:05.869000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] raise BypassFxGraphCache(f"Unable to pickle non-standard op: {name}") 2025-12-04T11:21:33.5754056Z W1204 11:17:05.869000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] torch._inductor.codecache.BypassFxGraphCache: Unable to pickle non-standard op: torch.ops.prims.rev.default 2025-12-04T11:21:33.5754390Z W1204 11:17:06.591000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] Unable to pickle input graph or example inputs 2025-12-04T11:21:33.5754650Z W1204 11:17:06.591000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] Traceback (most recent call last): 2025-12-04T11:21:33.5755157Z W1204 11:17:06.591000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/compile_fx_ext.py", line 486, in serialize_compile 2025-12-04T11:21:33.5755371Z W1204 11:17:06.591000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] ).serialize() 2025-12-04T11:21:33.5755845Z W1204 11:17:06.591000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/compile_fx_ext.py", line 210, in serialize 2025-12-04T11:21:33.5756193Z W1204 11:17:06.591000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] return _WireProtocolPickledInput(GraphPickler.dumps(self)) 2025-12-04T11:21:33.5756708Z W1204 11:17:06.591000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 127, in dumps 2025-12-04T11:21:33.5756942Z W1204 11:17:06.591000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] pickler.dump(obj) 2025-12-04T11:21:33.5757412Z W1204 11:17:06.591000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 94, in reducer_override 2025-12-04T11:21:33.5757772Z W1204 11:17:06.591000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] return _GraphModulePickleData.reduce_helper(self, obj) 2025-12-04T11:21:33.5758240Z W1204 11:17:06.591000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 352, in reduce_helper 2025-12-04T11:21:33.5758499Z W1204 11:17:06.591000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] cls(obj, pickler.options), 2025-12-04T11:21:33.5758950Z W1204 11:17:06.591000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 364, in __init__ 2025-12-04T11:21:33.5759248Z W1204 11:17:06.591000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] self.graph = _GraphPickleData(gm._graph, options) 2025-12-04T11:21:33.5759702Z W1204 11:17:06.591000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 571, in __init__ 2025-12-04T11:21:33.5760000Z W1204 11:17:06.591000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] nodes[node] = _NodePickleData(node, nodes, options) 2025-12-04T11:21:33.5760468Z W1204 11:17:06.591000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 387, in __init__ 2025-12-04T11:21:33.5760784Z W1204 11:17:06.591000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] self.target = _OpPickleData.pickle(node.target, options) 2025-12-04T11:21:33.5761227Z W1204 11:17:06.591000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 435, in pickle 2025-12-04T11:21:33.5761557Z W1204 11:17:06.591000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] return cls._pickle_op(name, _OpOverloadPickleData, options) 2025-12-04T11:21:33.5762018Z W1204 11:17:06.591000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 456, in _pickle_op 2025-12-04T11:21:33.5762417Z W1204 11:17:06.591000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] raise BypassFxGraphCache(f"Unable to pickle non-standard op: {name}") 2025-12-04T11:21:33.5762889Z W1204 11:17:06.591000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] torch._inductor.codecache.BypassFxGraphCache: Unable to pickle non-standard op: torch.ops.prims.rev.default 2025-12-04T11:21:33.5762958Z PASSED [1.6367s] [ 49%] 2025-12-04T11:21:33.5763292Z inductor/test_compile_subprocess.py::GPUTests::test_inplace_resize_as_cuda <- test/inductor/test_torchinductor.py PASSED [0.0416s] [ 49%] 2025-12-04T11:21:33.5763616Z inductor/test_compile_subprocess.py::GPUTests::test_input_mutation2_cuda <- test/inductor/test_torchinductor.py PASSED [0.2096s] [ 49%] 2025-12-04T11:21:33.5763937Z inductor/test_compile_subprocess.py::GPUTests::test_input_mutation5_cuda <- test/inductor/test_torchinductor.py PASSED [0.1526s] [ 50%] 2025-12-04T11:21:33.5764278Z inductor/test_compile_subprocess.py::GPUTests::test_insignificant_strides_cuda <- test/inductor/test_torchinductor.py PASSED [0.1538s] [ 50%] 2025-12-04T11:21:33.5764685Z inductor/test_compile_subprocess.py::GPUTests::test_int8_weight_only_quant_cuda <- test/inductor/test_torchinductor.py PASSED [0.3604s] [ 50%] 2025-12-04T11:21:33.5765009Z inductor/test_compile_subprocess.py::GPUTests::test_isin_tensor_scalar_cuda <- test/inductor/test_torchinductor.py PASSED [0.7863s] [ 50%] 2025-12-04T11:21:33.5765322Z inductor/test_compile_subprocess.py::GPUTests::test_issue102546_cuda <- test/inductor/test_torchinductor.py PASSED [0.2814s] [ 51%] 2025-12-04T11:21:33.5765636Z inductor/test_compile_subprocess.py::GPUTests::test_kernel_names_cuda <- test/inductor/test_torchinductor.py PASSED [0.1447s] [ 51%] 2025-12-04T11:21:33.5766071Z inductor/test_compile_subprocess.py::GPUTests::test_kwargs_cuda <- test/inductor/test_torchinductor.py SKIPPED [0.0025s] (histogramdd only supports cpu) [ 51%] 2025-12-04T11:21:33.5766373Z inductor/test_compile_subprocess.py::GPUTests::test_l1_loss_cuda <- test/inductor/test_torchinductor.py PASSED [0.2466s] [ 51%] 2025-12-04T11:21:33.5766698Z inductor/test_compile_subprocess.py::GPUTests::test_large_block_sizes_cuda <- test/inductor/test_torchinductor.py PASSED [5.0514s] [ 51%] 2025-12-04T11:21:33.5767048Z inductor/test_compile_subprocess.py::GPUTests::test_large_broadcast_reduction_cuda <- test/inductor/test_torchinductor.py PASSED [0.5155s] [ 52%] 2025-12-04T11:21:33.5767388Z inductor/test_compile_subprocess.py::GPUTests::test_large_offset_pointwise_cuda <- test/inductor/test_torchinductor.py PASSED [0.7486s] [ 52%] 2025-12-04T11:21:33.5767723Z inductor/test_compile_subprocess.py::GPUTests::test_large_tensor_reduction_cuda <- test/inductor/test_torchinductor.py PASSED [0.9549s] [ 52%] 2025-12-04T11:21:33.5768086Z inductor/test_compile_subprocess.py::GPUTests::test_linalg_eig_stride_consistency_cuda <- test/inductor/test_torchinductor.py PASSED [0.1234s] [ 52%] 2025-12-04T11:21:33.5768680Z inductor/test_compile_subprocess.py::GPUTests::test_linspace2_cuda <- test/inductor/test_torchinductor.py W1204 11:17:17.170000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] Unable to pickle input graph or example inputs 2025-12-04T11:21:33.5768950Z W1204 11:17:17.170000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] Traceback (most recent call last): 2025-12-04T11:21:33.5769456Z W1204 11:17:17.170000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/compile_fx_ext.py", line 486, in serialize_compile 2025-12-04T11:21:33.5769673Z W1204 11:17:17.170000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] ).serialize() 2025-12-04T11:21:33.5770155Z W1204 11:17:17.170000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/compile_fx_ext.py", line 210, in serialize 2025-12-04T11:21:33.5770523Z W1204 11:17:17.170000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] return _WireProtocolPickledInput(GraphPickler.dumps(self)) 2025-12-04T11:21:33.5770975Z W1204 11:17:17.170000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 127, in dumps 2025-12-04T11:21:33.5771206Z W1204 11:17:17.170000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] pickler.dump(obj) 2025-12-04T11:21:33.5771676Z W1204 11:17:17.170000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 94, in reducer_override 2025-12-04T11:21:33.5772011Z W1204 11:17:17.170000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] return _GraphModulePickleData.reduce_helper(self, obj) 2025-12-04T11:21:33.5772480Z W1204 11:17:17.170000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 352, in reduce_helper 2025-12-04T11:21:33.5772809Z W1204 11:17:17.170000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] cls(obj, pickler.options), 2025-12-04T11:21:33.5773265Z W1204 11:17:17.170000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 364, in __init__ 2025-12-04T11:21:33.5773566Z W1204 11:17:17.170000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] self.graph = _GraphPickleData(gm._graph, options) 2025-12-04T11:21:33.5774059Z W1204 11:17:17.170000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 571, in __init__ 2025-12-04T11:21:33.5774358Z W1204 11:17:17.170000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] nodes[node] = _NodePickleData(node, nodes, options) 2025-12-04T11:21:33.5774815Z W1204 11:17:17.170000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 387, in __init__ 2025-12-04T11:21:33.5775132Z W1204 11:17:17.170000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] self.target = _OpPickleData.pickle(node.target, options) 2025-12-04T11:21:33.5775579Z W1204 11:17:17.170000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 435, in pickle 2025-12-04T11:21:33.5775906Z W1204 11:17:17.170000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] return cls._pickle_op(name, _OpOverloadPickleData, options) 2025-12-04T11:21:33.5776370Z W1204 11:17:17.170000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 456, in _pickle_op 2025-12-04T11:21:33.5776730Z W1204 11:17:17.170000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] raise BypassFxGraphCache(f"Unable to pickle non-standard op: {name}") 2025-12-04T11:21:33.5777252Z W1204 11:17:17.170000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] torch._inductor.codecache.BypassFxGraphCache: Unable to pickle non-standard op: torch.ops.prims.convert_element_type.default 2025-12-04T11:21:33.5777319Z PASSED [0.2189s] [ 53%] 2025-12-04T11:21:33.5777729Z inductor/test_compile_subprocess.py::GPUTests::test_linspace4_cuda <- test/inductor/test_torchinductor.py SKIPPED [0.0002s] (requires multiple cuda devices) [ 53%] 2025-12-04T11:21:33.5778367Z inductor/test_compile_subprocess.py::GPUTests::test_list_clearing_cuda <- test/inductor/test_torchinductor.py SKIPPED [0.0004s] (test is slow; run with PYTORCH_TEST_WITH_SLOW to enable test) [ 53%] 2025-12-04T11:21:33.5778782Z inductor/test_compile_subprocess.py::GPUTests::test_lite_mode_not_decompose_cuda <- test/inductor/test_torchinductor.py PASSED [0.3558s] [ 53%] 2025-12-04T11:21:33.5779175Z inductor/test_compile_subprocess.py::GPUTests::test_lite_triton_kernel_wrapper_functional_cuda <- test/inductor/test_torchinductor.py PASSED [0.4337s] [ 54%] 2025-12-04T11:21:33.5779467Z inductor/test_compile_subprocess.py::GPUTests::test_log2_cuda <- test/inductor/test_torchinductor.py PASSED [0.4158s] [ 54%] 2025-12-04T11:21:33.5779766Z inductor/test_compile_subprocess.py::GPUTests::test_log_fp64_cuda <- test/inductor/test_torchinductor.py PASSED [0.4086s] [ 54%] 2025-12-04T11:21:33.5780151Z inductor/test_compile_subprocess.py::GPUTests::test_logaddexp_cuda <- test/inductor/test_torchinductor.py SKIPPED [0.0025s] (Not implemented for CUDA) [ 54%] 2025-12-04T11:21:33.5780470Z inductor/test_compile_subprocess.py::GPUTests::test_logcumsumexp_cuda <- test/inductor/test_torchinductor.py PASSED [10.3025s] [ 54%] 2025-12-04T11:21:33.5780811Z inductor/test_compile_subprocess.py::GPUTests::test_logcumsumexp_zero_dim_cuda <- test/inductor/test_torchinductor.py PASSED [0.2857s] [ 55%] 2025-12-04T11:21:33.5781522Z inductor/test_compile_subprocess.py::GPUTests::test_logsumexp_cuda <- test/inductor/test_torchinductor.py W1204 11:17:30.136000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] Unable to pickle input graph or example inputs 2025-12-04T11:21:33.5781787Z W1204 11:17:30.136000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] Traceback (most recent call last): 2025-12-04T11:21:33.5782296Z W1204 11:17:30.136000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/compile_fx_ext.py", line 486, in serialize_compile 2025-12-04T11:21:33.5782560Z W1204 11:17:30.136000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] ).serialize() 2025-12-04T11:21:33.5783037Z W1204 11:17:30.136000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/compile_fx_ext.py", line 210, in serialize 2025-12-04T11:21:33.5783372Z W1204 11:17:30.136000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] return _WireProtocolPickledInput(GraphPickler.dumps(self)) 2025-12-04T11:21:33.5783874Z W1204 11:17:30.136000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 127, in dumps 2025-12-04T11:21:33.5784108Z W1204 11:17:30.136000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] pickler.dump(obj) 2025-12-04T11:21:33.5784587Z W1204 11:17:30.136000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 94, in reducer_override 2025-12-04T11:21:33.5784920Z W1204 11:17:30.136000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] return _GraphModulePickleData.reduce_helper(self, obj) 2025-12-04T11:21:33.5785405Z W1204 11:17:30.136000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 352, in reduce_helper 2025-12-04T11:21:33.5785667Z W1204 11:17:30.136000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] cls(obj, pickler.options), 2025-12-04T11:21:33.5786130Z W1204 11:17:30.136000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 364, in __init__ 2025-12-04T11:21:33.5786428Z W1204 11:17:30.136000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] self.graph = _GraphPickleData(gm._graph, options) 2025-12-04T11:21:33.5786882Z W1204 11:17:30.136000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 571, in __init__ 2025-12-04T11:21:33.5787235Z W1204 11:17:30.136000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] nodes[node] = _NodePickleData(node, nodes, options) 2025-12-04T11:21:33.5787688Z W1204 11:17:30.136000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 387, in __init__ 2025-12-04T11:21:33.5788002Z W1204 11:17:30.136000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] self.target = _OpPickleData.pickle(node.target, options) 2025-12-04T11:21:33.5788456Z W1204 11:17:30.136000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 435, in pickle 2025-12-04T11:21:33.5788782Z W1204 11:17:30.136000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] return cls._pickle_op(name, _OpOverloadPickleData, options) 2025-12-04T11:21:33.5789322Z W1204 11:17:30.136000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 456, in _pickle_op 2025-12-04T11:21:33.5789685Z W1204 11:17:30.136000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] raise BypassFxGraphCache(f"Unable to pickle non-standard op: {name}") 2025-12-04T11:21:33.5790206Z W1204 11:17:30.136000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] torch._inductor.codecache.BypassFxGraphCache: Unable to pickle non-standard op: torch.ops.prims.convert_element_type.default 2025-12-04T11:21:33.5790272Z PASSED [1.1464s] [ 55%] 2025-12-04T11:21:33.5790962Z inductor/test_compile_subprocess.py::GPUTests::test_low_memory_max_pool_dilation_1_dim_3_cuda <- test/inductor/test_torchinductor.py W1204 11:17:30.726000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] Unable to pickle input graph or example inputs 2025-12-04T11:21:33.5791236Z W1204 11:17:30.726000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] Traceback (most recent call last): 2025-12-04T11:21:33.5791744Z W1204 11:17:30.726000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/compile_fx_ext.py", line 486, in serialize_compile 2025-12-04T11:21:33.5791959Z W1204 11:17:30.726000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] ).serialize() 2025-12-04T11:21:33.5792446Z W1204 11:17:30.726000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/compile_fx_ext.py", line 210, in serialize 2025-12-04T11:21:33.5792774Z W1204 11:17:30.726000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] return _WireProtocolPickledInput(GraphPickler.dumps(self)) 2025-12-04T11:21:33.5793224Z W1204 11:17:30.726000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 127, in dumps 2025-12-04T11:21:33.5793454Z W1204 11:17:30.726000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] pickler.dump(obj) 2025-12-04T11:21:33.5793926Z W1204 11:17:30.726000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 94, in reducer_override 2025-12-04T11:21:33.5794242Z W1204 11:17:30.726000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] return _GraphModulePickleData.reduce_helper(self, obj) 2025-12-04T11:21:33.5794707Z W1204 11:17:30.726000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 352, in reduce_helper 2025-12-04T11:21:33.5794963Z W1204 11:17:30.726000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] cls(obj, pickler.options), 2025-12-04T11:21:33.5795461Z W1204 11:17:30.726000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 364, in __init__ 2025-12-04T11:21:33.5795763Z W1204 11:17:30.726000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] self.graph = _GraphPickleData(gm._graph, options) 2025-12-04T11:21:33.5796211Z W1204 11:17:30.726000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 571, in __init__ 2025-12-04T11:21:33.5796511Z W1204 11:17:30.726000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] nodes[node] = _NodePickleData(node, nodes, options) 2025-12-04T11:21:33.5796960Z W1204 11:17:30.726000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 387, in __init__ 2025-12-04T11:21:33.5797775Z W1204 11:17:30.726000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] self.target = _OpPickleData.pickle(node.target, options) 2025-12-04T11:21:33.5798235Z W1204 11:17:30.726000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 435, in pickle 2025-12-04T11:21:33.5798558Z W1204 11:17:30.726000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] return cls._pickle_op(name, _OpOverloadPickleData, options) 2025-12-04T11:21:33.5799019Z W1204 11:17:30.726000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 456, in _pickle_op 2025-12-04T11:21:33.5799406Z W1204 11:17:30.726000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] raise BypassFxGraphCache(f"Unable to pickle non-standard op: {name}") 2025-12-04T11:21:33.5799962Z W1204 11:17:30.726000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] torch._inductor.codecache.BypassFxGraphCache: Unable to pickle non-standard op: torch.ops.prims._low_memory_max_pool_with_offsets.default 2025-12-04T11:21:33.5800255Z W1204 11:17:32.714000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] Unable to pickle input graph or example inputs 2025-12-04T11:21:33.5800514Z W1204 11:17:32.714000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] Traceback (most recent call last): 2025-12-04T11:21:33.5801019Z W1204 11:17:32.714000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/compile_fx_ext.py", line 486, in serialize_compile 2025-12-04T11:21:33.5801237Z W1204 11:17:32.714000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] ).serialize() 2025-12-04T11:21:33.5801715Z W1204 11:17:32.714000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/compile_fx_ext.py", line 210, in serialize 2025-12-04T11:21:33.5802049Z W1204 11:17:32.714000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] return _WireProtocolPickledInput(GraphPickler.dumps(self)) 2025-12-04T11:21:33.5802490Z W1204 11:17:32.714000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 127, in dumps 2025-12-04T11:21:33.5802721Z W1204 11:17:32.714000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] pickler.dump(obj) 2025-12-04T11:21:33.5803190Z W1204 11:17:32.714000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 94, in reducer_override 2025-12-04T11:21:33.5803556Z W1204 11:17:32.714000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] return _GraphModulePickleData.reduce_helper(self, obj) 2025-12-04T11:21:33.5804024Z W1204 11:17:32.714000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 352, in reduce_helper 2025-12-04T11:21:33.5804281Z W1204 11:17:32.714000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] cls(obj, pickler.options), 2025-12-04T11:21:33.5804731Z W1204 11:17:32.714000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 364, in __init__ 2025-12-04T11:21:33.5805027Z W1204 11:17:32.714000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] self.graph = _GraphPickleData(gm._graph, options) 2025-12-04T11:21:33.5805483Z W1204 11:17:32.714000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 571, in __init__ 2025-12-04T11:21:33.5805865Z W1204 11:17:32.714000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] nodes[node] = _NodePickleData(node, nodes, options) 2025-12-04T11:21:33.5806323Z W1204 11:17:32.714000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 387, in __init__ 2025-12-04T11:21:33.5806638Z W1204 11:17:32.714000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] self.target = _OpPickleData.pickle(node.target, options) 2025-12-04T11:21:33.5807122Z W1204 11:17:32.714000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 435, in pickle 2025-12-04T11:21:33.5807449Z W1204 11:17:32.714000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] return cls._pickle_op(name, _OpOverloadPickleData, options) 2025-12-04T11:21:33.5807923Z W1204 11:17:32.714000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 456, in _pickle_op 2025-12-04T11:21:33.5808281Z W1204 11:17:32.714000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] raise BypassFxGraphCache(f"Unable to pickle non-standard op: {name}") 2025-12-04T11:21:33.5808831Z W1204 11:17:32.714000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] torch._inductor.codecache.BypassFxGraphCache: Unable to pickle non-standard op: torch.ops.prims._low_memory_max_pool_with_offsets.default 2025-12-04T11:21:33.5808901Z PASSED [3.9825s] [ 55%] 2025-12-04T11:21:33.5809558Z inductor/test_compile_subprocess.py::GPUTests::test_low_memory_max_pool_dilation_2_dim_2_cuda <- test/inductor/test_torchinductor.py W1204 11:17:34.691000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] Unable to pickle input graph or example inputs 2025-12-04T11:21:33.5809822Z W1204 11:17:34.691000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] Traceback (most recent call last): 2025-12-04T11:21:33.5810332Z W1204 11:17:34.691000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/compile_fx_ext.py", line 486, in serialize_compile 2025-12-04T11:21:33.5810547Z W1204 11:17:34.691000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] ).serialize() 2025-12-04T11:21:33.5811030Z W1204 11:17:34.691000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/compile_fx_ext.py", line 210, in serialize 2025-12-04T11:21:33.5811362Z W1204 11:17:34.691000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] return _WireProtocolPickledInput(GraphPickler.dumps(self)) 2025-12-04T11:21:33.5811861Z W1204 11:17:34.691000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 127, in dumps 2025-12-04T11:21:33.5812096Z W1204 11:17:34.691000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] pickler.dump(obj) 2025-12-04T11:21:33.5812569Z W1204 11:17:34.691000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 94, in reducer_override 2025-12-04T11:21:33.5812889Z W1204 11:17:34.691000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] return _GraphModulePickleData.reduce_helper(self, obj) 2025-12-04T11:21:33.5813355Z W1204 11:17:34.691000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 352, in reduce_helper 2025-12-04T11:21:33.5813615Z W1204 11:17:34.691000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] cls(obj, pickler.options), 2025-12-04T11:21:33.5814160Z W1204 11:17:34.691000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 364, in __init__ 2025-12-04T11:21:33.5814466Z W1204 11:17:34.691000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] self.graph = _GraphPickleData(gm._graph, options) 2025-12-04T11:21:33.5814918Z W1204 11:17:34.691000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 571, in __init__ 2025-12-04T11:21:33.5815257Z W1204 11:17:34.691000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] nodes[node] = _NodePickleData(node, nodes, options) 2025-12-04T11:21:33.5815712Z W1204 11:17:34.691000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 387, in __init__ 2025-12-04T11:21:33.5816031Z W1204 11:17:34.691000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] self.target = _OpPickleData.pickle(node.target, options) 2025-12-04T11:21:33.5816480Z W1204 11:17:34.691000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 435, in pickle 2025-12-04T11:21:33.5816808Z W1204 11:17:34.691000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] return cls._pickle_op(name, _OpOverloadPickleData, options) 2025-12-04T11:21:33.5817269Z W1204 11:17:34.691000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 456, in _pickle_op 2025-12-04T11:21:33.5817629Z W1204 11:17:34.691000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] raise BypassFxGraphCache(f"Unable to pickle non-standard op: {name}") 2025-12-04T11:21:33.5818193Z W1204 11:17:34.691000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] torch._inductor.codecache.BypassFxGraphCache: Unable to pickle non-standard op: torch.ops.prims._low_memory_max_pool_with_offsets.default 2025-12-04T11:21:33.5818488Z W1204 11:17:35.326000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] Unable to pickle input graph or example inputs 2025-12-04T11:21:33.5818749Z W1204 11:17:35.326000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] Traceback (most recent call last): 2025-12-04T11:21:33.5819255Z W1204 11:17:35.326000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/compile_fx_ext.py", line 486, in serialize_compile 2025-12-04T11:21:33.5819470Z W1204 11:17:35.326000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] ).serialize() 2025-12-04T11:21:33.5819985Z W1204 11:17:35.326000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/compile_fx_ext.py", line 210, in serialize 2025-12-04T11:21:33.5820312Z W1204 11:17:35.326000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] return _WireProtocolPickledInput(GraphPickler.dumps(self)) 2025-12-04T11:21:33.5820754Z W1204 11:17:35.326000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 127, in dumps 2025-12-04T11:21:33.5820991Z W1204 11:17:35.326000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] pickler.dump(obj) 2025-12-04T11:21:33.5821474Z W1204 11:17:35.326000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 94, in reducer_override 2025-12-04T11:21:33.5821797Z W1204 11:17:35.326000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] return _GraphModulePickleData.reduce_helper(self, obj) 2025-12-04T11:21:33.5822331Z W1204 11:17:35.326000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 352, in reduce_helper 2025-12-04T11:21:33.5822588Z W1204 11:17:35.326000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] cls(obj, pickler.options), 2025-12-04T11:21:33.5823042Z W1204 11:17:35.326000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 364, in __init__ 2025-12-04T11:21:33.5823371Z W1204 11:17:35.326000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] self.graph = _GraphPickleData(gm._graph, options) 2025-12-04T11:21:33.5823887Z W1204 11:17:35.326000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 571, in __init__ 2025-12-04T11:21:33.5824188Z W1204 11:17:35.326000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] nodes[node] = _NodePickleData(node, nodes, options) 2025-12-04T11:21:33.5824643Z W1204 11:17:35.326000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 387, in __init__ 2025-12-04T11:21:33.5824961Z W1204 11:17:35.326000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] self.target = _OpPickleData.pickle(node.target, options) 2025-12-04T11:21:33.5825408Z W1204 11:17:35.326000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 435, in pickle 2025-12-04T11:21:33.5825736Z W1204 11:17:35.326000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] return cls._pickle_op(name, _OpOverloadPickleData, options) 2025-12-04T11:21:33.5826195Z W1204 11:17:35.326000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 456, in _pickle_op 2025-12-04T11:21:33.5826553Z W1204 11:17:35.326000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] raise BypassFxGraphCache(f"Unable to pickle non-standard op: {name}") 2025-12-04T11:21:33.5827099Z W1204 11:17:35.326000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] torch._inductor.codecache.BypassFxGraphCache: Unable to pickle non-standard op: torch.ops.prims._low_memory_max_pool_with_offsets.default 2025-12-04T11:21:33.5827167Z PASSED [1.2674s] [ 55%] 2025-12-04T11:21:33.5827818Z inductor/test_compile_subprocess.py::GPUTests::test_low_memory_max_pool_dilation_2_dim_3_cuda <- test/inductor/test_torchinductor.py W1204 11:17:35.971000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] Unable to pickle input graph or example inputs 2025-12-04T11:21:33.5828135Z W1204 11:17:35.971000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] Traceback (most recent call last): 2025-12-04T11:21:33.5828644Z W1204 11:17:35.971000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/compile_fx_ext.py", line 486, in serialize_compile 2025-12-04T11:21:33.5828861Z W1204 11:17:35.971000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] ).serialize() 2025-12-04T11:21:33.5829335Z W1204 11:17:35.971000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/compile_fx_ext.py", line 210, in serialize 2025-12-04T11:21:33.5829663Z W1204 11:17:35.971000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] return _WireProtocolPickledInput(GraphPickler.dumps(self)) 2025-12-04T11:21:33.5830189Z W1204 11:17:35.971000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 127, in dumps 2025-12-04T11:21:33.5830420Z W1204 11:17:35.971000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] pickler.dump(obj) 2025-12-04T11:21:33.5830893Z W1204 11:17:35.971000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 94, in reducer_override 2025-12-04T11:21:33.5831211Z W1204 11:17:35.971000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] return _GraphModulePickleData.reduce_helper(self, obj) 2025-12-04T11:21:33.5831719Z W1204 11:17:35.971000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 352, in reduce_helper 2025-12-04T11:21:33.5831984Z W1204 11:17:35.971000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] cls(obj, pickler.options), 2025-12-04T11:21:33.5832437Z W1204 11:17:35.971000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 364, in __init__ 2025-12-04T11:21:33.5832735Z W1204 11:17:35.971000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] self.graph = _GraphPickleData(gm._graph, options) 2025-12-04T11:21:33.5833190Z W1204 11:17:35.971000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 571, in __init__ 2025-12-04T11:21:33.5833488Z W1204 11:17:35.971000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] nodes[node] = _NodePickleData(node, nodes, options) 2025-12-04T11:21:33.5833942Z W1204 11:17:35.971000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 387, in __init__ 2025-12-04T11:21:33.5834264Z W1204 11:17:35.971000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] self.target = _OpPickleData.pickle(node.target, options) 2025-12-04T11:21:33.5834711Z W1204 11:17:35.971000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 435, in pickle 2025-12-04T11:21:33.5835037Z W1204 11:17:35.971000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] return cls._pickle_op(name, _OpOverloadPickleData, options) 2025-12-04T11:21:33.5835496Z W1204 11:17:35.971000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 456, in _pickle_op 2025-12-04T11:21:33.5835906Z W1204 11:17:35.971000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] raise BypassFxGraphCache(f"Unable to pickle non-standard op: {name}") 2025-12-04T11:21:33.5836459Z W1204 11:17:35.971000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] torch._inductor.codecache.BypassFxGraphCache: Unable to pickle non-standard op: torch.ops.prims._low_memory_max_pool_with_offsets.default 2025-12-04T11:21:33.5836758Z W1204 11:17:37.984000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] Unable to pickle input graph or example inputs 2025-12-04T11:21:33.5837021Z W1204 11:17:37.984000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] Traceback (most recent call last): 2025-12-04T11:21:33.5837530Z W1204 11:17:37.984000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/compile_fx_ext.py", line 486, in serialize_compile 2025-12-04T11:21:33.5837747Z W1204 11:17:37.984000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] ).serialize() 2025-12-04T11:21:33.5838289Z W1204 11:17:37.984000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/compile_fx_ext.py", line 210, in serialize 2025-12-04T11:21:33.5838623Z W1204 11:17:37.984000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] return _WireProtocolPickledInput(GraphPickler.dumps(self)) 2025-12-04T11:21:33.5839067Z W1204 11:17:37.984000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 127, in dumps 2025-12-04T11:21:33.5839304Z W1204 11:17:37.984000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] pickler.dump(obj) 2025-12-04T11:21:33.5839809Z W1204 11:17:37.984000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 94, in reducer_override 2025-12-04T11:21:33.5840134Z W1204 11:17:37.984000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] return _GraphModulePickleData.reduce_helper(self, obj) 2025-12-04T11:21:33.5840602Z W1204 11:17:37.984000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 352, in reduce_helper 2025-12-04T11:21:33.5840855Z W1204 11:17:37.984000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] cls(obj, pickler.options), 2025-12-04T11:21:33.5841309Z W1204 11:17:37.984000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 364, in __init__ 2025-12-04T11:21:33.5841605Z W1204 11:17:37.984000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] self.graph = _GraphPickleData(gm._graph, options) 2025-12-04T11:21:33.5842063Z W1204 11:17:37.984000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 571, in __init__ 2025-12-04T11:21:33.5842364Z W1204 11:17:37.984000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] nodes[node] = _NodePickleData(node, nodes, options) 2025-12-04T11:21:33.5842814Z W1204 11:17:37.984000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 387, in __init__ 2025-12-04T11:21:33.5843128Z W1204 11:17:37.984000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] self.target = _OpPickleData.pickle(node.target, options) 2025-12-04T11:21:33.5843576Z W1204 11:17:37.984000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 435, in pickle 2025-12-04T11:21:33.5844245Z W1204 11:17:37.984000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] return cls._pickle_op(name, _OpOverloadPickleData, options) 2025-12-04T11:21:33.5844738Z W1204 11:17:37.984000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 456, in _pickle_op 2025-12-04T11:21:33.5845099Z W1204 11:17:37.984000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] raise BypassFxGraphCache(f"Unable to pickle non-standard op: {name}") 2025-12-04T11:21:33.5845649Z W1204 11:17:37.984000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] torch._inductor.codecache.BypassFxGraphCache: Unable to pickle non-standard op: torch.ops.prims._low_memory_max_pool_with_offsets.default 2025-12-04T11:21:33.5845715Z PASSED [4.0722s] [ 56%] 2025-12-04T11:21:33.5846055Z inductor/test_compile_subprocess.py::GPUTests::test_masked_fill_cuda <- test/inductor/test_torchinductor.py PASSED [0.6706s] [ 56%] 2025-12-04T11:21:33.5846436Z inductor/test_compile_subprocess.py::GPUTests::test_masked_scatter_cuda <- test/inductor/test_torchinductor.py PASSED [0.4797s] [ 56%] 2025-12-04T11:21:33.5847069Z inductor/test_compile_subprocess.py::GPUTests::test_max_pool2d1_cuda <- test/inductor/test_torchinductor.py W1204 11:17:41.180000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] Unable to pickle input graph or example inputs 2025-12-04T11:21:33.5847334Z W1204 11:17:41.180000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] Traceback (most recent call last): 2025-12-04T11:21:33.5847843Z W1204 11:17:41.180000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/compile_fx_ext.py", line 486, in serialize_compile 2025-12-04T11:21:33.5848105Z W1204 11:17:41.180000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] ).serialize() 2025-12-04T11:21:33.5848592Z W1204 11:17:41.180000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/compile_fx_ext.py", line 210, in serialize 2025-12-04T11:21:33.5848930Z W1204 11:17:41.180000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] return _WireProtocolPickledInput(GraphPickler.dumps(self)) 2025-12-04T11:21:33.5849375Z W1204 11:17:41.180000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 127, in dumps 2025-12-04T11:21:33.5849608Z W1204 11:17:41.180000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] pickler.dump(obj) 2025-12-04T11:21:33.5850081Z W1204 11:17:41.180000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 94, in reducer_override 2025-12-04T11:21:33.5850403Z W1204 11:17:41.180000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] return _GraphModulePickleData.reduce_helper(self, obj) 2025-12-04T11:21:33.5850873Z W1204 11:17:41.180000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 352, in reduce_helper 2025-12-04T11:21:33.5851126Z W1204 11:17:41.180000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] cls(obj, pickler.options), 2025-12-04T11:21:33.5851583Z W1204 11:17:41.180000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 364, in __init__ 2025-12-04T11:21:33.5851884Z W1204 11:17:41.180000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] self.graph = _GraphPickleData(gm._graph, options) 2025-12-04T11:21:33.5852336Z W1204 11:17:41.180000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 571, in __init__ 2025-12-04T11:21:33.5852686Z W1204 11:17:41.180000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] nodes[node] = _NodePickleData(node, nodes, options) 2025-12-04T11:21:33.5853138Z W1204 11:17:41.180000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 387, in __init__ 2025-12-04T11:21:33.5853456Z W1204 11:17:41.180000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] self.target = _OpPickleData.pickle(node.target, options) 2025-12-04T11:21:33.5853900Z W1204 11:17:41.180000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 435, in pickle 2025-12-04T11:21:33.5854230Z W1204 11:17:41.180000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] return cls._pickle_op(name, _OpOverloadPickleData, options) 2025-12-04T11:21:33.5854842Z W1204 11:17:41.180000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 456, in _pickle_op 2025-12-04T11:21:33.5855197Z W1204 11:17:41.180000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] raise BypassFxGraphCache(f"Unable to pickle non-standard op: {name}") 2025-12-04T11:21:33.5855747Z W1204 11:17:41.180000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] torch._inductor.codecache.BypassFxGraphCache: Unable to pickle non-standard op: torch.ops.prims._low_memory_max_pool_with_offsets.default 2025-12-04T11:21:33.5856078Z W1204 11:17:41.899000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] Unable to pickle input graph or example inputs 2025-12-04T11:21:33.5856343Z W1204 11:17:41.899000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] Traceback (most recent call last): 2025-12-04T11:21:33.5856849Z W1204 11:17:41.899000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/compile_fx_ext.py", line 486, in serialize_compile 2025-12-04T11:21:33.5857072Z W1204 11:17:41.899000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] ).serialize() 2025-12-04T11:21:33.5857547Z W1204 11:17:41.899000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/compile_fx_ext.py", line 210, in serialize 2025-12-04T11:21:33.5857878Z W1204 11:17:41.899000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] return _WireProtocolPickledInput(GraphPickler.dumps(self)) 2025-12-04T11:21:33.5858323Z W1204 11:17:41.899000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 127, in dumps 2025-12-04T11:21:33.5858549Z W1204 11:17:41.899000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] pickler.dump(obj) 2025-12-04T11:21:33.5859020Z W1204 11:17:41.899000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 94, in reducer_override 2025-12-04T11:21:33.5859332Z W1204 11:17:41.899000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] return _GraphModulePickleData.reduce_helper(self, obj) 2025-12-04T11:21:33.5859801Z W1204 11:17:41.899000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 352, in reduce_helper 2025-12-04T11:21:33.5860055Z W1204 11:17:41.899000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] cls(obj, pickler.options), 2025-12-04T11:21:33.5860550Z W1204 11:17:41.899000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 364, in __init__ 2025-12-04T11:21:33.5860848Z W1204 11:17:41.899000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] self.graph = _GraphPickleData(gm._graph, options) 2025-12-04T11:21:33.5861295Z W1204 11:17:41.899000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 571, in __init__ 2025-12-04T11:21:33.5861596Z W1204 11:17:41.899000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] nodes[node] = _NodePickleData(node, nodes, options) 2025-12-04T11:21:33.5862043Z W1204 11:17:41.899000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 387, in __init__ 2025-12-04T11:21:33.5862358Z W1204 11:17:41.899000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] self.target = _OpPickleData.pickle(node.target, options) 2025-12-04T11:21:33.5862871Z W1204 11:17:41.899000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 435, in pickle 2025-12-04T11:21:33.5863200Z W1204 11:17:41.899000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] return cls._pickle_op(name, _OpOverloadPickleData, options) 2025-12-04T11:21:33.5863712Z W1204 11:17:41.899000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 456, in _pickle_op 2025-12-04T11:21:33.5864101Z W1204 11:17:41.899000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] raise BypassFxGraphCache(f"Unable to pickle non-standard op: {name}") 2025-12-04T11:21:33.5864659Z W1204 11:17:41.899000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] torch._inductor.codecache.BypassFxGraphCache: Unable to pickle non-standard op: torch.ops.prims._low_memory_max_pool_with_offsets.default 2025-12-04T11:21:33.5864724Z PASSED [1.4240s] [ 56%] 2025-12-04T11:21:33.5865327Z inductor/test_compile_subprocess.py::GPUTests::test_max_pool2d3_cuda <- test/inductor/test_torchinductor.py W1204 11:17:42.605000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] Unable to pickle input graph or example inputs 2025-12-04T11:21:33.5865594Z W1204 11:17:42.605000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] Traceback (most recent call last): 2025-12-04T11:21:33.5866284Z W1204 11:17:42.605000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/compile_fx_ext.py", line 486, in serialize_compile 2025-12-04T11:21:33.5866515Z W1204 11:17:42.605000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] ).serialize() 2025-12-04T11:21:33.5866998Z W1204 11:17:42.605000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/compile_fx_ext.py", line 210, in serialize 2025-12-04T11:21:33.5867331Z W1204 11:17:42.605000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] return _WireProtocolPickledInput(GraphPickler.dumps(self)) 2025-12-04T11:21:33.5867785Z W1204 11:17:42.605000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 127, in dumps 2025-12-04T11:21:33.5868022Z W1204 11:17:42.605000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] pickler.dump(obj) 2025-12-04T11:21:33.5868501Z W1204 11:17:42.605000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 94, in reducer_override 2025-12-04T11:21:33.5868871Z W1204 11:17:42.605000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] return _GraphModulePickleData.reduce_helper(self, obj) 2025-12-04T11:21:33.5869344Z W1204 11:17:42.605000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 352, in reduce_helper 2025-12-04T11:21:33.5869598Z W1204 11:17:42.605000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] cls(obj, pickler.options), 2025-12-04T11:21:33.5870053Z W1204 11:17:42.605000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 364, in __init__ 2025-12-04T11:21:33.5870352Z W1204 11:17:42.605000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] self.graph = _GraphPickleData(gm._graph, options) 2025-12-04T11:21:33.5870810Z W1204 11:17:42.605000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 571, in __init__ 2025-12-04T11:21:33.5871184Z W1204 11:17:42.605000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] nodes[node] = _NodePickleData(node, nodes, options) 2025-12-04T11:21:33.5871635Z W1204 11:17:42.605000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 387, in __init__ 2025-12-04T11:21:33.5871950Z W1204 11:17:42.605000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] self.target = _OpPickleData.pickle(node.target, options) 2025-12-04T11:21:33.5872448Z W1204 11:17:42.605000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 435, in pickle 2025-12-04T11:21:33.5872780Z W1204 11:17:42.605000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] return cls._pickle_op(name, _OpOverloadPickleData, options) 2025-12-04T11:21:33.5873241Z W1204 11:17:42.605000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 456, in _pickle_op 2025-12-04T11:21:33.5873595Z W1204 11:17:42.605000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] raise BypassFxGraphCache(f"Unable to pickle non-standard op: {name}") 2025-12-04T11:21:33.5874142Z W1204 11:17:42.605000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] torch._inductor.codecache.BypassFxGraphCache: Unable to pickle non-standard op: torch.ops.prims._low_memory_max_pool_with_offsets.default 2025-12-04T11:21:33.5874436Z W1204 11:17:43.808000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] Unable to pickle input graph or example inputs 2025-12-04T11:21:33.5874704Z W1204 11:17:43.808000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] Traceback (most recent call last): 2025-12-04T11:21:33.5875207Z W1204 11:17:43.808000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/compile_fx_ext.py", line 486, in serialize_compile 2025-12-04T11:21:33.5875426Z W1204 11:17:43.808000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] ).serialize() 2025-12-04T11:21:33.5875899Z W1204 11:17:43.808000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/compile_fx_ext.py", line 210, in serialize 2025-12-04T11:21:33.5876224Z W1204 11:17:43.808000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] return _WireProtocolPickledInput(GraphPickler.dumps(self)) 2025-12-04T11:21:33.5876670Z W1204 11:17:43.808000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 127, in dumps 2025-12-04T11:21:33.5876942Z W1204 11:17:43.808000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] pickler.dump(obj) 2025-12-04T11:21:33.5877418Z W1204 11:17:43.808000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 94, in reducer_override 2025-12-04T11:21:33.5877729Z W1204 11:17:43.808000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] return _GraphModulePickleData.reduce_helper(self, obj) 2025-12-04T11:21:33.5878372Z W1204 11:17:43.808000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 352, in reduce_helper 2025-12-04T11:21:33.5878626Z W1204 11:17:43.808000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] cls(obj, pickler.options), 2025-12-04T11:21:33.5879078Z W1204 11:17:43.808000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 364, in __init__ 2025-12-04T11:21:33.5879511Z W1204 11:17:43.808000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] self.graph = _GraphPickleData(gm._graph, options) 2025-12-04T11:21:33.5879959Z W1204 11:17:43.808000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 571, in __init__ 2025-12-04T11:21:33.5880256Z W1204 11:17:43.808000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] nodes[node] = _NodePickleData(node, nodes, options) 2025-12-04T11:21:33.5880748Z W1204 11:17:43.808000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 387, in __init__ 2025-12-04T11:21:33.5881082Z W1204 11:17:43.808000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] self.target = _OpPickleData.pickle(node.target, options) 2025-12-04T11:21:33.5881537Z W1204 11:17:43.808000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 435, in pickle 2025-12-04T11:21:33.5881863Z W1204 11:17:43.808000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] return cls._pickle_op(name, _OpOverloadPickleData, options) 2025-12-04T11:21:33.5882324Z W1204 11:17:43.808000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 456, in _pickle_op 2025-12-04T11:21:33.5882682Z W1204 11:17:43.808000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] raise BypassFxGraphCache(f"Unable to pickle non-standard op: {name}") 2025-12-04T11:21:33.5883238Z W1204 11:17:43.808000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] torch._inductor.codecache.BypassFxGraphCache: Unable to pickle non-standard op: torch.ops.prims._low_memory_max_pool_with_offsets.default 2025-12-04T11:21:33.5883305Z PASSED [1.9932s] [ 57%] 2025-12-04T11:21:33.5883910Z inductor/test_compile_subprocess.py::GPUTests::test_max_pool2d4_cuda <- test/inductor/test_torchinductor.py W1204 11:17:44.605000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] Unable to pickle input graph or example inputs 2025-12-04T11:21:33.5884171Z W1204 11:17:44.605000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] Traceback (most recent call last): 2025-12-04T11:21:33.5884678Z W1204 11:17:44.605000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/compile_fx_ext.py", line 486, in serialize_compile 2025-12-04T11:21:33.5884908Z W1204 11:17:44.605000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] ).serialize() 2025-12-04T11:21:33.5885440Z W1204 11:17:44.605000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/compile_fx_ext.py", line 210, in serialize 2025-12-04T11:21:33.5885768Z W1204 11:17:44.605000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] return _WireProtocolPickledInput(GraphPickler.dumps(self)) 2025-12-04T11:21:33.5886213Z W1204 11:17:44.605000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 127, in dumps 2025-12-04T11:21:33.5886442Z W1204 11:17:44.605000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] pickler.dump(obj) 2025-12-04T11:21:33.5886917Z W1204 11:17:44.605000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 94, in reducer_override 2025-12-04T11:21:33.5887280Z W1204 11:17:44.605000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] return _GraphModulePickleData.reduce_helper(self, obj) 2025-12-04T11:21:33.5887784Z W1204 11:17:44.605000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 352, in reduce_helper 2025-12-04T11:21:33.5888036Z W1204 11:17:44.605000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] cls(obj, pickler.options), 2025-12-04T11:21:33.5888487Z W1204 11:17:44.605000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 364, in __init__ 2025-12-04T11:21:33.5888825Z W1204 11:17:44.605000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] self.graph = _GraphPickleData(gm._graph, options) 2025-12-04T11:21:33.5889278Z W1204 11:17:44.605000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 571, in __init__ 2025-12-04T11:21:33.5889579Z W1204 11:17:44.605000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] nodes[node] = _NodePickleData(node, nodes, options) 2025-12-04T11:21:33.5890023Z W1204 11:17:44.605000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 387, in __init__ 2025-12-04T11:21:33.5890342Z W1204 11:17:44.605000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] self.target = _OpPickleData.pickle(node.target, options) 2025-12-04T11:21:33.5890784Z W1204 11:17:44.605000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 435, in pickle 2025-12-04T11:21:33.5891111Z W1204 11:17:44.605000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] return cls._pickle_op(name, _OpOverloadPickleData, options) 2025-12-04T11:21:33.5891568Z W1204 11:17:44.605000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 456, in _pickle_op 2025-12-04T11:21:33.5891923Z W1204 11:17:44.605000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] raise BypassFxGraphCache(f"Unable to pickle non-standard op: {name}") 2025-12-04T11:21:33.5892476Z W1204 11:17:44.605000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] torch._inductor.codecache.BypassFxGraphCache: Unable to pickle non-standard op: torch.ops.prims._low_memory_max_pool_with_offsets.default 2025-12-04T11:21:33.5892765Z W1204 11:17:45.425000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] Unable to pickle input graph or example inputs 2025-12-04T11:21:33.5893063Z W1204 11:17:45.425000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] Traceback (most recent call last): 2025-12-04T11:21:33.5893565Z W1204 11:17:45.425000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/compile_fx_ext.py", line 486, in serialize_compile 2025-12-04T11:21:33.5893784Z W1204 11:17:45.425000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] ).serialize() 2025-12-04T11:21:33.5894252Z W1204 11:17:45.425000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/compile_fx_ext.py", line 210, in serialize 2025-12-04T11:21:33.5894587Z W1204 11:17:45.425000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] return _WireProtocolPickledInput(GraphPickler.dumps(self)) 2025-12-04T11:21:33.5895036Z W1204 11:17:45.425000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 127, in dumps 2025-12-04T11:21:33.5895335Z W1204 11:17:45.425000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] pickler.dump(obj) 2025-12-04T11:21:33.5895809Z W1204 11:17:45.425000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 94, in reducer_override 2025-12-04T11:21:33.5896122Z W1204 11:17:45.425000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] return _GraphModulePickleData.reduce_helper(self, obj) 2025-12-04T11:21:33.5896585Z W1204 11:17:45.425000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 352, in reduce_helper 2025-12-04T11:21:33.5896878Z W1204 11:17:45.425000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] cls(obj, pickler.options), 2025-12-04T11:21:33.5897336Z W1204 11:17:45.425000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 364, in __init__ 2025-12-04T11:21:33.5897633Z W1204 11:17:45.425000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] self.graph = _GraphPickleData(gm._graph, options) 2025-12-04T11:21:33.5898079Z W1204 11:17:45.425000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 571, in __init__ 2025-12-04T11:21:33.5898377Z W1204 11:17:45.425000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] nodes[node] = _NodePickleData(node, nodes, options) 2025-12-04T11:21:33.5898825Z W1204 11:17:45.425000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 387, in __init__ 2025-12-04T11:21:33.5899142Z W1204 11:17:45.425000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] self.target = _OpPickleData.pickle(node.target, options) 2025-12-04T11:21:33.5899598Z W1204 11:17:45.425000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 435, in pickle 2025-12-04T11:21:33.5899921Z W1204 11:17:45.425000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] return cls._pickle_op(name, _OpOverloadPickleData, options) 2025-12-04T11:21:33.5900378Z W1204 11:17:45.425000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 456, in _pickle_op 2025-12-04T11:21:33.5900732Z W1204 11:17:45.425000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] raise BypassFxGraphCache(f"Unable to pickle non-standard op: {name}") 2025-12-04T11:21:33.5901323Z W1204 11:17:45.425000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] torch._inductor.codecache.BypassFxGraphCache: Unable to pickle non-standard op: torch.ops.prims._low_memory_max_pool_with_offsets.default 2025-12-04T11:21:33.5901389Z PASSED [1.6186s] [ 57%] 2025-12-04T11:21:33.5901991Z inductor/test_compile_subprocess.py::GPUTests::test_max_pool2d5_cuda <- test/inductor/test_torchinductor.py W1204 11:17:46.239000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] Unable to pickle input graph or example inputs 2025-12-04T11:21:33.5902256Z W1204 11:17:46.239000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] Traceback (most recent call last): 2025-12-04T11:21:33.5902765Z W1204 11:17:46.239000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/compile_fx_ext.py", line 486, in serialize_compile 2025-12-04T11:21:33.5902984Z W1204 11:17:46.239000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] ).serialize() 2025-12-04T11:21:33.5903609Z W1204 11:17:46.239000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/compile_fx_ext.py", line 210, in serialize 2025-12-04T11:21:33.5903947Z W1204 11:17:46.239000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] return _WireProtocolPickledInput(GraphPickler.dumps(self)) 2025-12-04T11:21:33.5904391Z W1204 11:17:46.239000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 127, in dumps 2025-12-04T11:21:33.5904624Z W1204 11:17:46.239000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] pickler.dump(obj) 2025-12-04T11:21:33.5905134Z W1204 11:17:46.239000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 94, in reducer_override 2025-12-04T11:21:33.5905456Z W1204 11:17:46.239000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] return _GraphModulePickleData.reduce_helper(self, obj) 2025-12-04T11:21:33.5905926Z W1204 11:17:46.239000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 352, in reduce_helper 2025-12-04T11:21:33.5906181Z W1204 11:17:46.239000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] cls(obj, pickler.options), 2025-12-04T11:21:33.5906632Z W1204 11:17:46.239000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 364, in __init__ 2025-12-04T11:21:33.5906931Z W1204 11:17:46.239000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] self.graph = _GraphPickleData(gm._graph, options) 2025-12-04T11:21:33.5907381Z W1204 11:17:46.239000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 571, in __init__ 2025-12-04T11:21:33.5907683Z W1204 11:17:46.239000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] nodes[node] = _NodePickleData(node, nodes, options) 2025-12-04T11:21:33.5908140Z W1204 11:17:46.239000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 387, in __init__ 2025-12-04T11:21:33.5908459Z W1204 11:17:46.239000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] self.target = _OpPickleData.pickle(node.target, options) 2025-12-04T11:21:33.5908905Z W1204 11:17:46.239000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 435, in pickle 2025-12-04T11:21:33.5909269Z W1204 11:17:46.239000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] return cls._pickle_op(name, _OpOverloadPickleData, options) 2025-12-04T11:21:33.5909727Z W1204 11:17:46.239000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 456, in _pickle_op 2025-12-04T11:21:33.5910084Z W1204 11:17:46.239000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] raise BypassFxGraphCache(f"Unable to pickle non-standard op: {name}") 2025-12-04T11:21:33.5910641Z W1204 11:17:46.239000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] torch._inductor.codecache.BypassFxGraphCache: Unable to pickle non-standard op: torch.ops.prims._low_memory_max_pool_with_offsets.default 2025-12-04T11:21:33.5910932Z W1204 11:17:47.122000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] Unable to pickle input graph or example inputs 2025-12-04T11:21:33.5911200Z W1204 11:17:47.122000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] Traceback (most recent call last): 2025-12-04T11:21:33.5911776Z W1204 11:17:47.122000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/compile_fx_ext.py", line 486, in serialize_compile 2025-12-04T11:21:33.5911999Z W1204 11:17:47.122000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] ).serialize() 2025-12-04T11:21:33.5912709Z W1204 11:17:47.122000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/compile_fx_ext.py", line 210, in serialize 2025-12-04T11:21:33.5913100Z W1204 11:17:47.122000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] return _WireProtocolPickledInput(GraphPickler.dumps(self)) 2025-12-04T11:21:33.5913553Z W1204 11:17:47.122000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 127, in dumps 2025-12-04T11:21:33.5913787Z W1204 11:17:47.122000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] pickler.dump(obj) 2025-12-04T11:21:33.5914258Z W1204 11:17:47.122000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 94, in reducer_override 2025-12-04T11:21:33.5914574Z W1204 11:17:47.122000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] return _GraphModulePickleData.reduce_helper(self, obj) 2025-12-04T11:21:33.5915040Z W1204 11:17:47.122000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 352, in reduce_helper 2025-12-04T11:21:33.5915299Z W1204 11:17:47.122000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] cls(obj, pickler.options), 2025-12-04T11:21:33.5915754Z W1204 11:17:47.122000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 364, in __init__ 2025-12-04T11:21:33.5916054Z W1204 11:17:47.122000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] self.graph = _GraphPickleData(gm._graph, options) 2025-12-04T11:21:33.5916501Z W1204 11:17:47.122000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 571, in __init__ 2025-12-04T11:21:33.5916800Z W1204 11:17:47.122000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] nodes[node] = _NodePickleData(node, nodes, options) 2025-12-04T11:21:33.5917264Z W1204 11:17:47.122000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 387, in __init__ 2025-12-04T11:21:33.5917626Z W1204 11:17:47.122000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] self.target = _OpPickleData.pickle(node.target, options) 2025-12-04T11:21:33.5918080Z W1204 11:17:47.122000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 435, in pickle 2025-12-04T11:21:33.5918403Z W1204 11:17:47.122000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] return cls._pickle_op(name, _OpOverloadPickleData, options) 2025-12-04T11:21:33.5918861Z W1204 11:17:47.122000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 456, in _pickle_op 2025-12-04T11:21:33.5919216Z W1204 11:17:47.122000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] raise BypassFxGraphCache(f"Unable to pickle non-standard op: {name}") 2025-12-04T11:21:33.5919846Z W1204 11:17:47.122000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] torch._inductor.codecache.BypassFxGraphCache: Unable to pickle non-standard op: torch.ops.prims._low_memory_max_pool_with_offsets.default 2025-12-04T11:21:33.5919912Z PASSED [1.7815s] [ 57%] 2025-12-04T11:21:33.5920265Z inductor/test_compile_subprocess.py::GPUTests::test_max_pool2d6_dilation_1_cuda <- test/inductor/test_torchinductor.py PASSED [2.4227s] [ 57%] 2025-12-04T11:21:33.5920640Z inductor/test_compile_subprocess.py::GPUTests::test_max_pool2d_with_indices_backward2_cuda <- test/inductor/test_torchinductor.py PASSED [6.3621s] [ 57%] 2025-12-04T11:21:33.5921005Z inductor/test_compile_subprocess.py::GPUTests::test_max_pool2d_with_indices_backward3_cuda <- test/inductor/test_torchinductor.py PASSED [3.2867s] [ 58%] 2025-12-04T11:21:33.5921410Z inductor/test_compile_subprocess.py::GPUTests::test_max_pool2d_with_indices_backward4_cuda <- test/inductor/test_torchinductor.py PASSED [11.1210s] [ 58%] 2025-12-04T11:21:33.5921778Z inductor/test_compile_subprocess.py::GPUTests::test_max_pool2d_with_indices_backward6_cuda <- test/inductor/test_torchinductor.py PASSED [0.2243s] [ 58%] 2025-12-04T11:21:33.5922140Z inductor/test_compile_subprocess.py::GPUTests::test_max_pool2d_with_indices_backward_cuda <- test/inductor/test_torchinductor.py PASSED [2.2677s] [ 58%] 2025-12-04T11:21:33.5922436Z inductor/test_compile_subprocess.py::GPUTests::test_mean_cuda <- test/inductor/test_torchinductor.py PASSED [0.7239s] [ 59%] 2025-12-04T11:21:33.5922758Z inductor/test_compile_subprocess.py::GPUTests::test_min_max_reduction_cuda <- test/inductor/test_torchinductor.py PASSED [1.0914s] [ 59%] 2025-12-04T11:21:33.5923111Z inductor/test_compile_subprocess.py::GPUTests::test_misaligned_address_issue1_cuda <- test/inductor/test_torchinductor.py PASSED [0.3641s] [ 59%] 2025-12-04T11:21:33.5923718Z inductor/test_compile_subprocess.py::GPUTests::test_mix_device_index_cuda <- test/inductor/test_torchinductor.py W1204 11:18:15.877000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] Unable to pickle input graph or example inputs 2025-12-04T11:21:33.5923990Z W1204 11:18:15.877000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] Traceback (most recent call last): 2025-12-04T11:21:33.5924496Z W1204 11:18:15.877000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/compile_fx_ext.py", line 486, in serialize_compile 2025-12-04T11:21:33.5924718Z W1204 11:18:15.877000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] ).serialize() 2025-12-04T11:21:33.5925201Z W1204 11:18:15.877000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/compile_fx_ext.py", line 210, in serialize 2025-12-04T11:21:33.5925532Z W1204 11:18:15.877000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] return _WireProtocolPickledInput(GraphPickler.dumps(self)) 2025-12-04T11:21:33.5926027Z W1204 11:18:15.877000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 127, in dumps 2025-12-04T11:21:33.5926257Z W1204 11:18:15.877000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] pickler.dump(obj) 2025-12-04T11:21:33.5926729Z W1204 11:18:15.877000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 94, in reducer_override 2025-12-04T11:21:33.5927045Z W1204 11:18:15.877000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] return _GraphModulePickleData.reduce_helper(self, obj) 2025-12-04T11:21:33.5927510Z W1204 11:18:15.877000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 352, in reduce_helper 2025-12-04T11:21:33.5927818Z W1204 11:18:15.877000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] cls(obj, pickler.options), 2025-12-04T11:21:33.5928301Z W1204 11:18:15.877000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 364, in __init__ 2025-12-04T11:21:33.5928602Z W1204 11:18:15.877000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] self.graph = _GraphPickleData(gm._graph, options) 2025-12-04T11:21:33.5929060Z W1204 11:18:15.877000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 571, in __init__ 2025-12-04T11:21:33.5929394Z W1204 11:18:15.877000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] nodes[node] = _NodePickleData(node, nodes, options) 2025-12-04T11:21:33.5929845Z W1204 11:18:15.877000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 387, in __init__ 2025-12-04T11:21:33.5930165Z W1204 11:18:15.877000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] self.target = _OpPickleData.pickle(node.target, options) 2025-12-04T11:21:33.5930610Z W1204 11:18:15.877000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 435, in pickle 2025-12-04T11:21:33.5930937Z W1204 11:18:15.877000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] return cls._pickle_op(name, _OpOverloadPickleData, options) 2025-12-04T11:21:33.5931401Z W1204 11:18:15.877000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 456, in _pickle_op 2025-12-04T11:21:33.5931760Z W1204 11:18:15.877000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] raise BypassFxGraphCache(f"Unable to pickle non-standard op: {name}") 2025-12-04T11:21:33.5932234Z W1204 11:18:15.877000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] torch._inductor.codecache.BypassFxGraphCache: Unable to pickle non-standard op: torch.ops.prims.iota.default 2025-12-04T11:21:33.5932302Z PASSED [0.1919s] [ 59%] 2025-12-04T11:21:33.5932890Z inductor/test_compile_subprocess.py::GPUTests::test_mixed_mm3_cuda <- test/inductor/test_torchinductor.py W1204 11:18:16.055000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] Unable to pickle input graph or example inputs 2025-12-04T11:21:33.5933156Z W1204 11:18:16.055000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] Traceback (most recent call last): 2025-12-04T11:21:33.5933654Z W1204 11:18:16.055000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/compile_fx_ext.py", line 486, in serialize_compile 2025-12-04T11:21:33.5933914Z W1204 11:18:16.055000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] ).serialize() 2025-12-04T11:21:33.5934391Z W1204 11:18:16.055000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/compile_fx_ext.py", line 210, in serialize 2025-12-04T11:21:33.5934721Z W1204 11:18:16.055000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] return _WireProtocolPickledInput(GraphPickler.dumps(self)) 2025-12-04T11:21:33.5935167Z W1204 11:18:16.055000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 127, in dumps 2025-12-04T11:21:33.5935398Z W1204 11:18:16.055000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] pickler.dump(obj) 2025-12-04T11:21:33.5935884Z W1204 11:18:16.055000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 94, in reducer_override 2025-12-04T11:21:33.5936287Z W1204 11:18:16.055000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] return _GraphModulePickleData.reduce_helper(self, obj) 2025-12-04T11:21:33.5936758Z W1204 11:18:16.055000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 352, in reduce_helper 2025-12-04T11:21:33.5937016Z W1204 11:18:16.055000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] cls(obj, pickler.options), 2025-12-04T11:21:33.5937509Z W1204 11:18:16.055000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 364, in __init__ 2025-12-04T11:21:33.5937814Z W1204 11:18:16.055000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] self.graph = _GraphPickleData(gm._graph, options) 2025-12-04T11:21:33.5938265Z W1204 11:18:16.055000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 571, in __init__ 2025-12-04T11:21:33.5938564Z W1204 11:18:16.055000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] nodes[node] = _NodePickleData(node, nodes, options) 2025-12-04T11:21:33.5939014Z W1204 11:18:16.055000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 387, in __init__ 2025-12-04T11:21:33.5939330Z W1204 11:18:16.055000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] self.target = _OpPickleData.pickle(node.target, options) 2025-12-04T11:21:33.5939777Z W1204 11:18:16.055000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 435, in pickle 2025-12-04T11:21:33.5940105Z W1204 11:18:16.055000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] return cls._pickle_op(name, _OpOverloadPickleData, options) 2025-12-04T11:21:33.5940566Z W1204 11:18:16.055000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 456, in _pickle_op 2025-12-04T11:21:33.5940920Z W1204 11:18:16.055000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] raise BypassFxGraphCache(f"Unable to pickle non-standard op: {name}") 2025-12-04T11:21:33.5941444Z W1204 11:18:16.055000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] torch._inductor.codecache.BypassFxGraphCache: Unable to pickle non-standard op: torch.ops.prims.convert_element_type.default 2025-12-04T11:21:33.5941774Z W1204 11:18:16.316000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] Unable to pickle input graph or example inputs 2025-12-04T11:21:33.5942036Z W1204 11:18:16.316000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] Traceback (most recent call last): 2025-12-04T11:21:33.5942552Z W1204 11:18:16.316000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/compile_fx_ext.py", line 486, in serialize_compile 2025-12-04T11:21:33.5942769Z W1204 11:18:16.316000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] ).serialize() 2025-12-04T11:21:33.5943249Z W1204 11:18:16.316000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/compile_fx_ext.py", line 210, in serialize 2025-12-04T11:21:33.5943659Z W1204 11:18:16.316000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] return _WireProtocolPickledInput(GraphPickler.dumps(self)) 2025-12-04T11:21:33.5944191Z W1204 11:18:16.316000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 127, in dumps 2025-12-04T11:21:33.5944424Z W1204 11:18:16.316000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] pickler.dump(obj) 2025-12-04T11:21:33.5944895Z W1204 11:18:16.316000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 94, in reducer_override 2025-12-04T11:21:33.5945218Z W1204 11:18:16.316000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] return _GraphModulePickleData.reduce_helper(self, obj) 2025-12-04T11:21:33.5945727Z W1204 11:18:16.316000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 352, in reduce_helper 2025-12-04T11:21:33.5945994Z W1204 11:18:16.316000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] cls(obj, pickler.options), 2025-12-04T11:21:33.5946447Z W1204 11:18:16.316000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 364, in __init__ 2025-12-04T11:21:33.5946743Z W1204 11:18:16.316000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] self.graph = _GraphPickleData(gm._graph, options) 2025-12-04T11:21:33.5947199Z W1204 11:18:16.316000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 571, in __init__ 2025-12-04T11:21:33.5947498Z W1204 11:18:16.316000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] nodes[node] = _NodePickleData(node, nodes, options) 2025-12-04T11:21:33.5947949Z W1204 11:18:16.316000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 387, in __init__ 2025-12-04T11:21:33.5948277Z W1204 11:18:16.316000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] self.target = _OpPickleData.pickle(node.target, options) 2025-12-04T11:21:33.5948727Z W1204 11:18:16.316000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 435, in pickle 2025-12-04T11:21:33.5949051Z W1204 11:18:16.316000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] return cls._pickle_op(name, _OpOverloadPickleData, options) 2025-12-04T11:21:33.5949507Z W1204 11:18:16.316000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 456, in _pickle_op 2025-12-04T11:21:33.5949905Z W1204 11:18:16.316000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] raise BypassFxGraphCache(f"Unable to pickle non-standard op: {name}") 2025-12-04T11:21:33.5950430Z W1204 11:18:16.316000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] torch._inductor.codecache.BypassFxGraphCache: Unable to pickle non-standard op: torch.ops.prims.convert_element_type.default 2025-12-04T11:21:33.5950499Z PASSED [0.5229s] [ 60%] 2025-12-04T11:21:33.5951086Z inductor/test_compile_subprocess.py::GPUTests::test_mixed_mm_cuda <- test/inductor/test_torchinductor.py W1204 11:18:16.580000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] Unable to pickle input graph or example inputs 2025-12-04T11:21:33.5951352Z W1204 11:18:16.580000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] Traceback (most recent call last): 2025-12-04T11:21:33.5951855Z W1204 11:18:16.580000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/compile_fx_ext.py", line 486, in serialize_compile 2025-12-04T11:21:33.5952152Z W1204 11:18:16.580000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] ).serialize() 2025-12-04T11:21:33.5952640Z W1204 11:18:16.580000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/compile_fx_ext.py", line 210, in serialize 2025-12-04T11:21:33.5952965Z W1204 11:18:16.580000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] return _WireProtocolPickledInput(GraphPickler.dumps(self)) 2025-12-04T11:21:33.5953417Z W1204 11:18:16.580000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 127, in dumps 2025-12-04T11:21:33.5953681Z W1204 11:18:16.580000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] pickler.dump(obj) 2025-12-04T11:21:33.5954162Z W1204 11:18:16.580000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 94, in reducer_override 2025-12-04T11:21:33.5954491Z W1204 11:18:16.580000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] return _GraphModulePickleData.reduce_helper(self, obj) 2025-12-04T11:21:33.5954961Z W1204 11:18:16.580000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 352, in reduce_helper 2025-12-04T11:21:33.5955224Z W1204 11:18:16.580000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] cls(obj, pickler.options), 2025-12-04T11:21:33.5955677Z W1204 11:18:16.580000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 364, in __init__ 2025-12-04T11:21:33.5955983Z W1204 11:18:16.580000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] self.graph = _GraphPickleData(gm._graph, options) 2025-12-04T11:21:33.5956441Z W1204 11:18:16.580000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 571, in __init__ 2025-12-04T11:21:33.5956744Z W1204 11:18:16.580000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] nodes[node] = _NodePickleData(node, nodes, options) 2025-12-04T11:21:33.5957208Z W1204 11:18:16.580000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 387, in __init__ 2025-12-04T11:21:33.5957525Z W1204 11:18:16.580000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] self.target = _OpPickleData.pickle(node.target, options) 2025-12-04T11:21:33.5958022Z W1204 11:18:16.580000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 435, in pickle 2025-12-04T11:21:33.5958349Z W1204 11:18:16.580000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] return cls._pickle_op(name, _OpOverloadPickleData, options) 2025-12-04T11:21:33.5958812Z W1204 11:18:16.580000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 456, in _pickle_op 2025-12-04T11:21:33.5959166Z W1204 11:18:16.580000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] raise BypassFxGraphCache(f"Unable to pickle non-standard op: {name}") 2025-12-04T11:21:33.5959687Z W1204 11:18:16.580000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] torch._inductor.codecache.BypassFxGraphCache: Unable to pickle non-standard op: torch.ops.prims.convert_element_type.default 2025-12-04T11:21:33.5959988Z W1204 11:18:16.723000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] Unable to pickle input graph or example inputs 2025-12-04T11:21:33.5960324Z W1204 11:18:16.723000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] Traceback (most recent call last): 2025-12-04T11:21:33.5960838Z W1204 11:18:16.723000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/compile_fx_ext.py", line 486, in serialize_compile 2025-12-04T11:21:33.5961059Z W1204 11:18:16.723000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] ).serialize() 2025-12-04T11:21:33.5961532Z W1204 11:18:16.723000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/compile_fx_ext.py", line 210, in serialize 2025-12-04T11:21:33.5961906Z W1204 11:18:16.723000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] return _WireProtocolPickledInput(GraphPickler.dumps(self)) 2025-12-04T11:21:33.5962359Z W1204 11:18:16.723000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 127, in dumps 2025-12-04T11:21:33.5962599Z W1204 11:18:16.723000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] pickler.dump(obj) 2025-12-04T11:21:33.5963068Z W1204 11:18:16.723000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 94, in reducer_override 2025-12-04T11:21:33.5963390Z W1204 11:18:16.723000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] return _GraphModulePickleData.reduce_helper(self, obj) 2025-12-04T11:21:33.5963856Z W1204 11:18:16.723000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 352, in reduce_helper 2025-12-04T11:21:33.5964118Z W1204 11:18:16.723000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] cls(obj, pickler.options), 2025-12-04T11:21:33.5964573Z W1204 11:18:16.723000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 364, in __init__ 2025-12-04T11:21:33.5964874Z W1204 11:18:16.723000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] self.graph = _GraphPickleData(gm._graph, options) 2025-12-04T11:21:33.5965328Z W1204 11:18:16.723000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 571, in __init__ 2025-12-04T11:21:33.5965630Z W1204 11:18:16.723000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] nodes[node] = _NodePickleData(node, nodes, options) 2025-12-04T11:21:33.5966138Z W1204 11:18:16.723000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 387, in __init__ 2025-12-04T11:21:33.5966456Z W1204 11:18:16.723000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] self.target = _OpPickleData.pickle(node.target, options) 2025-12-04T11:21:33.5966909Z W1204 11:18:16.723000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 435, in pickle 2025-12-04T11:21:33.5967234Z W1204 11:18:16.723000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] return cls._pickle_op(name, _OpOverloadPickleData, options) 2025-12-04T11:21:33.5967692Z W1204 11:18:16.723000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 456, in _pickle_op 2025-12-04T11:21:33.5968059Z W1204 11:18:16.723000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] raise BypassFxGraphCache(f"Unable to pickle non-standard op: {name}") 2025-12-04T11:21:33.5968662Z W1204 11:18:16.723000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] torch._inductor.codecache.BypassFxGraphCache: Unable to pickle non-standard op: torch.ops.prims.convert_element_type.default 2025-12-04T11:21:33.5968736Z PASSED [0.2855s] [ 60%] 2025-12-04T11:21:33.5969338Z inductor/test_compile_subprocess.py::GPUTests::test_move_arange_cuda <- test/inductor/test_torchinductor.py W1204 11:18:16.865000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] Unable to pickle input graph or example inputs 2025-12-04T11:21:33.5969607Z W1204 11:18:16.865000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] Traceback (most recent call last): 2025-12-04T11:21:33.5970147Z W1204 11:18:16.865000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/compile_fx_ext.py", line 486, in serialize_compile 2025-12-04T11:21:33.5970371Z W1204 11:18:16.865000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] ).serialize() 2025-12-04T11:21:33.5970850Z W1204 11:18:16.865000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/compile_fx_ext.py", line 210, in serialize 2025-12-04T11:21:33.5971178Z W1204 11:18:16.865000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] return _WireProtocolPickledInput(GraphPickler.dumps(self)) 2025-12-04T11:21:33.5971626Z W1204 11:18:16.865000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 127, in dumps 2025-12-04T11:21:33.5971859Z W1204 11:18:16.865000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] pickler.dump(obj) 2025-12-04T11:21:33.5972330Z W1204 11:18:16.865000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 94, in reducer_override 2025-12-04T11:21:33.5972652Z W1204 11:18:16.865000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] return _GraphModulePickleData.reduce_helper(self, obj) 2025-12-04T11:21:33.5973120Z W1204 11:18:16.865000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 352, in reduce_helper 2025-12-04T11:21:33.5973377Z W1204 11:18:16.865000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] cls(obj, pickler.options), 2025-12-04T11:21:33.5973829Z W1204 11:18:16.865000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 364, in __init__ 2025-12-04T11:21:33.5974169Z W1204 11:18:16.865000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] self.graph = _GraphPickleData(gm._graph, options) 2025-12-04T11:21:33.5974618Z W1204 11:18:16.865000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 571, in __init__ 2025-12-04T11:21:33.5974912Z W1204 11:18:16.865000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] nodes[node] = _NodePickleData(node, nodes, options) 2025-12-04T11:21:33.5975363Z W1204 11:18:16.865000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 387, in __init__ 2025-12-04T11:21:33.5975680Z W1204 11:18:16.865000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] self.target = _OpPickleData.pickle(node.target, options) 2025-12-04T11:21:33.5976135Z W1204 11:18:16.865000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 435, in pickle 2025-12-04T11:21:33.5976557Z W1204 11:18:16.865000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] return cls._pickle_op(name, _OpOverloadPickleData, options) 2025-12-04T11:21:33.5977024Z W1204 11:18:16.865000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 456, in _pickle_op 2025-12-04T11:21:33.5977381Z W1204 11:18:16.865000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] raise BypassFxGraphCache(f"Unable to pickle non-standard op: {name}") 2025-12-04T11:21:33.5978156Z W1204 11:18:16.865000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] torch._inductor.codecache.BypassFxGraphCache: Unable to pickle non-standard op: torch.ops.prims.iota.default 2025-12-04T11:21:33.5978245Z PASSED [0.1421s] [ 60%] 2025-12-04T11:21:33.5978866Z inductor/test_compile_subprocess.py::GPUTests::test_mul_index_expr_cuda <- test/inductor/test_torchinductor.py W1204 11:18:17.009000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] Unable to pickle input graph or example inputs 2025-12-04T11:21:33.5979144Z W1204 11:18:17.009000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] Traceback (most recent call last): 2025-12-04T11:21:33.5979658Z W1204 11:18:17.009000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/compile_fx_ext.py", line 486, in serialize_compile 2025-12-04T11:21:33.5979876Z W1204 11:18:17.009000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] ).serialize() 2025-12-04T11:21:33.5980353Z W1204 11:18:17.009000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/compile_fx_ext.py", line 210, in serialize 2025-12-04T11:21:33.5980685Z W1204 11:18:17.009000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] return _WireProtocolPickledInput(GraphPickler.dumps(self)) 2025-12-04T11:21:33.5981135Z W1204 11:18:17.009000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 127, in dumps 2025-12-04T11:21:33.5981366Z W1204 11:18:17.009000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] pickler.dump(obj) 2025-12-04T11:21:33.5981841Z W1204 11:18:17.009000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 94, in reducer_override 2025-12-04T11:21:33.5982157Z W1204 11:18:17.009000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] return _GraphModulePickleData.reduce_helper(self, obj) 2025-12-04T11:21:33.5982762Z W1204 11:18:17.009000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 352, in reduce_helper 2025-12-04T11:21:33.5983023Z W1204 11:18:17.009000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] cls(obj, pickler.options), 2025-12-04T11:21:33.5983473Z W1204 11:18:17.009000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 364, in __init__ 2025-12-04T11:21:33.5983816Z W1204 11:18:17.009000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] self.graph = _GraphPickleData(gm._graph, options) 2025-12-04T11:21:33.5984271Z W1204 11:18:17.009000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 571, in __init__ 2025-12-04T11:21:33.5984588Z W1204 11:18:17.009000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] nodes[node] = _NodePickleData(node, nodes, options) 2025-12-04T11:21:33.5985145Z W1204 11:18:17.009000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 387, in __init__ 2025-12-04T11:21:33.5985465Z W1204 11:18:17.009000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] self.target = _OpPickleData.pickle(node.target, options) 2025-12-04T11:21:33.5985915Z W1204 11:18:17.009000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 435, in pickle 2025-12-04T11:21:33.5986273Z W1204 11:18:17.009000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] return cls._pickle_op(name, _OpOverloadPickleData, options) 2025-12-04T11:21:33.5986739Z W1204 11:18:17.009000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 456, in _pickle_op 2025-12-04T11:21:33.5987101Z W1204 11:18:17.009000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] raise BypassFxGraphCache(f"Unable to pickle non-standard op: {name}") 2025-12-04T11:21:33.5987578Z W1204 11:18:17.009000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] torch._inductor.codecache.BypassFxGraphCache: Unable to pickle non-standard op: torch.ops.prims.iota.default 2025-12-04T11:21:33.5987641Z PASSED [0.3214s] [ 60%] 2025-12-04T11:21:33.5988266Z inductor/test_compile_subprocess.py::GPUTests::test_mul_softmax_symfloat_cuda <- test/inductor/test_torchinductor.py W1204 11:18:17.457000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0_1] Unable to pickle input graph or example inputs 2025-12-04T11:21:33.5988543Z W1204 11:18:17.457000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0_1] Traceback (most recent call last): 2025-12-04T11:21:33.5989051Z W1204 11:18:17.457000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0_1] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/compile_fx_ext.py", line 486, in serialize_compile 2025-12-04T11:21:33.5989281Z W1204 11:18:17.457000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0_1] ).serialize() 2025-12-04T11:21:33.5989768Z W1204 11:18:17.457000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0_1] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/compile_fx_ext.py", line 210, in serialize 2025-12-04T11:21:33.5990104Z W1204 11:18:17.457000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0_1] return _WireProtocolPickledInput(GraphPickler.dumps(self)) 2025-12-04T11:21:33.5990561Z W1204 11:18:17.457000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0_1] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 127, in dumps 2025-12-04T11:21:33.5990839Z W1204 11:18:17.457000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0_1] pickler.dump(obj) 2025-12-04T11:21:33.5991321Z W1204 11:18:17.457000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0_1] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 94, in reducer_override 2025-12-04T11:21:33.5991638Z W1204 11:18:17.457000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0_1] return _GraphModulePickleData.reduce_helper(self, obj) 2025-12-04T11:21:33.5992112Z W1204 11:18:17.457000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0_1] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 352, in reduce_helper 2025-12-04T11:21:33.5992374Z W1204 11:18:17.457000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0_1] cls(obj, pickler.options), 2025-12-04T11:21:33.5992825Z W1204 11:18:17.457000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0_1] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 364, in __init__ 2025-12-04T11:21:33.5993207Z W1204 11:18:17.457000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0_1] self.graph = _GraphPickleData(gm._graph, options) 2025-12-04T11:21:33.5993660Z W1204 11:18:17.457000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0_1] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 571, in __init__ 2025-12-04T11:21:33.5993975Z W1204 11:18:17.457000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0_1] nodes[node] = _NodePickleData(node, nodes, options) 2025-12-04T11:21:33.5994484Z W1204 11:18:17.457000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0_1] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 387, in __init__ 2025-12-04T11:21:33.5994816Z W1204 11:18:17.457000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0_1] self.target = _OpPickleData.pickle(node.target, options) 2025-12-04T11:21:33.5995276Z W1204 11:18:17.457000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0_1] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 435, in pickle 2025-12-04T11:21:33.5995605Z W1204 11:18:17.457000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0_1] return cls._pickle_op(name, _OpOverloadPickleData, options) 2025-12-04T11:21:33.5996072Z W1204 11:18:17.457000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0_1] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 456, in _pickle_op 2025-12-04T11:21:33.5996433Z W1204 11:18:17.457000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0_1] raise BypassFxGraphCache(f"Unable to pickle non-standard op: {name}") 2025-12-04T11:21:33.5996961Z W1204 11:18:17.457000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0_1] torch._inductor.codecache.BypassFxGraphCache: Unable to pickle non-standard op: torch.ops.prims.convert_element_type.default 2025-12-04T11:21:33.5997256Z W1204 11:18:18.325000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/1] Unable to pickle input graph or example inputs 2025-12-04T11:21:33.5997519Z W1204 11:18:18.325000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/1] Traceback (most recent call last): 2025-12-04T11:21:33.5998018Z W1204 11:18:18.325000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/1] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/compile_fx_ext.py", line 486, in serialize_compile 2025-12-04T11:21:33.5998235Z W1204 11:18:18.325000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/1] ).serialize() 2025-12-04T11:21:33.5998714Z W1204 11:18:18.325000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/1] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/compile_fx_ext.py", line 210, in serialize 2025-12-04T11:21:33.5999093Z W1204 11:18:18.325000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/1] return _WireProtocolPickledInput(GraphPickler.dumps(self)) 2025-12-04T11:21:33.5999542Z W1204 11:18:18.325000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/1] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 127, in dumps 2025-12-04T11:21:33.5999773Z W1204 11:18:18.325000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/1] pickler.dump(obj) 2025-12-04T11:21:33.6000243Z W1204 11:18:18.325000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/1] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 94, in reducer_override 2025-12-04T11:21:33.6000563Z W1204 11:18:18.325000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/1] return _GraphModulePickleData.reduce_helper(self, obj) 2025-12-04T11:21:33.6001065Z W1204 11:18:18.325000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/1] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 352, in reduce_helper 2025-12-04T11:21:33.6001357Z W1204 11:18:18.325000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/1] cls(obj, pickler.options), 2025-12-04T11:21:33.6001807Z W1204 11:18:18.325000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/1] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 364, in __init__ 2025-12-04T11:21:33.6002105Z W1204 11:18:18.325000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/1] self.graph = _GraphPickleData(gm._graph, options) 2025-12-04T11:21:33.6002601Z W1204 11:18:18.325000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/1] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 571, in __init__ 2025-12-04T11:21:33.6002906Z W1204 11:18:18.325000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/1] nodes[node] = _NodePickleData(node, nodes, options) 2025-12-04T11:21:33.6003364Z W1204 11:18:18.325000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/1] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 387, in __init__ 2025-12-04T11:21:33.6003679Z W1204 11:18:18.325000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/1] self.target = _OpPickleData.pickle(node.target, options) 2025-12-04T11:21:33.6004132Z W1204 11:18:18.325000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/1] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 435, in pickle 2025-12-04T11:21:33.6004457Z W1204 11:18:18.325000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/1] return cls._pickle_op(name, _OpOverloadPickleData, options) 2025-12-04T11:21:33.6004923Z W1204 11:18:18.325000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/1] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 456, in _pickle_op 2025-12-04T11:21:33.6005283Z W1204 11:18:18.325000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/1] raise BypassFxGraphCache(f"Unable to pickle non-standard op: {name}") 2025-12-04T11:21:33.6006044Z W1204 11:18:18.325000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/1] torch._inductor.codecache.BypassFxGraphCache: Unable to pickle non-standard op: torch.ops.prims.convert_element_type.default 2025-12-04T11:21:33.6006123Z PASSED [1.8081s] [ 60%] 2025-12-04T11:21:33.6006583Z inductor/test_compile_subprocess.py::GPUTests::test_multi_gpu_recompile_on_index_cuda <- test/inductor/test_torchinductor.py SKIPPED [0.0002s] (requires multiple cuda devices) [ 61%] 2025-12-04T11:21:33.6006918Z inductor/test_compile_subprocess.py::GPUTests::test_multi_threading_cuda <- test/inductor/test_torchinductor.py PASSED [0.1674s] [ 61%] 2025-12-04T11:21:33.6007324Z inductor/test_compile_subprocess.py::GPUTests::test_multilayer_prime_size_cuda <- test/inductor/test_torchinductor.py PASSED [0.6956s] [ 61%] 2025-12-04T11:21:33.6007649Z inductor/test_compile_subprocess.py::GPUTests::test_multilayer_var_cuda <- test/inductor/test_torchinductor.py PASSED [1.2333s] [ 61%] 2025-12-04T11:21:33.6008037Z inductor/test_compile_subprocess.py::GPUTests::test_nan_sort_stable_True_descending_False_cuda <- test/inductor/test_torchinductor.py PASSED [0.7277s] [ 62%] 2025-12-04T11:21:33.6008410Z inductor/test_compile_subprocess.py::GPUTests::test_nan_sort_stable_True_descending_True_cuda <- test/inductor/test_torchinductor.py PASSED [0.7167s] [ 62%] 2025-12-04T11:21:33.6009048Z inductor/test_compile_subprocess.py::GPUTests::test_needs_contiguous_strides_cuda <- test/inductor/test_torchinductor.py W1204 11:18:22.689000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] Unable to pickle input graph or example inputs 2025-12-04T11:21:33.6009317Z W1204 11:18:22.689000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] Traceback (most recent call last): 2025-12-04T11:21:33.6009901Z W1204 11:18:22.689000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/compile_fx_ext.py", line 486, in serialize_compile 2025-12-04T11:21:33.6010123Z W1204 11:18:22.689000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] ).serialize() 2025-12-04T11:21:33.6010597Z W1204 11:18:22.689000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/compile_fx_ext.py", line 210, in serialize 2025-12-04T11:21:33.6010967Z W1204 11:18:22.689000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] return _WireProtocolPickledInput(GraphPickler.dumps(self)) 2025-12-04T11:21:33.6011411Z W1204 11:18:22.689000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 127, in dumps 2025-12-04T11:21:33.6011655Z W1204 11:18:22.689000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] pickler.dump(obj) 2025-12-04T11:21:33.6012129Z W1204 11:18:22.689000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 94, in reducer_override 2025-12-04T11:21:33.6012448Z W1204 11:18:22.689000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] return _GraphModulePickleData.reduce_helper(self, obj) 2025-12-04T11:21:33.6012913Z W1204 11:18:22.689000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 352, in reduce_helper 2025-12-04T11:21:33.6013170Z W1204 11:18:22.689000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] cls(obj, pickler.options), 2025-12-04T11:21:33.6013635Z W1204 11:18:22.689000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 364, in __init__ 2025-12-04T11:21:33.6013933Z W1204 11:18:22.689000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] self.graph = _GraphPickleData(gm._graph, options) 2025-12-04T11:21:33.6014387Z W1204 11:18:22.689000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 571, in __init__ 2025-12-04T11:21:33.6014693Z W1204 11:18:22.689000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] nodes[node] = _NodePickleData(node, nodes, options) 2025-12-04T11:21:33.6015151Z W1204 11:18:22.689000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 387, in __init__ 2025-12-04T11:21:33.6015510Z W1204 11:18:22.689000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] self.target = _OpPickleData.pickle(node.target, options) 2025-12-04T11:21:33.6015957Z W1204 11:18:22.689000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 435, in pickle 2025-12-04T11:21:33.6016287Z W1204 11:18:22.689000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] return cls._pickle_op(name, _OpOverloadPickleData, options) 2025-12-04T11:21:33.6016748Z W1204 11:18:22.689000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 456, in _pickle_op 2025-12-04T11:21:33.6017111Z W1204 11:18:22.689000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] raise BypassFxGraphCache(f"Unable to pickle non-standard op: {name}") 2025-12-04T11:21:33.6017632Z W1204 11:18:22.689000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] torch._inductor.codecache.BypassFxGraphCache: Unable to pickle non-standard op: torch.ops.mylib.myop.default 2025-12-04T11:21:33.6017733Z PASSED [0.5914s] [ 62%] 2025-12-04T11:21:33.6018328Z inductor/test_compile_subprocess.py::GPUTests::test_neg_index_cuda <- test/inductor/test_torchinductor.py W1204 11:18:24.053000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] Unable to pickle input graph or example inputs 2025-12-04T11:21:33.6018592Z W1204 11:18:24.053000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] Traceback (most recent call last): 2025-12-04T11:21:33.6019138Z W1204 11:18:24.053000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/compile_fx_ext.py", line 486, in serialize_compile 2025-12-04T11:21:33.6019359Z W1204 11:18:24.053000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] ).serialize() 2025-12-04T11:21:33.6019849Z W1204 11:18:24.053000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/compile_fx_ext.py", line 210, in serialize 2025-12-04T11:21:33.6020176Z W1204 11:18:24.053000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] return _WireProtocolPickledInput(GraphPickler.dumps(self)) 2025-12-04T11:21:33.6020633Z W1204 11:18:24.053000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 127, in dumps 2025-12-04T11:21:33.6020870Z W1204 11:18:24.053000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] pickler.dump(obj) 2025-12-04T11:21:33.6021346Z W1204 11:18:24.053000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 94, in reducer_override 2025-12-04T11:21:33.6021668Z W1204 11:18:24.053000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] return _GraphModulePickleData.reduce_helper(self, obj) 2025-12-04T11:21:33.6022134Z W1204 11:18:24.053000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 352, in reduce_helper 2025-12-04T11:21:33.6022398Z W1204 11:18:24.053000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] cls(obj, pickler.options), 2025-12-04T11:21:33.6022853Z W1204 11:18:24.053000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 364, in __init__ 2025-12-04T11:21:33.6023157Z W1204 11:18:24.053000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] self.graph = _GraphPickleData(gm._graph, options) 2025-12-04T11:21:33.6023698Z W1204 11:18:24.053000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 571, in __init__ 2025-12-04T11:21:33.6023997Z W1204 11:18:24.053000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] nodes[node] = _NodePickleData(node, nodes, options) 2025-12-04T11:21:33.6024450Z W1204 11:18:24.053000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 387, in __init__ 2025-12-04T11:21:33.6024767Z W1204 11:18:24.053000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] self.target = _OpPickleData.pickle(node.target, options) 2025-12-04T11:21:33.6025226Z W1204 11:18:24.053000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 435, in pickle 2025-12-04T11:21:33.6025551Z W1204 11:18:24.053000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] return cls._pickle_op(name, _OpOverloadPickleData, options) 2025-12-04T11:21:33.6026104Z W1204 11:18:24.053000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 456, in _pickle_op 2025-12-04T11:21:33.6026472Z W1204 11:18:24.053000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] raise BypassFxGraphCache(f"Unable to pickle non-standard op: {name}") 2025-12-04T11:21:33.6026944Z W1204 11:18:24.053000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] torch._inductor.codecache.BypassFxGraphCache: Unable to pickle non-standard op: torch.ops.prims.iota.default 2025-12-04T11:21:33.6027478Z W1204 11:18:24.312000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] Unable to pickle input graph or example inputs 2025-12-04T11:21:33.6027754Z W1204 11:18:24.312000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] Traceback (most recent call last): 2025-12-04T11:21:33.6028266Z W1204 11:18:24.312000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/compile_fx_ext.py", line 486, in serialize_compile 2025-12-04T11:21:33.6028489Z W1204 11:18:24.312000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] ).serialize() 2025-12-04T11:21:33.6028976Z W1204 11:18:24.312000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/compile_fx_ext.py", line 210, in serialize 2025-12-04T11:21:33.6029312Z W1204 11:18:24.312000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] return _WireProtocolPickledInput(GraphPickler.dumps(self)) 2025-12-04T11:21:33.6029756Z W1204 11:18:24.312000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 127, in dumps 2025-12-04T11:21:33.6029995Z W1204 11:18:24.312000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] pickler.dump(obj) 2025-12-04T11:21:33.6030466Z W1204 11:18:24.312000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 94, in reducer_override 2025-12-04T11:21:33.6030781Z W1204 11:18:24.312000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] return _GraphModulePickleData.reduce_helper(self, obj) 2025-12-04T11:21:33.6031253Z W1204 11:18:24.312000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 352, in reduce_helper 2025-12-04T11:21:33.6031510Z W1204 11:18:24.312000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] cls(obj, pickler.options), 2025-12-04T11:21:33.6032010Z W1204 11:18:24.312000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 364, in __init__ 2025-12-04T11:21:33.6032311Z W1204 11:18:24.312000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] self.graph = _GraphPickleData(gm._graph, options) 2025-12-04T11:21:33.6032766Z W1204 11:18:24.312000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 571, in __init__ 2025-12-04T11:21:33.6033062Z W1204 11:18:24.312000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] nodes[node] = _NodePickleData(node, nodes, options) 2025-12-04T11:21:33.6033513Z W1204 11:18:24.312000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 387, in __init__ 2025-12-04T11:21:33.6033837Z W1204 11:18:24.312000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] self.target = _OpPickleData.pickle(node.target, options) 2025-12-04T11:21:33.6034359Z W1204 11:18:24.312000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 435, in pickle 2025-12-04T11:21:33.6034689Z W1204 11:18:24.312000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] return cls._pickle_op(name, _OpOverloadPickleData, options) 2025-12-04T11:21:33.6035146Z W1204 11:18:24.312000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 456, in _pickle_op 2025-12-04T11:21:33.6035538Z W1204 11:18:24.312000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] raise BypassFxGraphCache(f"Unable to pickle non-standard op: {name}") 2025-12-04T11:21:33.6036017Z W1204 11:18:24.312000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] torch._inductor.codecache.BypassFxGraphCache: Unable to pickle non-standard op: torch.ops.prims.iota.default 2025-12-04T11:21:33.6036312Z W1204 11:18:24.588000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] Unable to pickle input graph or example inputs 2025-12-04T11:21:33.6036576Z W1204 11:18:24.588000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] Traceback (most recent call last): 2025-12-04T11:21:33.6037077Z W1204 11:18:24.588000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/compile_fx_ext.py", line 486, in serialize_compile 2025-12-04T11:21:33.6037300Z W1204 11:18:24.588000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] ).serialize() 2025-12-04T11:21:33.6037774Z W1204 11:18:24.588000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/compile_fx_ext.py", line 210, in serialize 2025-12-04T11:21:33.6038103Z W1204 11:18:24.588000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] return _WireProtocolPickledInput(GraphPickler.dumps(self)) 2025-12-04T11:21:33.6038552Z W1204 11:18:24.588000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 127, in dumps 2025-12-04T11:21:33.6038785Z W1204 11:18:24.588000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] pickler.dump(obj) 2025-12-04T11:21:33.6039261Z W1204 11:18:24.588000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 94, in reducer_override 2025-12-04T11:21:33.6039575Z W1204 11:18:24.588000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] return _GraphModulePickleData.reduce_helper(self, obj) 2025-12-04T11:21:33.6040100Z W1204 11:18:24.588000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 352, in reduce_helper 2025-12-04T11:21:33.6040358Z W1204 11:18:24.588000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] cls(obj, pickler.options), 2025-12-04T11:21:33.6040810Z W1204 11:18:24.588000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 364, in __init__ 2025-12-04T11:21:33.6041113Z W1204 11:18:24.588000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] self.graph = _GraphPickleData(gm._graph, options) 2025-12-04T11:21:33.6041563Z W1204 11:18:24.588000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 571, in __init__ 2025-12-04T11:21:33.6041865Z W1204 11:18:24.588000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] nodes[node] = _NodePickleData(node, nodes, options) 2025-12-04T11:21:33.6042387Z W1204 11:18:24.588000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 387, in __init__ 2025-12-04T11:21:33.6042710Z W1204 11:18:24.588000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] self.target = _OpPickleData.pickle(node.target, options) 2025-12-04T11:21:33.6043155Z W1204 11:18:24.588000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 435, in pickle 2025-12-04T11:21:33.6043512Z W1204 11:18:24.588000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] return cls._pickle_op(name, _OpOverloadPickleData, options) 2025-12-04T11:21:33.6043979Z W1204 11:18:24.588000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 456, in _pickle_op 2025-12-04T11:21:33.6044336Z W1204 11:18:24.588000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] raise BypassFxGraphCache(f"Unable to pickle non-standard op: {name}") 2025-12-04T11:21:33.6044820Z W1204 11:18:24.588000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] torch._inductor.codecache.BypassFxGraphCache: Unable to pickle non-standard op: torch.ops.prims.iota.default 2025-12-04T11:21:33.6044888Z PASSED [2.1362s] [ 62%] 2025-12-04T11:21:33.6045206Z inductor/test_compile_subprocess.py::GPUTests::test_new_empty_cuda <- test/inductor/test_torchinductor.py PASSED [0.1845s] [ 63%] 2025-12-04T11:21:33.6045542Z inductor/test_compile_subprocess.py::GPUTests::test_new_empty_strided_cuda <- test/inductor/test_torchinductor.py PASSED [0.1802s] [ 63%] 2025-12-04T11:21:33.6045849Z inductor/test_compile_subprocess.py::GPUTests::test_new_ones_cuda <- test/inductor/test_torchinductor.py PASSED [0.2637s] [ 63%] 2025-12-04T11:21:33.6046216Z inductor/test_compile_subprocess.py::GPUTests::test_no_mega_fusion_during_lowering_cuda <- test/inductor/test_torchinductor.py PASSED [0.6467s] [ 63%] 2025-12-04T11:21:33.6046534Z inductor/test_compile_subprocess.py::GPUTests::test_no_op_reduction_cuda <- test/inductor/test_torchinductor.py PASSED [0.3198s] [ 63%] 2025-12-04T11:21:33.6047161Z inductor/test_compile_subprocess.py::GPUTests::test_norm_constant_overflow_cuda <- test/inductor/test_torchinductor.py W1204 11:18:27.295000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] Unable to pickle input graph or example inputs 2025-12-04T11:21:33.6047424Z W1204 11:18:27.295000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] Traceback (most recent call last): 2025-12-04T11:21:33.6047935Z W1204 11:18:27.295000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/compile_fx_ext.py", line 486, in serialize_compile 2025-12-04T11:21:33.6048215Z W1204 11:18:27.295000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] ).serialize() 2025-12-04T11:21:33.6048689Z W1204 11:18:27.295000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/compile_fx_ext.py", line 210, in serialize 2025-12-04T11:21:33.6049021Z W1204 11:18:27.295000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] return _WireProtocolPickledInput(GraphPickler.dumps(self)) 2025-12-04T11:21:33.6049471Z W1204 11:18:27.295000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 127, in dumps 2025-12-04T11:21:33.6049706Z W1204 11:18:27.295000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] pickler.dump(obj) 2025-12-04T11:21:33.6050218Z W1204 11:18:27.295000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 94, in reducer_override 2025-12-04T11:21:33.6050575Z W1204 11:18:27.295000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] return _GraphModulePickleData.reduce_helper(self, obj) 2025-12-04T11:21:33.6051046Z W1204 11:18:27.295000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 352, in reduce_helper 2025-12-04T11:21:33.6051304Z W1204 11:18:27.295000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] cls(obj, pickler.options), 2025-12-04T11:21:33.6051793Z W1204 11:18:27.295000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 364, in __init__ 2025-12-04T11:21:33.6052095Z W1204 11:18:27.295000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] self.graph = _GraphPickleData(gm._graph, options) 2025-12-04T11:21:33.6052544Z W1204 11:18:27.295000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 571, in __init__ 2025-12-04T11:21:33.6052845Z W1204 11:18:27.295000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] nodes[node] = _NodePickleData(node, nodes, options) 2025-12-04T11:21:33.6053294Z W1204 11:18:27.295000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 387, in __init__ 2025-12-04T11:21:33.6053613Z W1204 11:18:27.295000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] self.target = _OpPickleData.pickle(node.target, options) 2025-12-04T11:21:33.6054061Z W1204 11:18:27.295000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 435, in pickle 2025-12-04T11:21:33.6054392Z W1204 11:18:27.295000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] return cls._pickle_op(name, _OpOverloadPickleData, options) 2025-12-04T11:21:33.6054851Z W1204 11:18:27.295000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 456, in _pickle_op 2025-12-04T11:21:33.6055206Z W1204 11:18:27.295000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] raise BypassFxGraphCache(f"Unable to pickle non-standard op: {name}") 2025-12-04T11:21:33.6055748Z W1204 11:18:27.295000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] torch._inductor.codecache.BypassFxGraphCache: Unable to pickle non-standard op: torch.ops.prims.convert_element_type.default 2025-12-04T11:21:33.6055855Z PASSED [0.5228s] [ 64%] 2025-12-04T11:21:33.6056439Z inductor/test_compile_subprocess.py::GPUTests::test_one_hot_cuda <- test/inductor/test_torchinductor.py W1204 11:18:27.555000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] Unable to pickle input graph or example inputs 2025-12-04T11:21:33.6056700Z W1204 11:18:27.555000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] Traceback (most recent call last): 2025-12-04T11:21:33.6057203Z W1204 11:18:27.555000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/compile_fx_ext.py", line 486, in serialize_compile 2025-12-04T11:21:33.6057419Z W1204 11:18:27.555000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] ).serialize() 2025-12-04T11:21:33.6057900Z W1204 11:18:27.555000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/compile_fx_ext.py", line 210, in serialize 2025-12-04T11:21:33.6058333Z W1204 11:18:27.555000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] return _WireProtocolPickledInput(GraphPickler.dumps(self)) 2025-12-04T11:21:33.6058778Z W1204 11:18:27.555000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 127, in dumps 2025-12-04T11:21:33.6059012Z W1204 11:18:27.555000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] pickler.dump(obj) 2025-12-04T11:21:33.6059483Z W1204 11:18:27.555000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 94, in reducer_override 2025-12-04T11:21:33.6059834Z W1204 11:18:27.555000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] return _GraphModulePickleData.reduce_helper(self, obj) 2025-12-04T11:21:33.6060303Z W1204 11:18:27.555000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 352, in reduce_helper 2025-12-04T11:21:33.6060560Z W1204 11:18:27.555000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] cls(obj, pickler.options), 2025-12-04T11:21:33.6061028Z W1204 11:18:27.555000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 364, in __init__ 2025-12-04T11:21:33.6061327Z W1204 11:18:27.555000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] self.graph = _GraphPickleData(gm._graph, options) 2025-12-04T11:21:33.6061787Z W1204 11:18:27.555000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 571, in __init__ 2025-12-04T11:21:33.6062088Z W1204 11:18:27.555000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] nodes[node] = _NodePickleData(node, nodes, options) 2025-12-04T11:21:33.6062539Z W1204 11:18:27.555000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 387, in __init__ 2025-12-04T11:21:33.6062858Z W1204 11:18:27.555000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] self.target = _OpPickleData.pickle(node.target, options) 2025-12-04T11:21:33.6063303Z W1204 11:18:27.555000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 435, in pickle 2025-12-04T11:21:33.6063686Z W1204 11:18:27.555000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] return cls._pickle_op(name, _OpOverloadPickleData, options) 2025-12-04T11:21:33.6064185Z W1204 11:18:27.555000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 456, in _pickle_op 2025-12-04T11:21:33.6064554Z W1204 11:18:27.555000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] raise BypassFxGraphCache(f"Unable to pickle non-standard op: {name}") 2025-12-04T11:21:33.6065027Z W1204 11:18:27.555000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] torch._inductor.codecache.BypassFxGraphCache: Unable to pickle non-standard op: torch.ops.prims.iota.default 2025-12-04T11:21:33.6065089Z PASSED [0.1935s] [ 64%] 2025-12-04T11:21:33.6065409Z inductor/test_compile_subprocess.py::GPUTests::test_pad_single_cuda <- test/inductor/test_torchinductor.py PASSED [0.3163s] [ 64%] 2025-12-04T11:21:33.6066055Z inductor/test_compile_subprocess.py::GPUTests::test_pattern_matcher_unbacked_cuda <- test/inductor/test_torchinductor.py W1204 11:18:28.122000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] Unable to pickle input graph or example inputs 2025-12-04T11:21:33.6066361Z W1204 11:18:28.122000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] Traceback (most recent call last): 2025-12-04T11:21:33.6066894Z W1204 11:18:28.122000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/compile_fx_ext.py", line 486, in serialize_compile 2025-12-04T11:21:33.6067120Z W1204 11:18:28.122000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] ).serialize() 2025-12-04T11:21:33.6067662Z W1204 11:18:28.122000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/compile_fx_ext.py", line 210, in serialize 2025-12-04T11:21:33.6068144Z W1204 11:18:28.122000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] return _WireProtocolPickledInput(GraphPickler.dumps(self)) 2025-12-04T11:21:33.6068604Z W1204 11:18:28.122000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 127, in dumps 2025-12-04T11:21:33.6068839Z W1204 11:18:28.122000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] pickler.dump(obj) 2025-12-04T11:21:33.6069316Z W1204 11:18:28.122000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 94, in reducer_override 2025-12-04T11:21:33.6069630Z W1204 11:18:28.122000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] return _GraphModulePickleData.reduce_helper(self, obj) 2025-12-04T11:21:33.6070098Z W1204 11:18:28.122000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 352, in reduce_helper 2025-12-04T11:21:33.6070355Z W1204 11:18:28.122000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] cls(obj, pickler.options), 2025-12-04T11:21:33.6070810Z W1204 11:18:28.122000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 364, in __init__ 2025-12-04T11:21:33.6071124Z W1204 11:18:28.122000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] self.graph = _GraphPickleData(gm._graph, options) 2025-12-04T11:21:33.6071575Z W1204 11:18:28.122000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 571, in __init__ 2025-12-04T11:21:33.6071877Z W1204 11:18:28.122000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] nodes[node] = _NodePickleData(node, nodes, options) 2025-12-04T11:21:33.6072331Z W1204 11:18:28.122000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 387, in __init__ 2025-12-04T11:21:33.6072696Z W1204 11:18:28.122000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] self.target = _OpPickleData.pickle(node.target, options) 2025-12-04T11:21:33.6073145Z W1204 11:18:28.122000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 435, in pickle 2025-12-04T11:21:33.6073469Z W1204 11:18:28.122000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] return cls._pickle_op(name, _OpOverloadPickleData, options) 2025-12-04T11:21:33.6073944Z W1204 11:18:28.122000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 456, in _pickle_op 2025-12-04T11:21:33.6074301Z W1204 11:18:28.122000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] raise BypassFxGraphCache(f"Unable to pickle non-standard op: {name}") 2025-12-04T11:21:33.6074899Z W1204 11:18:28.122000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] torch._inductor.codecache.BypassFxGraphCache: Unable to pickle non-standard op: torch.ops.prims.convert_element_type.default 2025-12-04T11:21:33.6074975Z PASSED [0.6145s] [ 64%] 2025-12-04T11:21:33.6075292Z inductor/test_compile_subprocess.py::GPUTests::test_permute1_cuda <- test/inductor/test_torchinductor.py PASSED [0.3702s] [ 65%] 2025-12-04T11:21:33.6075890Z inductor/test_compile_subprocess.py::GPUTests::test_philox_rand_cuda <- test/inductor/test_torchinductor.py W1204 11:18:29.071000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] Unable to pickle input graph or example inputs 2025-12-04T11:21:33.6076186Z W1204 11:18:29.071000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] Traceback (most recent call last): 2025-12-04T11:21:33.6076691Z W1204 11:18:29.071000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/compile_fx_ext.py", line 486, in serialize_compile 2025-12-04T11:21:33.6076911Z W1204 11:18:29.071000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] ).serialize() 2025-12-04T11:21:33.6077384Z W1204 11:18:29.071000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/compile_fx_ext.py", line 210, in serialize 2025-12-04T11:21:33.6077716Z W1204 11:18:29.071000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] return _WireProtocolPickledInput(GraphPickler.dumps(self)) 2025-12-04T11:21:33.6078339Z W1204 11:18:29.071000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 127, in dumps 2025-12-04T11:21:33.6078578Z W1204 11:18:29.071000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] pickler.dump(obj) 2025-12-04T11:21:33.6079054Z W1204 11:18:29.071000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 94, in reducer_override 2025-12-04T11:21:33.6079377Z W1204 11:18:29.071000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] return _GraphModulePickleData.reduce_helper(self, obj) 2025-12-04T11:21:33.6079841Z W1204 11:18:29.071000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 352, in reduce_helper 2025-12-04T11:21:33.6080094Z W1204 11:18:29.071000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] cls(obj, pickler.options), 2025-12-04T11:21:33.6080561Z W1204 11:18:29.071000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 364, in __init__ 2025-12-04T11:21:33.6080938Z W1204 11:18:29.071000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] self.graph = _GraphPickleData(gm._graph, options) 2025-12-04T11:21:33.6081388Z W1204 11:18:29.071000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 571, in __init__ 2025-12-04T11:21:33.6081683Z W1204 11:18:29.071000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] nodes[node] = _NodePickleData(node, nodes, options) 2025-12-04T11:21:33.6082136Z W1204 11:18:29.071000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 387, in __init__ 2025-12-04T11:21:33.6082453Z W1204 11:18:29.071000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] self.target = _OpPickleData.pickle(node.target, options) 2025-12-04T11:21:33.6082975Z W1204 11:18:29.071000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 435, in pickle 2025-12-04T11:21:33.6083355Z W1204 11:18:29.071000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] return cls._pickle_op(name, _OpOverloadPickleData, options) 2025-12-04T11:21:33.6083814Z W1204 11:18:29.071000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 456, in _pickle_op 2025-12-04T11:21:33.6084172Z W1204 11:18:29.071000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] raise BypassFxGraphCache(f"Unable to pickle non-standard op: {name}") 2025-12-04T11:21:33.6084720Z W1204 11:18:29.071000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] torch._inductor.codecache.BypassFxGraphCache: Unable to pickle non-standard op: torch.ops.rngprims.philox_rand.default 2025-12-04T11:21:33.6085019Z W1204 11:18:29.744000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/1] Unable to pickle input graph or example inputs 2025-12-04T11:21:33.6085281Z W1204 11:18:29.744000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/1] Traceback (most recent call last): 2025-12-04T11:21:33.6085781Z W1204 11:18:29.744000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/1] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/compile_fx_ext.py", line 486, in serialize_compile 2025-12-04T11:21:33.6086003Z W1204 11:18:29.744000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/1] ).serialize() 2025-12-04T11:21:33.6086477Z W1204 11:18:29.744000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/1] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/compile_fx_ext.py", line 210, in serialize 2025-12-04T11:21:33.6086810Z W1204 11:18:29.744000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/1] return _WireProtocolPickledInput(GraphPickler.dumps(self)) 2025-12-04T11:21:33.6087269Z W1204 11:18:29.744000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/1] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 127, in dumps 2025-12-04T11:21:33.6087505Z W1204 11:18:29.744000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/1] pickler.dump(obj) 2025-12-04T11:21:33.6087975Z W1204 11:18:29.744000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/1] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 94, in reducer_override 2025-12-04T11:21:33.6088292Z W1204 11:18:29.744000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/1] return _GraphModulePickleData.reduce_helper(self, obj) 2025-12-04T11:21:33.6088763Z W1204 11:18:29.744000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/1] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 352, in reduce_helper 2025-12-04T11:21:33.6089077Z W1204 11:18:29.744000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/1] cls(obj, pickler.options), 2025-12-04T11:21:33.6089531Z W1204 11:18:29.744000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/1] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 364, in __init__ 2025-12-04T11:21:33.6089831Z W1204 11:18:29.744000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/1] self.graph = _GraphPickleData(gm._graph, options) 2025-12-04T11:21:33.6090282Z W1204 11:18:29.744000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/1] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 571, in __init__ 2025-12-04T11:21:33.6090584Z W1204 11:18:29.744000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/1] nodes[node] = _NodePickleData(node, nodes, options) 2025-12-04T11:21:33.6091114Z W1204 11:18:29.744000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/1] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 387, in __init__ 2025-12-04T11:21:33.6091435Z W1204 11:18:29.744000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/1] self.target = _OpPickleData.pickle(node.target, options) 2025-12-04T11:21:33.6091880Z W1204 11:18:29.744000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/1] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 435, in pickle 2025-12-04T11:21:33.6092207Z W1204 11:18:29.744000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/1] return cls._pickle_op(name, _OpOverloadPickleData, options) 2025-12-04T11:21:33.6092697Z W1204 11:18:29.744000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/1] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 456, in _pickle_op 2025-12-04T11:21:33.6093070Z W1204 11:18:29.744000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/1] raise BypassFxGraphCache(f"Unable to pickle non-standard op: {name}") 2025-12-04T11:21:33.6093574Z W1204 11:18:29.744000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/1] torch._inductor.codecache.BypassFxGraphCache: Unable to pickle non-standard op: torch.ops.rngprims.philox_rand.default 2025-12-04T11:21:33.6093640Z PASSED [1.2783s] [ 65%] 2025-12-04T11:21:33.6093988Z inductor/test_compile_subprocess.py::GPUTests::test_pointwise_bessel_j0_cuda <- test/inductor/test_torchinductor.py PASSED [0.6775s] [ 65%] 2025-12-04T11:21:33.6094322Z inductor/test_compile_subprocess.py::GPUTests::test_pointwise_bessel_j1_cuda <- test/inductor/test_torchinductor.py PASSED [0.2985s] [ 65%] 2025-12-04T11:21:33.6094651Z inductor/test_compile_subprocess.py::GPUTests::test_pointwise_bessel_y0_cuda <- test/inductor/test_torchinductor.py PASSED [0.2607s] [ 66%] 2025-12-04T11:21:33.6094982Z inductor/test_compile_subprocess.py::GPUTests::test_pointwise_bessel_y1_cuda <- test/inductor/test_torchinductor.py PASSED [0.2648s] [ 66%] 2025-12-04T11:21:33.6095358Z inductor/test_compile_subprocess.py::GPUTests::test_pointwise_chebyshev_polynomial_t_cuda <- test/inductor/test_torchinductor.py PASSED [0.5306s] [ 66%] 2025-12-04T11:21:33.6095735Z inductor/test_compile_subprocess.py::GPUTests::test_pointwise_chebyshev_polynomial_u_cuda <- test/inductor/test_torchinductor.py PASSED [0.8177s] [ 66%] 2025-12-04T11:21:33.6096067Z inductor/test_compile_subprocess.py::GPUTests::test_pointwise_digamma_cuda <- test/inductor/test_torchinductor.py PASSED [0.1900s] [ 66%] 2025-12-04T11:21:33.6096679Z inductor/test_compile_subprocess.py::GPUTests::test_pointwise_entr_cuda <- test/inductor/test_torchinductor.py W1204 11:18:33.767000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] Unable to pickle input graph or example inputs 2025-12-04T11:21:33.6096981Z W1204 11:18:33.767000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] Traceback (most recent call last): 2025-12-04T11:21:33.6097491Z W1204 11:18:33.767000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/compile_fx_ext.py", line 486, in serialize_compile 2025-12-04T11:21:33.6097715Z W1204 11:18:33.767000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] ).serialize() 2025-12-04T11:21:33.6098190Z W1204 11:18:33.767000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/compile_fx_ext.py", line 210, in serialize 2025-12-04T11:21:33.6098523Z W1204 11:18:33.767000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] return _WireProtocolPickledInput(GraphPickler.dumps(self)) 2025-12-04T11:21:33.6098972Z W1204 11:18:33.767000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 127, in dumps 2025-12-04T11:21:33.6099279Z W1204 11:18:33.767000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] pickler.dump(obj) 2025-12-04T11:21:33.6099765Z W1204 11:18:33.767000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 94, in reducer_override 2025-12-04T11:21:33.6100082Z W1204 11:18:33.767000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] return _GraphModulePickleData.reduce_helper(self, obj) 2025-12-04T11:21:33.6100550Z W1204 11:18:33.767000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 352, in reduce_helper 2025-12-04T11:21:33.6100835Z W1204 11:18:33.767000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] cls(obj, pickler.options), 2025-12-04T11:21:33.6101291Z W1204 11:18:33.767000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 364, in __init__ 2025-12-04T11:21:33.6101589Z W1204 11:18:33.767000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] self.graph = _GraphPickleData(gm._graph, options) 2025-12-04T11:21:33.6102047Z W1204 11:18:33.767000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 571, in __init__ 2025-12-04T11:21:33.6102351Z W1204 11:18:33.767000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] nodes[node] = _NodePickleData(node, nodes, options) 2025-12-04T11:21:33.6102806Z W1204 11:18:33.767000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 387, in __init__ 2025-12-04T11:21:33.6103126Z W1204 11:18:33.767000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] self.target = _OpPickleData.pickle(node.target, options) 2025-12-04T11:21:33.6103637Z W1204 11:18:33.767000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 435, in pickle 2025-12-04T11:21:33.6103969Z W1204 11:18:33.767000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] return cls._pickle_op(name, _OpOverloadPickleData, options) 2025-12-04T11:21:33.6104429Z W1204 11:18:33.767000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 456, in _pickle_op 2025-12-04T11:21:33.6104803Z W1204 11:18:33.767000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] raise BypassFxGraphCache(f"Unable to pickle non-standard op: {name}") 2025-12-04T11:21:33.6105374Z W1204 11:18:33.767000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] torch._inductor.codecache.BypassFxGraphCache: Unable to pickle non-standard op: torch.ops.prims.convert_element_type.default 2025-12-04T11:21:33.6105443Z PASSED [0.5837s] [ 67%] 2025-12-04T11:21:33.6105777Z inductor/test_compile_subprocess.py::GPUTests::test_pointwise_erfc_cuda <- test/inductor/test_torchinductor.py PASSED [0.3693s] [ 67%] 2025-12-04T11:21:33.6106108Z inductor/test_compile_subprocess.py::GPUTests::test_pointwise_erfinv_cuda <- test/inductor/test_torchinductor.py PASSED [0.3737s] [ 67%] 2025-12-04T11:21:33.6106446Z inductor/test_compile_subprocess.py::GPUTests::test_pointwise_exp2_cuda <- test/inductor/test_torchinductor.py PASSED [0.3420s] [ 67%] 2025-12-04T11:21:33.6106768Z inductor/test_compile_subprocess.py::GPUTests::test_pointwise_expm1_cuda <- test/inductor/test_torchinductor.py PASSED [0.3548s] [ 68%] 2025-12-04T11:21:33.6107097Z inductor/test_compile_subprocess.py::GPUTests::test_pointwise_gammainc_cuda <- test/inductor/test_torchinductor.py PASSED [0.1074s] [ 68%] 2025-12-04T11:21:33.6107508Z inductor/test_compile_subprocess.py::GPUTests::test_pointwise_gammaln_cuda <- test/inductor/test_torchinductor.py PASSED [0.4251s] [ 68%] 2025-12-04T11:21:33.6107875Z inductor/test_compile_subprocess.py::GPUTests::test_pointwise_hermite_polynomial_h_cuda <- test/inductor/test_torchinductor.py PASSED [0.0994s] [ 68%] 2025-12-04T11:21:33.6108244Z inductor/test_compile_subprocess.py::GPUTests::test_pointwise_laguerre_polynomial_l_cuda <- test/inductor/test_torchinductor.py PASSED [0.0970s] [ 69%] 2025-12-04T11:21:33.6108615Z inductor/test_compile_subprocess.py::GPUTests::test_pointwise_legendre_polynomial_p_cuda <- test/inductor/test_torchinductor.py PASSED [0.0961s] [ 69%] 2025-12-04T11:21:33.6109049Z inductor/test_compile_subprocess.py::GPUTests::test_pointwise_log1p_cuda <- test/inductor/test_torchinductor.py PASSED [0.3558s] [ 69%] 2025-12-04T11:21:33.6109412Z inductor/test_compile_subprocess.py::GPUTests::test_pointwise_modified_bessel_i0_cuda <- test/inductor/test_torchinductor.py PASSED [0.1856s] [ 69%] 2025-12-04T11:21:33.6109765Z inductor/test_compile_subprocess.py::GPUTests::test_pointwise_modified_bessel_i1_cuda <- test/inductor/test_torchinductor.py PASSED [0.1867s] [ 69%] 2025-12-04T11:21:33.6110118Z inductor/test_compile_subprocess.py::GPUTests::test_pointwise_modified_bessel_k0_cuda <- test/inductor/test_torchinductor.py PASSED [0.0985s] [ 70%] 2025-12-04T11:21:33.6110465Z inductor/test_compile_subprocess.py::GPUTests::test_pointwise_modified_bessel_k1_cuda <- test/inductor/test_torchinductor.py PASSED [0.0971s] [ 70%] 2025-12-04T11:21:33.6110782Z inductor/test_compile_subprocess.py::GPUTests::test_pointwise_ndtri_cuda <- test/inductor/test_torchinductor.py PASSED [0.0965s] [ 70%] 2025-12-04T11:21:33.6111118Z inductor/test_compile_subprocess.py::GPUTests::test_pointwise_polygamma_cuda <- test/inductor/test_torchinductor.py PASSED [0.1898s] [ 70%] 2025-12-04T11:21:33.6111434Z inductor/test_compile_subprocess.py::GPUTests::test_pointwise_round_cuda <- test/inductor/test_torchinductor.py PASSED [0.3573s] [ 71%] 2025-12-04T11:21:33.6111820Z inductor/test_compile_subprocess.py::GPUTests::test_pointwise_scaled_modified_bessel_k0_cuda <- test/inductor/test_torchinductor.py PASSED [0.0982s] [ 71%] 2025-12-04T11:21:33.6112210Z inductor/test_compile_subprocess.py::GPUTests::test_pointwise_scaled_modified_bessel_k1_cuda <- test/inductor/test_torchinductor.py PASSED [0.0980s] [ 71%] 2025-12-04T11:21:33.6112824Z inductor/test_compile_subprocess.py::GPUTests::test_pointwise_sinc_cuda <- test/inductor/test_torchinductor.py W1204 11:18:38.636000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] Unable to pickle input graph or example inputs 2025-12-04T11:21:33.6113088Z W1204 11:18:38.636000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] Traceback (most recent call last): 2025-12-04T11:21:33.6113590Z W1204 11:18:38.636000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/compile_fx_ext.py", line 486, in serialize_compile 2025-12-04T11:21:33.6113854Z W1204 11:18:38.636000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] ).serialize() 2025-12-04T11:21:33.6114333Z W1204 11:18:38.636000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/compile_fx_ext.py", line 210, in serialize 2025-12-04T11:21:33.6114663Z W1204 11:18:38.636000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] return _WireProtocolPickledInput(GraphPickler.dumps(self)) 2025-12-04T11:21:33.6115110Z W1204 11:18:38.636000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 127, in dumps 2025-12-04T11:21:33.6115345Z W1204 11:18:38.636000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] pickler.dump(obj) 2025-12-04T11:21:33.6115892Z W1204 11:18:38.636000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 94, in reducer_override 2025-12-04T11:21:33.6116209Z W1204 11:18:38.636000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] return _GraphModulePickleData.reduce_helper(self, obj) 2025-12-04T11:21:33.6116683Z W1204 11:18:38.636000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 352, in reduce_helper 2025-12-04T11:21:33.6116941Z W1204 11:18:38.636000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] cls(obj, pickler.options), 2025-12-04T11:21:33.6117449Z W1204 11:18:38.636000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 364, in __init__ 2025-12-04T11:21:33.6117766Z W1204 11:18:38.636000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] self.graph = _GraphPickleData(gm._graph, options) 2025-12-04T11:21:33.6118228Z W1204 11:18:38.636000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 571, in __init__ 2025-12-04T11:21:33.6118527Z W1204 11:18:38.636000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] nodes[node] = _NodePickleData(node, nodes, options) 2025-12-04T11:21:33.6118978Z W1204 11:18:38.636000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 387, in __init__ 2025-12-04T11:21:33.6119301Z W1204 11:18:38.636000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] self.target = _OpPickleData.pickle(node.target, options) 2025-12-04T11:21:33.6119749Z W1204 11:18:38.636000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 435, in pickle 2025-12-04T11:21:33.6120083Z W1204 11:18:38.636000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] return cls._pickle_op(name, _OpOverloadPickleData, options) 2025-12-04T11:21:33.6120545Z W1204 11:18:38.636000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 456, in _pickle_op 2025-12-04T11:21:33.6120904Z W1204 11:18:38.636000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] raise BypassFxGraphCache(f"Unable to pickle non-standard op: {name}") 2025-12-04T11:21:33.6121428Z W1204 11:18:38.636000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] torch._inductor.codecache.BypassFxGraphCache: Unable to pickle non-standard op: torch.ops.prims.convert_element_type.default 2025-12-04T11:21:33.6121531Z PASSED [0.8114s] [ 71%] 2025-12-04T11:21:33.6121905Z inductor/test_compile_subprocess.py::GPUTests::test_pointwise_spherical_bessel_j0_cuda <- test/inductor/test_torchinductor.py PASSED [0.0991s] [ 72%] 2025-12-04T11:21:33.6122519Z inductor/test_compile_subprocess.py::GPUTests::test_pointwise_xlog1py_cuda <- test/inductor/test_torchinductor.py W1204 11:18:39.160000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] Unable to pickle input graph or example inputs 2025-12-04T11:21:33.6122786Z W1204 11:18:39.160000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] Traceback (most recent call last): 2025-12-04T11:21:33.6123290Z W1204 11:18:39.160000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/compile_fx_ext.py", line 486, in serialize_compile 2025-12-04T11:21:33.6123507Z W1204 11:18:39.160000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] ).serialize() 2025-12-04T11:21:33.6124055Z W1204 11:18:39.160000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/compile_fx_ext.py", line 210, in serialize 2025-12-04T11:21:33.6124383Z W1204 11:18:39.160000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] return _WireProtocolPickledInput(GraphPickler.dumps(self)) 2025-12-04T11:21:33.6124833Z W1204 11:18:39.160000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 127, in dumps 2025-12-04T11:21:33.6125063Z W1204 11:18:39.160000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] pickler.dump(obj) 2025-12-04T11:21:33.6125575Z W1204 11:18:39.160000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 94, in reducer_override 2025-12-04T11:21:33.6125898Z W1204 11:18:39.160000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] return _GraphModulePickleData.reduce_helper(self, obj) 2025-12-04T11:21:33.6126367Z W1204 11:18:39.160000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 352, in reduce_helper 2025-12-04T11:21:33.6126628Z W1204 11:18:39.160000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] cls(obj, pickler.options), 2025-12-04T11:21:33.6127084Z W1204 11:18:39.160000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 364, in __init__ 2025-12-04T11:21:33.6127388Z W1204 11:18:39.160000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] self.graph = _GraphPickleData(gm._graph, options) 2025-12-04T11:21:33.6127849Z W1204 11:18:39.160000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 571, in __init__ 2025-12-04T11:21:33.6128147Z W1204 11:18:39.160000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] nodes[node] = _NodePickleData(node, nodes, options) 2025-12-04T11:21:33.6128597Z W1204 11:18:39.160000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 387, in __init__ 2025-12-04T11:21:33.6128912Z W1204 11:18:39.160000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] self.target = _OpPickleData.pickle(node.target, options) 2025-12-04T11:21:33.6129363Z W1204 11:18:39.160000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 435, in pickle 2025-12-04T11:21:33.6129734Z W1204 11:18:39.160000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] return cls._pickle_op(name, _OpOverloadPickleData, options) 2025-12-04T11:21:33.6130203Z W1204 11:18:39.160000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 456, in _pickle_op 2025-12-04T11:21:33.6130560Z W1204 11:18:39.160000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] raise BypassFxGraphCache(f"Unable to pickle non-standard op: {name}") 2025-12-04T11:21:33.6131078Z W1204 11:18:39.160000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] torch._inductor.codecache.BypassFxGraphCache: Unable to pickle non-standard op: torch.ops.prims.convert_element_type.default 2025-12-04T11:21:33.6131147Z PASSED [0.4147s] [ 72%] 2025-12-04T11:21:33.6131751Z inductor/test_compile_subprocess.py::GPUTests::test_pointwise_xlogy_cuda <- test/inductor/test_torchinductor.py W1204 11:18:39.580000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] Unable to pickle input graph or example inputs 2025-12-04T11:21:33.6132090Z W1204 11:18:39.580000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] Traceback (most recent call last): 2025-12-04T11:21:33.6132592Z W1204 11:18:39.580000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/compile_fx_ext.py", line 486, in serialize_compile 2025-12-04T11:21:33.6132823Z W1204 11:18:39.580000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] ).serialize() 2025-12-04T11:21:33.6133296Z W1204 11:18:39.580000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/compile_fx_ext.py", line 210, in serialize 2025-12-04T11:21:33.6133658Z W1204 11:18:39.580000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] return _WireProtocolPickledInput(GraphPickler.dumps(self)) 2025-12-04T11:21:33.6134110Z W1204 11:18:39.580000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 127, in dumps 2025-12-04T11:21:33.6134352Z W1204 11:18:39.580000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] pickler.dump(obj) 2025-12-04T11:21:33.6134828Z W1204 11:18:39.580000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 94, in reducer_override 2025-12-04T11:21:33.6135142Z W1204 11:18:39.580000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] return _GraphModulePickleData.reduce_helper(self, obj) 2025-12-04T11:21:33.6135612Z W1204 11:18:39.580000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 352, in reduce_helper 2025-12-04T11:21:33.6135867Z W1204 11:18:39.580000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] cls(obj, pickler.options), 2025-12-04T11:21:33.6136321Z W1204 11:18:39.580000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 364, in __init__ 2025-12-04T11:21:33.6136626Z W1204 11:18:39.580000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] self.graph = _GraphPickleData(gm._graph, options) 2025-12-04T11:21:33.6137074Z W1204 11:18:39.580000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 571, in __init__ 2025-12-04T11:21:33.6137374Z W1204 11:18:39.580000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] nodes[node] = _NodePickleData(node, nodes, options) 2025-12-04T11:21:33.6137824Z W1204 11:18:39.580000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 387, in __init__ 2025-12-04T11:21:33.6138181Z W1204 11:18:39.580000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] self.target = _OpPickleData.pickle(node.target, options) 2025-12-04T11:21:33.6138629Z W1204 11:18:39.580000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 435, in pickle 2025-12-04T11:21:33.6138954Z W1204 11:18:39.580000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] return cls._pickle_op(name, _OpOverloadPickleData, options) 2025-12-04T11:21:33.6139413Z W1204 11:18:39.580000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 456, in _pickle_op 2025-12-04T11:21:33.6139768Z W1204 11:18:39.580000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] raise BypassFxGraphCache(f"Unable to pickle non-standard op: {name}") 2025-12-04T11:21:33.6140357Z W1204 11:18:39.580000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] torch._inductor.codecache.BypassFxGraphCache: Unable to pickle non-standard op: torch.ops.prims.convert_element_type.default 2025-12-04T11:21:33.6140420Z PASSED [0.4144s] [ 72%] 2025-12-04T11:21:33.6140744Z inductor/test_compile_subprocess.py::GPUTests::test_pointwise_zeta_cuda <- test/inductor/test_torchinductor.py PASSED [0.1065s] [ 72%] 2025-12-04T11:21:33.6141043Z inductor/test_compile_subprocess.py::GPUTests::test_pow3_cuda <- test/inductor/test_torchinductor.py PASSED [0.2656s] [ 72%] 2025-12-04T11:21:33.6141344Z inductor/test_compile_subprocess.py::GPUTests::test_pow_int_cuda <- test/inductor/test_torchinductor.py PASSED [1.6879s] [ 73%] 2025-12-04T11:21:33.6141981Z inductor/test_compile_subprocess.py::GPUTests::test_pow_symfloat_cuda <- test/inductor/test_torchinductor.py W1204 11:18:41.862000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] Unable to pickle input graph or example inputs 2025-12-04T11:21:33.6142253Z W1204 11:18:41.862000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] Traceback (most recent call last): 2025-12-04T11:21:33.6142755Z W1204 11:18:41.862000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/compile_fx_ext.py", line 486, in serialize_compile 2025-12-04T11:21:33.6142973Z W1204 11:18:41.862000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] ).serialize() 2025-12-04T11:21:33.6143447Z W1204 11:18:41.862000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/compile_fx_ext.py", line 210, in serialize 2025-12-04T11:21:33.6143821Z W1204 11:18:41.862000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] return _WireProtocolPickledInput(GraphPickler.dumps(self)) 2025-12-04T11:21:33.6144281Z W1204 11:18:41.862000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 127, in dumps 2025-12-04T11:21:33.6144518Z W1204 11:18:41.862000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] pickler.dump(obj) 2025-12-04T11:21:33.6144988Z W1204 11:18:41.862000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 94, in reducer_override 2025-12-04T11:21:33.6145301Z W1204 11:18:41.862000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] return _GraphModulePickleData.reduce_helper(self, obj) 2025-12-04T11:21:33.6145771Z W1204 11:18:41.862000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 352, in reduce_helper 2025-12-04T11:21:33.6146068Z W1204 11:18:41.862000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] cls(obj, pickler.options), 2025-12-04T11:21:33.6146526Z W1204 11:18:41.862000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 364, in __init__ 2025-12-04T11:21:33.6146826Z W1204 11:18:41.862000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] self.graph = _GraphPickleData(gm._graph, options) 2025-12-04T11:21:33.6147283Z W1204 11:18:41.862000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 571, in __init__ 2025-12-04T11:21:33.6147579Z W1204 11:18:41.862000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] nodes[node] = _NodePickleData(node, nodes, options) 2025-12-04T11:21:33.6148031Z W1204 11:18:41.862000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 387, in __init__ 2025-12-04T11:21:33.6148455Z W1204 11:18:41.862000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] self.target = _OpPickleData.pickle(node.target, options) 2025-12-04T11:21:33.6148903Z W1204 11:18:41.862000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 435, in pickle 2025-12-04T11:21:33.6149229Z W1204 11:18:41.862000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] return cls._pickle_op(name, _OpOverloadPickleData, options) 2025-12-04T11:21:33.6149718Z W1204 11:18:41.862000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 456, in _pickle_op 2025-12-04T11:21:33.6150087Z W1204 11:18:41.862000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] raise BypassFxGraphCache(f"Unable to pickle non-standard op: {name}") 2025-12-04T11:21:33.6150621Z W1204 11:18:41.862000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] torch._inductor.codecache.BypassFxGraphCache: Unable to pickle non-standard op: torch.ops.prims.convert_element_type.default 2025-12-04T11:21:33.6150685Z PASSED [0.4785s] [ 73%] 2025-12-04T11:21:33.6150990Z inductor/test_compile_subprocess.py::GPUTests::test_prod_cuda <- test/inductor/test_torchinductor.py PASSED [1.2117s] [ 73%] 2025-12-04T11:21:33.6151348Z inductor/test_compile_subprocess.py::GPUTests::test_profiler_mark_wrapper_call_cuda <- test/inductor/test_torchinductor.py PASSED [0.1738s] [ 73%] 2025-12-04T11:21:33.6151979Z inductor/test_compile_subprocess.py::GPUTests::test_rand_like_deterministic_cuda <- test/inductor/test_torchinductor.py W1204 11:18:43.712000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] Unable to pickle input graph or example inputs 2025-12-04T11:21:33.6152246Z W1204 11:18:43.712000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] Traceback (most recent call last): 2025-12-04T11:21:33.6152751Z W1204 11:18:43.712000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/compile_fx_ext.py", line 486, in serialize_compile 2025-12-04T11:21:33.6152982Z W1204 11:18:43.712000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] ).serialize() 2025-12-04T11:21:33.6153454Z W1204 11:18:43.712000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/compile_fx_ext.py", line 210, in serialize 2025-12-04T11:21:33.6153790Z W1204 11:18:43.712000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] return _WireProtocolPickledInput(GraphPickler.dumps(self)) 2025-12-04T11:21:33.6154271Z W1204 11:18:43.712000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 127, in dumps 2025-12-04T11:21:33.6154510Z W1204 11:18:43.712000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] pickler.dump(obj) 2025-12-04T11:21:33.6154979Z W1204 11:18:43.712000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 94, in reducer_override 2025-12-04T11:21:33.6155294Z W1204 11:18:43.712000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] return _GraphModulePickleData.reduce_helper(self, obj) 2025-12-04T11:21:33.6155764Z W1204 11:18:43.712000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 352, in reduce_helper 2025-12-04T11:21:33.6156026Z W1204 11:18:43.712000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] cls(obj, pickler.options), 2025-12-04T11:21:33.6156560Z W1204 11:18:43.712000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 364, in __init__ 2025-12-04T11:21:33.6156862Z W1204 11:18:43.712000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] self.graph = _GraphPickleData(gm._graph, options) 2025-12-04T11:21:33.6157318Z W1204 11:18:43.712000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 571, in __init__ 2025-12-04T11:21:33.6157697Z W1204 11:18:43.712000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] nodes[node] = _NodePickleData(node, nodes, options) 2025-12-04T11:21:33.6158244Z W1204 11:18:43.712000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 387, in __init__ 2025-12-04T11:21:33.6158628Z W1204 11:18:43.712000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] self.target = _OpPickleData.pickle(node.target, options) 2025-12-04T11:21:33.6159161Z W1204 11:18:43.712000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 435, in pickle 2025-12-04T11:21:33.6159548Z W1204 11:18:43.712000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] return cls._pickle_op(name, _OpOverloadPickleData, options) 2025-12-04T11:21:33.6160011Z W1204 11:18:43.712000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 456, in _pickle_op 2025-12-04T11:21:33.6160377Z W1204 11:18:43.712000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] raise BypassFxGraphCache(f"Unable to pickle non-standard op: {name}") 2025-12-04T11:21:33.6160882Z W1204 11:18:43.712000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] torch._inductor.codecache.BypassFxGraphCache: Unable to pickle non-standard op: torch.ops.prims.inductor_seeds.default 2025-12-04T11:21:33.6160946Z PASSED [0.5065s] [ 74%] 2025-12-04T11:21:33.6167221Z inductor/test_compile_subprocess.py::GPUTests::test_randint_cuda <- test/inductor/test_torchinductor.py W1204 11:18:44.235000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] Unable to pickle input graph or example inputs 2025-12-04T11:21:33.6167570Z W1204 11:18:44.235000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] Traceback (most recent call last): 2025-12-04T11:21:33.6168135Z W1204 11:18:44.235000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/compile_fx_ext.py", line 486, in serialize_compile 2025-12-04T11:21:33.6168446Z W1204 11:18:44.235000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] ).serialize() 2025-12-04T11:21:33.6168960Z W1204 11:18:44.235000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/compile_fx_ext.py", line 210, in serialize 2025-12-04T11:21:33.6169312Z W1204 11:18:44.235000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] return _WireProtocolPickledInput(GraphPickler.dumps(self)) 2025-12-04T11:21:33.6169787Z W1204 11:18:44.235000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 127, in dumps 2025-12-04T11:21:33.6170030Z W1204 11:18:44.235000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] pickler.dump(obj) 2025-12-04T11:21:33.6170528Z W1204 11:18:44.235000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 94, in reducer_override 2025-12-04T11:21:33.6170938Z W1204 11:18:44.235000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] return _GraphModulePickleData.reduce_helper(self, obj) 2025-12-04T11:21:33.6171420Z W1204 11:18:44.235000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 352, in reduce_helper 2025-12-04T11:21:33.6171691Z W1204 11:18:44.235000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] cls(obj, pickler.options), 2025-12-04T11:21:33.6172181Z W1204 11:18:44.235000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 364, in __init__ 2025-12-04T11:21:33.6172492Z W1204 11:18:44.235000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] self.graph = _GraphPickleData(gm._graph, options) 2025-12-04T11:21:33.6172948Z W1204 11:18:44.235000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 571, in __init__ 2025-12-04T11:21:33.6173246Z W1204 11:18:44.235000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] nodes[node] = _NodePickleData(node, nodes, options) 2025-12-04T11:21:33.6173701Z W1204 11:18:44.235000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 387, in __init__ 2025-12-04T11:21:33.6174019Z W1204 11:18:44.235000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] self.target = _OpPickleData.pickle(node.target, options) 2025-12-04T11:21:33.6174468Z W1204 11:18:44.235000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 435, in pickle 2025-12-04T11:21:33.6174811Z W1204 11:18:44.235000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] return cls._pickle_op(name, _OpOverloadPickleData, options) 2025-12-04T11:21:33.6175278Z W1204 11:18:44.235000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 456, in _pickle_op 2025-12-04T11:21:33.6175635Z W1204 11:18:44.235000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] raise BypassFxGraphCache(f"Unable to pickle non-standard op: {name}") 2025-12-04T11:21:33.6176136Z W1204 11:18:44.235000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] torch._inductor.codecache.BypassFxGraphCache: Unable to pickle non-standard op: torch.ops.prims.inductor_seeds.default 2025-12-04T11:21:33.6176207Z PASSED [0.6244s] [ 74%] 2025-12-04T11:21:33.6176863Z inductor/test_compile_subprocess.py::GPUTests::test_randn_generator_cuda <- test/inductor/test_torchinductor.py W1204 11:18:44.843000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] Unable to pickle input graph or example inputs 2025-12-04T11:21:33.6177148Z W1204 11:18:44.843000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] Traceback (most recent call last): 2025-12-04T11:21:33.6177656Z W1204 11:18:44.843000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/compile_fx_ext.py", line 486, in serialize_compile 2025-12-04T11:21:33.6178060Z W1204 11:18:44.843000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] ).serialize() 2025-12-04T11:21:33.6178537Z W1204 11:18:44.843000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/compile_fx_ext.py", line 210, in serialize 2025-12-04T11:21:33.6178869Z W1204 11:18:44.843000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] return _WireProtocolPickledInput(GraphPickler.dumps(self)) 2025-12-04T11:21:33.6179443Z W1204 11:18:44.843000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 127, in dumps 2025-12-04T11:21:33.6179678Z W1204 11:18:44.843000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] pickler.dump(obj) 2025-12-04T11:21:33.6180164Z W1204 11:18:44.843000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 94, in reducer_override 2025-12-04T11:21:33.6180530Z W1204 11:18:44.843000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] return _GraphModulePickleData.reduce_helper(self, obj) 2025-12-04T11:21:33.6181003Z W1204 11:18:44.843000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 352, in reduce_helper 2025-12-04T11:21:33.6181265Z W1204 11:18:44.843000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] cls(obj, pickler.options), 2025-12-04T11:21:33.6181717Z W1204 11:18:44.843000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 364, in __init__ 2025-12-04T11:21:33.6182021Z W1204 11:18:44.843000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] self.graph = _GraphPickleData(gm._graph, options) 2025-12-04T11:21:33.6182466Z W1204 11:18:44.843000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 571, in __init__ 2025-12-04T11:21:33.6182770Z W1204 11:18:44.843000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] nodes[node] = _NodePickleData(node, nodes, options) 2025-12-04T11:21:33.6183557Z W1204 11:18:44.843000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 387, in __init__ 2025-12-04T11:21:33.6183910Z W1204 11:18:44.843000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] self.target = _OpPickleData.pickle(node.target, options) 2025-12-04T11:21:33.6184391Z W1204 11:18:44.843000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 435, in pickle 2025-12-04T11:21:33.6184730Z W1204 11:18:44.843000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] return cls._pickle_op(name, _OpOverloadPickleData, options) 2025-12-04T11:21:33.6185193Z W1204 11:18:44.843000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 456, in _pickle_op 2025-12-04T11:21:33.6185655Z W1204 11:18:44.843000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] raise BypassFxGraphCache(f"Unable to pickle non-standard op: {name}") 2025-12-04T11:21:33.6186163Z W1204 11:18:44.843000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] torch._inductor.codecache.BypassFxGraphCache: Unable to pickle non-standard op: torch.ops.prims.inductor_seeds.default 2025-12-04T11:21:33.6186232Z PASSED [0.5316s] [ 74%] 2025-12-04T11:21:33.6186568Z inductor/test_compile_subprocess.py::GPUTests::test_reduction1_cuda <- test/inductor/test_torchinductor.py PASSED [0.4379s] [ 74%] 2025-12-04T11:21:33.6186880Z inductor/test_compile_subprocess.py::GPUTests::test_reduction3_cuda <- test/inductor/test_torchinductor.py PASSED [0.4211s] [ 75%] 2025-12-04T11:21:33.6187234Z inductor/test_compile_subprocess.py::GPUTests::test_reduction_config_limit_cuda <- test/inductor/test_torchinductor.py PASSED [0.0026s] [ 75%] 2025-12-04T11:21:33.6187935Z inductor/test_compile_subprocess.py::GPUTests::test_reflection_pad2d_backward_cuda <- test/inductor/test_torchinductor.py W1204 11:18:46.286000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] Unable to pickle input graph or example inputs 2025-12-04T11:21:33.6188248Z W1204 11:18:46.286000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] Traceback (most recent call last): 2025-12-04T11:21:33.6188766Z W1204 11:18:46.286000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/compile_fx_ext.py", line 486, in serialize_compile 2025-12-04T11:21:33.6188988Z W1204 11:18:46.286000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] ).serialize() 2025-12-04T11:21:33.6189504Z W1204 11:18:46.286000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/compile_fx_ext.py", line 210, in serialize 2025-12-04T11:21:33.6189835Z W1204 11:18:46.286000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] return _WireProtocolPickledInput(GraphPickler.dumps(self)) 2025-12-04T11:21:33.6190285Z W1204 11:18:46.286000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 127, in dumps 2025-12-04T11:21:33.6190518Z W1204 11:18:46.286000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] pickler.dump(obj) 2025-12-04T11:21:33.6190991Z W1204 11:18:46.286000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 94, in reducer_override 2025-12-04T11:21:33.6191310Z W1204 11:18:46.286000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] return _GraphModulePickleData.reduce_helper(self, obj) 2025-12-04T11:21:33.6191788Z W1204 11:18:46.286000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 352, in reduce_helper 2025-12-04T11:21:33.6192059Z W1204 11:18:46.286000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] cls(obj, pickler.options), 2025-12-04T11:21:33.6192514Z W1204 11:18:46.286000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 364, in __init__ 2025-12-04T11:21:33.6192816Z W1204 11:18:46.286000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] self.graph = _GraphPickleData(gm._graph, options) 2025-12-04T11:21:33.6193267Z W1204 11:18:46.286000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 571, in __init__ 2025-12-04T11:21:33.6193604Z W1204 11:18:46.286000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] nodes[node] = _NodePickleData(node, nodes, options) 2025-12-04T11:21:33.6194064Z W1204 11:18:46.286000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 387, in __init__ 2025-12-04T11:21:33.6194384Z W1204 11:18:46.286000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] self.target = _OpPickleData.pickle(node.target, options) 2025-12-04T11:21:33.6194837Z W1204 11:18:46.286000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 435, in pickle 2025-12-04T11:21:33.6195163Z W1204 11:18:46.286000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] return cls._pickle_op(name, _OpOverloadPickleData, options) 2025-12-04T11:21:33.6195633Z W1204 11:18:46.286000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 456, in _pickle_op 2025-12-04T11:21:33.6196076Z W1204 11:18:46.286000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] raise BypassFxGraphCache(f"Unable to pickle non-standard op: {name}") 2025-12-04T11:21:33.6196553Z W1204 11:18:46.286000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] torch._inductor.codecache.BypassFxGraphCache: Unable to pickle non-standard op: torch.ops.prims.iota.default 2025-12-04T11:21:33.6196847Z W1204 11:18:46.475000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] Unable to pickle input graph or example inputs 2025-12-04T11:21:33.6197108Z W1204 11:18:46.475000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] Traceback (most recent call last): 2025-12-04T11:21:33.6197647Z W1204 11:18:46.475000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/compile_fx_ext.py", line 486, in serialize_compile 2025-12-04T11:21:33.6197869Z W1204 11:18:46.475000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] ).serialize() 2025-12-04T11:21:33.6198358Z W1204 11:18:46.475000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/compile_fx_ext.py", line 210, in serialize 2025-12-04T11:21:33.6198684Z W1204 11:18:46.475000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] return _WireProtocolPickledInput(GraphPickler.dumps(self)) 2025-12-04T11:21:33.6199124Z W1204 11:18:46.475000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 127, in dumps 2025-12-04T11:21:33.6199359Z W1204 11:18:46.475000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] pickler.dump(obj) 2025-12-04T11:21:33.6199832Z W1204 11:18:46.475000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 94, in reducer_override 2025-12-04T11:21:33.6200153Z W1204 11:18:46.475000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] return _GraphModulePickleData.reduce_helper(self, obj) 2025-12-04T11:21:33.6200617Z W1204 11:18:46.475000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 352, in reduce_helper 2025-12-04T11:21:33.6200875Z W1204 11:18:46.475000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] cls(obj, pickler.options), 2025-12-04T11:21:33.6201324Z W1204 11:18:46.475000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 364, in __init__ 2025-12-04T11:21:33.6201661Z W1204 11:18:46.475000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] self.graph = _GraphPickleData(gm._graph, options) 2025-12-04T11:21:33.6202118Z W1204 11:18:46.475000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 571, in __init__ 2025-12-04T11:21:33.6202415Z W1204 11:18:46.475000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] nodes[node] = _NodePickleData(node, nodes, options) 2025-12-04T11:21:33.6202867Z W1204 11:18:46.475000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 387, in __init__ 2025-12-04T11:21:33.6203184Z W1204 11:18:46.475000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] self.target = _OpPickleData.pickle(node.target, options) 2025-12-04T11:21:33.6203630Z W1204 11:18:46.475000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 435, in pickle 2025-12-04T11:21:33.6204031Z W1204 11:18:46.475000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] return cls._pickle_op(name, _OpOverloadPickleData, options) 2025-12-04T11:21:33.6204492Z W1204 11:18:46.475000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 456, in _pickle_op 2025-12-04T11:21:33.6204850Z W1204 11:18:46.475000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] raise BypassFxGraphCache(f"Unable to pickle non-standard op: {name}") 2025-12-04T11:21:33.6205366Z W1204 11:18:46.475000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] torch._inductor.codecache.BypassFxGraphCache: Unable to pickle non-standard op: torch.ops.prims.iota.default 2025-12-04T11:21:33.6205665Z W1204 11:18:46.724000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] Unable to pickle input graph or example inputs 2025-12-04T11:21:33.6205925Z W1204 11:18:46.724000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] Traceback (most recent call last): 2025-12-04T11:21:33.6206424Z W1204 11:18:46.724000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/compile_fx_ext.py", line 486, in serialize_compile 2025-12-04T11:21:33.6206645Z W1204 11:18:46.724000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] ).serialize() 2025-12-04T11:21:33.6207116Z W1204 11:18:46.724000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/compile_fx_ext.py", line 210, in serialize 2025-12-04T11:21:33.6207451Z W1204 11:18:46.724000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] return _WireProtocolPickledInput(GraphPickler.dumps(self)) 2025-12-04T11:21:33.6207903Z W1204 11:18:46.724000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 127, in dumps 2025-12-04T11:21:33.6208146Z W1204 11:18:46.724000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] pickler.dump(obj) 2025-12-04T11:21:33.6208624Z W1204 11:18:46.724000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 94, in reducer_override 2025-12-04T11:21:33.6208941Z W1204 11:18:46.724000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] return _GraphModulePickleData.reduce_helper(self, obj) 2025-12-04T11:21:33.6209413Z W1204 11:18:46.724000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 352, in reduce_helper 2025-12-04T11:21:33.6209706Z W1204 11:18:46.724000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] cls(obj, pickler.options), 2025-12-04T11:21:33.6210164Z W1204 11:18:46.724000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 364, in __init__ 2025-12-04T11:21:33.6210460Z W1204 11:18:46.724000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] self.graph = _GraphPickleData(gm._graph, options) 2025-12-04T11:21:33.6210915Z W1204 11:18:46.724000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 571, in __init__ 2025-12-04T11:21:33.6211212Z W1204 11:18:46.724000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] nodes[node] = _NodePickleData(node, nodes, options) 2025-12-04T11:21:33.6211658Z W1204 11:18:46.724000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 387, in __init__ 2025-12-04T11:21:33.6212046Z W1204 11:18:46.724000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] self.target = _OpPickleData.pickle(node.target, options) 2025-12-04T11:21:33.6212499Z W1204 11:18:46.724000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 435, in pickle 2025-12-04T11:21:33.6212827Z W1204 11:18:46.724000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] return cls._pickle_op(name, _OpOverloadPickleData, options) 2025-12-04T11:21:33.6213317Z W1204 11:18:46.724000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 456, in _pickle_op 2025-12-04T11:21:33.6213675Z W1204 11:18:46.724000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] raise BypassFxGraphCache(f"Unable to pickle non-standard op: {name}") 2025-12-04T11:21:33.6214155Z W1204 11:18:46.724000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] torch._inductor.codecache.BypassFxGraphCache: Unable to pickle non-standard op: torch.ops.prims.iota.default 2025-12-04T11:21:33.6214443Z W1204 11:18:47.271000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] Unable to pickle input graph or example inputs 2025-12-04T11:21:33.6214707Z W1204 11:18:47.271000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] Traceback (most recent call last): 2025-12-04T11:21:33.6215206Z W1204 11:18:47.271000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/compile_fx_ext.py", line 486, in serialize_compile 2025-12-04T11:21:33.6215427Z W1204 11:18:47.271000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] ).serialize() 2025-12-04T11:21:33.6215911Z W1204 11:18:47.271000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/compile_fx_ext.py", line 210, in serialize 2025-12-04T11:21:33.6216240Z W1204 11:18:47.271000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] return _WireProtocolPickledInput(GraphPickler.dumps(self)) 2025-12-04T11:21:33.6216693Z W1204 11:18:47.271000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 127, in dumps 2025-12-04T11:21:33.6216921Z W1204 11:18:47.271000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] pickler.dump(obj) 2025-12-04T11:21:33.6217397Z W1204 11:18:47.271000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 94, in reducer_override 2025-12-04T11:21:33.6217768Z W1204 11:18:47.271000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] return _GraphModulePickleData.reduce_helper(self, obj) 2025-12-04T11:21:33.6218247Z W1204 11:18:47.271000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 352, in reduce_helper 2025-12-04T11:21:33.6218502Z W1204 11:18:47.271000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] cls(obj, pickler.options), 2025-12-04T11:21:33.6218952Z W1204 11:18:47.271000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 364, in __init__ 2025-12-04T11:21:33.6219253Z W1204 11:18:47.271000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] self.graph = _GraphPickleData(gm._graph, options) 2025-12-04T11:21:33.6219703Z W1204 11:18:47.271000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 571, in __init__ 2025-12-04T11:21:33.6220071Z W1204 11:18:47.271000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] nodes[node] = _NodePickleData(node, nodes, options) 2025-12-04T11:21:33.6220533Z W1204 11:18:47.271000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 387, in __init__ 2025-12-04T11:21:33.6220855Z W1204 11:18:47.271000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] self.target = _OpPickleData.pickle(node.target, options) 2025-12-04T11:21:33.6221335Z W1204 11:18:47.271000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 435, in pickle 2025-12-04T11:21:33.6221662Z W1204 11:18:47.271000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] return cls._pickle_op(name, _OpOverloadPickleData, options) 2025-12-04T11:21:33.6222129Z W1204 11:18:47.271000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 456, in _pickle_op 2025-12-04T11:21:33.6222484Z W1204 11:18:47.271000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] raise BypassFxGraphCache(f"Unable to pickle non-standard op: {name}") 2025-12-04T11:21:33.6222961Z W1204 11:18:47.271000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] torch._inductor.codecache.BypassFxGraphCache: Unable to pickle non-standard op: torch.ops.prims.iota.default 2025-12-04T11:21:33.6223251Z W1204 11:18:47.778000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] Unable to pickle input graph or example inputs 2025-12-04T11:21:33.6223582Z W1204 11:18:47.778000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] Traceback (most recent call last): 2025-12-04T11:21:33.6224092Z W1204 11:18:47.778000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/compile_fx_ext.py", line 486, in serialize_compile 2025-12-04T11:21:33.6224309Z W1204 11:18:47.778000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] ).serialize() 2025-12-04T11:21:33.6224796Z W1204 11:18:47.778000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/compile_fx_ext.py", line 210, in serialize 2025-12-04T11:21:33.6225125Z W1204 11:18:47.778000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] return _WireProtocolPickledInput(GraphPickler.dumps(self)) 2025-12-04T11:21:33.6225573Z W1204 11:18:47.778000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 127, in dumps 2025-12-04T11:21:33.6225848Z W1204 11:18:47.778000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] pickler.dump(obj) 2025-12-04T11:21:33.6226321Z W1204 11:18:47.778000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 94, in reducer_override 2025-12-04T11:21:33.6226648Z W1204 11:18:47.778000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] return _GraphModulePickleData.reduce_helper(self, obj) 2025-12-04T11:21:33.6227114Z W1204 11:18:47.778000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 352, in reduce_helper 2025-12-04T11:21:33.6227379Z W1204 11:18:47.778000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] cls(obj, pickler.options), 2025-12-04T11:21:33.6227829Z W1204 11:18:47.778000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 364, in __init__ 2025-12-04T11:21:33.6228528Z W1204 11:18:47.778000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] self.graph = _GraphPickleData(gm._graph, options) 2025-12-04T11:21:33.6228981Z W1204 11:18:47.778000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 571, in __init__ 2025-12-04T11:21:33.6229283Z W1204 11:18:47.778000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] nodes[node] = _NodePickleData(node, nodes, options) 2025-12-04T11:21:33.6229780Z W1204 11:18:47.778000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 387, in __init__ 2025-12-04T11:21:33.6230102Z W1204 11:18:47.778000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] self.target = _OpPickleData.pickle(node.target, options) 2025-12-04T11:21:33.6230555Z W1204 11:18:47.778000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 435, in pickle 2025-12-04T11:21:33.6230880Z W1204 11:18:47.778000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] return cls._pickle_op(name, _OpOverloadPickleData, options) 2025-12-04T11:21:33.6231343Z W1204 11:18:47.778000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 456, in _pickle_op 2025-12-04T11:21:33.6231702Z W1204 11:18:47.778000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] raise BypassFxGraphCache(f"Unable to pickle non-standard op: {name}") 2025-12-04T11:21:33.6232177Z W1204 11:18:47.778000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] torch._inductor.codecache.BypassFxGraphCache: Unable to pickle non-standard op: torch.ops.prims.iota.default 2025-12-04T11:21:33.6232478Z W1204 11:18:48.318000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] Unable to pickle input graph or example inputs 2025-12-04T11:21:33.6232739Z W1204 11:18:48.318000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] Traceback (most recent call last): 2025-12-04T11:21:33.6233248Z W1204 11:18:48.318000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/compile_fx_ext.py", line 486, in serialize_compile 2025-12-04T11:21:33.6233466Z W1204 11:18:48.318000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] ).serialize() 2025-12-04T11:21:33.6233940Z W1204 11:18:48.318000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/compile_fx_ext.py", line 210, in serialize 2025-12-04T11:21:33.6234315Z W1204 11:18:48.318000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] return _WireProtocolPickledInput(GraphPickler.dumps(self)) 2025-12-04T11:21:33.6234760Z W1204 11:18:48.318000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 127, in dumps 2025-12-04T11:21:33.6234994Z W1204 11:18:48.318000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] pickler.dump(obj) 2025-12-04T11:21:33.6235469Z W1204 11:18:48.318000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 94, in reducer_override 2025-12-04T11:21:33.6235801Z W1204 11:18:48.318000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] return _GraphModulePickleData.reduce_helper(self, obj) 2025-12-04T11:21:33.6236305Z W1204 11:18:48.318000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 352, in reduce_helper 2025-12-04T11:21:33.6236595Z W1204 11:18:48.318000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] cls(obj, pickler.options), 2025-12-04T11:21:33.6237051Z W1204 11:18:48.318000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 364, in __init__ 2025-12-04T11:21:33.6237347Z W1204 11:18:48.318000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] self.graph = _GraphPickleData(gm._graph, options) 2025-12-04T11:21:33.6237848Z W1204 11:18:48.318000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 571, in __init__ 2025-12-04T11:21:33.6238149Z W1204 11:18:48.318000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] nodes[node] = _NodePickleData(node, nodes, options) 2025-12-04T11:21:33.6238605Z W1204 11:18:48.318000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 387, in __init__ 2025-12-04T11:21:33.6238922Z W1204 11:18:48.318000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] self.target = _OpPickleData.pickle(node.target, options) 2025-12-04T11:21:33.6239369Z W1204 11:18:48.318000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 435, in pickle 2025-12-04T11:21:33.6239696Z W1204 11:18:48.318000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] return cls._pickle_op(name, _OpOverloadPickleData, options) 2025-12-04T11:21:33.6240153Z W1204 11:18:48.318000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 456, in _pickle_op 2025-12-04T11:21:33.6240525Z W1204 11:18:48.318000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] raise BypassFxGraphCache(f"Unable to pickle non-standard op: {name}") 2025-12-04T11:21:33.6240999Z W1204 11:18:48.318000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] torch._inductor.codecache.BypassFxGraphCache: Unable to pickle non-standard op: torch.ops.prims.iota.default 2025-12-04T11:21:33.6241292Z W1204 11:18:48.782000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] Unable to pickle input graph or example inputs 2025-12-04T11:21:33.6241551Z W1204 11:18:48.782000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] Traceback (most recent call last): 2025-12-04T11:21:33.6242055Z W1204 11:18:48.782000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/compile_fx_ext.py", line 486, in serialize_compile 2025-12-04T11:21:33.6242328Z W1204 11:18:48.782000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] ).serialize() 2025-12-04T11:21:33.6242801Z W1204 11:18:48.782000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/compile_fx_ext.py", line 210, in serialize 2025-12-04T11:21:33.6243130Z W1204 11:18:48.782000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] return _WireProtocolPickledInput(GraphPickler.dumps(self)) 2025-12-04T11:21:33.6243571Z W1204 11:18:48.782000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 127, in dumps 2025-12-04T11:21:33.6243802Z W1204 11:18:48.782000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] pickler.dump(obj) 2025-12-04T11:21:33.6244312Z W1204 11:18:48.782000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 94, in reducer_override 2025-12-04T11:21:33.6244740Z W1204 11:18:48.782000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] return _GraphModulePickleData.reduce_helper(self, obj) 2025-12-04T11:21:33.6245210Z W1204 11:18:48.782000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 352, in reduce_helper 2025-12-04T11:21:33.6245466Z W1204 11:18:48.782000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] cls(obj, pickler.options), 2025-12-04T11:21:33.6245981Z W1204 11:18:48.782000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 364, in __init__ 2025-12-04T11:21:33.6246282Z W1204 11:18:48.782000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] self.graph = _GraphPickleData(gm._graph, options) 2025-12-04T11:21:33.6246732Z W1204 11:18:48.782000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 571, in __init__ 2025-12-04T11:21:33.6247032Z W1204 11:18:48.782000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] nodes[node] = _NodePickleData(node, nodes, options) 2025-12-04T11:21:33.6247480Z W1204 11:18:48.782000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 387, in __init__ 2025-12-04T11:21:33.6247804Z W1204 11:18:48.782000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] self.target = _OpPickleData.pickle(node.target, options) 2025-12-04T11:21:33.6248259Z W1204 11:18:48.782000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 435, in pickle 2025-12-04T11:21:33.6248593Z W1204 11:18:48.782000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] return cls._pickle_op(name, _OpOverloadPickleData, options) 2025-12-04T11:21:33.6249053Z W1204 11:18:48.782000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 456, in _pickle_op 2025-12-04T11:21:33.6249409Z W1204 11:18:48.782000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] raise BypassFxGraphCache(f"Unable to pickle non-standard op: {name}") 2025-12-04T11:21:33.6249886Z W1204 11:18:48.782000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] torch._inductor.codecache.BypassFxGraphCache: Unable to pickle non-standard op: torch.ops.prims.iota.default 2025-12-04T11:21:33.6250214Z W1204 11:18:49.056000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] Unable to pickle input graph or example inputs 2025-12-04T11:21:33.6250480Z W1204 11:18:49.056000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] Traceback (most recent call last): 2025-12-04T11:21:33.6250980Z W1204 11:18:49.056000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/compile_fx_ext.py", line 486, in serialize_compile 2025-12-04T11:21:33.6251200Z W1204 11:18:49.056000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] ).serialize() 2025-12-04T11:21:33.6251673Z W1204 11:18:49.056000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/compile_fx_ext.py", line 210, in serialize 2025-12-04T11:21:33.6252000Z W1204 11:18:49.056000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] return _WireProtocolPickledInput(GraphPickler.dumps(self)) 2025-12-04T11:21:33.6252529Z W1204 11:18:49.056000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 127, in dumps 2025-12-04T11:21:33.6252761Z W1204 11:18:49.056000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] pickler.dump(obj) 2025-12-04T11:21:33.6253235Z W1204 11:18:49.056000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 94, in reducer_override 2025-12-04T11:21:33.6253549Z W1204 11:18:49.056000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] return _GraphModulePickleData.reduce_helper(self, obj) 2025-12-04T11:21:33.6254048Z W1204 11:18:49.056000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 352, in reduce_helper 2025-12-04T11:21:33.6254305Z W1204 11:18:49.056000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] cls(obj, pickler.options), 2025-12-04T11:21:33.6254756Z W1204 11:18:49.056000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 364, in __init__ 2025-12-04T11:21:33.6255058Z W1204 11:18:49.056000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] self.graph = _GraphPickleData(gm._graph, options) 2025-12-04T11:21:33.6255507Z W1204 11:18:49.056000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 571, in __init__ 2025-12-04T11:21:33.6255810Z W1204 11:18:49.056000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] nodes[node] = _NodePickleData(node, nodes, options) 2025-12-04T11:21:33.6256260Z W1204 11:18:49.056000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 387, in __init__ 2025-12-04T11:21:33.6256575Z W1204 11:18:49.056000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] self.target = _OpPickleData.pickle(node.target, options) 2025-12-04T11:21:33.6257027Z W1204 11:18:49.056000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 435, in pickle 2025-12-04T11:21:33.6257356Z W1204 11:18:49.056000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] return cls._pickle_op(name, _OpOverloadPickleData, options) 2025-12-04T11:21:33.6257816Z W1204 11:18:49.056000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 456, in _pickle_op 2025-12-04T11:21:33.6258213Z W1204 11:18:49.056000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] raise BypassFxGraphCache(f"Unable to pickle non-standard op: {name}") 2025-12-04T11:21:33.6258691Z W1204 11:18:49.056000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] torch._inductor.codecache.BypassFxGraphCache: Unable to pickle non-standard op: torch.ops.prims.iota.default 2025-12-04T11:21:33.6258991Z W1204 11:18:49.320000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] Unable to pickle input graph or example inputs 2025-12-04T11:21:33.6259252Z W1204 11:18:49.320000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] Traceback (most recent call last): 2025-12-04T11:21:33.6259763Z W1204 11:18:49.320000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/compile_fx_ext.py", line 486, in serialize_compile 2025-12-04T11:21:33.6259980Z W1204 11:18:49.320000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] ).serialize() 2025-12-04T11:21:33.6260525Z W1204 11:18:49.320000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/compile_fx_ext.py", line 210, in serialize 2025-12-04T11:21:33.6260853Z W1204 11:18:49.320000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] return _WireProtocolPickledInput(GraphPickler.dumps(self)) 2025-12-04T11:21:33.6261305Z W1204 11:18:49.320000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 127, in dumps 2025-12-04T11:21:33.6261534Z W1204 11:18:49.320000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] pickler.dump(obj) 2025-12-04T11:21:33.6262044Z W1204 11:18:49.320000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 94, in reducer_override 2025-12-04T11:21:33.6262372Z W1204 11:18:49.320000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] return _GraphModulePickleData.reduce_helper(self, obj) 2025-12-04T11:21:33.6262846Z W1204 11:18:49.320000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 352, in reduce_helper 2025-12-04T11:21:33.6263102Z W1204 11:18:49.320000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] cls(obj, pickler.options), 2025-12-04T11:21:33.6263610Z W1204 11:18:49.320000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 364, in __init__ 2025-12-04T11:21:33.6263916Z W1204 11:18:49.320000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] self.graph = _GraphPickleData(gm._graph, options) 2025-12-04T11:21:33.6264368Z W1204 11:18:49.320000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 571, in __init__ 2025-12-04T11:21:33.6264667Z W1204 11:18:49.320000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] nodes[node] = _NodePickleData(node, nodes, options) 2025-12-04T11:21:33.6265119Z W1204 11:18:49.320000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 387, in __init__ 2025-12-04T11:21:33.6265432Z W1204 11:18:49.320000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] self.target = _OpPickleData.pickle(node.target, options) 2025-12-04T11:21:33.6265882Z W1204 11:18:49.320000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 435, in pickle 2025-12-04T11:21:33.6266253Z W1204 11:18:49.320000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] return cls._pickle_op(name, _OpOverloadPickleData, options) 2025-12-04T11:21:33.6266716Z W1204 11:18:49.320000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 456, in _pickle_op 2025-12-04T11:21:33.6267075Z W1204 11:18:49.320000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] raise BypassFxGraphCache(f"Unable to pickle non-standard op: {name}") 2025-12-04T11:21:33.6267548Z W1204 11:18:49.320000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] torch._inductor.codecache.BypassFxGraphCache: Unable to pickle non-standard op: torch.ops.prims.iota.default 2025-12-04T11:21:33.6267845Z W1204 11:18:49.585000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] Unable to pickle input graph or example inputs 2025-12-04T11:21:33.6268108Z W1204 11:18:49.585000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] Traceback (most recent call last): 2025-12-04T11:21:33.6268695Z W1204 11:18:49.585000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/compile_fx_ext.py", line 486, in serialize_compile 2025-12-04T11:21:33.6268913Z W1204 11:18:49.585000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] ).serialize() 2025-12-04T11:21:33.6269382Z W1204 11:18:49.585000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/compile_fx_ext.py", line 210, in serialize 2025-12-04T11:21:33.6269709Z W1204 11:18:49.585000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] return _WireProtocolPickledInput(GraphPickler.dumps(self)) 2025-12-04T11:21:33.6270194Z W1204 11:18:49.585000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 127, in dumps 2025-12-04T11:21:33.6270436Z W1204 11:18:49.585000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] pickler.dump(obj) 2025-12-04T11:21:33.6270910Z W1204 11:18:49.585000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 94, in reducer_override 2025-12-04T11:21:33.6271226Z W1204 11:18:49.585000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] return _GraphModulePickleData.reduce_helper(self, obj) 2025-12-04T11:21:33.6271729Z W1204 11:18:49.585000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 352, in reduce_helper 2025-12-04T11:21:33.6272017Z W1204 11:18:49.585000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] cls(obj, pickler.options), 2025-12-04T11:21:33.6272483Z W1204 11:18:49.585000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 364, in __init__ 2025-12-04T11:21:33.6272788Z W1204 11:18:49.585000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] self.graph = _GraphPickleData(gm._graph, options) 2025-12-04T11:21:33.6273246Z W1204 11:18:49.585000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 571, in __init__ 2025-12-04T11:21:33.6273544Z W1204 11:18:49.585000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] nodes[node] = _NodePickleData(node, nodes, options) 2025-12-04T11:21:33.6273998Z W1204 11:18:49.585000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 387, in __init__ 2025-12-04T11:21:33.6274365Z W1204 11:18:49.585000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] self.target = _OpPickleData.pickle(node.target, options) 2025-12-04T11:21:33.6274817Z W1204 11:18:49.585000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 435, in pickle 2025-12-04T11:21:33.6275146Z W1204 11:18:49.585000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] return cls._pickle_op(name, _OpOverloadPickleData, options) 2025-12-04T11:21:33.6275602Z W1204 11:18:49.585000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 456, in _pickle_op 2025-12-04T11:21:33.6275964Z W1204 11:18:49.585000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] raise BypassFxGraphCache(f"Unable to pickle non-standard op: {name}") 2025-12-04T11:21:33.6276439Z W1204 11:18:49.585000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] torch._inductor.codecache.BypassFxGraphCache: Unable to pickle non-standard op: torch.ops.prims.iota.default 2025-12-04T11:21:33.6276831Z W1204 11:18:49.857000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] Unable to pickle input graph or example inputs 2025-12-04T11:21:33.6277099Z W1204 11:18:49.857000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] Traceback (most recent call last): 2025-12-04T11:21:33.6277600Z W1204 11:18:49.857000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/compile_fx_ext.py", line 486, in serialize_compile 2025-12-04T11:21:33.6277988Z W1204 11:18:49.857000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] ).serialize() 2025-12-04T11:21:33.6278546Z W1204 11:18:49.857000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/compile_fx_ext.py", line 210, in serialize 2025-12-04T11:21:33.6278889Z W1204 11:18:49.857000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] return _WireProtocolPickledInput(GraphPickler.dumps(self)) 2025-12-04T11:21:33.6279335Z W1204 11:18:49.857000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 127, in dumps 2025-12-04T11:21:33.6279567Z W1204 11:18:49.857000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] pickler.dump(obj) 2025-12-04T11:21:33.6280042Z W1204 11:18:49.857000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 94, in reducer_override 2025-12-04T11:21:33.6280361Z W1204 11:18:49.857000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] return _GraphModulePickleData.reduce_helper(self, obj) 2025-12-04T11:21:33.6280834Z W1204 11:18:49.857000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 352, in reduce_helper 2025-12-04T11:21:33.6281093Z W1204 11:18:49.857000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] cls(obj, pickler.options), 2025-12-04T11:21:33.6281546Z W1204 11:18:49.857000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 364, in __init__ 2025-12-04T11:21:33.6281848Z W1204 11:18:49.857000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] self.graph = _GraphPickleData(gm._graph, options) 2025-12-04T11:21:33.6282300Z W1204 11:18:49.857000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 571, in __init__ 2025-12-04T11:21:33.6282655Z W1204 11:18:49.857000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] nodes[node] = _NodePickleData(node, nodes, options) 2025-12-04T11:21:33.6283108Z W1204 11:18:49.857000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 387, in __init__ 2025-12-04T11:21:33.6283437Z W1204 11:18:49.857000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] self.target = _OpPickleData.pickle(node.target, options) 2025-12-04T11:21:33.6283887Z W1204 11:18:49.857000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 435, in pickle 2025-12-04T11:21:33.6284219Z W1204 11:18:49.857000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] return cls._pickle_op(name, _OpOverloadPickleData, options) 2025-12-04T11:21:33.6284673Z W1204 11:18:49.857000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 456, in _pickle_op 2025-12-04T11:21:33.6285122Z W1204 11:18:49.857000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] raise BypassFxGraphCache(f"Unable to pickle non-standard op: {name}") 2025-12-04T11:21:33.6285618Z W1204 11:18:49.857000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] torch._inductor.codecache.BypassFxGraphCache: Unable to pickle non-standard op: torch.ops.prims.iota.default 2025-12-04T11:21:33.6285913Z W1204 11:18:50.577000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] Unable to pickle input graph or example inputs 2025-12-04T11:21:33.6286216Z W1204 11:18:50.577000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] Traceback (most recent call last): 2025-12-04T11:21:33.6286722Z W1204 11:18:50.577000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/compile_fx_ext.py", line 486, in serialize_compile 2025-12-04T11:21:33.6286944Z W1204 11:18:50.577000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] ).serialize() 2025-12-04T11:21:33.6287420Z W1204 11:18:50.577000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/compile_fx_ext.py", line 210, in serialize 2025-12-04T11:21:33.6287747Z W1204 11:18:50.577000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] return _WireProtocolPickledInput(GraphPickler.dumps(self)) 2025-12-04T11:21:33.6288198Z W1204 11:18:50.577000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 127, in dumps 2025-12-04T11:21:33.6288431Z W1204 11:18:50.577000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] pickler.dump(obj) 2025-12-04T11:21:33.6288912Z W1204 11:18:50.577000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 94, in reducer_override 2025-12-04T11:21:33.6289227Z W1204 11:18:50.577000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] return _GraphModulePickleData.reduce_helper(self, obj) 2025-12-04T11:21:33.6289692Z W1204 11:18:50.577000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 352, in reduce_helper 2025-12-04T11:21:33.6289960Z W1204 11:18:50.577000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] cls(obj, pickler.options), 2025-12-04T11:21:33.6290421Z W1204 11:18:50.577000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 364, in __init__ 2025-12-04T11:21:33.6290764Z W1204 11:18:50.577000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] self.graph = _GraphPickleData(gm._graph, options) 2025-12-04T11:21:33.6291218Z W1204 11:18:50.577000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 571, in __init__ 2025-12-04T11:21:33.6291518Z W1204 11:18:50.577000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] nodes[node] = _NodePickleData(node, nodes, options) 2025-12-04T11:21:33.6291967Z W1204 11:18:50.577000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 387, in __init__ 2025-12-04T11:21:33.6292282Z W1204 11:18:50.577000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] self.target = _OpPickleData.pickle(node.target, options) 2025-12-04T11:21:33.6292735Z W1204 11:18:50.577000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 435, in pickle 2025-12-04T11:21:33.6293143Z W1204 11:18:50.577000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] return cls._pickle_op(name, _OpOverloadPickleData, options) 2025-12-04T11:21:33.6293618Z W1204 11:18:50.577000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 456, in _pickle_op 2025-12-04T11:21:33.6293979Z W1204 11:18:50.577000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] raise BypassFxGraphCache(f"Unable to pickle non-standard op: {name}") 2025-12-04T11:21:33.6294487Z W1204 11:18:50.577000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] torch._inductor.codecache.BypassFxGraphCache: Unable to pickle non-standard op: torch.ops.prims.iota.default 2025-12-04T11:21:33.6294783Z W1204 11:18:50.843000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] Unable to pickle input graph or example inputs 2025-12-04T11:21:33.6295047Z W1204 11:18:50.843000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] Traceback (most recent call last): 2025-12-04T11:21:33.6295548Z W1204 11:18:50.843000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/compile_fx_ext.py", line 486, in serialize_compile 2025-12-04T11:21:33.6295765Z W1204 11:18:50.843000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] ).serialize() 2025-12-04T11:21:33.6296243Z W1204 11:18:50.843000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/compile_fx_ext.py", line 210, in serialize 2025-12-04T11:21:33.6296573Z W1204 11:18:50.843000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] return _WireProtocolPickledInput(GraphPickler.dumps(self)) 2025-12-04T11:21:33.6297034Z W1204 11:18:50.843000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 127, in dumps 2025-12-04T11:21:33.6297269Z W1204 11:18:50.843000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] pickler.dump(obj) 2025-12-04T11:21:33.6297736Z W1204 11:18:50.843000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 94, in reducer_override 2025-12-04T11:21:33.6298060Z W1204 11:18:50.843000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] return _GraphModulePickleData.reduce_helper(self, obj) 2025-12-04T11:21:33.6298526Z W1204 11:18:50.843000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 352, in reduce_helper 2025-12-04T11:21:33.6298827Z W1204 11:18:50.843000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] cls(obj, pickler.options), 2025-12-04T11:21:33.6299275Z W1204 11:18:50.843000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 364, in __init__ 2025-12-04T11:21:33.6299574Z W1204 11:18:50.843000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] self.graph = _GraphPickleData(gm._graph, options) 2025-12-04T11:21:33.6300026Z W1204 11:18:50.843000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 571, in __init__ 2025-12-04T11:21:33.6300326Z W1204 11:18:50.843000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] nodes[node] = _NodePickleData(node, nodes, options) 2025-12-04T11:21:33.6300817Z W1204 11:18:50.843000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 387, in __init__ 2025-12-04T11:21:33.6301164Z W1204 11:18:50.843000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] self.target = _OpPickleData.pickle(node.target, options) 2025-12-04T11:21:33.6301620Z W1204 11:18:50.843000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 435, in pickle 2025-12-04T11:21:33.6301945Z W1204 11:18:50.843000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] return cls._pickle_op(name, _OpOverloadPickleData, options) 2025-12-04T11:21:33.6302433Z W1204 11:18:50.843000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 456, in _pickle_op 2025-12-04T11:21:33.6302820Z W1204 11:18:50.843000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] raise BypassFxGraphCache(f"Unable to pickle non-standard op: {name}") 2025-12-04T11:21:33.6303685Z W1204 11:18:50.843000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] torch._inductor.codecache.BypassFxGraphCache: Unable to pickle non-standard op: torch.ops.prims.iota.default 2025-12-04T11:21:33.6304003Z W1204 11:18:51.115000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] Unable to pickle input graph or example inputs 2025-12-04T11:21:33.6304271Z W1204 11:18:51.115000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] Traceback (most recent call last): 2025-12-04T11:21:33.6304791Z W1204 11:18:51.115000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/compile_fx_ext.py", line 486, in serialize_compile 2025-12-04T11:21:33.6305014Z W1204 11:18:51.115000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] ).serialize() 2025-12-04T11:21:33.6305491Z W1204 11:18:51.115000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/compile_fx_ext.py", line 210, in serialize 2025-12-04T11:21:33.6305830Z W1204 11:18:51.115000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] return _WireProtocolPickledInput(GraphPickler.dumps(self)) 2025-12-04T11:21:33.6306275Z W1204 11:18:51.115000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 127, in dumps 2025-12-04T11:21:33.6306518Z W1204 11:18:51.115000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] pickler.dump(obj) 2025-12-04T11:21:33.6306995Z W1204 11:18:51.115000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 94, in reducer_override 2025-12-04T11:21:33.6307424Z W1204 11:18:51.115000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] return _GraphModulePickleData.reduce_helper(self, obj) 2025-12-04T11:21:33.6307889Z W1204 11:18:51.115000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 352, in reduce_helper 2025-12-04T11:21:33.6308149Z W1204 11:18:51.115000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] cls(obj, pickler.options), 2025-12-04T11:21:33.6308615Z W1204 11:18:51.115000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 364, in __init__ 2025-12-04T11:21:33.6308919Z W1204 11:18:51.115000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] self.graph = _GraphPickleData(gm._graph, options) 2025-12-04T11:21:33.6309468Z W1204 11:18:51.115000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 571, in __init__ 2025-12-04T11:21:33.6309769Z W1204 11:18:51.115000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] nodes[node] = _NodePickleData(node, nodes, options) 2025-12-04T11:21:33.6310224Z W1204 11:18:51.115000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 387, in __init__ 2025-12-04T11:21:33.6310550Z W1204 11:18:51.115000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] self.target = _OpPickleData.pickle(node.target, options) 2025-12-04T11:21:33.6311035Z W1204 11:18:51.115000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 435, in pickle 2025-12-04T11:21:33.6311372Z W1204 11:18:51.115000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] return cls._pickle_op(name, _OpOverloadPickleData, options) 2025-12-04T11:21:33.6311832Z W1204 11:18:51.115000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 456, in _pickle_op 2025-12-04T11:21:33.6312193Z W1204 11:18:51.115000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] raise BypassFxGraphCache(f"Unable to pickle non-standard op: {name}") 2025-12-04T11:21:33.6312811Z W1204 11:18:51.115000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] torch._inductor.codecache.BypassFxGraphCache: Unable to pickle non-standard op: torch.ops.prims.iota.default 2025-12-04T11:21:33.6312889Z PASSED [5.1230s] [ 75%] 2025-12-04T11:21:33.6313621Z inductor/test_compile_subprocess.py::GPUTests::test_reflection_pad2d_cuda <- test/inductor/test_torchinductor.py W1204 11:18:51.390000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] Unable to pickle input graph or example inputs 2025-12-04T11:21:33.6314032Z W1204 11:18:51.390000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] Traceback (most recent call last): 2025-12-04T11:21:33.6314548Z W1204 11:18:51.390000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/compile_fx_ext.py", line 486, in serialize_compile 2025-12-04T11:21:33.6314772Z W1204 11:18:51.390000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] ).serialize() 2025-12-04T11:21:33.6315249Z W1204 11:18:51.390000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/compile_fx_ext.py", line 210, in serialize 2025-12-04T11:21:33.6315579Z W1204 11:18:51.390000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] return _WireProtocolPickledInput(GraphPickler.dumps(self)) 2025-12-04T11:21:33.6316098Z W1204 11:18:51.390000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 127, in dumps 2025-12-04T11:21:33.6316328Z W1204 11:18:51.390000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] pickler.dump(obj) 2025-12-04T11:21:33.6316808Z W1204 11:18:51.390000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 94, in reducer_override 2025-12-04T11:21:33.6317136Z W1204 11:18:51.390000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] return _GraphModulePickleData.reduce_helper(self, obj) 2025-12-04T11:21:33.6317603Z W1204 11:18:51.390000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 352, in reduce_helper 2025-12-04T11:21:33.6317904Z W1204 11:18:51.390000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] cls(obj, pickler.options), 2025-12-04T11:21:33.6318398Z W1204 11:18:51.390000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 364, in __init__ 2025-12-04T11:21:33.6318698Z W1204 11:18:51.390000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] self.graph = _GraphPickleData(gm._graph, options) 2025-12-04T11:21:33.6319218Z W1204 11:18:51.390000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 571, in __init__ 2025-12-04T11:21:33.6319648Z W1204 11:18:51.390000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] nodes[node] = _NodePickleData(node, nodes, options) 2025-12-04T11:21:33.6320112Z W1204 11:18:51.390000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 387, in __init__ 2025-12-04T11:21:33.6320432Z W1204 11:18:51.390000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] self.target = _OpPickleData.pickle(node.target, options) 2025-12-04T11:21:33.6320882Z W1204 11:18:51.390000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 435, in pickle 2025-12-04T11:21:33.6321212Z W1204 11:18:51.390000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] return cls._pickle_op(name, _OpOverloadPickleData, options) 2025-12-04T11:21:33.6321668Z W1204 11:18:51.390000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 456, in _pickle_op 2025-12-04T11:21:33.6322028Z W1204 11:18:51.390000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] raise BypassFxGraphCache(f"Unable to pickle non-standard op: {name}") 2025-12-04T11:21:33.6322507Z W1204 11:18:51.390000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] torch._inductor.codecache.BypassFxGraphCache: Unable to pickle non-standard op: torch.ops.prims.iota.default 2025-12-04T11:21:33.6322801Z W1204 11:18:51.852000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] Unable to pickle input graph or example inputs 2025-12-04T11:21:33.6323064Z W1204 11:18:51.852000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] Traceback (most recent call last): 2025-12-04T11:21:33.6323574Z W1204 11:18:51.852000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/compile_fx_ext.py", line 486, in serialize_compile 2025-12-04T11:21:33.6323832Z W1204 11:18:51.852000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] ).serialize() 2025-12-04T11:21:33.6324306Z W1204 11:18:51.852000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/compile_fx_ext.py", line 210, in serialize 2025-12-04T11:21:33.6324639Z W1204 11:18:51.852000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] return _WireProtocolPickledInput(GraphPickler.dumps(self)) 2025-12-04T11:21:33.6325082Z W1204 11:18:51.852000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 127, in dumps 2025-12-04T11:21:33.6325316Z W1204 11:18:51.852000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] pickler.dump(obj) 2025-12-04T11:21:33.6325789Z W1204 11:18:51.852000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 94, in reducer_override 2025-12-04T11:21:33.6326191Z W1204 11:18:51.852000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] return _GraphModulePickleData.reduce_helper(self, obj) 2025-12-04T11:21:33.6326662Z W1204 11:18:51.852000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 352, in reduce_helper 2025-12-04T11:21:33.6326920Z W1204 11:18:51.852000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] cls(obj, pickler.options), 2025-12-04T11:21:33.6327375Z W1204 11:18:51.852000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 364, in __init__ 2025-12-04T11:21:33.6327705Z W1204 11:18:51.852000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] self.graph = _GraphPickleData(gm._graph, options) 2025-12-04T11:21:33.6328171Z W1204 11:18:51.852000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 571, in __init__ 2025-12-04T11:21:33.6328467Z W1204 11:18:51.852000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] nodes[node] = _NodePickleData(node, nodes, options) 2025-12-04T11:21:33.6328924Z W1204 11:18:51.852000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 387, in __init__ 2025-12-04T11:21:33.6329245Z W1204 11:18:51.852000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] self.target = _OpPickleData.pickle(node.target, options) 2025-12-04T11:21:33.6329694Z W1204 11:18:51.852000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 435, in pickle 2025-12-04T11:21:33.6330025Z W1204 11:18:51.852000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] return cls._pickle_op(name, _OpOverloadPickleData, options) 2025-12-04T11:21:33.6330485Z W1204 11:18:51.852000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 456, in _pickle_op 2025-12-04T11:21:33.6330848Z W1204 11:18:51.852000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] raise BypassFxGraphCache(f"Unable to pickle non-standard op: {name}") 2025-12-04T11:21:33.6331367Z W1204 11:18:51.852000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] torch._inductor.codecache.BypassFxGraphCache: Unable to pickle non-standard op: torch.ops.prims.convert_element_type.default 2025-12-04T11:21:33.6331437Z PASSED [0.7992s] [ 75%] 2025-12-04T11:21:33.6331808Z inductor/test_compile_subprocess.py::GPUTests::test_reinterpret_dtypeview_cuda <- test/inductor/test_torchinductor.py PASSED [0.2575s] [ 75%] 2025-12-04T11:21:33.6332142Z inductor/test_compile_subprocess.py::GPUTests::test_relu_cuda <- test/inductor/test_torchinductor.py PASSED [0.3684s] [ 76%] 2025-12-04T11:21:33.6332745Z inductor/test_compile_subprocess.py::GPUTests::test_remove_no_ops_cuda <- test/inductor/test_torchinductor.py W1204 11:18:53.154000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] Unable to pickle input graph or example inputs 2025-12-04T11:21:33.6333010Z W1204 11:18:53.154000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] Traceback (most recent call last): 2025-12-04T11:21:33.6333519Z W1204 11:18:53.154000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/compile_fx_ext.py", line 486, in serialize_compile 2025-12-04T11:21:33.6333741Z W1204 11:18:53.154000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] ).serialize() 2025-12-04T11:21:33.6334216Z W1204 11:18:53.154000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/compile_fx_ext.py", line 210, in serialize 2025-12-04T11:21:33.6334618Z W1204 11:18:53.154000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] return _WireProtocolPickledInput(GraphPickler.dumps(self)) 2025-12-04T11:21:33.6335067Z W1204 11:18:53.154000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 127, in dumps 2025-12-04T11:21:33.6335304Z W1204 11:18:53.154000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] pickler.dump(obj) 2025-12-04T11:21:33.6335814Z W1204 11:18:53.154000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 94, in reducer_override 2025-12-04T11:21:33.6336134Z W1204 11:18:53.154000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] return _GraphModulePickleData.reduce_helper(self, obj) 2025-12-04T11:21:33.6336606Z W1204 11:18:53.154000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 352, in reduce_helper 2025-12-04T11:21:33.6336860Z W1204 11:18:53.154000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] cls(obj, pickler.options), 2025-12-04T11:21:33.6337312Z W1204 11:18:53.154000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 364, in __init__ 2025-12-04T11:21:33.6337613Z W1204 11:18:53.154000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] self.graph = _GraphPickleData(gm._graph, options) 2025-12-04T11:21:33.6338068Z W1204 11:18:53.154000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 571, in __init__ 2025-12-04T11:21:33.6338380Z W1204 11:18:53.154000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] nodes[node] = _NodePickleData(node, nodes, options) 2025-12-04T11:21:33.6338828Z W1204 11:18:53.154000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 387, in __init__ 2025-12-04T11:21:33.6339146Z W1204 11:18:53.154000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] self.target = _OpPickleData.pickle(node.target, options) 2025-12-04T11:21:33.6339592Z W1204 11:18:53.154000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 435, in pickle 2025-12-04T11:21:33.6339922Z W1204 11:18:53.154000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] return cls._pickle_op(name, _OpOverloadPickleData, options) 2025-12-04T11:21:33.6340418Z W1204 11:18:53.154000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 456, in _pickle_op 2025-12-04T11:21:33.6340774Z W1204 11:18:53.154000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] raise BypassFxGraphCache(f"Unable to pickle non-standard op: {name}") 2025-12-04T11:21:33.6341292Z W1204 11:18:53.154000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] torch._inductor.codecache.BypassFxGraphCache: Unable to pickle non-standard op: torch.ops.prims.convert_element_type.default 2025-12-04T11:21:33.6341592Z W1204 11:18:53.488000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] Unable to pickle input graph or example inputs 2025-12-04T11:21:33.6341861Z W1204 11:18:53.488000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] Traceback (most recent call last): 2025-12-04T11:21:33.6342457Z W1204 11:18:53.488000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/compile_fx_ext.py", line 486, in serialize_compile 2025-12-04T11:21:33.6342680Z W1204 11:18:53.488000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] ).serialize() 2025-12-04T11:21:33.6343148Z W1204 11:18:53.488000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/compile_fx_ext.py", line 210, in serialize 2025-12-04T11:21:33.6343478Z W1204 11:18:53.488000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] return _WireProtocolPickledInput(GraphPickler.dumps(self)) 2025-12-04T11:21:33.6344071Z W1204 11:18:53.488000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 127, in dumps 2025-12-04T11:21:33.6344305Z W1204 11:18:53.488000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] pickler.dump(obj) 2025-12-04T11:21:33.6344784Z W1204 11:18:53.488000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 94, in reducer_override 2025-12-04T11:21:33.6345098Z W1204 11:18:53.488000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] return _GraphModulePickleData.reduce_helper(self, obj) 2025-12-04T11:21:33.6345569Z W1204 11:18:53.488000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 352, in reduce_helper 2025-12-04T11:21:33.6345822Z W1204 11:18:53.488000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] cls(obj, pickler.options), 2025-12-04T11:21:33.6346279Z W1204 11:18:53.488000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 364, in __init__ 2025-12-04T11:21:33.6346581Z W1204 11:18:53.488000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] self.graph = _GraphPickleData(gm._graph, options) 2025-12-04T11:21:33.6347030Z W1204 11:18:53.488000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 571, in __init__ 2025-12-04T11:21:33.6347329Z W1204 11:18:53.488000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] nodes[node] = _NodePickleData(node, nodes, options) 2025-12-04T11:21:33.6347775Z W1204 11:18:53.488000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 387, in __init__ 2025-12-04T11:21:33.6348133Z W1204 11:18:53.488000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] self.target = _OpPickleData.pickle(node.target, options) 2025-12-04T11:21:33.6348582Z W1204 11:18:53.488000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 435, in pickle 2025-12-04T11:21:33.6348905Z W1204 11:18:53.488000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] return cls._pickle_op(name, _OpOverloadPickleData, options) 2025-12-04T11:21:33.6349364Z W1204 11:18:53.488000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 456, in _pickle_op 2025-12-04T11:21:33.6349716Z W1204 11:18:53.488000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] raise BypassFxGraphCache(f"Unable to pickle non-standard op: {name}") 2025-12-04T11:21:33.6350240Z W1204 11:18:53.488000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] torch._inductor.codecache.BypassFxGraphCache: Unable to pickle non-standard op: torch.ops.prims.convert_element_type.default 2025-12-04T11:21:33.6350600Z W1204 11:18:54.151000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] Unable to pickle input graph or example inputs 2025-12-04T11:21:33.6350862Z W1204 11:18:54.151000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] Traceback (most recent call last): 2025-12-04T11:21:33.6351359Z W1204 11:18:54.151000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/compile_fx_ext.py", line 486, in serialize_compile 2025-12-04T11:21:33.6351584Z W1204 11:18:54.151000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] ).serialize() 2025-12-04T11:21:33.6352102Z W1204 11:18:54.151000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/compile_fx_ext.py", line 210, in serialize 2025-12-04T11:21:33.6352435Z W1204 11:18:54.151000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] return _WireProtocolPickledInput(GraphPickler.dumps(self)) 2025-12-04T11:21:33.6352883Z W1204 11:18:54.151000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 127, in dumps 2025-12-04T11:21:33.6353112Z W1204 11:18:54.151000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] pickler.dump(obj) 2025-12-04T11:21:33.6353585Z W1204 11:18:54.151000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 94, in reducer_override 2025-12-04T11:21:33.6353900Z W1204 11:18:54.151000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] return _GraphModulePickleData.reduce_helper(self, obj) 2025-12-04T11:21:33.6354363Z W1204 11:18:54.151000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 352, in reduce_helper 2025-12-04T11:21:33.6354623Z W1204 11:18:54.151000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] cls(obj, pickler.options), 2025-12-04T11:21:33.6355071Z W1204 11:18:54.151000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 364, in __init__ 2025-12-04T11:21:33.6355375Z W1204 11:18:54.151000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] self.graph = _GraphPickleData(gm._graph, options) 2025-12-04T11:21:33.6355823Z W1204 11:18:54.151000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 571, in __init__ 2025-12-04T11:21:33.6356162Z W1204 11:18:54.151000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] nodes[node] = _NodePickleData(node, nodes, options) 2025-12-04T11:21:33.6356611Z W1204 11:18:54.151000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 387, in __init__ 2025-12-04T11:21:33.6356929Z W1204 11:18:54.151000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] self.target = _OpPickleData.pickle(node.target, options) 2025-12-04T11:21:33.6357379Z W1204 11:18:54.151000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 435, in pickle 2025-12-04T11:21:33.6357702Z W1204 11:18:54.151000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] return cls._pickle_op(name, _OpOverloadPickleData, options) 2025-12-04T11:21:33.6358182Z W1204 11:18:54.151000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 456, in _pickle_op 2025-12-04T11:21:33.6358608Z W1204 11:18:54.151000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] raise BypassFxGraphCache(f"Unable to pickle non-standard op: {name}") 2025-12-04T11:21:33.6359135Z W1204 11:18:54.151000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] torch._inductor.codecache.BypassFxGraphCache: Unable to pickle non-standard op: torch.ops.prims.convert_element_type.default 2025-12-04T11:21:33.6359422Z W1204 11:18:54.481000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] Unable to pickle input graph or example inputs 2025-12-04T11:21:33.6359721Z W1204 11:18:54.481000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] Traceback (most recent call last): 2025-12-04T11:21:33.6360228Z W1204 11:18:54.481000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/compile_fx_ext.py", line 486, in serialize_compile 2025-12-04T11:21:33.6360448Z W1204 11:18:54.481000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] ).serialize() 2025-12-04T11:21:33.6360922Z W1204 11:18:54.481000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/compile_fx_ext.py", line 210, in serialize 2025-12-04T11:21:33.6361249Z W1204 11:18:54.481000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] return _WireProtocolPickledInput(GraphPickler.dumps(self)) 2025-12-04T11:21:33.6361693Z W1204 11:18:54.481000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 127, in dumps 2025-12-04T11:21:33.6361928Z W1204 11:18:54.481000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] pickler.dump(obj) 2025-12-04T11:21:33.6362396Z W1204 11:18:54.481000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 94, in reducer_override 2025-12-04T11:21:33.6362714Z W1204 11:18:54.481000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] return _GraphModulePickleData.reduce_helper(self, obj) 2025-12-04T11:21:33.6363177Z W1204 11:18:54.481000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 352, in reduce_helper 2025-12-04T11:21:33.6363432Z W1204 11:18:54.481000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] cls(obj, pickler.options), 2025-12-04T11:21:33.6363881Z W1204 11:18:54.481000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 364, in __init__ 2025-12-04T11:21:33.6364221Z W1204 11:18:54.481000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] self.graph = _GraphPickleData(gm._graph, options) 2025-12-04T11:21:33.6364680Z W1204 11:18:54.481000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 571, in __init__ 2025-12-04T11:21:33.6364974Z W1204 11:18:54.481000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] nodes[node] = _NodePickleData(node, nodes, options) 2025-12-04T11:21:33.6365427Z W1204 11:18:54.481000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 387, in __init__ 2025-12-04T11:21:33.6365742Z W1204 11:18:54.481000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] self.target = _OpPickleData.pickle(node.target, options) 2025-12-04T11:21:33.6366192Z W1204 11:18:54.481000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 435, in pickle 2025-12-04T11:21:33.6366600Z W1204 11:18:54.481000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] return cls._pickle_op(name, _OpOverloadPickleData, options) 2025-12-04T11:21:33.6367058Z W1204 11:18:54.481000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 456, in _pickle_op 2025-12-04T11:21:33.6367420Z W1204 11:18:54.481000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] raise BypassFxGraphCache(f"Unable to pickle non-standard op: {name}") 2025-12-04T11:21:33.6367970Z W1204 11:18:54.481000 89367 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] torch._inductor.codecache.BypassFxGraphCache: Unable to pickle non-standard op: torch.ops.prims.convert_element_type.default 2025-12-04T11:21:33.6368044Z PASSED [2.3380s] [ 76%] 2025-12-04T11:21:33.6368453Z inductor/test_compile_subprocess.py::GPUTests::test_remove_noop_slice_cuda <- test/inductor/test_torchinductor.py ('RERUN', {'yellow': True}) [0.5732s] [ 76%] 2025-12-04T11:21:33.6368832Z inductor/test_compile_subprocess.py::GPUTests::test_remove_noop_slice_cuda <- test/inductor/test_torchinductor.py ('RERUN', {'yellow': True}) [0.2235s] [ 76%] 2025-12-04T11:21:33.6369165Z inductor/test_compile_subprocess.py::GPUTests::test_remove_noop_slice_cuda <- test/inductor/test_torchinductor.py FAILED [0.2189s] [ 76%] 2025-12-04T11:21:33.6369171Z 2025-12-04T11:21:33.6369267Z ==================================== RERUNS ==================================== 2025-12-04T11:21:33.6369415Z _____________________ GPUTests.test_remove_noop_slice_cuda _____________________ 2025-12-04T11:21:33.6369494Z Traceback (most recent call last): 2025-12-04T11:21:33.6369732Z File "/var/lib/jenkins/workspace/test/inductor/test_torchinductor.py", line 14842, in new_test 2025-12-04T11:21:33.6369808Z return value(self) 2025-12-04T11:21:33.6370083Z File "/var/lib/jenkins/workspace/test/inductor/test_torchinductor.py", line 6708, in test_remove_noop_slice 2025-12-04T11:21:33.6370170Z self.assertExpectedInline( 2025-12-04T11:21:33.6370514Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3272, in assertExpectedInline 2025-12-04T11:21:33.6370764Z return super().assertExpectedInline(actual if isinstance(actual, str) else str(actual), expect, skip + 1) 2025-12-04T11:21:33.6371054Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/expecttest/__init__.py", line 413, in assertExpectedInline 2025-12-04T11:21:33.6371125Z assert_expected_inline( 2025-12-04T11:21:33.6371414Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/expecttest/__init__.py", line 378, in assert_expected_inline 2025-12-04T11:21:33.6371507Z assert_eq(expect, actual, msg=help_text) 2025-12-04T11:21:33.6371869Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/expecttest/__init__.py", line 450, in assertMultiLineEqualMaybeCppStack 2025-12-04T11:21:33.6372010Z self.assertMultiLineEqual(expect, actual, *args, **kwargs) 2025-12-04T11:21:33.6372233Z File "/opt/conda/envs/py_3.10/lib/python3.10/unittest/case.py", line 1226, in assertMultiLineEqual 2025-12-04T11:21:33.6372336Z self.fail(self._formatMessage(msg, standardMsg)) 2025-12-04T11:21:33.6372513Z File "/opt/conda/envs/py_3.10/lib/python3.10/unittest/case.py", line 675, in fail 2025-12-04T11:21:33.6372601Z raise self.failureException(msg) 2025-12-04T11:21:33.6372775Z AssertionError: 'def forward(self, arg0_1: "Sym(s77)", arg[333 chars]_9,)' != '' 2025-12-04T11:21:33.6373004Z - def forward(self, arg0_1: "Sym(s77)", arg1_1: "Sym(s27)", arg2_1: "Sym(s53)", arg3_1: "f32[s77, s27, s53][s27*s53, s53, 1]cuda:0"): 2025-12-04T11:21:33.6373199Z - add: "f32[s77, s27, s53][s27*s53, s53, 1]cuda:0" = torch.ops.aten.add.Tensor(arg3_1, 1); arg3_1 = None 2025-12-04T11:21:33.6373387Z - add_9: "f32[s77, s27, s53][s27*s53, s53, 1]cuda:0" = torch.ops.aten.add.Tensor(add, 1); add = None 2025-12-04T11:21:33.6373905Z - return (add_9,) : To accept the new output, re-run test with envvar EXPECTTEST_ACCEPT=1 (we recommend staging/committing your changes before doing this) 2025-12-04T11:21:33.6373910Z 2025-12-04T11:21:33.6374047Z To execute this test, run the following from the base repo dir: 2025-12-04T11:21:33.6374342Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_compile_subprocess.py GPUTests.test_remove_noop_slice_cuda 2025-12-04T11:21:33.6374345Z 2025-12-04T11:21:33.6374511Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:21:33.6374655Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:21:33.6374774Z frames [('total', 1), ('ok', 1)] 2025-12-04T11:21:33.6374879Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:21:33.6375070Z aot_autograd [('total', 1), ('autograd_cache_miss', 1), ('autograd_cache_saved', 1), ('ok', 1)] 2025-12-04T11:21:33.6375387Z inductor [('triton_bundler_save_kernel', 8), ('fxgraph_cache_miss', 1), ('async_compile_cache_miss', 1), ('triton_bundler_save_static_autotuner', 1)] 2025-12-04T11:21:33.6375526Z ----------------------------- Captured stderr call ----------------------------- 2025-12-04T11:21:33.6375955Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/ops_handler.py:772: UserWarning: undefined OpHandler.__getstate__, please add missing op schema 2025-12-04T11:21:33.6376124Z warnings.warn(f"undefined OpHandler.{name}, please add missing op schema") 2025-12-04T11:21:33.6376543Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/ops_handler.py:772: UserWarning: undefined OpHandler.__setstate__, please add missing op schema 2025-12-04T11:21:33.6376701Z warnings.warn(f"undefined OpHandler.{name}, please add missing op schema") 2025-12-04T11:21:33.6376845Z _____________________ GPUTests.test_remove_noop_slice_cuda _____________________ 2025-12-04T11:21:33.6376926Z Traceback (most recent call last): 2025-12-04T11:21:33.6377164Z File "/var/lib/jenkins/workspace/test/inductor/test_torchinductor.py", line 14842, in new_test 2025-12-04T11:21:33.6377235Z return value(self) 2025-12-04T11:21:33.6377510Z File "/var/lib/jenkins/workspace/test/inductor/test_torchinductor.py", line 6708, in test_remove_noop_slice 2025-12-04T11:21:33.6377592Z self.assertExpectedInline( 2025-12-04T11:21:33.6378270Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3272, in assertExpectedInline 2025-12-04T11:21:33.6378537Z return super().assertExpectedInline(actual if isinstance(actual, str) else str(actual), expect, skip + 1) 2025-12-04T11:21:33.6378828Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/expecttest/__init__.py", line 413, in assertExpectedInline 2025-12-04T11:21:33.6378980Z assert_expected_inline( 2025-12-04T11:21:33.6379276Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/expecttest/__init__.py", line 378, in assert_expected_inline 2025-12-04T11:21:33.6379374Z assert_eq(expect, actual, msg=help_text) 2025-12-04T11:21:33.6379693Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/expecttest/__init__.py", line 450, in assertMultiLineEqualMaybeCppStack 2025-12-04T11:21:33.6379829Z self.assertMultiLineEqual(expect, actual, *args, **kwargs) 2025-12-04T11:21:33.6380053Z File "/opt/conda/envs/py_3.10/lib/python3.10/unittest/case.py", line 1226, in assertMultiLineEqual 2025-12-04T11:21:33.6380155Z self.fail(self._formatMessage(msg, standardMsg)) 2025-12-04T11:21:33.6380328Z File "/opt/conda/envs/py_3.10/lib/python3.10/unittest/case.py", line 675, in fail 2025-12-04T11:21:33.6380407Z raise self.failureException(msg) 2025-12-04T11:21:33.6380579Z AssertionError: 'def forward(self, arg0_1: "Sym(s77)", arg[333 chars]_9,)' != '' 2025-12-04T11:21:33.6380805Z - def forward(self, arg0_1: "Sym(s77)", arg1_1: "Sym(s27)", arg2_1: "Sym(s53)", arg3_1: "f32[s77, s27, s53][s27*s53, s53, 1]cuda:0"): 2025-12-04T11:21:33.6381107Z - add: "f32[s77, s27, s53][s27*s53, s53, 1]cuda:0" = torch.ops.aten.add.Tensor(arg3_1, 1); arg3_1 = None 2025-12-04T11:21:33.6381296Z - add_9: "f32[s77, s27, s53][s27*s53, s53, 1]cuda:0" = torch.ops.aten.add.Tensor(add, 1); add = None 2025-12-04T11:21:33.6381645Z - return (add_9,) : To accept the new output, re-run test with envvar EXPECTTEST_ACCEPT=1 (we recommend staging/committing your changes before doing this) 2025-12-04T11:21:33.6381650Z 2025-12-04T11:21:33.6381785Z To execute this test, run the following from the base repo dir: 2025-12-04T11:21:33.6382083Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_compile_subprocess.py GPUTests.test_remove_noop_slice_cuda 2025-12-04T11:21:33.6382144Z 2025-12-04T11:21:33.6382311Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:21:33.6382451Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:21:33.6382524Z frames [('total', 1), ('ok', 1)] 2025-12-04T11:21:33.6382625Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:21:33.6382819Z aot_autograd [('total', 1), ('autograd_cache_miss', 1), ('autograd_cache_saved', 1), ('ok', 1)] 2025-12-04T11:21:33.6383134Z inductor [('triton_bundler_save_kernel', 8), ('fxgraph_cache_miss', 1), ('async_compile_cache_miss', 1), ('triton_bundler_save_static_autotuner', 1)] 2025-12-04T11:21:33.6383266Z ----------------------------- Captured stderr call ----------------------------- 2025-12-04T11:21:33.6383781Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/ops_handler.py:772: UserWarning: undefined OpHandler.__getstate__, please add missing op schema 2025-12-04T11:21:33.6383953Z warnings.warn(f"undefined OpHandler.{name}, please add missing op schema") 2025-12-04T11:21:33.6384375Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/ops_handler.py:772: UserWarning: undefined OpHandler.__setstate__, please add missing op schema 2025-12-04T11:21:33.6384538Z warnings.warn(f"undefined OpHandler.{name}, please add missing op schema") 2025-12-04T11:21:33.6384668Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:21:33.6384736Z frames [('total', 1), ('ok', 1)] 2025-12-04T11:21:33.6384830Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:21:33.6385014Z aot_autograd [('total', 1), ('autograd_cache_miss', 1), ('autograd_cache_saved', 1), ('ok', 1)] 2025-12-04T11:21:33.6385329Z inductor [('triton_bundler_save_kernel', 8), ('fxgraph_cache_miss', 1), ('async_compile_cache_miss', 1), ('triton_bundler_save_static_autotuner', 1)] 2025-12-04T11:21:33.6385458Z ----------------------------- Captured stderr call ----------------------------- 2025-12-04T11:21:33.6385872Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/ops_handler.py:772: UserWarning: undefined OpHandler.__getstate__, please add missing op schema 2025-12-04T11:21:33.6386080Z warnings.warn(f"undefined OpHandler.{name}, please add missing op schema") 2025-12-04T11:21:33.6386493Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/ops_handler.py:772: UserWarning: undefined OpHandler.__setstate__, please add missing op schema 2025-12-04T11:21:33.6386651Z warnings.warn(f"undefined OpHandler.{name}, please add missing op schema") 2025-12-04T11:21:33.6386743Z =================================== FAILURES =================================== 2025-12-04T11:21:33.6386881Z _____________________ GPUTests.test_remove_noop_slice_cuda _____________________ 2025-12-04T11:21:33.6386955Z Traceback (most recent call last): 2025-12-04T11:21:33.6387197Z File "/var/lib/jenkins/workspace/test/inductor/test_torchinductor.py", line 14842, in new_test 2025-12-04T11:21:33.6387263Z return value(self) 2025-12-04T11:21:33.6387542Z File "/var/lib/jenkins/workspace/test/inductor/test_torchinductor.py", line 6708, in test_remove_noop_slice 2025-12-04T11:21:33.6387667Z self.assertExpectedInline( 2025-12-04T11:21:33.6388044Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3272, in assertExpectedInline 2025-12-04T11:21:33.6388302Z return super().assertExpectedInline(actual if isinstance(actual, str) else str(actual), expect, skip + 1) 2025-12-04T11:21:33.6388590Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/expecttest/__init__.py", line 413, in assertExpectedInline 2025-12-04T11:21:33.6388660Z assert_expected_inline( 2025-12-04T11:21:33.6388952Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/expecttest/__init__.py", line 378, in assert_expected_inline 2025-12-04T11:21:33.6389038Z assert_eq(expect, actual, msg=help_text) 2025-12-04T11:21:33.6389391Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/expecttest/__init__.py", line 450, in assertMultiLineEqualMaybeCppStack 2025-12-04T11:21:33.6389535Z self.assertMultiLineEqual(expect, actual, *args, **kwargs) 2025-12-04T11:21:33.6389775Z File "/opt/conda/envs/py_3.10/lib/python3.10/unittest/case.py", line 1226, in assertMultiLineEqual 2025-12-04T11:21:33.6389888Z self.fail(self._formatMessage(msg, standardMsg)) 2025-12-04T11:21:33.6390060Z File "/opt/conda/envs/py_3.10/lib/python3.10/unittest/case.py", line 675, in fail 2025-12-04T11:21:33.6390140Z raise self.failureException(msg) 2025-12-04T11:21:33.6390307Z AssertionError: 'def forward(self, arg0_1: "Sym(s77)", arg[333 chars]_9,)' != '' 2025-12-04T11:21:33.6390539Z - def forward(self, arg0_1: "Sym(s77)", arg1_1: "Sym(s27)", arg2_1: "Sym(s53)", arg3_1: "f32[s77, s27, s53][s27*s53, s53, 1]cuda:0"): 2025-12-04T11:21:33.6390740Z - add: "f32[s77, s27, s53][s27*s53, s53, 1]cuda:0" = torch.ops.aten.add.Tensor(arg3_1, 1); arg3_1 = None 2025-12-04T11:21:33.6390930Z - add_9: "f32[s77, s27, s53][s27*s53, s53, 1]cuda:0" = torch.ops.aten.add.Tensor(add, 1); add = None 2025-12-04T11:21:33.6391284Z - return (add_9,) : To accept the new output, re-run test with envvar EXPECTTEST_ACCEPT=1 (we recommend staging/committing your changes before doing this) 2025-12-04T11:21:33.6391290Z 2025-12-04T11:21:33.6391430Z To execute this test, run the following from the base repo dir: 2025-12-04T11:21:33.6391723Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_compile_subprocess.py GPUTests.test_remove_noop_slice_cuda 2025-12-04T11:21:33.6391727Z 2025-12-04T11:21:33.6391893Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:21:33.6392021Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:21:33.6392091Z frames [('total', 1), ('ok', 1)] 2025-12-04T11:21:33.6392190Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:21:33.6392377Z aot_autograd [('total', 1), ('autograd_cache_miss', 1), ('autograd_cache_saved', 1), ('ok', 1)] 2025-12-04T11:21:33.6392731Z inductor [('triton_bundler_save_kernel', 8), ('fxgraph_cache_miss', 1), ('async_compile_cache_miss', 1), ('triton_bundler_save_static_autotuner', 1)] 2025-12-04T11:21:33.6392863Z ----------------------------- Captured stderr call ----------------------------- 2025-12-04T11:21:33.6393299Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/ops_handler.py:772: UserWarning: undefined OpHandler.__getstate__, please add missing op schema 2025-12-04T11:21:33.6393470Z warnings.warn(f"undefined OpHandler.{name}, please add missing op schema") 2025-12-04T11:21:33.6393886Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/ops_handler.py:772: UserWarning: undefined OpHandler.__setstate__, please add missing op schema 2025-12-04T11:21:33.6394045Z warnings.warn(f"undefined OpHandler.{name}, please add missing op schema") 2025-12-04T11:21:33.6394178Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:21:33.6394247Z frames [('total', 1), ('ok', 1)] 2025-12-04T11:21:33.6394346Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:21:33.6394564Z aot_autograd [('total', 1), ('autograd_cache_miss', 1), ('autograd_cache_saved', 1), ('ok', 1)] 2025-12-04T11:21:33.6394918Z inductor [('triton_bundler_save_kernel', 8), ('fxgraph_cache_miss', 1), ('async_compile_cache_miss', 1), ('triton_bundler_save_static_autotuner', 1)] 2025-12-04T11:21:33.6395056Z ----------------------------- Captured stderr call ----------------------------- 2025-12-04T11:21:33.6395471Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/ops_handler.py:772: UserWarning: undefined OpHandler.__getstate__, please add missing op schema 2025-12-04T11:21:33.6395634Z warnings.warn(f"undefined OpHandler.{name}, please add missing op schema") 2025-12-04T11:21:33.6396076Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/ops_handler.py:772: UserWarning: undefined OpHandler.__setstate__, please add missing op schema 2025-12-04T11:21:33.6396238Z warnings.warn(f"undefined OpHandler.{name}, please add missing op schema") 2025-12-04T11:21:33.6396367Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:21:33.6396435Z frames [('total', 1), ('ok', 1)] 2025-12-04T11:21:33.6396527Z stats [('calls_captured', 3), ('unique_graphs', 1)] 2025-12-04T11:21:33.6396706Z aot_autograd [('total', 1), ('autograd_cache_miss', 1), ('autograd_cache_saved', 1), ('ok', 1)] 2025-12-04T11:21:33.6397016Z inductor [('triton_bundler_save_kernel', 8), ('fxgraph_cache_miss', 1), ('async_compile_cache_miss', 1), ('triton_bundler_save_static_autotuner', 1)] 2025-12-04T11:21:33.6397145Z ----------------------------- Captured stderr call ----------------------------- 2025-12-04T11:21:33.6397556Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/ops_handler.py:772: UserWarning: undefined OpHandler.__getstate__, please add missing op schema 2025-12-04T11:21:33.6397714Z warnings.warn(f"undefined OpHandler.{name}, please add missing op schema") 2025-12-04T11:21:33.6398126Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/ops_handler.py:772: UserWarning: undefined OpHandler.__setstate__, please add missing op schema 2025-12-04T11:21:33.6398291Z warnings.warn(f"undefined OpHandler.{name}, please add missing op schema") 2025-12-04T11:21:33.6398782Z - generated xml file: /var/lib/jenkins/workspace/test/test-reports/python-pytest/inductor.test_compile_subprocess/inductor.test_compile_subprocess-576cab637f4fea56.xml - 2025-12-04T11:21:33.6398887Z =========================== short test summary info ============================ 2025-12-04T11:21:33.6399323Z FAILED [0.2189s] inductor/test_compile_subprocess.py::GPUTests::test_remove_noop_slice_cuda - AssertionError: 'def forward(self, arg0_1: "Sym(s77)", arg[333 chars]_9,)' != '' 2025-12-04T11:21:33.6399563Z - def forward(self, arg0_1: "Sym(s77)", arg1_1: "Sym(s27)", arg2_1: "Sym(s53)", arg3_1: "f32[s77, s27, s53][s27*s53, s53, 1]cuda:0"): 2025-12-04T11:21:33.6399814Z - add: "f32[s77, s27, s53][s27*s53, s53, 1]cuda:0" = torch.ops.aten.add.Tensor(arg3_1, 1); arg3_1 = None 2025-12-04T11:21:33.6399997Z - add_9: "f32[s77, s27, s53][s27*s53, s53, 1]cuda:0" = torch.ops.aten.add.Tensor(add, 1); add = None 2025-12-04T11:21:33.6400339Z - return (add_9,) : To accept the new output, re-run test with envvar EXPECTTEST_ACCEPT=1 (we recommend staging/committing your changes before doing this) 2025-12-04T11:21:33.6400344Z 2025-12-04T11:21:33.6400476Z To execute this test, run the following from the base repo dir: 2025-12-04T11:21:33.6400765Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_compile_subprocess.py GPUTests.test_remove_noop_slice_cuda 2025-12-04T11:21:33.6400768Z 2025-12-04T11:21:33.6400932Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:21:33.6401045Z !!!!!!!!!!!!!!!!!!!!!!!!!! stopping after 1 failures !!!!!!!!!!!!!!!!!!!!!!!!!!! 2025-12-04T11:21:33.6401179Z ======== 1 failed, 316 passed, 15 skipped, 2 rerun in 294.64s (0:04:54) ======== 2025-12-04T11:21:33.6401278Z Got exit code 1 2025-12-04T11:21:33.6401348Z Retrying single test... 2025-12-04T11:21:33.6401654Z W1204 11:19:05.736000 99861 site-packages/torch/_inductor/utils.py:1703] Not enough SMs to use max_autotune_gemm mode 2025-12-04T11:21:33.6402035Z Test results will be stored in test-reports/python-pytest/inductor.test_compile_subprocess/inductor.test_compile_subprocess-c5e45fed58b5f3dc.xml 2025-12-04T11:21:33.6402133Z ============================= test session starts ============================== 2025-12-04T11:21:33.6402345Z platform linux -- Python 3.10.14, pytest-7.3.2, pluggy-1.6.0 -- /opt/conda/envs/py_3.10/bin/python 2025-12-04T11:21:33.6402418Z cachedir: .pytest_cache 2025-12-04T11:21:33.6402759Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow] 2025-12-04T11:21:33.6402844Z rootdir: /var/lib/jenkins/workspace 2025-12-04T11:21:33.6402912Z configfile: pytest.ini 2025-12-04T11:21:33.6403226Z plugins: hypothesis-6.56.4, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-14.0, subtests-0.13.1, xdist-3.3.1, xdoctest-1.3.0, typeguard-4.3.0 2025-12-04T11:21:33.6403371Z collecting ... collected 897 items / 432 deselected / 465 selected 2025-12-04T11:21:33.6403712Z stepcurrent: skipping 331 already run items. Running only test/inductor/test_compile_subprocess.py::GPUTests::test_remove_noop_slice_cuda 2025-12-04T11:21:33.6403784Z Running 1 items in this shard 2025-12-04T11:21:33.6403788Z 2025-12-04T11:21:33.6404124Z inductor/test_compile_subprocess.py::GPUTests::test_remove_noop_slice_cuda <- test/inductor/test_torchinductor.py PASSED [8.3704s] [100%] 2025-12-04T11:21:33.6404127Z 2025-12-04T11:21:33.6404599Z - generated xml file: /var/lib/jenkins/workspace/test/test-reports/python-pytest/inductor.test_compile_subprocess/inductor.test_compile_subprocess-c5e45fed58b5f3dc.xml - 2025-12-04T11:21:33.6404722Z ====================== 1 passed, 432 deselected in 8.43s ======================= 2025-12-04T11:21:33.6404784Z Got exit code 0 2025-12-04T11:21:33.6404935Z Test succeeded in new process, continuing with the rest of the tests 2025-12-04T11:21:33.6405210Z W1204 11:19:25.072000 100145 site-packages/torch/_inductor/utils.py:1703] Not enough SMs to use max_autotune_gemm mode 2025-12-04T11:21:33.6405579Z Test results will be stored in test-reports/python-pytest/inductor.test_compile_subprocess/inductor.test_compile_subprocess-825047c883c2cfeb.xml 2025-12-04T11:21:33.6405679Z ============================= test session starts ============================== 2025-12-04T11:21:33.6405887Z platform linux -- Python 3.10.14, pytest-7.3.2, pluggy-1.6.0 -- /opt/conda/envs/py_3.10/bin/python 2025-12-04T11:21:33.6405954Z cachedir: .pytest_cache 2025-12-04T11:21:33.6406274Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow] 2025-12-04T11:21:33.6406352Z rootdir: /var/lib/jenkins/workspace 2025-12-04T11:21:33.6406463Z configfile: pytest.ini 2025-12-04T11:21:33.6406779Z plugins: hypothesis-6.56.4, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-14.0, subtests-0.13.1, xdist-3.3.1, xdoctest-1.3.0, typeguard-4.3.0 2025-12-04T11:21:33.6406919Z collecting ... collected 897 items / 332 deselected / 565 selected 2025-12-04T11:21:33.6407014Z stepcurrent: skipping 332 already run items. 2025-12-04T11:21:33.6407086Z Running 101 items in this shard 2025-12-04T11:21:33.6407089Z 2025-12-04T11:21:33.6408264Z inductor/test_compile_subprocess.py::GPUTests::test_remove_noop_view_default_cuda <- test/inductor/test_torchinductor.py SKIPPED [0.0007s] (Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/151511 for platform(s) linux, rocm, slow. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests.) [ 0%] 2025-12-04T11:21:33.6409444Z inductor/test_compile_subprocess.py::GPUTests::test_remove_noop_view_dtype_cuda <- test/inductor/test_torchinductor.py SKIPPED [0.0006s] (Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/151541 for platform(s) linux, rocm, slow. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests.) [ 1%] 2025-12-04T11:21:33.6409817Z inductor/test_compile_subprocess.py::GPUTests::test_repeat_as_strided_cuda <- test/inductor/test_torchinductor.py PASSED [8.3194s] [ 2%] 2025-12-04T11:21:33.6410441Z inductor/test_compile_subprocess.py::GPUTests::test_repeat_interleave_2_cuda <- test/inductor/test_torchinductor.py W1204 11:19:33.803000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] Unable to pickle input graph or example inputs 2025-12-04T11:21:33.6410755Z W1204 11:19:33.803000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] Traceback (most recent call last): 2025-12-04T11:21:33.6411284Z W1204 11:19:33.803000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/compile_fx_ext.py", line 486, in serialize_compile 2025-12-04T11:21:33.6411514Z W1204 11:19:33.803000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] ).serialize() 2025-12-04T11:21:33.6412001Z W1204 11:19:33.803000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/compile_fx_ext.py", line 210, in serialize 2025-12-04T11:21:33.6412335Z W1204 11:19:33.803000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] return _WireProtocolPickledInput(GraphPickler.dumps(self)) 2025-12-04T11:21:33.6412791Z W1204 11:19:33.803000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 127, in dumps 2025-12-04T11:21:33.6413027Z W1204 11:19:33.803000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] pickler.dump(obj) 2025-12-04T11:21:33.6413506Z W1204 11:19:33.803000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 94, in reducer_override 2025-12-04T11:21:33.6413831Z W1204 11:19:33.803000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] return _GraphModulePickleData.reduce_helper(self, obj) 2025-12-04T11:21:33.6414303Z W1204 11:19:33.803000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 352, in reduce_helper 2025-12-04T11:21:33.6414568Z W1204 11:19:33.803000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] cls(obj, pickler.options), 2025-12-04T11:21:33.6415023Z W1204 11:19:33.803000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 364, in __init__ 2025-12-04T11:21:33.6415374Z W1204 11:19:33.803000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] self.graph = _GraphPickleData(gm._graph, options) 2025-12-04T11:21:33.6415832Z W1204 11:19:33.803000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 571, in __init__ 2025-12-04T11:21:33.6416138Z W1204 11:19:33.803000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] nodes[node] = _NodePickleData(node, nodes, options) 2025-12-04T11:21:33.6416600Z W1204 11:19:33.803000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 387, in __init__ 2025-12-04T11:21:33.6416931Z W1204 11:19:33.803000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] self.target = _OpPickleData.pickle(node.target, options) 2025-12-04T11:21:33.6417458Z W1204 11:19:33.803000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 435, in pickle 2025-12-04T11:21:33.6417793Z W1204 11:19:33.803000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] return cls._pickle_op(name, _OpOverloadPickleData, options) 2025-12-04T11:21:33.6418265Z W1204 11:19:33.803000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 456, in _pickle_op 2025-12-04T11:21:33.6418628Z W1204 11:19:33.803000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] raise BypassFxGraphCache(f"Unable to pickle non-standard op: {name}") 2025-12-04T11:21:33.6419139Z W1204 11:19:33.803000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] torch._inductor.codecache.BypassFxGraphCache: Unable to pickle non-standard op: torch.ops.prims.iota.default 2025-12-04T11:21:33.6419223Z PASSED [0.5702s] [ 3%] 2025-12-04T11:21:33.6419913Z inductor/test_compile_subprocess.py::GPUTests::test_repeat_interleave_Tensor_decomp_int64_nd_1_cuda <- test/inductor/test_torchinductor.py W1204 11:19:34.377000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] Unable to pickle input graph or example inputs 2025-12-04T11:21:33.6420187Z W1204 11:19:34.377000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] Traceback (most recent call last): 2025-12-04T11:21:33.6420692Z W1204 11:19:34.377000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/compile_fx_ext.py", line 486, in serialize_compile 2025-12-04T11:21:33.6420913Z W1204 11:19:34.377000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] ).serialize() 2025-12-04T11:21:33.6421400Z W1204 11:19:34.377000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/compile_fx_ext.py", line 210, in serialize 2025-12-04T11:21:33.6421737Z W1204 11:19:34.377000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] return _WireProtocolPickledInput(GraphPickler.dumps(self)) 2025-12-04T11:21:33.6422192Z W1204 11:19:34.377000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 127, in dumps 2025-12-04T11:21:33.6422426Z W1204 11:19:34.377000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] pickler.dump(obj) 2025-12-04T11:21:33.6422907Z W1204 11:19:34.377000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 94, in reducer_override 2025-12-04T11:21:33.6423266Z W1204 11:19:34.377000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] return _GraphModulePickleData.reduce_helper(self, obj) 2025-12-04T11:21:33.6423816Z W1204 11:19:34.377000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 352, in reduce_helper 2025-12-04T11:21:33.6424083Z W1204 11:19:34.377000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] cls(obj, pickler.options), 2025-12-04T11:21:33.6424541Z W1204 11:19:34.377000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 364, in __init__ 2025-12-04T11:21:33.6424842Z W1204 11:19:34.377000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] self.graph = _GraphPickleData(gm._graph, options) 2025-12-04T11:21:33.6425296Z W1204 11:19:34.377000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 571, in __init__ 2025-12-04T11:21:33.6425677Z W1204 11:19:34.377000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] nodes[node] = _NodePickleData(node, nodes, options) 2025-12-04T11:21:33.6426134Z W1204 11:19:34.377000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 387, in __init__ 2025-12-04T11:21:33.6426455Z W1204 11:19:34.377000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] self.target = _OpPickleData.pickle(node.target, options) 2025-12-04T11:21:33.6426957Z W1204 11:19:34.377000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 435, in pickle 2025-12-04T11:21:33.6427288Z W1204 11:19:34.377000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] return cls._pickle_op(name, _OpOverloadPickleData, options) 2025-12-04T11:21:33.6427757Z W1204 11:19:34.377000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 456, in _pickle_op 2025-12-04T11:21:33.6428115Z W1204 11:19:34.377000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] raise BypassFxGraphCache(f"Unable to pickle non-standard op: {name}") 2025-12-04T11:21:33.6428600Z W1204 11:19:34.377000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] torch._inductor.codecache.BypassFxGraphCache: Unable to pickle non-standard op: torch.ops.prims.iota.default 2025-12-04T11:21:33.6428665Z PASSED [0.5083s] [ 4%] 2025-12-04T11:21:33.6429353Z inductor/test_compile_subprocess.py::GPUTests::test_repeat_interleave_Tensor_decomp_int64_nd_2_cuda <- test/inductor/test_torchinductor.py W1204 11:19:34.881000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] Unable to pickle input graph or example inputs 2025-12-04T11:21:33.6429629Z W1204 11:19:34.881000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] Traceback (most recent call last): 2025-12-04T11:21:33.6430137Z W1204 11:19:34.881000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/compile_fx_ext.py", line 486, in serialize_compile 2025-12-04T11:21:33.6430362Z W1204 11:19:34.881000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] ).serialize() 2025-12-04T11:21:33.6430837Z W1204 11:19:34.881000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/compile_fx_ext.py", line 210, in serialize 2025-12-04T11:21:33.6431175Z W1204 11:19:34.881000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] return _WireProtocolPickledInput(GraphPickler.dumps(self)) 2025-12-04T11:21:33.6431672Z W1204 11:19:34.881000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 127, in dumps 2025-12-04T11:21:33.6431905Z W1204 11:19:34.881000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] pickler.dump(obj) 2025-12-04T11:21:33.6432384Z W1204 11:19:34.881000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 94, in reducer_override 2025-12-04T11:21:33.6432702Z W1204 11:19:34.881000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] return _GraphModulePickleData.reduce_helper(self, obj) 2025-12-04T11:21:33.6433174Z W1204 11:19:34.881000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 352, in reduce_helper 2025-12-04T11:21:33.6433437Z W1204 11:19:34.881000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] cls(obj, pickler.options), 2025-12-04T11:21:33.6433960Z W1204 11:19:34.881000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 364, in __init__ 2025-12-04T11:21:33.6434267Z W1204 11:19:34.881000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] self.graph = _GraphPickleData(gm._graph, options) 2025-12-04T11:21:33.6434721Z W1204 11:19:34.881000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 571, in __init__ 2025-12-04T11:21:33.6435066Z W1204 11:19:34.881000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] nodes[node] = _NodePickleData(node, nodes, options) 2025-12-04T11:21:33.6435520Z W1204 11:19:34.881000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 387, in __init__ 2025-12-04T11:21:33.6435848Z W1204 11:19:34.881000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] self.target = _OpPickleData.pickle(node.target, options) 2025-12-04T11:21:33.6436298Z W1204 11:19:34.881000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 435, in pickle 2025-12-04T11:21:33.6436626Z W1204 11:19:34.881000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] return cls._pickle_op(name, _OpOverloadPickleData, options) 2025-12-04T11:21:33.6437093Z W1204 11:19:34.881000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 456, in _pickle_op 2025-12-04T11:21:33.6437450Z W1204 11:19:34.881000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] raise BypassFxGraphCache(f"Unable to pickle non-standard op: {name}") 2025-12-04T11:21:33.6437938Z W1204 11:19:34.881000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] torch._inductor.codecache.BypassFxGraphCache: Unable to pickle non-standard op: torch.ops.prims.iota.default 2025-12-04T11:21:33.6438002Z PASSED [0.4741s] [ 5%] 2025-12-04T11:21:33.6438336Z inductor/test_compile_subprocess.py::GPUTests::test_repeat_interleave_cuda <- test/inductor/test_torchinductor.py PASSED [1.3924s] [ 6%] 2025-12-04T11:21:33.6438696Z inductor/test_compile_subprocess.py::GPUTests::test_reuse_buffers_with_aliasing_cuda <- test/inductor/test_torchinductor.py PASSED [4.5495s] [ 7%] 2025-12-04T11:21:33.6438989Z inductor/test_compile_subprocess.py::GPUTests::test_rsqrt_cuda <- test/inductor/test_torchinductor.py PASSED [0.7237s] [ 8%] 2025-12-04T11:21:33.6439616Z inductor/test_compile_subprocess.py::GPUTests::test_rsqrt_dynamic_shapes_cuda <- test/inductor/test_torchinductor.py W1204 11:19:42.101000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] Unable to pickle input graph or example inputs 2025-12-04T11:21:33.6439926Z W1204 11:19:42.101000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] Traceback (most recent call last): 2025-12-04T11:21:33.6440437Z W1204 11:19:42.101000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/compile_fx_ext.py", line 486, in serialize_compile 2025-12-04T11:21:33.6440657Z W1204 11:19:42.101000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] ).serialize() 2025-12-04T11:21:33.6441133Z W1204 11:19:42.101000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/compile_fx_ext.py", line 210, in serialize 2025-12-04T11:21:33.6441468Z W1204 11:19:42.101000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] return _WireProtocolPickledInput(GraphPickler.dumps(self)) 2025-12-04T11:21:33.6441997Z W1204 11:19:42.101000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 127, in dumps 2025-12-04T11:21:33.6442243Z W1204 11:19:42.101000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] pickler.dump(obj) 2025-12-04T11:21:33.6442722Z W1204 11:19:42.101000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 94, in reducer_override 2025-12-04T11:21:33.6443042Z W1204 11:19:42.101000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] return _GraphModulePickleData.reduce_helper(self, obj) 2025-12-04T11:21:33.6443546Z W1204 11:19:42.101000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 352, in reduce_helper 2025-12-04T11:21:33.6443806Z W1204 11:19:42.101000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] cls(obj, pickler.options), 2025-12-04T11:21:33.6444271Z W1204 11:19:42.101000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 364, in __init__ 2025-12-04T11:21:33.6444572Z W1204 11:19:42.101000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] self.graph = _GraphPickleData(gm._graph, options) 2025-12-04T11:21:33.6445028Z W1204 11:19:42.101000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 571, in __init__ 2025-12-04T11:21:33.6445329Z W1204 11:19:42.101000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] nodes[node] = _NodePickleData(node, nodes, options) 2025-12-04T11:21:33.6445789Z W1204 11:19:42.101000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 387, in __init__ 2025-12-04T11:21:33.6446109Z W1204 11:19:42.101000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] self.target = _OpPickleData.pickle(node.target, options) 2025-12-04T11:21:33.6446562Z W1204 11:19:42.101000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 435, in pickle 2025-12-04T11:21:33.6446891Z W1204 11:19:42.101000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] return cls._pickle_op(name, _OpOverloadPickleData, options) 2025-12-04T11:21:33.6447357Z W1204 11:19:42.101000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 456, in _pickle_op 2025-12-04T11:21:33.6447756Z W1204 11:19:42.101000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] raise BypassFxGraphCache(f"Unable to pickle non-standard op: {name}") 2025-12-04T11:21:33.6448286Z W1204 11:19:42.101000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] torch._inductor.codecache.BypassFxGraphCache: Unable to pickle non-standard op: torch.ops.prims.convert_element_type.default 2025-12-04T11:21:33.6448563Z W1204 11:19:42.113000 100145 site-packages/torch/_inductor/utils.py:1703] [0/0] Not enough SMs to use max_autotune_gemm mode 2025-12-04T11:21:33.6448827Z W1204 11:19:42.562000 100314 site-packages/torch/_inductor/utils.py:1703] [1/0] Not enough SMs to use max_autotune_gemm mode 2025-12-04T11:21:33.6449122Z W1204 11:19:43.013000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] Unable to pickle input graph or example inputs 2025-12-04T11:21:33.6449391Z W1204 11:19:43.013000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] Traceback (most recent call last): 2025-12-04T11:21:33.6449964Z W1204 11:19:43.013000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/compile_fx_ext.py", line 486, in serialize_compile 2025-12-04T11:21:33.6450190Z W1204 11:19:43.013000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] ).serialize() 2025-12-04T11:21:33.6450669Z W1204 11:19:43.013000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/compile_fx_ext.py", line 210, in serialize 2025-12-04T11:21:33.6451002Z W1204 11:19:43.013000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] return _WireProtocolPickledInput(GraphPickler.dumps(self)) 2025-12-04T11:21:33.6451500Z W1204 11:19:43.013000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 127, in dumps 2025-12-04T11:21:33.6451740Z W1204 11:19:43.013000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] pickler.dump(obj) 2025-12-04T11:21:33.6452222Z W1204 11:19:43.013000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 94, in reducer_override 2025-12-04T11:21:33.6452544Z W1204 11:19:43.013000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] return _GraphModulePickleData.reduce_helper(self, obj) 2025-12-04T11:21:33.6453016Z W1204 11:19:43.013000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 352, in reduce_helper 2025-12-04T11:21:33.6453273Z W1204 11:19:43.013000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] cls(obj, pickler.options), 2025-12-04T11:21:33.6453733Z W1204 11:19:43.013000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 364, in __init__ 2025-12-04T11:21:33.6454040Z W1204 11:19:43.013000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] self.graph = _GraphPickleData(gm._graph, options) 2025-12-04T11:21:33.6454491Z W1204 11:19:43.013000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 571, in __init__ 2025-12-04T11:21:33.6454796Z W1204 11:19:43.013000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] nodes[node] = _NodePickleData(node, nodes, options) 2025-12-04T11:21:33.6455248Z W1204 11:19:43.013000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 387, in __init__ 2025-12-04T11:21:33.6455607Z W1204 11:19:43.013000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] self.target = _OpPickleData.pickle(node.target, options) 2025-12-04T11:21:33.6456060Z W1204 11:19:43.013000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 435, in pickle 2025-12-04T11:21:33.6456387Z W1204 11:19:43.013000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] return cls._pickle_op(name, _OpOverloadPickleData, options) 2025-12-04T11:21:33.6456850Z W1204 11:19:43.013000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 456, in _pickle_op 2025-12-04T11:21:33.6457208Z W1204 11:19:43.013000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] raise BypassFxGraphCache(f"Unable to pickle non-standard op: {name}") 2025-12-04T11:21:33.6457733Z W1204 11:19:43.013000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] torch._inductor.codecache.BypassFxGraphCache: Unable to pickle non-standard op: torch.ops.prims.convert_element_type.default 2025-12-04T11:21:33.6457831Z PASSED [1.7929s] [ 9%] 2025-12-04T11:21:33.6458252Z inductor/test_compile_subprocess.py::GPUTests::test_scatter2_cuda <- test/inductor/test_torchinductor.py SKIPPED [0.0025s] (unstable on sm86) [ 10%] 2025-12-04T11:21:33.6458575Z inductor/test_compile_subprocess.py::GPUTests::test_scatter3_cuda <- test/inductor/test_torchinductor.py PASSED [1.0154s] [ 11%] 2025-12-04T11:21:33.6458878Z inductor/test_compile_subprocess.py::GPUTests::test_scatter6_cuda <- test/inductor/test_torchinductor.py PASSED [1.4196s] [ 12%] 2025-12-04T11:21:33.6459317Z inductor/test_compile_subprocess.py::GPUTests::test_scatter_add1_cuda <- test/inductor/test_torchinductor.py SKIPPED [0.0002s] (Flaky test, needs debugging) [ 13%] 2025-12-04T11:21:33.6459644Z inductor/test_compile_subprocess.py::GPUTests::test_scatter_bf16_cuda <- test/inductor/test_torchinductor.py PASSED [0.6345s] [ 14%] 2025-12-04T11:21:33.6459975Z inductor/test_compile_subprocess.py::GPUTests::test_scatter_reduce1_cuda <- test/inductor/test_torchinductor.py PASSED [0.5723s] [ 15%] 2025-12-04T11:21:33.6460296Z inductor/test_compile_subprocess.py::GPUTests::test_scatter_reduce3_cuda <- test/inductor/test_torchinductor.py PASSED [0.8087s] [ 16%] 2025-12-04T11:21:33.6460692Z inductor/test_compile_subprocess.py::GPUTests::test_sdpa_prefer_nd_tiling_True_use_block_ptr_True_cuda <- test/inductor/test_torchinductor.py PASSED [0.6786s] [ 17%] 2025-12-04T11:21:33.6461027Z inductor/test_compile_subprocess.py::GPUTests::test_searchsorted_cuda <- test/inductor/test_torchinductor.py PASSED [12.8041s] [ 18%] 2025-12-04T11:21:33.6461340Z inductor/test_compile_subprocess.py::GPUTests::test_select_scatter_cuda <- test/inductor/test_torchinductor.py PASSED [0.7050s] [ 19%] 2025-12-04T11:21:33.6461695Z inductor/test_compile_subprocess.py::GPUTests::test_setitem_with_int_parameter_cuda <- test/inductor/test_torchinductor.py PASSED [0.5336s] [ 20%] 2025-12-04T11:21:33.6462273Z inductor/test_compile_subprocess.py::GPUTests::test_sgn_cuda <- test/inductor/test_torchinductor.py W1204 11:20:03.194000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] Unable to pickle input graph or example inputs 2025-12-04T11:21:33.6462543Z W1204 11:20:03.194000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] Traceback (most recent call last): 2025-12-04T11:21:33.6463058Z W1204 11:20:03.194000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/compile_fx_ext.py", line 486, in serialize_compile 2025-12-04T11:21:33.6463281Z W1204 11:20:03.194000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] ).serialize() 2025-12-04T11:21:33.6463810Z W1204 11:20:03.194000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/compile_fx_ext.py", line 210, in serialize 2025-12-04T11:21:33.6464200Z W1204 11:20:03.194000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] return _WireProtocolPickledInput(GraphPickler.dumps(self)) 2025-12-04T11:21:33.6464655Z W1204 11:20:03.194000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 127, in dumps 2025-12-04T11:21:33.6464889Z W1204 11:20:03.194000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] pickler.dump(obj) 2025-12-04T11:21:33.6465366Z W1204 11:20:03.194000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 94, in reducer_override 2025-12-04T11:21:33.6465692Z W1204 11:20:03.194000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] return _GraphModulePickleData.reduce_helper(self, obj) 2025-12-04T11:21:33.6466200Z W1204 11:20:03.194000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 352, in reduce_helper 2025-12-04T11:21:33.6466497Z W1204 11:20:03.194000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] cls(obj, pickler.options), 2025-12-04T11:21:33.6466956Z W1204 11:20:03.194000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 364, in __init__ 2025-12-04T11:21:33.6467259Z W1204 11:20:03.194000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] self.graph = _GraphPickleData(gm._graph, options) 2025-12-04T11:21:33.6467755Z W1204 11:20:03.194000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 571, in __init__ 2025-12-04T11:21:33.6468059Z W1204 11:20:03.194000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] nodes[node] = _NodePickleData(node, nodes, options) 2025-12-04T11:21:33.6468520Z W1204 11:20:03.194000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 387, in __init__ 2025-12-04T11:21:33.6468839Z W1204 11:20:03.194000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] self.target = _OpPickleData.pickle(node.target, options) 2025-12-04T11:21:33.6469295Z W1204 11:20:03.194000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 435, in pickle 2025-12-04T11:21:33.6469625Z W1204 11:20:03.194000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] return cls._pickle_op(name, _OpOverloadPickleData, options) 2025-12-04T11:21:33.6470098Z W1204 11:20:03.194000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 456, in _pickle_op 2025-12-04T11:21:33.6470471Z W1204 11:20:03.194000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] raise BypassFxGraphCache(f"Unable to pickle non-standard op: {name}") 2025-12-04T11:21:33.6470992Z W1204 11:20:03.194000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] torch._inductor.codecache.BypassFxGraphCache: Unable to pickle non-standard op: torch.ops.prims.convert_element_type.default 2025-12-04T11:21:33.6471061Z PASSED [0.3568s] [ 21%] 2025-12-04T11:21:33.6471666Z inductor/test_compile_subprocess.py::GPUTests::test_sgn_extremal_cuda <- test/inductor/test_torchinductor.py W1204 11:20:03.509000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] Unable to pickle input graph or example inputs 2025-12-04T11:21:33.6471942Z W1204 11:20:03.509000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] Traceback (most recent call last): 2025-12-04T11:21:33.6472502Z W1204 11:20:03.509000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/compile_fx_ext.py", line 486, in serialize_compile 2025-12-04T11:21:33.6472723Z W1204 11:20:03.509000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] ).serialize() 2025-12-04T11:21:33.6473205Z W1204 11:20:03.509000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/compile_fx_ext.py", line 210, in serialize 2025-12-04T11:21:33.6473534Z W1204 11:20:03.509000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] return _WireProtocolPickledInput(GraphPickler.dumps(self)) 2025-12-04T11:21:33.6473987Z W1204 11:20:03.509000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 127, in dumps 2025-12-04T11:21:33.6474259Z W1204 11:20:03.509000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] pickler.dump(obj) 2025-12-04T11:21:33.6474778Z W1204 11:20:03.509000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 94, in reducer_override 2025-12-04T11:21:33.6475096Z W1204 11:20:03.509000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] return _GraphModulePickleData.reduce_helper(self, obj) 2025-12-04T11:21:33.6475564Z W1204 11:20:03.509000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 352, in reduce_helper 2025-12-04T11:21:33.6475859Z W1204 11:20:03.509000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] cls(obj, pickler.options), 2025-12-04T11:21:33.6476320Z W1204 11:20:03.509000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 364, in __init__ 2025-12-04T11:21:33.6476629Z W1204 11:20:03.509000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] self.graph = _GraphPickleData(gm._graph, options) 2025-12-04T11:21:33.6477082Z W1204 11:20:03.509000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 571, in __init__ 2025-12-04T11:21:33.6477386Z W1204 11:20:03.509000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] nodes[node] = _NodePickleData(node, nodes, options) 2025-12-04T11:21:33.6478028Z W1204 11:20:03.509000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 387, in __init__ 2025-12-04T11:21:33.6478353Z W1204 11:20:03.509000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] self.target = _OpPickleData.pickle(node.target, options) 2025-12-04T11:21:33.6478816Z W1204 11:20:03.509000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 435, in pickle 2025-12-04T11:21:33.6479143Z W1204 11:20:03.509000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] return cls._pickle_op(name, _OpOverloadPickleData, options) 2025-12-04T11:21:33.6479618Z W1204 11:20:03.509000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 456, in _pickle_op 2025-12-04T11:21:33.6479981Z W1204 11:20:03.509000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] raise BypassFxGraphCache(f"Unable to pickle non-standard op: {name}") 2025-12-04T11:21:33.6480568Z W1204 11:20:03.509000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] torch._inductor.codecache.BypassFxGraphCache: Unable to pickle non-standard op: torch.ops.prims.convert_element_type.default 2025-12-04T11:21:33.6480642Z PASSED [0.2884s] [ 22%] 2025-12-04T11:21:33.6480978Z inductor/test_compile_subprocess.py::GPUTests::test_shape_padding_cuda <- test/inductor/test_torchinductor.py PASSED [2.0210s] [ 23%] 2025-12-04T11:21:33.6481561Z inductor/test_compile_subprocess.py::GPUTests::test_silu_cuda <- test/inductor/test_torchinductor.py W1204 11:20:05.835000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] Unable to pickle input graph or example inputs 2025-12-04T11:21:33.6481832Z W1204 11:20:05.835000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] Traceback (most recent call last): 2025-12-04T11:21:33.6482347Z W1204 11:20:05.835000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/compile_fx_ext.py", line 486, in serialize_compile 2025-12-04T11:21:33.6482627Z W1204 11:20:05.835000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] ).serialize() 2025-12-04T11:21:33.6483153Z W1204 11:20:05.835000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/compile_fx_ext.py", line 210, in serialize 2025-12-04T11:21:33.6483503Z W1204 11:20:05.835000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] return _WireProtocolPickledInput(GraphPickler.dumps(self)) 2025-12-04T11:21:33.6483959Z W1204 11:20:05.835000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 127, in dumps 2025-12-04T11:21:33.6484257Z W1204 11:20:05.835000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] pickler.dump(obj) 2025-12-04T11:21:33.6484742Z W1204 11:20:05.835000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 94, in reducer_override 2025-12-04T11:21:33.6485081Z W1204 11:20:05.835000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] return _GraphModulePickleData.reduce_helper(self, obj) 2025-12-04T11:21:33.6485554Z W1204 11:20:05.835000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 352, in reduce_helper 2025-12-04T11:21:33.6485815Z W1204 11:20:05.835000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] cls(obj, pickler.options), 2025-12-04T11:21:33.6486280Z W1204 11:20:05.835000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 364, in __init__ 2025-12-04T11:21:33.6486586Z W1204 11:20:05.835000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] self.graph = _GraphPickleData(gm._graph, options) 2025-12-04T11:21:33.6487048Z W1204 11:20:05.835000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 571, in __init__ 2025-12-04T11:21:33.6487352Z W1204 11:20:05.835000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] nodes[node] = _NodePickleData(node, nodes, options) 2025-12-04T11:21:33.6487809Z W1204 11:20:05.835000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 387, in __init__ 2025-12-04T11:21:33.6488138Z W1204 11:20:05.835000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] self.target = _OpPickleData.pickle(node.target, options) 2025-12-04T11:21:33.6488593Z W1204 11:20:05.835000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 435, in pickle 2025-12-04T11:21:33.6488972Z W1204 11:20:05.835000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] return cls._pickle_op(name, _OpOverloadPickleData, options) 2025-12-04T11:21:33.6489434Z W1204 11:20:05.835000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 456, in _pickle_op 2025-12-04T11:21:33.6489809Z W1204 11:20:05.835000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] raise BypassFxGraphCache(f"Unable to pickle non-standard op: {name}") 2025-12-04T11:21:33.6490334Z W1204 11:20:05.835000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] torch._inductor.codecache.BypassFxGraphCache: Unable to pickle non-standard op: torch.ops.prims.convert_element_type.default 2025-12-04T11:21:33.6490401Z PASSED [0.3132s] [ 24%] 2025-12-04T11:21:33.6490795Z inductor/test_compile_subprocess.py::GPUTests::test_size_asserts_for_multi_output_fallback_cuda <- test/inductor/test_torchinductor.py PASSED [0.1154s] [ 25%] 2025-12-04T11:21:33.6491190Z inductor/test_compile_subprocess.py::GPUTests::test_slice2_cuda <- test/inductor/test_torchinductor.py PASSED [0.4550s] [ 26%] 2025-12-04T11:21:33.6491498Z inductor/test_compile_subprocess.py::GPUTests::test_slice4_cuda <- test/inductor/test_torchinductor.py PASSED [0.2222s] [ 27%] 2025-12-04T11:21:33.6491826Z inductor/test_compile_subprocess.py::GPUTests::test_slice_mutation3_cuda <- test/inductor/test_torchinductor.py PASSED [0.1710s] [ 28%] 2025-12-04T11:21:33.6492145Z inductor/test_compile_subprocess.py::GPUTests::test_slice_scatter3_cuda <- test/inductor/test_torchinductor.py PASSED [0.3673s] [ 29%] 2025-12-04T11:21:33.6492499Z inductor/test_compile_subprocess.py::GPUTests::test_slice_scatter4_cuda <- test/inductor/test_torchinductor.py PASSED [0.3686s] [ 30%] 2025-12-04T11:21:33.6492817Z inductor/test_compile_subprocess.py::GPUTests::test_slice_scatter_cuda <- test/inductor/test_torchinductor.py PASSED [0.7493s] [ 31%] 2025-12-04T11:21:33.6493197Z inductor/test_compile_subprocess.py::GPUTests::test_slice_scatter_dtype_consistency_cuda <- test/inductor/test_torchinductor.py PASSED [0.5911s] [ 32%] 2025-12-04T11:21:33.6493549Z inductor/test_compile_subprocess.py::GPUTests::test_slice_view_with_graph_break_cuda <- test/inductor/test_torchinductor.py PASSED [0.3677s] [ 33%] 2025-12-04T11:21:33.6494177Z inductor/test_compile_subprocess.py::GPUTests::test_softmax_backward_data_cuda <- test/inductor/test_torchinductor.py W1204 11:20:09.433000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] Unable to pickle input graph or example inputs 2025-12-04T11:21:33.6494447Z W1204 11:20:09.433000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] Traceback (most recent call last): 2025-12-04T11:21:33.6494956Z W1204 11:20:09.433000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/compile_fx_ext.py", line 486, in serialize_compile 2025-12-04T11:21:33.6495188Z W1204 11:20:09.433000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] ).serialize() 2025-12-04T11:21:33.6495667Z W1204 11:20:09.433000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/compile_fx_ext.py", line 210, in serialize 2025-12-04T11:21:33.6496006Z W1204 11:20:09.433000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] return _WireProtocolPickledInput(GraphPickler.dumps(self)) 2025-12-04T11:21:33.6496460Z W1204 11:20:09.433000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 127, in dumps 2025-12-04T11:21:33.6496698Z W1204 11:20:09.433000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] pickler.dump(obj) 2025-12-04T11:21:33.6497306Z W1204 11:20:09.433000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 94, in reducer_override 2025-12-04T11:21:33.6497624Z W1204 11:20:09.433000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] return _GraphModulePickleData.reduce_helper(self, obj) 2025-12-04T11:21:33.6498100Z W1204 11:20:09.433000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 352, in reduce_helper 2025-12-04T11:21:33.6498360Z W1204 11:20:09.433000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] cls(obj, pickler.options), 2025-12-04T11:21:33.6498822Z W1204 11:20:09.433000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 364, in __init__ 2025-12-04T11:21:33.6499177Z W1204 11:20:09.433000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] self.graph = _GraphPickleData(gm._graph, options) 2025-12-04T11:21:33.6499665Z W1204 11:20:09.433000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 571, in __init__ 2025-12-04T11:21:33.6499970Z W1204 11:20:09.433000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] nodes[node] = _NodePickleData(node, nodes, options) 2025-12-04T11:21:33.6500427Z W1204 11:20:09.433000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 387, in __init__ 2025-12-04T11:21:33.6500788Z W1204 11:20:09.433000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] self.target = _OpPickleData.pickle(node.target, options) 2025-12-04T11:21:33.6501246Z W1204 11:20:09.433000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 435, in pickle 2025-12-04T11:21:33.6501582Z W1204 11:20:09.433000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] return cls._pickle_op(name, _OpOverloadPickleData, options) 2025-12-04T11:21:33.6502044Z W1204 11:20:09.433000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 456, in _pickle_op 2025-12-04T11:21:33.6502403Z W1204 11:20:09.433000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] raise BypassFxGraphCache(f"Unable to pickle non-standard op: {name}") 2025-12-04T11:21:33.6502883Z W1204 11:20:09.433000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] torch._inductor.codecache.BypassFxGraphCache: Unable to pickle non-standard op: torch.ops.prims.fma.default 2025-12-04T11:21:33.6503178Z W1204 11:20:09.799000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] Unable to pickle input graph or example inputs 2025-12-04T11:21:33.6503449Z W1204 11:20:09.799000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] Traceback (most recent call last): 2025-12-04T11:21:33.6504004Z W1204 11:20:09.799000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/compile_fx_ext.py", line 486, in serialize_compile 2025-12-04T11:21:33.6504233Z W1204 11:20:09.799000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] ).serialize() 2025-12-04T11:21:33.6504711Z W1204 11:20:09.799000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/compile_fx_ext.py", line 210, in serialize 2025-12-04T11:21:33.6505083Z W1204 11:20:09.799000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] return _WireProtocolPickledInput(GraphPickler.dumps(self)) 2025-12-04T11:21:33.6505555Z W1204 11:20:09.799000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 127, in dumps 2025-12-04T11:21:33.6505789Z W1204 11:20:09.799000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] pickler.dump(obj) 2025-12-04T11:21:33.6506269Z W1204 11:20:09.799000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 94, in reducer_override 2025-12-04T11:21:33.6506591Z W1204 11:20:09.799000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] return _GraphModulePickleData.reduce_helper(self, obj) 2025-12-04T11:21:33.6507065Z W1204 11:20:09.799000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 352, in reduce_helper 2025-12-04T11:21:33.6507392Z W1204 11:20:09.799000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] cls(obj, pickler.options), 2025-12-04T11:21:33.6507844Z W1204 11:20:09.799000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 364, in __init__ 2025-12-04T11:21:33.6508151Z W1204 11:20:09.799000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] self.graph = _GraphPickleData(gm._graph, options) 2025-12-04T11:21:33.6508606Z W1204 11:20:09.799000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 571, in __init__ 2025-12-04T11:21:33.6508949Z W1204 11:20:09.799000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] nodes[node] = _NodePickleData(node, nodes, options) 2025-12-04T11:21:33.6509419Z W1204 11:20:09.799000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 387, in __init__ 2025-12-04T11:21:33.6509744Z W1204 11:20:09.799000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] self.target = _OpPickleData.pickle(node.target, options) 2025-12-04T11:21:33.6510196Z W1204 11:20:09.799000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 435, in pickle 2025-12-04T11:21:33.6510523Z W1204 11:20:09.799000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] return cls._pickle_op(name, _OpOverloadPickleData, options) 2025-12-04T11:21:33.6510994Z W1204 11:20:09.799000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 456, in _pickle_op 2025-12-04T11:21:33.6511355Z W1204 11:20:09.799000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] raise BypassFxGraphCache(f"Unable to pickle non-standard op: {name}") 2025-12-04T11:21:33.6511883Z W1204 11:20:09.799000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] torch._inductor.codecache.BypassFxGraphCache: Unable to pickle non-standard op: torch.ops.prims.convert_element_type.default 2025-12-04T11:21:33.6511948Z PASSED [0.7579s] [ 34%] 2025-12-04T11:21:33.6512297Z inductor/test_compile_subprocess.py::GPUTests::test_softmax_one_kernel_loop_cuda <- test/inductor/test_torchinductor.py PASSED [0.3090s] [ 35%] 2025-12-04T11:21:33.6512889Z inductor/test_compile_subprocess.py::GPUTests::test_sort_bool_cuda <- test/inductor/test_torchinductor.py W1204 11:20:10.485000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] Unable to pickle input graph or example inputs 2025-12-04T11:21:33.6513197Z W1204 11:20:10.485000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] Traceback (most recent call last): 2025-12-04T11:21:33.6513710Z W1204 11:20:10.485000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/compile_fx_ext.py", line 486, in serialize_compile 2025-12-04T11:21:33.6513933Z W1204 11:20:10.485000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] ).serialize() 2025-12-04T11:21:33.6514427Z W1204 11:20:10.485000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/compile_fx_ext.py", line 210, in serialize 2025-12-04T11:21:33.6514762Z W1204 11:20:10.485000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] return _WireProtocolPickledInput(GraphPickler.dumps(self)) 2025-12-04T11:21:33.6515213Z W1204 11:20:10.485000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 127, in dumps 2025-12-04T11:21:33.6515526Z W1204 11:20:10.485000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] pickler.dump(obj) 2025-12-04T11:21:33.6516005Z W1204 11:20:10.485000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 94, in reducer_override 2025-12-04T11:21:33.6516328Z W1204 11:20:10.485000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] return _GraphModulePickleData.reduce_helper(self, obj) 2025-12-04T11:21:33.6516799Z W1204 11:20:10.485000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 352, in reduce_helper 2025-12-04T11:21:33.6517106Z W1204 11:20:10.485000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] cls(obj, pickler.options), 2025-12-04T11:21:33.6517566Z W1204 11:20:10.485000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 364, in __init__ 2025-12-04T11:21:33.6517871Z W1204 11:20:10.485000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] self.graph = _GraphPickleData(gm._graph, options) 2025-12-04T11:21:33.6518345Z W1204 11:20:10.485000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 571, in __init__ 2025-12-04T11:21:33.6518645Z W1204 11:20:10.485000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] nodes[node] = _NodePickleData(node, nodes, options) 2025-12-04T11:21:33.6519106Z W1204 11:20:10.485000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 387, in __init__ 2025-12-04T11:21:33.6519426Z W1204 11:20:10.485000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] self.target = _OpPickleData.pickle(node.target, options) 2025-12-04T11:21:33.6519885Z W1204 11:20:10.485000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 435, in pickle 2025-12-04T11:21:33.6520217Z W1204 11:20:10.485000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] return cls._pickle_op(name, _OpOverloadPickleData, options) 2025-12-04T11:21:33.6520679Z W1204 11:20:10.485000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 456, in _pickle_op 2025-12-04T11:21:33.6521049Z W1204 11:20:10.485000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] raise BypassFxGraphCache(f"Unable to pickle non-standard op: {name}") 2025-12-04T11:21:33.6521625Z W1204 11:20:10.485000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] torch._inductor.codecache.BypassFxGraphCache: Unable to pickle non-standard op: torch.ops.prims.convert_element_type.default 2025-12-04T11:21:33.6521926Z W1204 11:20:15.758000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] Unable to pickle input graph or example inputs 2025-12-04T11:21:33.6522192Z W1204 11:20:15.758000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] Traceback (most recent call last): 2025-12-04T11:21:33.6522701Z W1204 11:20:15.758000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/compile_fx_ext.py", line 486, in serialize_compile 2025-12-04T11:21:33.6522928Z W1204 11:20:15.758000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] ).serialize() 2025-12-04T11:21:33.6523403Z W1204 11:20:15.758000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/compile_fx_ext.py", line 210, in serialize 2025-12-04T11:21:33.6523813Z W1204 11:20:15.758000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] return _WireProtocolPickledInput(GraphPickler.dumps(self)) 2025-12-04T11:21:33.6524268Z W1204 11:20:15.758000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 127, in dumps 2025-12-04T11:21:33.6524510Z W1204 11:20:15.758000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] pickler.dump(obj) 2025-12-04T11:21:33.6525028Z W1204 11:20:15.758000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 94, in reducer_override 2025-12-04T11:21:33.6525351Z W1204 11:20:15.758000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] return _GraphModulePickleData.reduce_helper(self, obj) 2025-12-04T11:21:33.6525830Z W1204 11:20:15.758000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 352, in reduce_helper 2025-12-04T11:21:33.6526089Z W1204 11:20:15.758000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] cls(obj, pickler.options), 2025-12-04T11:21:33.6526558Z W1204 11:20:15.758000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 364, in __init__ 2025-12-04T11:21:33.6526861Z W1204 11:20:15.758000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] self.graph = _GraphPickleData(gm._graph, options) 2025-12-04T11:21:33.6527324Z W1204 11:20:15.758000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 571, in __init__ 2025-12-04T11:21:33.6527629Z W1204 11:20:15.758000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] nodes[node] = _NodePickleData(node, nodes, options) 2025-12-04T11:21:33.6528089Z W1204 11:20:15.758000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 387, in __init__ 2025-12-04T11:21:33.6528412Z W1204 11:20:15.758000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] self.target = _OpPickleData.pickle(node.target, options) 2025-12-04T11:21:33.6528861Z W1204 11:20:15.758000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 435, in pickle 2025-12-04T11:21:33.6529196Z W1204 11:20:15.758000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] return cls._pickle_op(name, _OpOverloadPickleData, options) 2025-12-04T11:21:33.6529712Z W1204 11:20:15.758000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 456, in _pickle_op 2025-12-04T11:21:33.6530073Z W1204 11:20:15.758000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] raise BypassFxGraphCache(f"Unable to pickle non-standard op: {name}") 2025-12-04T11:21:33.6530596Z W1204 11:20:15.758000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] torch._inductor.codecache.BypassFxGraphCache: Unable to pickle non-standard op: torch.ops.prims.convert_element_type.default 2025-12-04T11:21:33.6530664Z PASSED [10.5463s] [ 36%] 2025-12-04T11:21:33.6530969Z inductor/test_compile_subprocess.py::GPUTests::test_sort_cuda <- test/inductor/test_torchinductor.py PASSED [1.6682s] [ 37%] 2025-12-04T11:21:33.6531444Z inductor/test_compile_subprocess.py::GPUTests::test_sort_stable_cuda <- test/inductor/test_torchinductor.py SKIPPED [0.0005s] (test is slow; run with PYTORCH_TEST_WITH_SLOW to enable test) [ 38%] 2025-12-04T11:21:33.6531852Z inductor/test_compile_subprocess.py::GPUTests::test_sort_transpose_cuda <- test/inductor/test_torchinductor.py PASSED [20.0425s] [ 39%] 2025-12-04T11:21:33.6532185Z inductor/test_compile_subprocess.py::GPUTests::test_special_polygamma_cuda <- test/inductor/test_torchinductor.py PASSED [0.5559s] [ 40%] 2025-12-04T11:21:33.6532502Z inductor/test_compile_subprocess.py::GPUTests::test_split_cumprod_cuda <- test/inductor/test_torchinductor.py PASSED [0.9668s] [ 41%] 2025-12-04T11:21:33.6532838Z inductor/test_compile_subprocess.py::GPUTests::test_split_cumsum_low_prec_cuda <- test/inductor/test_torchinductor.py PASSED [0.2431s] [ 42%] 2025-12-04T11:21:33.6533533Z inductor/test_compile_subprocess.py::GPUTests::test_split_failed_cuda <- test/inductor/test_torchinductor.py E1204 11:20:44.477000 100145 site-packages/torch/_subclasses/fake_tensor.py:2827] [0/0] failed while attempting to run meta for aten.split_with_sizes.default 2025-12-04T11:21:33.6533816Z E1204 11:20:44.477000 100145 site-packages/torch/_subclasses/fake_tensor.py:2827] [0/0] Traceback (most recent call last): 2025-12-04T11:21:33.6534323Z E1204 11:20:44.477000 100145 site-packages/torch/_subclasses/fake_tensor.py:2827] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_subclasses/fake_tensor.py", line 2823, in _dispatch_impl 2025-12-04T11:21:33.6534581Z E1204 11:20:44.477000 100145 site-packages/torch/_subclasses/fake_tensor.py:2827] [0/0] r = func(*args, **kwargs) 2025-12-04T11:21:33.6535003Z E1204 11:20:44.477000 100145 site-packages/torch/_subclasses/fake_tensor.py:2827] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_ops.py", line 836, in __call__ 2025-12-04T11:21:33.6535274Z E1204 11:20:44.477000 100145 site-packages/torch/_subclasses/fake_tensor.py:2827] [0/0] return self._op(*args, **kwargs) 2025-12-04T11:21:33.6535752Z E1204 11:20:44.477000 100145 site-packages/torch/_subclasses/fake_tensor.py:2827] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_refs/__init__.py", line 4359, in split_with_sizes 2025-12-04T11:21:33.6535990Z E1204 11:20:44.477000 100145 site-packages/torch/_subclasses/fake_tensor.py:2827] [0/0] torch._check_with( 2025-12-04T11:21:33.6536441Z E1204 11:20:44.477000 100145 site-packages/torch/_subclasses/fake_tensor.py:2827] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/__init__.py", line 1716, in _check_with 2025-12-04T11:21:33.6536720Z E1204 11:20:44.477000 100145 site-packages/torch/_subclasses/fake_tensor.py:2827] [0/0] raise error_type(message_evaluated) 2025-12-04T11:21:33.6537067Z E1204 11:20:44.477000 100145 site-packages/torch/_subclasses/fake_tensor.py:2827] [0/0] ValueError: Split sizes add up to 4 but got the tensor's size of 5 2025-12-04T11:21:33.6537137Z PASSED [0.0143s] [ 43%] 2025-12-04T11:21:33.6537462Z inductor/test_compile_subprocess.py::GPUTests::test_split_with_list_cuda <- test/inductor/test_torchinductor.py PASSED [1.3204s] [ 44%] 2025-12-04T11:21:33.6538180Z inductor/test_compile_subprocess.py::GPUTests::test_split_with_sizes_with_unbacked_symints_cuda <- test/inductor/test_torchinductor.py W1204 11:20:46.092000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [1/0] Unable to pickle input graph or example inputs 2025-12-04T11:21:33.6538446Z W1204 11:20:46.092000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [1/0] Traceback (most recent call last): 2025-12-04T11:21:33.6538959Z W1204 11:20:46.092000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [1/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/compile_fx_ext.py", line 486, in serialize_compile 2025-12-04T11:21:33.6539180Z W1204 11:20:46.092000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [1/0] ).serialize() 2025-12-04T11:21:33.6539658Z W1204 11:20:46.092000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [1/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/compile_fx_ext.py", line 210, in serialize 2025-12-04T11:21:33.6540069Z W1204 11:20:46.092000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [1/0] return _WireProtocolPickledInput(GraphPickler.dumps(self)) 2025-12-04T11:21:33.6540521Z W1204 11:20:46.092000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [1/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 127, in dumps 2025-12-04T11:21:33.6540760Z W1204 11:20:46.092000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [1/0] pickler.dump(obj) 2025-12-04T11:21:33.6541280Z W1204 11:20:46.092000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [1/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 94, in reducer_override 2025-12-04T11:21:33.6541608Z W1204 11:20:46.092000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [1/0] return _GraphModulePickleData.reduce_helper(self, obj) 2025-12-04T11:21:33.6542085Z W1204 11:20:46.092000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [1/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 352, in reduce_helper 2025-12-04T11:21:33.6542346Z W1204 11:20:46.092000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [1/0] cls(obj, pickler.options), 2025-12-04T11:21:33.6542816Z W1204 11:20:46.092000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [1/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 364, in __init__ 2025-12-04T11:21:33.6543119Z W1204 11:20:46.092000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [1/0] self.graph = _GraphPickleData(gm._graph, options) 2025-12-04T11:21:33.6543628Z W1204 11:20:46.092000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [1/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 571, in __init__ 2025-12-04T11:21:33.6543932Z W1204 11:20:46.092000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [1/0] nodes[node] = _NodePickleData(node, nodes, options) 2025-12-04T11:21:33.6544388Z W1204 11:20:46.092000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [1/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 387, in __init__ 2025-12-04T11:21:33.6544715Z W1204 11:20:46.092000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [1/0] self.target = _OpPickleData.pickle(node.target, options) 2025-12-04T11:21:33.6545167Z W1204 11:20:46.092000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [1/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 435, in pickle 2025-12-04T11:21:33.6545524Z W1204 11:20:46.092000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [1/0] return cls._pickle_op(name, _OpOverloadPickleData, options) 2025-12-04T11:21:33.6546036Z W1204 11:20:46.092000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [1/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 456, in _pickle_op 2025-12-04T11:21:33.6546404Z W1204 11:20:46.092000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [1/0] raise BypassFxGraphCache(f"Unable to pickle non-standard op: {name}") 2025-12-04T11:21:33.6546885Z W1204 11:20:46.092000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [1/0] torch._inductor.codecache.BypassFxGraphCache: Unable to pickle non-standard op: torch.ops.prims.iota.default 2025-12-04T11:21:33.6546951Z PASSED [0.7881s] [ 45%] 2025-12-04T11:21:33.6547548Z inductor/test_compile_subprocess.py::GPUTests::test_std_cuda <- test/inductor/test_torchinductor.py W1204 11:20:47.535000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] Unable to pickle input graph or example inputs 2025-12-04T11:21:33.6547820Z W1204 11:20:47.535000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] Traceback (most recent call last): 2025-12-04T11:21:33.6548412Z W1204 11:20:47.535000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/compile_fx_ext.py", line 486, in serialize_compile 2025-12-04T11:21:33.6548638Z W1204 11:20:47.535000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] ).serialize() 2025-12-04T11:21:33.6549125Z W1204 11:20:47.535000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/compile_fx_ext.py", line 210, in serialize 2025-12-04T11:21:33.6549461Z W1204 11:20:47.535000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] return _WireProtocolPickledInput(GraphPickler.dumps(self)) 2025-12-04T11:21:33.6549962Z W1204 11:20:47.535000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 127, in dumps 2025-12-04T11:21:33.6550207Z W1204 11:20:47.535000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] pickler.dump(obj) 2025-12-04T11:21:33.6550695Z W1204 11:20:47.535000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 94, in reducer_override 2025-12-04T11:21:33.6551021Z W1204 11:20:47.535000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] return _GraphModulePickleData.reduce_helper(self, obj) 2025-12-04T11:21:33.6551491Z W1204 11:20:47.535000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 352, in reduce_helper 2025-12-04T11:21:33.6551759Z W1204 11:20:47.535000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] cls(obj, pickler.options), 2025-12-04T11:21:33.6552218Z W1204 11:20:47.535000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 364, in __init__ 2025-12-04T11:21:33.6552522Z W1204 11:20:47.535000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] self.graph = _GraphPickleData(gm._graph, options) 2025-12-04T11:21:33.6552985Z W1204 11:20:47.535000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 571, in __init__ 2025-12-04T11:21:33.6553288Z W1204 11:20:47.535000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] nodes[node] = _NodePickleData(node, nodes, options) 2025-12-04T11:21:33.6553745Z W1204 11:20:47.535000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 387, in __init__ 2025-12-04T11:21:33.6554099Z W1204 11:20:47.535000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] self.target = _OpPickleData.pickle(node.target, options) 2025-12-04T11:21:33.6554549Z W1204 11:20:47.535000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 435, in pickle 2025-12-04T11:21:33.6554893Z W1204 11:20:47.535000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] return cls._pickle_op(name, _OpOverloadPickleData, options) 2025-12-04T11:21:33.6555355Z W1204 11:20:47.535000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 456, in _pickle_op 2025-12-04T11:21:33.6555721Z W1204 11:20:47.535000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] raise BypassFxGraphCache(f"Unable to pickle non-standard op: {name}") 2025-12-04T11:21:33.6556285Z W1204 11:20:47.535000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] torch._inductor.codecache.BypassFxGraphCache: Unable to pickle non-standard op: torch.ops.prims.convert_element_type.default 2025-12-04T11:21:33.6556391Z PASSED [1.7620s] [ 46%] 2025-12-04T11:21:33.6556817Z inductor/test_compile_subprocess.py::GPUTests::test_stride_preservation_with_stride_modifying_fx_pass_cuda <- test/inductor/test_torchinductor.py PASSED [0.1359s] [ 47%] 2025-12-04T11:21:33.6557141Z inductor/test_compile_subprocess.py::GPUTests::test_strided_inputs_cuda <- test/inductor/test_torchinductor.py PASSED [0.1703s] [ 48%] 2025-12-04T11:21:33.6557441Z inductor/test_compile_subprocess.py::GPUTests::test_sum1_cuda <- test/inductor/test_torchinductor.py PASSED [0.6183s] [ 49%] 2025-12-04T11:21:33.6557768Z inductor/test_compile_subprocess.py::GPUTests::test_sum3_cuda <- test/inductor/test_torchinductor.py PASSED [0.6970s] [ 50%] 2025-12-04T11:21:33.6558085Z inductor/test_compile_subprocess.py::GPUTests::test_sum_dtype_cuda <- test/inductor/test_torchinductor.py PASSED [0.8722s] [ 51%] 2025-12-04T11:21:33.6558386Z inductor/test_compile_subprocess.py::GPUTests::test_sum_int_cuda <- test/inductor/test_torchinductor.py PASSED [0.5384s] [ 52%] 2025-12-04T11:21:33.6558696Z inductor/test_compile_subprocess.py::GPUTests::test_sum_keepdims_cuda <- test/inductor/test_torchinductor.py PASSED [0.6237s] [ 53%] 2025-12-04T11:21:33.6558993Z inductor/test_compile_subprocess.py::GPUTests::test_tanh_cuda <- test/inductor/test_torchinductor.py PASSED [0.7241s] [ 54%] 2025-12-04T11:21:33.6559296Z inductor/test_compile_subprocess.py::GPUTests::test_tensor1_cuda <- test/inductor/test_torchinductor.py PASSED [0.3290s] [ 55%] 2025-12-04T11:21:33.6559599Z inductor/test_compile_subprocess.py::GPUTests::test_tensor2_cuda <- test/inductor/test_torchinductor.py PASSED [0.3172s] [ 56%] 2025-12-04T11:21:33.6559895Z inductor/test_compile_subprocess.py::GPUTests::test_tensor3_cuda <- test/inductor/test_torchinductor.py PASSED [0.6684s] [ 57%] 2025-12-04T11:21:33.6560235Z inductor/test_compile_subprocess.py::GPUTests::test_tensor_index_put_slice_cuda <- test/inductor/test_torchinductor.py PASSED [3.0230s] [ 58%] 2025-12-04T11:21:33.6560930Z inductor/test_compile_subprocess.py::GPUTests::test_tmp_not_defined_issue1_use_block_ptr_True_cuda <- test/inductor/test_torchinductor.py W1204 11:20:57.560000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] Unable to pickle input graph or example inputs 2025-12-04T11:21:33.6561199Z W1204 11:20:57.560000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] Traceback (most recent call last): 2025-12-04T11:21:33.6561715Z W1204 11:20:57.560000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/compile_fx_ext.py", line 486, in serialize_compile 2025-12-04T11:21:33.6561941Z W1204 11:20:57.560000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] ).serialize() 2025-12-04T11:21:33.6562470Z W1204 11:20:57.560000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/compile_fx_ext.py", line 210, in serialize 2025-12-04T11:21:33.6562802Z W1204 11:20:57.560000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] return _WireProtocolPickledInput(GraphPickler.dumps(self)) 2025-12-04T11:21:33.6563251Z W1204 11:20:57.560000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 127, in dumps 2025-12-04T11:21:33.6563490Z W1204 11:20:57.560000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] pickler.dump(obj) 2025-12-04T11:21:33.6563968Z W1204 11:20:57.560000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 94, in reducer_override 2025-12-04T11:21:33.6564293Z W1204 11:20:57.560000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] return _GraphModulePickleData.reduce_helper(self, obj) 2025-12-04T11:21:33.6564834Z W1204 11:20:57.560000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 352, in reduce_helper 2025-12-04T11:21:33.6565102Z W1204 11:20:57.560000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] cls(obj, pickler.options), 2025-12-04T11:21:33.6565558Z W1204 11:20:57.560000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 364, in __init__ 2025-12-04T11:21:33.6565905Z W1204 11:20:57.560000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] self.graph = _GraphPickleData(gm._graph, options) 2025-12-04T11:21:33.6566373Z W1204 11:20:57.560000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 571, in __init__ 2025-12-04T11:21:33.6566678Z W1204 11:20:57.560000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] nodes[node] = _NodePickleData(node, nodes, options) 2025-12-04T11:21:33.6567135Z W1204 11:20:57.560000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 387, in __init__ 2025-12-04T11:21:33.6567452Z W1204 11:20:57.560000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] self.target = _OpPickleData.pickle(node.target, options) 2025-12-04T11:21:33.6567909Z W1204 11:20:57.560000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 435, in pickle 2025-12-04T11:21:33.6568239Z W1204 11:20:57.560000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] return cls._pickle_op(name, _OpOverloadPickleData, options) 2025-12-04T11:21:33.6568704Z W1204 11:20:57.560000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 456, in _pickle_op 2025-12-04T11:21:33.6569075Z W1204 11:20:57.560000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] raise BypassFxGraphCache(f"Unable to pickle non-standard op: {name}") 2025-12-04T11:21:33.6569811Z W1204 11:20:57.560000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] torch._inductor.codecache.BypassFxGraphCache: Unable to pickle non-standard op: torch.ops.prims.convert_element_type.default 2025-12-04T11:21:33.6569885Z PASSED [0.7283s] [ 59%] 2025-12-04T11:21:33.6570245Z inductor/test_compile_subprocess.py::GPUTests::test_tmp_not_defined_issue2_cuda <- test/inductor/test_torchinductor.py PASSED [0.6526s] [ 60%] 2025-12-04T11:21:33.6570953Z inductor/test_compile_subprocess.py::GPUTests::test_tmp_not_defined_issue3_cuda <- test/inductor/test_torchinductor.py W1204 11:20:58.648000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] Unable to pickle input graph or example inputs 2025-12-04T11:21:33.6571242Z W1204 11:20:58.648000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] Traceback (most recent call last): 2025-12-04T11:21:33.6571763Z W1204 11:20:58.648000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/compile_fx_ext.py", line 486, in serialize_compile 2025-12-04T11:21:33.6571994Z W1204 11:20:58.648000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] ).serialize() 2025-12-04T11:21:33.6572476Z W1204 11:20:58.648000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/compile_fx_ext.py", line 210, in serialize 2025-12-04T11:21:33.6572817Z W1204 11:20:58.648000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] return _WireProtocolPickledInput(GraphPickler.dumps(self)) 2025-12-04T11:21:33.6573346Z W1204 11:20:58.648000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 127, in dumps 2025-12-04T11:21:33.6573585Z W1204 11:20:58.648000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] pickler.dump(obj) 2025-12-04T11:21:33.6574070Z W1204 11:20:58.648000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 94, in reducer_override 2025-12-04T11:21:33.6574434Z W1204 11:20:58.648000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] return _GraphModulePickleData.reduce_helper(self, obj) 2025-12-04T11:21:33.6574918Z W1204 11:20:58.648000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 352, in reduce_helper 2025-12-04T11:21:33.6575184Z W1204 11:20:58.648000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] cls(obj, pickler.options), 2025-12-04T11:21:33.6575647Z W1204 11:20:58.648000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 364, in __init__ 2025-12-04T11:21:33.6575953Z W1204 11:20:58.648000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] self.graph = _GraphPickleData(gm._graph, options) 2025-12-04T11:21:33.6576410Z W1204 11:20:58.648000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 571, in __init__ 2025-12-04T11:21:33.6576724Z W1204 11:20:58.648000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] nodes[node] = _NodePickleData(node, nodes, options) 2025-12-04T11:21:33.6577185Z W1204 11:20:58.648000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 387, in __init__ 2025-12-04T11:21:33.6577510Z W1204 11:20:58.648000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] self.target = _OpPickleData.pickle(node.target, options) 2025-12-04T11:21:33.6578161Z W1204 11:20:58.648000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 435, in pickle 2025-12-04T11:21:33.6578516Z W1204 11:20:58.648000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] return cls._pickle_op(name, _OpOverloadPickleData, options) 2025-12-04T11:21:33.6578985Z W1204 11:20:58.648000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 456, in _pickle_op 2025-12-04T11:21:33.6579421Z W1204 11:20:58.648000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] raise BypassFxGraphCache(f"Unable to pickle non-standard op: {name}") 2025-12-04T11:21:33.6579910Z W1204 11:20:58.648000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] torch._inductor.codecache.BypassFxGraphCache: Unable to pickle non-standard op: torch.ops.prims.iota.default 2025-12-04T11:21:33.6579979Z PASSED [2.8328s] [ 61%] 2025-12-04T11:21:33.6580614Z inductor/test_compile_subprocess.py::GPUTests::test_to_device_constant_cuda <- test/inductor/test_torchinductor.py W1204 11:21:01.375000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] Unable to pickle input graph or example inputs 2025-12-04T11:21:33.6580891Z W1204 11:21:01.375000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] Traceback (most recent call last): 2025-12-04T11:21:33.6581459Z W1204 11:21:01.375000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/compile_fx_ext.py", line 486, in serialize_compile 2025-12-04T11:21:33.6581754Z W1204 11:21:01.375000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] ).serialize() 2025-12-04T11:21:33.6582239Z W1204 11:21:01.375000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/compile_fx_ext.py", line 210, in serialize 2025-12-04T11:21:33.6582582Z W1204 11:21:01.375000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] return _WireProtocolPickledInput(GraphPickler.dumps(self)) 2025-12-04T11:21:33.6583093Z W1204 11:21:01.375000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 127, in dumps 2025-12-04T11:21:33.6583342Z W1204 11:21:01.375000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] pickler.dump(obj) 2025-12-04T11:21:33.6583920Z W1204 11:21:01.375000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 94, in reducer_override 2025-12-04T11:21:33.6584242Z W1204 11:21:01.375000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] return _GraphModulePickleData.reduce_helper(self, obj) 2025-12-04T11:21:33.6584720Z W1204 11:21:01.375000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 352, in reduce_helper 2025-12-04T11:21:33.6584985Z W1204 11:21:01.375000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] cls(obj, pickler.options), 2025-12-04T11:21:33.6585448Z W1204 11:21:01.375000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 364, in __init__ 2025-12-04T11:21:33.6585758Z W1204 11:21:01.375000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] self.graph = _GraphPickleData(gm._graph, options) 2025-12-04T11:21:33.6586219Z W1204 11:21:01.375000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 571, in __init__ 2025-12-04T11:21:33.6586523Z W1204 11:21:01.375000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] nodes[node] = _NodePickleData(node, nodes, options) 2025-12-04T11:21:33.6586977Z W1204 11:21:01.375000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 387, in __init__ 2025-12-04T11:21:33.6587308Z W1204 11:21:01.375000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] self.target = _OpPickleData.pickle(node.target, options) 2025-12-04T11:21:33.6587805Z W1204 11:21:01.375000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 435, in pickle 2025-12-04T11:21:33.6588151Z W1204 11:21:01.375000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] return cls._pickle_op(name, _OpOverloadPickleData, options) 2025-12-04T11:21:33.6588612Z W1204 11:21:01.375000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 456, in _pickle_op 2025-12-04T11:21:33.6588980Z W1204 11:21:01.375000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] raise BypassFxGraphCache(f"Unable to pickle non-standard op: {name}") 2025-12-04T11:21:33.6589466Z W1204 11:21:01.375000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] torch._inductor.codecache.BypassFxGraphCache: Unable to pickle non-standard op: torch.ops.prims.iota.default 2025-12-04T11:21:33.6589847Z W1204 11:21:01.539000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] Unable to pickle input graph or example inputs 2025-12-04T11:21:33.6590125Z W1204 11:21:01.539000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] Traceback (most recent call last): 2025-12-04T11:21:33.6590633Z W1204 11:21:01.539000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/compile_fx_ext.py", line 486, in serialize_compile 2025-12-04T11:21:33.6590859Z W1204 11:21:01.539000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] ).serialize() 2025-12-04T11:21:33.6591372Z W1204 11:21:01.539000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/compile_fx_ext.py", line 210, in serialize 2025-12-04T11:21:33.6591716Z W1204 11:21:01.539000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] return _WireProtocolPickledInput(GraphPickler.dumps(self)) 2025-12-04T11:21:33.6592164Z W1204 11:21:01.539000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 127, in dumps 2025-12-04T11:21:33.6592399Z W1204 11:21:01.539000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] pickler.dump(obj) 2025-12-04T11:21:33.6592883Z W1204 11:21:01.539000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 94, in reducer_override 2025-12-04T11:21:33.6593204Z W1204 11:21:01.539000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] return _GraphModulePickleData.reduce_helper(self, obj) 2025-12-04T11:21:33.6593685Z W1204 11:21:01.539000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 352, in reduce_helper 2025-12-04T11:21:33.6593948Z W1204 11:21:01.539000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] cls(obj, pickler.options), 2025-12-04T11:21:33.6594401Z W1204 11:21:01.539000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 364, in __init__ 2025-12-04T11:21:33.6594712Z W1204 11:21:01.539000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] self.graph = _GraphPickleData(gm._graph, options) 2025-12-04T11:21:33.6595166Z W1204 11:21:01.539000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 571, in __init__ 2025-12-04T11:21:33.6595514Z W1204 11:21:01.539000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] nodes[node] = _NodePickleData(node, nodes, options) 2025-12-04T11:21:33.6595969Z W1204 11:21:01.539000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 387, in __init__ 2025-12-04T11:21:33.6596302Z W1204 11:21:01.539000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] self.target = _OpPickleData.pickle(node.target, options) 2025-12-04T11:21:33.6596763Z W1204 11:21:01.539000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 435, in pickle 2025-12-04T11:21:33.6597097Z W1204 11:21:01.539000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] return cls._pickle_op(name, _OpOverloadPickleData, options) 2025-12-04T11:21:33.6597564Z W1204 11:21:01.539000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 456, in _pickle_op 2025-12-04T11:21:33.6598006Z W1204 11:21:01.539000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] raise BypassFxGraphCache(f"Unable to pickle non-standard op: {name}") 2025-12-04T11:21:33.6598489Z W1204 11:21:01.539000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] torch._inductor.codecache.BypassFxGraphCache: Unable to pickle non-standard op: torch.ops.prims.iota.default 2025-12-04T11:21:33.6598556Z PASSED [0.3619s] [ 62%] 2025-12-04T11:21:33.6598895Z inductor/test_compile_subprocess.py::GPUTests::test_to_memory_format_cuda <- test/inductor/test_torchinductor.py PASSED [0.7002s] [ 63%] 2025-12-04T11:21:33.6599233Z inductor/test_compile_subprocess.py::GPUTests::test_topk_cuda <- test/inductor/test_torchinductor.py PASSED [0.2098s] [ 64%] 2025-12-04T11:21:33.6599555Z inductor/test_compile_subprocess.py::GPUTests::test_transpose_add_cuda <- test/inductor/test_torchinductor.py PASSED [0.4419s] [ 65%] 2025-12-04T11:21:33.6599908Z inductor/test_compile_subprocess.py::GPUTests::test_transposed_propagates_cuda <- test/inductor/test_torchinductor.py PASSED [0.1590s] [ 66%] 2025-12-04T11:21:33.6600309Z inductor/test_compile_subprocess.py::GPUTests::test_triton_argmin_argmax_transpose_logical_index_cuda <- test/inductor/test_torchinductor.py PASSED [3.5082s] [ 67%] 2025-12-04T11:21:33.6600947Z inductor/test_compile_subprocess.py::GPUTests::test_triton_kernel_bool_param_cuda <- test/inductor/test_torchinductor.py W1204 11:21:06.973000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] Unable to pickle input graph or example inputs 2025-12-04T11:21:33.6601216Z W1204 11:21:06.973000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] Traceback (most recent call last): 2025-12-04T11:21:33.6601732Z W1204 11:21:06.973000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/compile_fx_ext.py", line 486, in serialize_compile 2025-12-04T11:21:33.6601966Z W1204 11:21:06.973000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] ).serialize() 2025-12-04T11:21:33.6602445Z W1204 11:21:06.973000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/compile_fx_ext.py", line 210, in serialize 2025-12-04T11:21:33.6602780Z W1204 11:21:06.973000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] return _WireProtocolPickledInput(GraphPickler.dumps(self)) 2025-12-04T11:21:33.6603229Z W1204 11:21:06.973000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 127, in dumps 2025-12-04T11:21:33.6603470Z W1204 11:21:06.973000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] pickler.dump(obj) 2025-12-04T11:21:33.6603957Z W1204 11:21:06.973000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] AttributeError: Can't pickle local object 'CommonTemplate.test_triton_kernel_bool_param..Model' 2025-12-04T11:21:33.6604251Z W1204 11:21:07.349000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] Unable to pickle input graph or example inputs 2025-12-04T11:21:33.6604520Z W1204 11:21:07.349000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] Traceback (most recent call last): 2025-12-04T11:21:33.6605028Z W1204 11:21:07.349000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/compile_fx_ext.py", line 486, in serialize_compile 2025-12-04T11:21:33.6605252Z W1204 11:21:07.349000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] ).serialize() 2025-12-04T11:21:33.6605731Z W1204 11:21:07.349000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/compile_fx_ext.py", line 210, in serialize 2025-12-04T11:21:33.6606148Z W1204 11:21:07.349000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] return _WireProtocolPickledInput(GraphPickler.dumps(self)) 2025-12-04T11:21:33.6606595Z W1204 11:21:07.349000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 127, in dumps 2025-12-04T11:21:33.6606826Z W1204 11:21:07.349000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] pickler.dump(obj) 2025-12-04T11:21:33.6607285Z W1204 11:21:07.349000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] AttributeError: Can't pickle local object 'CommonTemplate.test_triton_kernel_bool_param..Model' 2025-12-04T11:21:33.6607383Z PASSED [0.7621s] [ 68%] 2025-12-04T11:21:33.6607973Z inductor/test_compile_subprocess.py::GPUTests::test_triu_cuda <- test/inductor/test_torchinductor.py W1204 11:21:07.541000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] Unable to pickle input graph or example inputs 2025-12-04T11:21:33.6608249Z W1204 11:21:07.541000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] Traceback (most recent call last): 2025-12-04T11:21:33.6608754Z W1204 11:21:07.541000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/compile_fx_ext.py", line 486, in serialize_compile 2025-12-04T11:21:33.6608978Z W1204 11:21:07.541000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] ).serialize() 2025-12-04T11:21:33.6609456Z W1204 11:21:07.541000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/compile_fx_ext.py", line 210, in serialize 2025-12-04T11:21:33.6609791Z W1204 11:21:07.541000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] return _WireProtocolPickledInput(GraphPickler.dumps(self)) 2025-12-04T11:21:33.6610239Z W1204 11:21:07.541000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 127, in dumps 2025-12-04T11:21:33.6610480Z W1204 11:21:07.541000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] pickler.dump(obj) 2025-12-04T11:21:33.6610956Z W1204 11:21:07.541000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 94, in reducer_override 2025-12-04T11:21:33.6611274Z W1204 11:21:07.541000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] return _GraphModulePickleData.reduce_helper(self, obj) 2025-12-04T11:21:33.6611751Z W1204 11:21:07.541000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 352, in reduce_helper 2025-12-04T11:21:33.6612063Z W1204 11:21:07.541000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] cls(obj, pickler.options), 2025-12-04T11:21:33.6612527Z W1204 11:21:07.541000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 364, in __init__ 2025-12-04T11:21:33.6612829Z W1204 11:21:07.541000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] self.graph = _GraphPickleData(gm._graph, options) 2025-12-04T11:21:33.6613284Z W1204 11:21:07.541000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 571, in __init__ 2025-12-04T11:21:33.6613594Z W1204 11:21:07.541000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] nodes[node] = _NodePickleData(node, nodes, options) 2025-12-04T11:21:33.6614142Z W1204 11:21:07.541000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 387, in __init__ 2025-12-04T11:21:33.6614473Z W1204 11:21:07.541000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] self.target = _OpPickleData.pickle(node.target, options) 2025-12-04T11:21:33.6614928Z W1204 11:21:07.541000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 435, in pickle 2025-12-04T11:21:33.6615265Z W1204 11:21:07.541000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] return cls._pickle_op(name, _OpOverloadPickleData, options) 2025-12-04T11:21:33.6615776Z W1204 11:21:07.541000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 456, in _pickle_op 2025-12-04T11:21:33.6616147Z W1204 11:21:07.541000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] raise BypassFxGraphCache(f"Unable to pickle non-standard op: {name}") 2025-12-04T11:21:33.6616633Z W1204 11:21:07.541000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] torch._inductor.codecache.BypassFxGraphCache: Unable to pickle non-standard op: torch.ops.prims.iota.default 2025-12-04T11:21:33.6616928Z W1204 11:21:07.901000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] Unable to pickle input graph or example inputs 2025-12-04T11:21:33.6617199Z W1204 11:21:07.901000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] Traceback (most recent call last): 2025-12-04T11:21:33.6617711Z W1204 11:21:07.901000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/compile_fx_ext.py", line 486, in serialize_compile 2025-12-04T11:21:33.6617939Z W1204 11:21:07.901000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] ).serialize() 2025-12-04T11:21:33.6618419Z W1204 11:21:07.901000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/compile_fx_ext.py", line 210, in serialize 2025-12-04T11:21:33.6618749Z W1204 11:21:07.901000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] return _WireProtocolPickledInput(GraphPickler.dumps(self)) 2025-12-04T11:21:33.6619203Z W1204 11:21:07.901000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 127, in dumps 2025-12-04T11:21:33.6619439Z W1204 11:21:07.901000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] pickler.dump(obj) 2025-12-04T11:21:33.6619920Z W1204 11:21:07.901000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 94, in reducer_override 2025-12-04T11:21:33.6620278Z W1204 11:21:07.901000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] return _GraphModulePickleData.reduce_helper(self, obj) 2025-12-04T11:21:33.6620752Z W1204 11:21:07.901000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 352, in reduce_helper 2025-12-04T11:21:33.6621009Z W1204 11:21:07.901000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] cls(obj, pickler.options), 2025-12-04T11:21:33.6621468Z W1204 11:21:07.901000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 364, in __init__ 2025-12-04T11:21:33.6621777Z W1204 11:21:07.901000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] self.graph = _GraphPickleData(gm._graph, options) 2025-12-04T11:21:33.6622312Z W1204 11:21:07.901000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 571, in __init__ 2025-12-04T11:21:33.6622620Z W1204 11:21:07.901000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] nodes[node] = _NodePickleData(node, nodes, options) 2025-12-04T11:21:33.6623076Z W1204 11:21:07.901000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 387, in __init__ 2025-12-04T11:21:33.6623402Z W1204 11:21:07.901000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] self.target = _OpPickleData.pickle(node.target, options) 2025-12-04T11:21:33.6624044Z W1204 11:21:07.901000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 435, in pickle 2025-12-04T11:21:33.6624390Z W1204 11:21:07.901000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] return cls._pickle_op(name, _OpOverloadPickleData, options) 2025-12-04T11:21:33.6624859Z W1204 11:21:07.901000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 456, in _pickle_op 2025-12-04T11:21:33.6625217Z W1204 11:21:07.901000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] raise BypassFxGraphCache(f"Unable to pickle non-standard op: {name}") 2025-12-04T11:21:33.6625698Z W1204 11:21:07.901000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] torch._inductor.codecache.BypassFxGraphCache: Unable to pickle non-standard op: torch.ops.prims.iota.default 2025-12-04T11:21:33.6625763Z PASSED [0.7163s] [ 69%] 2025-12-04T11:21:33.6626391Z inductor/test_compile_subprocess.py::GPUTests::test_uint4x2_mixed_mm_cuda <- test/inductor/test_torchinductor.py W1204 11:21:08.257000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] Unable to pickle input graph or example inputs 2025-12-04T11:21:33.6626668Z W1204 11:21:08.257000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] Traceback (most recent call last): 2025-12-04T11:21:33.6627185Z W1204 11:21:08.257000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/compile_fx_ext.py", line 486, in serialize_compile 2025-12-04T11:21:33.6627415Z W1204 11:21:08.257000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] ).serialize() 2025-12-04T11:21:33.6627903Z W1204 11:21:08.257000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/compile_fx_ext.py", line 210, in serialize 2025-12-04T11:21:33.6628286Z W1204 11:21:08.257000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] return _WireProtocolPickledInput(GraphPickler.dumps(self)) 2025-12-04T11:21:33.6628742Z W1204 11:21:08.257000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 127, in dumps 2025-12-04T11:21:33.6628978Z W1204 11:21:08.257000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] pickler.dump(obj) 2025-12-04T11:21:33.6629462Z W1204 11:21:08.257000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 94, in reducer_override 2025-12-04T11:21:33.6629782Z W1204 11:21:08.257000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] return _GraphModulePickleData.reduce_helper(self, obj) 2025-12-04T11:21:33.6630264Z W1204 11:21:08.257000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 352, in reduce_helper 2025-12-04T11:21:33.6630607Z W1204 11:21:08.257000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] cls(obj, pickler.options), 2025-12-04T11:21:33.6631073Z W1204 11:21:08.257000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 364, in __init__ 2025-12-04T11:21:33.6631377Z W1204 11:21:08.257000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] self.graph = _GraphPickleData(gm._graph, options) 2025-12-04T11:21:33.6631866Z W1204 11:21:08.257000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 571, in __init__ 2025-12-04T11:21:33.6632180Z W1204 11:21:08.257000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] nodes[node] = _NodePickleData(node, nodes, options) 2025-12-04T11:21:33.6632650Z W1204 11:21:08.257000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 387, in __init__ 2025-12-04T11:21:33.6632975Z W1204 11:21:08.257000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] self.target = _OpPickleData.pickle(node.target, options) 2025-12-04T11:21:33.6633426Z W1204 11:21:08.257000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 435, in pickle 2025-12-04T11:21:33.6633762Z W1204 11:21:08.257000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] return cls._pickle_op(name, _OpOverloadPickleData, options) 2025-12-04T11:21:33.6634222Z W1204 11:21:08.257000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 456, in _pickle_op 2025-12-04T11:21:33.6634586Z W1204 11:21:08.257000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] raise BypassFxGraphCache(f"Unable to pickle non-standard op: {name}") 2025-12-04T11:21:33.6635111Z W1204 11:21:08.257000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] torch._inductor.codecache.BypassFxGraphCache: Unable to pickle non-standard op: torch.ops.prims.convert_element_type.default 2025-12-04T11:21:33.6635403Z W1204 11:21:08.570000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] Unable to pickle input graph or example inputs 2025-12-04T11:21:33.6635672Z W1204 11:21:08.570000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] Traceback (most recent call last): 2025-12-04T11:21:33.6636181Z W1204 11:21:08.570000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/compile_fx_ext.py", line 486, in serialize_compile 2025-12-04T11:21:33.6636442Z W1204 11:21:08.570000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] ).serialize() 2025-12-04T11:21:33.6636929Z W1204 11:21:08.570000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/compile_fx_ext.py", line 210, in serialize 2025-12-04T11:21:33.6637261Z W1204 11:21:08.570000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] return _WireProtocolPickledInput(GraphPickler.dumps(self)) 2025-12-04T11:21:33.6637719Z W1204 11:21:08.570000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 127, in dumps 2025-12-04T11:21:33.6637953Z W1204 11:21:08.570000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] pickler.dump(obj) 2025-12-04T11:21:33.6638432Z W1204 11:21:08.570000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 94, in reducer_override 2025-12-04T11:21:33.6638829Z W1204 11:21:08.570000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] return _GraphModulePickleData.reduce_helper(self, obj) 2025-12-04T11:21:33.6639301Z W1204 11:21:08.570000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 352, in reduce_helper 2025-12-04T11:21:33.6639567Z W1204 11:21:08.570000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] cls(obj, pickler.options), 2025-12-04T11:21:33.6640059Z W1204 11:21:08.570000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 364, in __init__ 2025-12-04T11:21:33.6640372Z W1204 11:21:08.570000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] self.graph = _GraphPickleData(gm._graph, options) 2025-12-04T11:21:33.6640829Z W1204 11:21:08.570000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 571, in __init__ 2025-12-04T11:21:33.6641141Z W1204 11:21:08.570000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] nodes[node] = _NodePickleData(node, nodes, options) 2025-12-04T11:21:33.6641601Z W1204 11:21:08.570000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 387, in __init__ 2025-12-04T11:21:33.6641920Z W1204 11:21:08.570000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] self.target = _OpPickleData.pickle(node.target, options) 2025-12-04T11:21:33.6642375Z W1204 11:21:08.570000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 435, in pickle 2025-12-04T11:21:33.6642707Z W1204 11:21:08.570000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] return cls._pickle_op(name, _OpOverloadPickleData, options) 2025-12-04T11:21:33.6643173Z W1204 11:21:08.570000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 456, in _pickle_op 2025-12-04T11:21:33.6643529Z W1204 11:21:08.570000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] raise BypassFxGraphCache(f"Unable to pickle non-standard op: {name}") 2025-12-04T11:21:33.6644069Z W1204 11:21:08.570000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] torch._inductor.codecache.BypassFxGraphCache: Unable to pickle non-standard op: torch.ops.prims.convert_element_type.default 2025-12-04T11:21:33.6644192Z PASSED [0.5644s] [ 70%] 2025-12-04T11:21:33.6644816Z inductor/test_compile_subprocess.py::GPUTests::test_unbacked_float_item_cuda <- test/inductor/test_torchinductor.py W1204 11:21:09.132000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] Unable to pickle input graph or example inputs 2025-12-04T11:21:33.6645096Z W1204 11:21:09.132000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] Traceback (most recent call last): 2025-12-04T11:21:33.6645598Z W1204 11:21:09.132000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/compile_fx_ext.py", line 486, in serialize_compile 2025-12-04T11:21:33.6645823Z W1204 11:21:09.132000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] ).serialize() 2025-12-04T11:21:33.6646302Z W1204 11:21:09.132000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/compile_fx_ext.py", line 210, in serialize 2025-12-04T11:21:33.6646637Z W1204 11:21:09.132000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] return _WireProtocolPickledInput(GraphPickler.dumps(self)) 2025-12-04T11:21:33.6647362Z W1204 11:21:09.132000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 127, in dumps 2025-12-04T11:21:33.6647598Z W1204 11:21:09.132000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] pickler.dump(obj) 2025-12-04T11:21:33.6648090Z W1204 11:21:09.132000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 94, in reducer_override 2025-12-04T11:21:33.6648448Z W1204 11:21:09.132000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] return _GraphModulePickleData.reduce_helper(self, obj) 2025-12-04T11:21:33.6648929Z W1204 11:21:09.132000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 352, in reduce_helper 2025-12-04T11:21:33.6649190Z W1204 11:21:09.132000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] cls(obj, pickler.options), 2025-12-04T11:21:33.6649642Z W1204 11:21:09.132000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 364, in __init__ 2025-12-04T11:21:33.6649947Z W1204 11:21:09.132000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] self.graph = _GraphPickleData(gm._graph, options) 2025-12-04T11:21:33.6650403Z W1204 11:21:09.132000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 571, in __init__ 2025-12-04T11:21:33.6650708Z W1204 11:21:09.132000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] nodes[node] = _NodePickleData(node, nodes, options) 2025-12-04T11:21:33.6651339Z W1204 11:21:09.132000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 387, in __init__ 2025-12-04T11:21:33.6651733Z W1204 11:21:09.132000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] self.target = _OpPickleData.pickle(node.target, options) 2025-12-04T11:21:33.6652187Z W1204 11:21:09.132000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 435, in pickle 2025-12-04T11:21:33.6652522Z W1204 11:21:09.132000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] return cls._pickle_op(name, _OpOverloadPickleData, options) 2025-12-04T11:21:33.6652992Z W1204 11:21:09.132000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 456, in _pickle_op 2025-12-04T11:21:33.6653410Z W1204 11:21:09.132000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] raise BypassFxGraphCache(f"Unable to pickle non-standard op: {name}") 2025-12-04T11:21:33.6653937Z W1204 11:21:09.132000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] torch._inductor.codecache.BypassFxGraphCache: Unable to pickle non-standard op: torch.ops.prims.convert_element_type.default 2025-12-04T11:21:33.6654007Z PASSED [0.6319s] [ 71%] 2025-12-04T11:21:33.6654378Z inductor/test_compile_subprocess.py::GPUTests::test_unbacked_floordiv_simplify_cuda <- test/inductor/test_torchinductor.py PASSED [0.7635s] [ 72%] 2025-12-04T11:21:33.6654764Z inductor/test_compile_subprocess.py::GPUTests::test_unbacked_floordiv_simplify_errors_cuda <- test/inductor/test_torchinductor.py PASSED [0.0161s] [ 73%] 2025-12-04T11:21:33.6655112Z inductor/test_compile_subprocess.py::GPUTests::test_unroll_small_reduction_cuda <- test/inductor/test_torchinductor.py PASSED [1.8797s] [ 74%] 2025-12-04T11:21:33.6655526Z inductor/test_compile_subprocess.py::GPUTests::test_unspec_inputs_float32_cuda <- test/inductor/test_torchinductor.py PASSED [0.5492s] [ 75%] 2025-12-04T11:21:33.6655861Z inductor/test_compile_subprocess.py::GPUTests::test_unspec_inputs_float64_cuda <- test/inductor/test_torchinductor.py PASSED [0.7793s] [ 76%] 2025-12-04T11:21:33.6656198Z inductor/test_compile_subprocess.py::GPUTests::test_unspec_inputs_int16_cuda <- test/inductor/test_torchinductor.py PASSED [0.4080s] [ 77%] 2025-12-04T11:21:33.6656526Z inductor/test_compile_subprocess.py::GPUTests::test_unspec_inputs_int32_cuda <- test/inductor/test_torchinductor.py PASSED [0.4074s] [ 78%] 2025-12-04T11:21:33.6656892Z inductor/test_compile_subprocess.py::GPUTests::test_unspec_inputs_int64_cuda <- test/inductor/test_torchinductor.py PASSED [0.4169s] [ 79%] 2025-12-04T11:21:33.6657231Z inductor/test_compile_subprocess.py::GPUTests::test_unspec_inputs_int8_cuda <- test/inductor/test_torchinductor.py PASSED [0.4097s] [ 80%] 2025-12-04T11:21:33.6657559Z inductor/test_compile_subprocess.py::GPUTests::test_unsqueeze_inplace_cuda <- test/inductor/test_torchinductor.py PASSED [0.3744s] [ 81%] 2025-12-04T11:21:33.6658203Z inductor/test_compile_subprocess.py::GPUTests::test_upsample_bilinear2d_a_cuda <- test/inductor/test_torchinductor.py W1204 11:21:15.610000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] Unable to pickle input graph or example inputs 2025-12-04T11:21:33.6658473Z W1204 11:21:15.610000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] Traceback (most recent call last): 2025-12-04T11:21:33.6658992Z W1204 11:21:15.610000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/compile_fx_ext.py", line 486, in serialize_compile 2025-12-04T11:21:33.6659219Z W1204 11:21:15.610000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] ).serialize() 2025-12-04T11:21:33.6659700Z W1204 11:21:15.610000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/compile_fx_ext.py", line 210, in serialize 2025-12-04T11:21:33.6660039Z W1204 11:21:15.610000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] return _WireProtocolPickledInput(GraphPickler.dumps(self)) 2025-12-04T11:21:33.6660495Z W1204 11:21:15.610000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 127, in dumps 2025-12-04T11:21:33.6660736Z W1204 11:21:15.610000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] pickler.dump(obj) 2025-12-04T11:21:33.6661216Z W1204 11:21:15.610000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 94, in reducer_override 2025-12-04T11:21:33.6661587Z W1204 11:21:15.610000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] return _GraphModulePickleData.reduce_helper(self, obj) 2025-12-04T11:21:33.6662064Z W1204 11:21:15.610000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 352, in reduce_helper 2025-12-04T11:21:33.6662321Z W1204 11:21:15.610000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] cls(obj, pickler.options), 2025-12-04T11:21:33.6665313Z W1204 11:21:15.610000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 364, in __init__ 2025-12-04T11:21:33.6665705Z W1204 11:21:15.610000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] self.graph = _GraphPickleData(gm._graph, options) 2025-12-04T11:21:33.6666272Z W1204 11:21:15.610000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 571, in __init__ 2025-12-04T11:21:33.6666635Z W1204 11:21:15.610000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] nodes[node] = _NodePickleData(node, nodes, options) 2025-12-04T11:21:33.6667104Z W1204 11:21:15.610000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 387, in __init__ 2025-12-04T11:21:33.6667442Z W1204 11:21:15.610000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] self.target = _OpPickleData.pickle(node.target, options) 2025-12-04T11:21:33.6667929Z W1204 11:21:15.610000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 435, in pickle 2025-12-04T11:21:33.6668268Z W1204 11:21:15.610000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] return cls._pickle_op(name, _OpOverloadPickleData, options) 2025-12-04T11:21:33.6668762Z W1204 11:21:15.610000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 456, in _pickle_op 2025-12-04T11:21:33.6669137Z W1204 11:21:15.610000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] raise BypassFxGraphCache(f"Unable to pickle non-standard op: {name}") 2025-12-04T11:21:33.6669636Z W1204 11:21:15.610000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] torch._inductor.codecache.BypassFxGraphCache: Unable to pickle non-standard op: torch.ops.prims.iota.default 2025-12-04T11:21:33.6669938Z W1204 11:21:16.702000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] Unable to pickle input graph or example inputs 2025-12-04T11:21:33.6670209Z W1204 11:21:16.702000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] Traceback (most recent call last): 2025-12-04T11:21:33.6670737Z W1204 11:21:16.702000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/compile_fx_ext.py", line 486, in serialize_compile 2025-12-04T11:21:33.6670966Z W1204 11:21:16.702000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] ).serialize() 2025-12-04T11:21:33.6671452Z W1204 11:21:16.702000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/compile_fx_ext.py", line 210, in serialize 2025-12-04T11:21:33.6671791Z W1204 11:21:16.702000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] return _WireProtocolPickledInput(GraphPickler.dumps(self)) 2025-12-04T11:21:33.6672247Z W1204 11:21:16.702000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 127, in dumps 2025-12-04T11:21:33.6672532Z W1204 11:21:16.702000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] pickler.dump(obj) 2025-12-04T11:21:33.6673010Z W1204 11:21:16.702000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 94, in reducer_override 2025-12-04T11:21:33.6673336Z W1204 11:21:16.702000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] return _GraphModulePickleData.reduce_helper(self, obj) 2025-12-04T11:21:33.6673876Z W1204 11:21:16.702000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 352, in reduce_helper 2025-12-04T11:21:33.6674148Z W1204 11:21:16.702000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] cls(obj, pickler.options), 2025-12-04T11:21:33.6674716Z W1204 11:21:16.702000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 364, in __init__ 2025-12-04T11:21:33.6675028Z W1204 11:21:16.702000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] self.graph = _GraphPickleData(gm._graph, options) 2025-12-04T11:21:33.6675491Z W1204 11:21:16.702000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 571, in __init__ 2025-12-04T11:21:33.6675793Z W1204 11:21:16.702000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] nodes[node] = _NodePickleData(node, nodes, options) 2025-12-04T11:21:33.6676256Z W1204 11:21:16.702000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 387, in __init__ 2025-12-04T11:21:33.6676580Z W1204 11:21:16.702000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] self.target = _OpPickleData.pickle(node.target, options) 2025-12-04T11:21:33.6677035Z W1204 11:21:16.702000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 435, in pickle 2025-12-04T11:21:33.6677366Z W1204 11:21:16.702000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] return cls._pickle_op(name, _OpOverloadPickleData, options) 2025-12-04T11:21:33.6678485Z W1204 11:21:16.702000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 456, in _pickle_op 2025-12-04T11:21:33.6678890Z W1204 11:21:16.702000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] raise BypassFxGraphCache(f"Unable to pickle non-standard op: {name}") 2025-12-04T11:21:33.6679436Z W1204 11:21:16.702000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] torch._inductor.codecache.BypassFxGraphCache: Unable to pickle non-standard op: torch.ops.prims.convert_element_type.default 2025-12-04T11:21:33.6679511Z PASSED [2.2207s] [ 82%] 2025-12-04T11:21:33.6679969Z inductor/test_compile_subprocess.py::GPUTests::test_upsample_cat_conv_cuda <- test/inductor/test_torchinductor.py SKIPPED [0.0026s] (only support cpu upsample_cat_conv test) [ 83%] 2025-12-04T11:21:33.6680596Z inductor/test_compile_subprocess.py::GPUTests::test_upsample_nearest1d_cuda <- test/inductor/test_torchinductor.py W1204 11:21:17.741000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] Unable to pickle input graph or example inputs 2025-12-04T11:21:33.6680870Z W1204 11:21:17.741000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] Traceback (most recent call last): 2025-12-04T11:21:33.6681480Z W1204 11:21:17.741000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/compile_fx_ext.py", line 486, in serialize_compile 2025-12-04T11:21:33.6681718Z W1204 11:21:17.741000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] ).serialize() 2025-12-04T11:21:33.6682206Z W1204 11:21:17.741000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/compile_fx_ext.py", line 210, in serialize 2025-12-04T11:21:33.6682549Z W1204 11:21:17.741000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] return _WireProtocolPickledInput(GraphPickler.dumps(self)) 2025-12-04T11:21:33.6683085Z W1204 11:21:17.741000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 127, in dumps 2025-12-04T11:21:33.6683339Z W1204 11:21:17.741000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] pickler.dump(obj) 2025-12-04T11:21:33.6683928Z W1204 11:21:17.741000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 94, in reducer_override 2025-12-04T11:21:33.6684254Z W1204 11:21:17.741000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] return _GraphModulePickleData.reduce_helper(self, obj) 2025-12-04T11:21:33.6684734Z W1204 11:21:17.741000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 352, in reduce_helper 2025-12-04T11:21:33.6685002Z W1204 11:21:17.741000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] cls(obj, pickler.options), 2025-12-04T11:21:33.6685465Z W1204 11:21:17.741000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 364, in __init__ 2025-12-04T11:21:33.6685774Z W1204 11:21:17.741000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] self.graph = _GraphPickleData(gm._graph, options) 2025-12-04T11:21:33.6686236Z W1204 11:21:17.741000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 571, in __init__ 2025-12-04T11:21:33.6686540Z W1204 11:21:17.741000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] nodes[node] = _NodePickleData(node, nodes, options) 2025-12-04T11:21:33.6686998Z W1204 11:21:17.741000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 387, in __init__ 2025-12-04T11:21:33.6687325Z W1204 11:21:17.741000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] self.target = _OpPickleData.pickle(node.target, options) 2025-12-04T11:21:33.6687782Z W1204 11:21:17.741000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 435, in pickle 2025-12-04T11:21:33.6688123Z W1204 11:21:17.741000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] return cls._pickle_op(name, _OpOverloadPickleData, options) 2025-12-04T11:21:33.6688587Z W1204 11:21:17.741000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 456, in _pickle_op 2025-12-04T11:21:33.6688958Z W1204 11:21:17.741000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] raise BypassFxGraphCache(f"Unable to pickle non-standard op: {name}") 2025-12-04T11:21:33.6689439Z W1204 11:21:17.741000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] torch._inductor.codecache.BypassFxGraphCache: Unable to pickle non-standard op: torch.ops.prims.iota.default 2025-12-04T11:21:33.6689778Z W1204 11:21:18.465000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] Unable to pickle input graph or example inputs 2025-12-04T11:21:33.6690052Z W1204 11:21:18.465000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] Traceback (most recent call last): 2025-12-04T11:21:33.6690560Z W1204 11:21:18.465000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/compile_fx_ext.py", line 486, in serialize_compile 2025-12-04T11:21:33.6690790Z W1204 11:21:18.465000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] ).serialize() 2025-12-04T11:21:33.6691311Z W1204 11:21:18.465000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/compile_fx_ext.py", line 210, in serialize 2025-12-04T11:21:33.6691646Z W1204 11:21:18.465000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] return _WireProtocolPickledInput(GraphPickler.dumps(self)) 2025-12-04T11:21:33.6692179Z W1204 11:21:18.465000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 127, in dumps 2025-12-04T11:21:33.6692418Z W1204 11:21:18.465000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] pickler.dump(obj) 2025-12-04T11:21:33.6692903Z W1204 11:21:18.465000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 94, in reducer_override 2025-12-04T11:21:33.6693223Z W1204 11:21:18.465000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] return _GraphModulePickleData.reduce_helper(self, obj) 2025-12-04T11:21:33.6693696Z W1204 11:21:18.465000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 352, in reduce_helper 2025-12-04T11:21:33.6693966Z W1204 11:21:18.465000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] cls(obj, pickler.options), 2025-12-04T11:21:33.6694418Z W1204 11:21:18.465000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 364, in __init__ 2025-12-04T11:21:33.6694725Z W1204 11:21:18.465000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] self.graph = _GraphPickleData(gm._graph, options) 2025-12-04T11:21:33.6695182Z W1204 11:21:18.465000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 571, in __init__ 2025-12-04T11:21:33.6695493Z W1204 11:21:18.465000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] nodes[node] = _NodePickleData(node, nodes, options) 2025-12-04T11:21:33.6695955Z W1204 11:21:18.465000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 387, in __init__ 2025-12-04T11:21:33.6696282Z W1204 11:21:18.465000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] self.target = _OpPickleData.pickle(node.target, options) 2025-12-04T11:21:33.6696736Z W1204 11:21:18.465000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 435, in pickle 2025-12-04T11:21:33.6697069Z W1204 11:21:18.465000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] return cls._pickle_op(name, _OpOverloadPickleData, options) 2025-12-04T11:21:33.6697541Z W1204 11:21:18.465000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 456, in _pickle_op 2025-12-04T11:21:33.6703579Z W1204 11:21:18.465000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] raise BypassFxGraphCache(f"Unable to pickle non-standard op: {name}") 2025-12-04T11:21:33.6704161Z W1204 11:21:18.465000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] torch._inductor.codecache.BypassFxGraphCache: Unable to pickle non-standard op: torch.ops.prims.convert_element_type.default 2025-12-04T11:21:33.6704240Z PASSED [1.4352s] [ 84%] 2025-12-04T11:21:33.6704594Z inductor/test_compile_subprocess.py::GPUTests::test_var_correction_cuda <- test/inductor/test_torchinductor.py PASSED [1.0130s] [ 85%] 2025-12-04T11:21:33.6705046Z inductor/test_compile_subprocess.py::GPUTests::test_var_mean_tile_reduction_False_cuda <- test/inductor/test_torchinductor.py PASSED [0.7551s] [ 86%] 2025-12-04T11:21:33.6705368Z inductor/test_compile_subprocess.py::GPUTests::test_vdd_clamp_cuda <- test/inductor/test_torchinductor.py PASSED [0.3489s] [ 87%] 2025-12-04T11:21:33.6705802Z inductor/test_compile_subprocess.py::GPUTests::test_view_as_complex_cuda <- test/inductor/test_torchinductor.py PASSED [0.2128s] [ 88%] 2025-12-04T11:21:33.6706120Z inductor/test_compile_subprocess.py::GPUTests::test_view_as_real_cuda <- test/inductor/test_torchinductor.py PASSED [0.1986s] [ 89%] 2025-12-04T11:21:33.6706430Z inductor/test_compile_subprocess.py::GPUTests::test_view_detach_cuda <- test/inductor/test_torchinductor.py PASSED [0.1893s] [ 90%] 2025-12-04T11:21:33.6706728Z inductor/test_compile_subprocess.py::GPUTests::test_views2_cuda <- test/inductor/test_torchinductor.py PASSED [2.2478s] [ 91%] 2025-12-04T11:21:33.6707031Z inductor/test_compile_subprocess.py::GPUTests::test_views4_cuda <- test/inductor/test_torchinductor.py PASSED [1.6900s] [ 92%] 2025-12-04T11:21:33.6707324Z inductor/test_compile_subprocess.py::GPUTests::test_views5_cuda <- test/inductor/test_torchinductor.py PASSED [0.2042s] [ 93%] 2025-12-04T11:21:33.6707634Z inductor/test_compile_subprocess.py::GPUTests::test_views6_cuda <- test/inductor/test_torchinductor.py PASSED [0.3537s] [ 94%] 2025-12-04T11:21:33.6708280Z inductor/test_compile_subprocess.py::GPUTests::test_weight_norm_bwd_cuda <- test/inductor/test_torchinductor.py W1204 11:21:26.546000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0_1] Unable to pickle input graph or example inputs 2025-12-04T11:21:33.6708554Z W1204 11:21:26.546000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0_1] Traceback (most recent call last): 2025-12-04T11:21:33.6709085Z W1204 11:21:26.546000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0_1] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/compile_fx_ext.py", line 486, in serialize_compile 2025-12-04T11:21:33.6709322Z W1204 11:21:26.546000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0_1] ).serialize() 2025-12-04T11:21:33.6709810Z W1204 11:21:26.546000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0_1] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/compile_fx_ext.py", line 210, in serialize 2025-12-04T11:21:33.6710156Z W1204 11:21:26.546000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0_1] return _WireProtocolPickledInput(GraphPickler.dumps(self)) 2025-12-04T11:21:33.6710642Z W1204 11:21:26.546000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0_1] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 127, in dumps 2025-12-04T11:21:33.6710898Z W1204 11:21:26.546000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0_1] pickler.dump(obj) 2025-12-04T11:21:33.6711390Z W1204 11:21:26.546000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0_1] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 94, in reducer_override 2025-12-04T11:21:33.6711783Z W1204 11:21:26.546000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0_1] return _GraphModulePickleData.reduce_helper(self, obj) 2025-12-04T11:21:33.6712265Z W1204 11:21:26.546000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0_1] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 352, in reduce_helper 2025-12-04T11:21:33.6712545Z W1204 11:21:26.546000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0_1] cls(obj, pickler.options), 2025-12-04T11:21:33.6713010Z W1204 11:21:26.546000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0_1] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 364, in __init__ 2025-12-04T11:21:33.6713376Z W1204 11:21:26.546000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0_1] self.graph = _GraphPickleData(gm._graph, options) 2025-12-04T11:21:33.6713842Z W1204 11:21:26.546000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0_1] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 571, in __init__ 2025-12-04T11:21:33.6714227Z W1204 11:21:26.546000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0_1] nodes[node] = _NodePickleData(node, nodes, options) 2025-12-04T11:21:33.6714704Z W1204 11:21:26.546000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0_1] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 387, in __init__ 2025-12-04T11:21:33.6715030Z W1204 11:21:26.546000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0_1] self.target = _OpPickleData.pickle(node.target, options) 2025-12-04T11:21:33.6715493Z W1204 11:21:26.546000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0_1] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 435, in pickle 2025-12-04T11:21:33.6715829Z W1204 11:21:26.546000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0_1] return cls._pickle_op(name, _OpOverloadPickleData, options) 2025-12-04T11:21:33.6716307Z W1204 11:21:26.546000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0_1] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 456, in _pickle_op 2025-12-04T11:21:33.6716674Z W1204 11:21:26.546000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0_1] raise BypassFxGraphCache(f"Unable to pickle non-standard op: {name}") 2025-12-04T11:21:33.6717201Z W1204 11:21:26.546000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0_1] torch._inductor.codecache.BypassFxGraphCache: Unable to pickle non-standard op: torch.ops.prims.convert_element_type.default 2025-12-04T11:21:33.6717510Z W1204 11:21:26.828000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0_1] Unable to pickle input graph or example inputs 2025-12-04T11:21:33.6717793Z W1204 11:21:26.828000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0_1] Traceback (most recent call last): 2025-12-04T11:21:33.6718316Z W1204 11:21:26.828000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0_1] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/compile_fx_ext.py", line 486, in serialize_compile 2025-12-04T11:21:33.6718545Z W1204 11:21:26.828000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0_1] ).serialize() 2025-12-04T11:21:33.6719033Z W1204 11:21:26.828000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0_1] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/compile_fx_ext.py", line 210, in serialize 2025-12-04T11:21:33.6719368Z W1204 11:21:26.828000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0_1] return _WireProtocolPickledInput(GraphPickler.dumps(self)) 2025-12-04T11:21:33.6719829Z W1204 11:21:26.828000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0_1] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 127, in dumps 2025-12-04T11:21:33.6720122Z W1204 11:21:26.828000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0_1] pickler.dump(obj) 2025-12-04T11:21:33.6720603Z W1204 11:21:26.828000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0_1] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 94, in reducer_override 2025-12-04T11:21:33.6720928Z W1204 11:21:26.828000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0_1] return _GraphModulePickleData.reduce_helper(self, obj) 2025-12-04T11:21:33.6721439Z W1204 11:21:26.828000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0_1] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 352, in reduce_helper 2025-12-04T11:21:33.6721703Z W1204 11:21:26.828000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0_1] cls(obj, pickler.options), 2025-12-04T11:21:33.6722164Z W1204 11:21:26.828000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0_1] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 364, in __init__ 2025-12-04T11:21:33.6722634Z W1204 11:21:26.828000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0_1] self.graph = _GraphPickleData(gm._graph, options) 2025-12-04T11:21:33.6723095Z W1204 11:21:26.828000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0_1] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 571, in __init__ 2025-12-04T11:21:33.6723402Z W1204 11:21:26.828000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0_1] nodes[node] = _NodePickleData(node, nodes, options) 2025-12-04T11:21:33.6723872Z W1204 11:21:26.828000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0_1] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 387, in __init__ 2025-12-04T11:21:33.6724199Z W1204 11:21:26.828000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0_1] self.target = _OpPickleData.pickle(node.target, options) 2025-12-04T11:21:33.6724664Z W1204 11:21:26.828000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0_1] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 435, in pickle 2025-12-04T11:21:33.6724996Z W1204 11:21:26.828000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0_1] return cls._pickle_op(name, _OpOverloadPickleData, options) 2025-12-04T11:21:33.6725461Z W1204 11:21:26.828000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0_1] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 456, in _pickle_op 2025-12-04T11:21:33.6725827Z W1204 11:21:26.828000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0_1] raise BypassFxGraphCache(f"Unable to pickle non-standard op: {name}") 2025-12-04T11:21:33.6726359Z W1204 11:21:26.828000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0_1] torch._inductor.codecache.BypassFxGraphCache: Unable to pickle non-standard op: torch.ops.prims.convert_element_type.default 2025-12-04T11:21:33.6726431Z PASSED [0.8519s] [ 95%] 2025-12-04T11:21:33.6726803Z inductor/test_compile_subprocess.py::GPUTests::test_weight_norm_conv2d_cuda <- test/inductor/test_torchinductor.py SKIPPED [0.0002s] (Skipped!) [ 96%] 2025-12-04T11:21:33.6727409Z inductor/test_compile_subprocess.py::GPUTests::test_where_broadcast_cuda <- test/inductor/test_torchinductor.py W1204 11:21:27.739000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] Unable to pickle input graph or example inputs 2025-12-04T11:21:33.6727671Z W1204 11:21:27.739000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] Traceback (most recent call last): 2025-12-04T11:21:33.6728185Z W1204 11:21:27.739000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/compile_fx_ext.py", line 486, in serialize_compile 2025-12-04T11:21:33.6728441Z W1204 11:21:27.739000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] ).serialize() 2025-12-04T11:21:33.6728919Z W1204 11:21:27.739000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/compile_fx_ext.py", line 210, in serialize 2025-12-04T11:21:33.6729246Z W1204 11:21:27.739000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] return _WireProtocolPickledInput(GraphPickler.dumps(self)) 2025-12-04T11:21:33.6729736Z W1204 11:21:27.739000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 127, in dumps 2025-12-04T11:21:33.6729968Z W1204 11:21:27.739000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] pickler.dump(obj) 2025-12-04T11:21:33.6730438Z W1204 11:21:27.739000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 94, in reducer_override 2025-12-04T11:21:33.6730832Z W1204 11:21:27.739000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] return _GraphModulePickleData.reduce_helper(self, obj) 2025-12-04T11:21:33.6731298Z W1204 11:21:27.739000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 352, in reduce_helper 2025-12-04T11:21:33.6731549Z W1204 11:21:27.739000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] cls(obj, pickler.options), 2025-12-04T11:21:33.6732003Z W1204 11:21:27.739000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 364, in __init__ 2025-12-04T11:21:33.6732304Z W1204 11:21:27.739000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] self.graph = _GraphPickleData(gm._graph, options) 2025-12-04T11:21:33.6732764Z W1204 11:21:27.739000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 571, in __init__ 2025-12-04T11:21:33.6733062Z W1204 11:21:27.739000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] nodes[node] = _NodePickleData(node, nodes, options) 2025-12-04T11:21:33.6733511Z W1204 11:21:27.739000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 387, in __init__ 2025-12-04T11:21:33.6733826Z W1204 11:21:27.739000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] self.target = _OpPickleData.pickle(node.target, options) 2025-12-04T11:21:33.6734272Z W1204 11:21:27.739000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 435, in pickle 2025-12-04T11:21:33.6734602Z W1204 11:21:27.739000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] return cls._pickle_op(name, _OpOverloadPickleData, options) 2025-12-04T11:21:33.6735060Z W1204 11:21:27.739000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 456, in _pickle_op 2025-12-04T11:21:33.6735422Z W1204 11:21:27.739000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] raise BypassFxGraphCache(f"Unable to pickle non-standard op: {name}") 2025-12-04T11:21:33.6735941Z W1204 11:21:27.739000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] torch._inductor.codecache.BypassFxGraphCache: Unable to pickle non-standard op: torch.ops.prims.convert_element_type.default 2025-12-04T11:21:33.6736012Z PASSED [1.1460s] [ 97%] 2025-12-04T11:21:33.6736395Z inductor/test_compile_subprocess.py::GPUTests::test_where_with_logical_op_cuda <- test/inductor/test_torchinductor.py PASSED [0.6817s] [ 98%] 2025-12-04T11:21:33.6736732Z inductor/test_compile_subprocess.py::GPUTests::test_zero_dim_reductions_cuda <- test/inductor/test_torchinductor.py PASSED [0.2887s] [ 99%] 2025-12-04T11:21:33.6737366Z inductor/test_compile_subprocess.py::GPUTests::test_zero_element_mutation_cuda <- test/inductor/test_torchinductor.py W1204 11:21:29.322000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] Unable to pickle input graph or example inputs 2025-12-04T11:21:33.6737635Z W1204 11:21:29.322000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] Traceback (most recent call last): 2025-12-04T11:21:33.6738203Z W1204 11:21:29.322000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/compile_fx_ext.py", line 486, in serialize_compile 2025-12-04T11:21:33.6738430Z W1204 11:21:29.322000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] ).serialize() 2025-12-04T11:21:33.6738983Z W1204 11:21:29.322000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/compile_fx_ext.py", line 210, in serialize 2025-12-04T11:21:33.6739326Z W1204 11:21:29.322000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] return _WireProtocolPickledInput(GraphPickler.dumps(self)) 2025-12-04T11:21:33.6739772Z W1204 11:21:29.322000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 127, in dumps 2025-12-04T11:21:33.6740010Z W1204 11:21:29.322000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] pickler.dump(obj) 2025-12-04T11:21:33.6740468Z W1204 11:21:29.322000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] AttributeError: Can't pickle local object 'CommonTemplate.test_zero_element_mutation..CustomModel' 2025-12-04T11:21:33.6740771Z W1204 11:21:29.417000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] Unable to pickle input graph or example inputs 2025-12-04T11:21:33.6741034Z W1204 11:21:29.417000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] Traceback (most recent call last): 2025-12-04T11:21:33.6741538Z W1204 11:21:29.417000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/compile_fx_ext.py", line 486, in serialize_compile 2025-12-04T11:21:33.6741763Z W1204 11:21:29.417000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] ).serialize() 2025-12-04T11:21:33.6742238Z W1204 11:21:29.417000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/compile_fx_ext.py", line 210, in serialize 2025-12-04T11:21:33.6742575Z W1204 11:21:29.417000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] return _WireProtocolPickledInput(GraphPickler.dumps(self)) 2025-12-04T11:21:33.6743023Z W1204 11:21:29.417000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 127, in dumps 2025-12-04T11:21:33.6743272Z W1204 11:21:29.417000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] pickler.dump(obj) 2025-12-04T11:21:33.6743804Z W1204 11:21:29.417000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 94, in reducer_override 2025-12-04T11:21:33.6744126Z W1204 11:21:29.417000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] return _GraphModulePickleData.reduce_helper(self, obj) 2025-12-04T11:21:33.6744596Z W1204 11:21:29.417000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 352, in reduce_helper 2025-12-04T11:21:33.6744911Z W1204 11:21:29.417000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] cls(obj, pickler.options), 2025-12-04T11:21:33.6745371Z W1204 11:21:29.417000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 364, in __init__ 2025-12-04T11:21:33.6745681Z W1204 11:21:29.417000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] self.graph = _GraphPickleData(gm._graph, options) 2025-12-04T11:21:33.6746196Z W1204 11:21:29.417000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 571, in __init__ 2025-12-04T11:21:33.6746500Z W1204 11:21:29.417000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] nodes[node] = _NodePickleData(node, nodes, options) 2025-12-04T11:21:33.6747026Z W1204 11:21:29.417000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 387, in __init__ 2025-12-04T11:21:33.6747350Z W1204 11:21:29.417000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] self.target = _OpPickleData.pickle(node.target, options) 2025-12-04T11:21:33.6747800Z W1204 11:21:29.417000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 435, in pickle 2025-12-04T11:21:33.6748142Z W1204 11:21:29.417000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] return cls._pickle_op(name, _OpOverloadPickleData, options) 2025-12-04T11:21:33.6748606Z W1204 11:21:29.417000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_graph_pickler.py", line 456, in _pickle_op 2025-12-04T11:21:33.6748971Z W1204 11:21:29.417000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] raise BypassFxGraphCache(f"Unable to pickle non-standard op: {name}") 2025-12-04T11:21:33.6749500Z W1204 11:21:29.417000 100145 site-packages/torch/_inductor/compile_fx_ext.py:493] [0/0] torch._inductor.codecache.BypassFxGraphCache: Unable to pickle non-standard op: torch.ops.prims.convert_element_type.default 2025-12-04T11:21:33.6749566Z PASSED [0.1940s] [100%] 2025-12-04T11:21:33.6749571Z 2025-12-04T11:21:33.6750059Z - generated xml file: /var/lib/jenkins/workspace/test/test-reports/python-pytest/inductor.test_compile_subprocess/inductor.test_compile_subprocess-825047c883c2cfeb.xml - 2025-12-04T11:21:33.6750204Z ========== 94 passed, 7 skipped, 332 deselected in 124.10s (0:02:04) =========== 2025-12-04T11:21:33.6750594Z The following tests failed and then succeeded when run in a new process['test/inductor/test_compile_subprocess.py::GPUTests::test_remove_noop_slice_cuda'] 2025-12-04T11:21:33.6750601Z 2025-12-04T11:21:33.6750969Z FINISHED PRINTING LOG FILE of inductor/test_compile_subprocess 1/2 (test/test-reports/inductor.test_compile_subprocess_1.2_abe6ea6a5cdf0a6c_.log) 2025-12-04T11:21:33.6750973Z 2025-12-04T11:21:33.6751203Z Finished inductor/test_compile_subprocess 1/2 ... [2025-12-04 11:21:33.431054][6141.373268019], took 7.62min 2025-12-04T11:21:33.6751712Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/inductor.test_compile_subprocess/inductor.test_compile_subprocess-576cab637f4fea56.xml 2025-12-04T11:21:33.6752248Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/inductor.test_compile_subprocess/inductor.test_compile_subprocess-c5e45fed58b5f3dc.xml 2025-12-04T11:21:33.6752745Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/inductor.test_compile_subprocess/inductor.test_compile_subprocess-825047c883c2cfeb.xml 2025-12-04T11:21:33.6752984Z Running inductor/test_deterministic 2/3 ... [2025-12-04 11:21:33.619283][6141.561499253] 2025-12-04T11:21:33.6753070Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2025-12-04T11:21:33.6753672Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'inductor/test_deterministic.py', '--shard-id=2', '--num-shards=3', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-12-04 11:21:33.619582] 2025-12-04T11:22:29.3251544Z 2025-12-04T11:22:29.3252692Z inductor/test_deterministic 2/3 was successful, full logs can be found in artifacts with path test/test-reports/inductor.test_deterministic_2.3_06e251a75554d673_.log 2025-12-04T11:22:29.3259228Z Running 7 items in this shard: test/inductor/test_deterministic.py::DeterministicTest::test_max_autotune_deterministic_False, test/inductor/test_deterministic.py::DeterministicTest::test_run2run_determinism_model_name_BertForMaskedLM_training_or_inference_training_precision_float16, test/inductor/test_deterministic.py::DeterministicTest::test_run2run_determinism_model_name_DistillGPT2_training_or_inference_training_precision_amp, test/inductor/test_deterministic.py::DeterministicTest::test_run2run_determinism_model_name_DistillGPT2_training_or_inference_training_precision_float16, test/inductor/test_deterministic.py::DeterministicTest::test_run2run_determinism_model_name_GoogleFnet_training_or_inference_inference_precision_amp, test/inductor/test_deterministic.py::DeterministicTest::test_run2run_determinism_model_name_GoogleFnet_training_or_inference_training_precision_float16, test/inductor/test_deterministic.py::DeterministicTest::test_use_deterministic_algorithsm 2025-12-04T11:22:29.3264773Z 2025-12-04T11:22:29.3265200Z Finished inductor/test_deterministic 2/3 ... [2025-12-04 11:22:29.324826][6197.267043247], took 0.93min 2025-12-04T11:22:29.3504358Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/inductor.test_deterministic/inductor.test_deterministic-3de4e0d14d6571c6.xml 2025-12-04T11:22:29.4346788Z Running dynamo/test_fx_graph_runnable 1/1 ... [2025-12-04 11:22:29.434407][6197.376624811] 2025-12-04T11:22:29.4347462Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2025-12-04T11:22:29.4350364Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'dynamo/test_fx_graph_runnable.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-12-04 11:22:29.434744] 2025-12-04T11:23:58.1459345Z 2025-12-04T11:23:58.1462547Z dynamo/test_fx_graph_runnable 1/1 was successful, full logs can be found in artifacts with path test/test-reports/dynamo.test_fx_graph_runnable_1.1_ccfc333edc67c9c4_.log 2025-12-04T11:23:58.1467813Z Running 17 items in this shard: test/dynamo/test_fx_graph_runnable.py::FxGraphRunnableTest::test_all_gather_collective, test/dynamo/test_fx_graph_runnable.py::FxGraphRunnableTest::test_all_reduce_collective, test/dynamo/test_fx_graph_runnable.py::FxGraphRunnableTest::test_basic_tensor_add, test/dynamo/test_fx_graph_runnable.py::FxGraphRunnableTest::test_broadcast_add_dynamic, test/dynamo/test_fx_graph_runnable.py::FxGraphRunnableTest::test_broadcast_collective, test/dynamo/test_fx_graph_runnable.py::FxGraphRunnableTest::test_dtensor_compile_redistribute, test/dynamo/test_fx_graph_runnable.py::FxGraphRunnableTest::test_dynamic_expression, test/dynamo/test_fx_graph_runnable.py::FxGraphRunnableTest::test_dynamic_shapes_run, test/dynamo/test_fx_graph_runnable.py::FxGraphRunnableTest::test_metrics_context, test/dynamo/test_fx_graph_runnable.py::FxGraphRunnableTest::test_reduce_scatter_collective, test/dynamo/test_fx_graph_runnable.py::FxGraphRunnableTest::test_scalar_multiply, test/dynamo/test_fx_graph_runnable.py::FxGraphRunnableTest::test_toy_model_basic, test/dynamo/test_fx_graph_runnable.py::FxGraphRunnableTest::test_toy_model_batch_processing, test/dynamo/test_fx_graph_runnable.py::FxGraphRunnableTest::test_toy_model_dynamic_batch, test/dynamo/test_fx_graph_runnable.py::FxGraphRunnableTest::test_two_inputs_matmul, test/dynamo/test_fx_graph_runnable.py::FxGraphRunnableTest::test_user_defined_triton_kernel, test/dynamo/test_fx_graph_runnable.py::FxGraphRunnableTest::test_user_defined_triton_kernel_autotune 2025-12-04T11:23:58.1472487Z 2025-12-04T11:23:58.1472722Z Finished dynamo/test_fx_graph_runnable 1/1 ... [2025-12-04 11:23:58.145619][6286.087835948], took 1.48min 2025-12-04T11:23:58.1704216Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/dynamo.test_fx_graph_runnable/dynamo.test_fx_graph_runnable-4db38b1d3487bc19.xml 2025-12-04T11:23:58.2431130Z Running inductor/test_scatter_optimization 1/1 ... [2025-12-04 11:23:58.242866][6286.185084732] 2025-12-04T11:23:58.2431829Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2025-12-04T11:23:58.2434432Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'inductor/test_scatter_optimization.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-12-04 11:23:58.243175] 2025-12-04T11:24:11.8930845Z 2025-12-04T11:24:11.8934483Z inductor/test_scatter_optimization 1/1 was successful, full logs can be found in artifacts with path test/test-reports/inductor.test_scatter_optimization_1.1_74b917c016dc1507_.log 2025-12-04T11:24:11.8937730Z Running 8 items in this shard: test/inductor/test_scatter_optimization.py::TestScatterOpt::test_3d_tensor, test/inductor/test_scatter_optimization.py::TestScatterOpt::test_can_not_optimize_due_to_dense, test/inductor/test_scatter_optimization.py::TestScatterOpt::test_can_not_optimize_due_to_non_const, test/inductor/test_scatter_optimization.py::TestScatterOpt::test_cross_entropy_loss, test/inductor/test_scatter_optimization.py::TestScatterOpt::test_neg_scatter_dim, test/inductor/test_scatter_optimization.py::TestScatterOpt::test_non_last_dim, test/inductor/test_scatter_optimization.py::TestScatterOpt::test_nonzero_const_tensor, test/inductor/test_scatter_optimization.py::TestScatterOpt::test_shorter_index_tensor 2025-12-04T11:24:11.8940268Z 2025-12-04T11:24:11.8940529Z Finished inductor/test_scatter_optimization 1/1 ... [2025-12-04 11:24:11.892688][6299.83489987], took 0.23min 2025-12-04T11:24:11.9173567Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/inductor.test_scatter_optimization/inductor.test_scatter_optimization-7db92b732e333421.xml 2025-12-04T11:24:11.9855256Z Running inductor/test_padding 1/1 ... [2025-12-04 11:24:11.985137][6299.927350303] 2025-12-04T11:24:11.9855693Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2025-12-04T11:24:11.9858537Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'inductor/test_padding.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-12-04 11:24:11.985589] 2025-12-04T11:24:47.6615307Z 2025-12-04T11:24:47.6616461Z inductor/test_padding 1/1 was successful, full logs can be found in artifacts with path test/test-reports/inductor.test_padding_1.1_52cec87b51539027_.log 2025-12-04T11:24:47.6634782Z Running 55 items in this shard: test/inductor/test_padding.py::PerfTestBetweenGoodAndBadShape::test_BertForMaskedLM, test/inductor/test_padding.py::PerfTestBetweenGoodAndBadShape::test_LinearAndSoftmax_both_shapes, test/inductor/test_padding.py::PerfTestBetweenGoodAndBadShape::test_nobias_LinearAndSoftmax_both_shapes, test/inductor/test_padding.py::PerfTestWithAndWithoutPadding::test_longformer, test/inductor/test_padding.py::PerfTestWithAndWithoutPadding::test_longformer_small_bs, test/inductor/test_padding.py::PerfTestWithAndWithoutPadding::test_nvidia_deeprecommender, test/inductor/test_padding.py::PaddingTest::test_LinearAndSoftmax_codegen, test/inductor/test_padding.py::PaddingTest::test_attention, test/inductor/test_padding.py::PaddingTest::test_cat, test/inductor/test_padding.py::PaddingTest::test_conv, test/inductor/test_padding.py::PaddingTest::test_dynamic_shape_padding_shape0_alignment_bytes_32_enable_pad_False, test/inductor/test_padding.py::PaddingTest::test_dynamic_shape_padding_shape1_alignment_bytes_32_enable_pad_True, test/inductor/test_padding.py::PaddingTest::test_dynamic_shape_padding_shape2_alignment_bytes_64_enable_pad_False, test/inductor/test_padding.py::PaddingTest::test_dynamic_shape_padding_shape3_alignment_bytes_64_enable_pad_True, test/inductor/test_padding.py::PaddingTest::test_dynamic_shape_padding_shape4_alignment_bytes_32_enable_pad_False, test/inductor/test_padding.py::PaddingTest::test_dynamic_shape_padding_shape5_alignment_bytes_32_enable_pad_True, test/inductor/test_padding.py::PaddingTest::test_dynamic_shape_padding_shape6_alignment_bytes_64_enable_pad_False, test/inductor/test_padding.py::PaddingTest::test_dynamic_shape_padding_shape7_alignment_bytes_64_enable_pad_True, test/inductor/test_padding.py::PaddingTest::test_matmul, test/inductor/test_padding.py::PaddingTest::test_mm_padding_perf, test/inductor/test_padding.py::PaddingTest::test_nobias_LinearAndSoftmax_codegen, test/inductor/test_padding.py::PaddingTest::test_noop_concat_output_padding_shape0_alignment_bytes_32_pad_output_False, test/inductor/test_padding.py::PaddingTest::test_noop_concat_output_padding_shape1_alignment_bytes_32_pad_output_True, test/inductor/test_padding.py::PaddingTest::test_noop_concat_output_padding_shape2_alignment_bytes_64_pad_output_False, test/inductor/test_padding.py::PaddingTest::test_noop_concat_output_padding_shape3_alignment_bytes_64_pad_output_True, test/inductor/test_padding.py::PaddingTest::test_outer_dynamic_shape_padding_shape0_alignment_bytes_32_enable_pad_False, test/inductor/test_padding.py::PaddingTest::test_outer_dynamic_shape_padding_shape1_alignment_bytes_32_enable_pad_True, test/inductor/test_padding.py::PaddingTest::test_outer_dynamic_shape_padding_shape2_alignment_bytes_64_enable_pad_False, test/inductor/test_padding.py::PaddingTest::test_outer_dynamic_shape_padding_shape3_alignment_bytes_64_enable_pad_True, test/inductor/test_padding.py::PaddingTest::test_outer_dynamic_shape_padding_shape4_alignment_bytes_32_enable_pad_False, test/inductor/test_padding.py::PaddingTest::test_outer_dynamic_shape_padding_shape5_alignment_bytes_32_enable_pad_True, test/inductor/test_padding.py::PaddingTest::test_outer_dynamic_shape_padding_shape6_alignment_bytes_64_enable_pad_False, test/inductor/test_padding.py::PaddingTest::test_outer_dynamic_shape_padding_shape7_alignment_bytes_64_enable_pad_True, test/inductor/test_padding.py::PaddingTest::test_pad_3d_tensor, test/inductor/test_padding.py::PaddingTest::test_pad_channels_last, test/inductor/test_padding.py::PaddingTest::test_pad_outputs_alignment_bytes_128_shape0_float16, test/inductor/test_padding.py::PaddingTest::test_pad_outputs_alignment_bytes_128_shape0_float32, test/inductor/test_padding.py::PaddingTest::test_pad_outputs_alignment_bytes_128_shape1_float16, test/inductor/test_padding.py::PaddingTest::test_pad_outputs_alignment_bytes_128_shape1_float32, test/inductor/test_padding.py::PaddingTest::test_pad_outputs_alignment_bytes_32_shape0_float16, test/inductor/test_padding.py::PaddingTest::test_pad_outputs_alignment_bytes_32_shape0_float32, test/inductor/test_padding.py::PaddingTest::test_pad_outputs_alignment_bytes_32_shape1_float16, test/inductor/test_padding.py::PaddingTest::test_pad_outputs_alignment_bytes_32_shape1_float32, test/inductor/test_padding.py::PaddingTest::test_pad_strides, test/inductor/test_padding.py::PaddingTest::test_pad_strides_skip, test/inductor/test_padding.py::PaddingTest::test_padmm, test/inductor/test_padding.py::PaddingTest::test_perm_outer_dynamic_shape_padding_shape0_perm0_alignment_bytes_32_enable_pad_False, test/inductor/test_padding.py::PaddingTest::test_perm_outer_dynamic_shape_padding_shape1_perm1_alignment_bytes_32_enable_pad_True, test/inductor/test_padding.py::PaddingTest::test_perm_outer_dynamic_shape_padding_shape2_perm2_alignment_bytes_64_enable_pad_True, test/inductor/test_padding.py::PaddingTest::test_perm_outer_dynamic_shape_padding_shape3_perm3_alignment_bytes_64_enable_pad_False, test/inductor/test_padding.py::PaddingTest::test_perm_outer_dynamic_shape_padding_shape4_perm4_alignment_bytes_32_enable_pad_False, test/inductor/test_padding.py::PaddingTest::test_perm_outer_dynamic_shape_padding_shape5_perm5_alignment_bytes_32_enable_pad_True, test/inductor/test_padding.py::PaddingTest::test_perm_outer_dynamic_shape_padding_shape6_perm6_alignment_bytes_64_enable_pad_True, test/inductor/test_padding.py::PaddingTest::test_perm_outer_dynamic_shape_padding_shape7_perm7_alignment_bytes_64_enable_pad_False, test/inductor/test_padding.py::PaddingTest::test_view 2025-12-04T11:24:47.6651292Z 2025-12-04T11:24:47.6651516Z Finished inductor/test_padding 1/1 ... [2025-12-04 11:24:47.661298][6335.603515471], took 0.59min 2025-12-04T11:24:47.6863215Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/inductor.test_padding/inductor.test_padding-2fcd8838081bfd5e.xml 2025-12-04T11:24:47.7645278Z Running dynamo/test_callback 1/1 ... [2025-12-04 11:24:47.764284][6335.706501335] 2025-12-04T11:24:47.7645900Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2025-12-04T11:24:47.7649136Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'dynamo/test_callback.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-12-04 11:24:47.764625] 2025-12-04T11:24:57.9472519Z 2025-12-04T11:24:57.9474085Z dynamo/test_callback 1/1 was successful, full logs can be found in artifacts with path test/test-reports/dynamo.test_callback_1.1_35bc12fe46f76cf6_.log 2025-12-04T11:24:57.9476103Z Running 4 items in this shard: test/dynamo/test_callback.py::CallbackTests::test_callbacks_with_duplicate_prevention, test/dynamo/test_callback.py::CallbackTests::test_counter, test/dynamo/test_callback.py::CallbackTests::test_counter_assertion, test/dynamo/test_callback.py::CallbackTests::test_triggers 2025-12-04T11:24:57.9477035Z 2025-12-04T11:24:57.9477254Z Finished dynamo/test_callback 1/1 ... [2025-12-04 11:24:57.946832][6345.889044579], took 0.17min 2025-12-04T11:24:57.9718426Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/dynamo.test_callback/dynamo.test_callback-dbddf90c7da99eb5.xml 2025-12-04T11:24:58.0465622Z Running inductor/test_cudagraph_trees_expandable_segments 1/1 ... [2025-12-04 11:24:58.046294][6345.988512396] 2025-12-04T11:24:58.0466185Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2025-12-04T11:24:58.0469183Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'inductor/test_cudagraph_trees_expandable_segments.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-12-04 11:24:58.046639] 2025-12-04T11:27:13.6713230Z 2025-12-04T11:27:13.6714379Z PRINTING LOG FILE of inductor/test_cudagraph_trees_expandable_segments 1/1 (test/test-reports/inductor.test_cudagraph_trees_expandable_segments_1.1_7984e53c82f556c8_.log) 2025-12-04T11:27:13.6716474Z /var/lib/jenkins/workspace/test/inductor/test_cudagraph_trees_expandable_segments.py:43: FutureWarning: torch.cuda._set_allocator_settings is deprecated. Use torch._C._accelerator_setAllocatorSettings instead. 2025-12-04T11:27:13.6717765Z torch.cuda.memory._set_allocator_settings("expandable_segments:True") 2025-12-04T11:27:13.6718734Z Downloading https://ossci-metrics.s3.amazonaws.com/disabled-tests-condensed.json to ./.pytorch-disabled-tests.json 2025-12-04T11:27:13.6720297Z Test results will be stored in test-reports/python-pytest/inductor.test_cudagraph_trees_expandable_segments/inductor.test_cudagraph_trees_expandable_segments-e8ff2a368a15c33d.xml 2025-12-04T11:27:13.6721519Z ============================= test session starts ============================== 2025-12-04T11:27:13.6722247Z platform linux -- Python 3.10.14, pytest-7.3.2, pluggy-1.6.0 -- /opt/conda/envs/py_3.10/bin/python 2025-12-04T11:27:13.6723307Z cachedir: .pytest_cache 2025-12-04T11:27:13.6724045Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow] 2025-12-04T11:27:13.6724847Z rootdir: /var/lib/jenkins/workspace 2025-12-04T11:27:13.6725211Z configfile: pytest.ini 2025-12-04T11:27:13.6726043Z plugins: hypothesis-6.56.4, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-14.0, subtests-0.13.1, xdist-3.3.1, xdoctest-1.3.0, anyio-4.12.0, typeguard-4.3.0 2025-12-04T11:27:13.6726911Z collecting ... collected 152 items 2025-12-04T11:27:13.6727360Z stepcurrent: Cannot find last run test, not skipping 2025-12-04T11:27:13.6822721Z Running 152 items in this shard: test/inductor/test_cudagraph_trees_expandable_segments.py::CudaGraphTreeTests::test_accumulate_grad, test/inductor/test_cudagraph_trees_expandable_segments.py::CudaGraphTreeTests::test_accumulate_multiple_recordings, test/inductor/test_cudagraph_trees_expandable_segments.py::CudaGraphTreeTests::test_alias_of_parameter, test/inductor/test_cudagraph_trees_expandable_segments.py::CudaGraphTreeTests::test_aliased_output_checkpoint, test/inductor/test_cudagraph_trees_expandable_segments.py::CudaGraphTreeTests::test_aliased_static_parameter, test/inductor/test_cudagraph_trees_expandable_segments.py::CudaGraphTreeTests::test_aliased_storage_single_weakref, test/inductor/test_cudagraph_trees_expandable_segments.py::CudaGraphTreeTests::test_aliasing_static_ref, test/inductor/test_cudagraph_trees_expandable_segments.py::CudaGraphTreeTests::test_amp_cache_disabled, test/inductor/test_cudagraph_trees_expandable_segments.py::CudaGraphTreeTests::test_backward_gets_cached_cudagraphs, test/inductor/test_cudagraph_trees_expandable_segments.py::CudaGraphTreeTests::test_cache_hit_forward_miss_backward, test/inductor/test_cudagraph_trees_expandable_segments.py::CudaGraphTreeTests::test_cached_boxed_forward_device_index, test/inductor/test_cudagraph_trees_expandable_segments.py::CudaGraphTreeTests::test_cached_forward_backward, test/inductor/test_cudagraph_trees_expandable_segments.py::CudaGraphTreeTests::test_checkpoint_shared_output_storage_deallocation, test/inductor/test_cudagraph_trees_expandable_segments.py::CudaGraphTreeTests::test_checkpointing_resets_persistent_refs, test/inductor/test_cudagraph_trees_expandable_segments.py::CudaGraphTreeTests::test_cleanup, test/inductor/test_cudagraph_trees_expandable_segments.py::CudaGraphTreeTests::test_compiled_autograd_static_input_params, test/inductor/test_cudagraph_trees_expandable_segments.py::CudaGraphTreeTests::test_constant_output, test/inductor/test_cudagraph_trees_expandable_segments.py::CudaGraphTreeTests::test_conv_benchmark, test/inductor/test_cudagraph_trees_expandable_segments.py::CudaGraphTreeTests::test_cpp_wrapper, test/inductor/test_cudagraph_trees_expandable_segments.py::CudaGraphTreeTests::test_cudagraph_capture_sizes, test/inductor/test_cudagraph_trees_expandable_segments.py::CudaGraphTreeTests::test_cudagraph_capture_sizes1, test/inductor/test_cudagraph_trees_expandable_segments.py::CudaGraphTreeTests::test_cudagraph_capture_sizes2, test/inductor/test_cudagraph_trees_expandable_segments.py::CudaGraphTreeTests::test_cudagraph_or_error, test/inductor/test_cudagraph_trees_expandable_segments.py::CudaGraphTreeTests::test_dynamic_backward, test/inductor/test_cudagraph_trees_expandable_segments.py::CudaGraphTreeTests::test_dynamic_warmup, test/inductor/test_cudagraph_trees_expandable_segments.py::CudaGraphTreeTests::test_empty_cpu_tensor, test/inductor/test_cudagraph_trees_expandable_segments.py::CudaGraphTreeTests::test_empty_storage, test/inductor/test_cudagraph_trees_expandable_segments.py::CudaGraphTreeTests::test_end_recording_early, test/inductor/test_cudagraph_trees_expandable_segments.py::CudaGraphTreeTests::test_error_on_dealloc_use, test/inductor/test_cudagraph_trees_expandable_segments.py::CudaGraphTreeTests::test_error_on_dealloc_use2, test/inductor/test_cudagraph_trees_expandable_segments.py::CudaGraphTreeTests::test_execution_into_recording, test/inductor/test_cudagraph_trees_expandable_segments.py::CudaGraphTreeTests::test_expanded_inputs, test/inductor/test_cudagraph_trees_expandable_segments.py::CudaGraphTreeTests::test_fallback_to_eager_if_recompiling_too_many_times, test/inductor/test_cudagraph_trees_expandable_segments.py::CudaGraphTreeTests::test_fallback_to_eager_if_recompiling_too_many_times_due_to_cudagraph_managed_tensor, test/inductor/test_cudagraph_trees_expandable_segments.py::CudaGraphTreeTests::test_fallback_to_eager_if_recompiling_too_many_times_warn_only_once, test/inductor/test_cudagraph_trees_expandable_segments.py::CudaGraphTreeTests::test_forward_backward, test/inductor/test_cudagraph_trees_expandable_segments.py::CudaGraphTreeTests::test_forward_backward_not_called_backend_cudagraphs, test/inductor/test_cudagraph_trees_expandable_segments.py::CudaGraphTreeTests::test_forward_backward_not_called_backend_inductor, test/inductor/test_cudagraph_trees_expandable_segments.py::CudaGraphTreeTests::test_forward_generation, test/inductor/test_cudagraph_trees_expandable_segments.py::CudaGraphTreeTests::test_forward_with_skipped_cudagraphed_backward, test/inductor/test_cudagraph_trees_expandable_segments.py::CudaGraphTreeTests::test_frozen_fn, test/inductor/test_cudagraph_trees_expandable_segments.py::CudaGraphTreeTests::test_function_compiled_multiple_times, test/inductor/test_cudagraph_trees_expandable_segments.py::CudaGraphTreeTests::test_graph_partition, test/inductor/test_cudagraph_trees_expandable_segments.py::CudaGraphTreeTests::test_graph_partition_buffer_reuse, test/inductor/test_cudagraph_trees_expandable_segments.py::CudaGraphTreeTests::test_graph_partition_condition_op, test/inductor/test_cudagraph_trees_expandable_segments.py::CudaGraphTreeTests::test_graph_partition_cpu_only, test/inductor/test_cudagraph_trees_expandable_segments.py::CudaGraphTreeTests::test_graph_partition_cpu_op_and_dynamic_shapes, test/inductor/test_cudagraph_trees_expandable_segments.py::CudaGraphTreeTests::test_graph_partition_cpu_scalar1, test/inductor/test_cudagraph_trees_expandable_segments.py::CudaGraphTreeTests::test_graph_partition_cpu_scalar2, test/inductor/test_cudagraph_trees_expandable_segments.py::CudaGraphTreeTests::test_graph_partition_cpu_scalar3, test/inductor/test_cudagraph_trees_expandable_segments.py::CudaGraphTreeTests::test_graph_partition_cpu_scalar4, test/inductor/test_cudagraph_trees_expandable_segments.py::CudaGraphTreeTests::test_graph_partition_cpu_scalar_device_put, test/inductor/test_cudagraph_trees_expandable_segments.py::CudaGraphTreeTests::test_graph_partition_cpu_scalar_multiple, test/inductor/test_cudagraph_trees_expandable_segments.py::CudaGraphTreeTests::test_graph_partition_cpu_scalar_mutation, test/inductor/test_cudagraph_trees_expandable_segments.py::CudaGraphTreeTests::test_graph_partition_cpu_tensor_symints, test/inductor/test_cudagraph_trees_expandable_segments.py::CudaGraphTreeTests::test_graph_partition_custom_op, test/inductor/test_cudagraph_trees_expandable_segments.py::CudaGraphTreeTests::test_graph_partition_custom_op_dynamoc_shapes, test/inductor/test_cudagraph_trees_expandable_segments.py::CudaGraphTreeTests::test_graph_partition_custom_op_mutation, test/inductor/test_cudagraph_trees_expandable_segments.py::CudaGraphTreeTests::test_graph_partition_custom_op_mutation_late_free, test/inductor/test_cudagraph_trees_expandable_segments.py::CudaGraphTreeTests::test_graph_partition_custom_op_no_split, test/inductor/test_cudagraph_trees_expandable_segments.py::CudaGraphTreeTests::test_graph_partition_custom_rule, test/inductor/test_cudagraph_trees_expandable_segments.py::CudaGraphTreeTests::test_graph_partition_dynamic_scalar_inputs, test/inductor/test_cudagraph_trees_expandable_segments.py::CudaGraphTreeTests::test_graph_partition_dynamic_shapes, test/inductor/test_cudagraph_trees_expandable_segments.py::CudaGraphTreeTests::test_graph_partition_foreach_op, test/inductor/test_cudagraph_trees_expandable_segments.py::CudaGraphTreeTests::test_graph_partition_forward_backward, test/inductor/test_cudagraph_trees_expandable_segments.py::CudaGraphTreeTests::test_graph_partition_forward_backward_not_called, test/inductor/test_cudagraph_trees_expandable_segments.py::CudaGraphTreeTests::test_graph_partition_forward_with_skipped_cudagraphed_backward, test/inductor/test_cudagraph_trees_expandable_segments.py::CudaGraphTreeTests::test_graph_partition_fused_scheduler_node, test/inductor/test_cudagraph_trees_expandable_segments.py::CudaGraphTreeTests::test_graph_partition_gc, test/inductor/test_cudagraph_trees_expandable_segments.py::CudaGraphTreeTests::test_graph_partition_item, test/inductor/test_cudagraph_trees_expandable_segments.py::CudaGraphTreeTests::test_graph_partition_kernel_reuse_autotune_at_compile_time_False, test/inductor/test_cudagraph_trees_expandable_segments.py::CudaGraphTreeTests::test_graph_partition_kernel_reuse_autotune_at_compile_time_True, test/inductor/test_cudagraph_trees_expandable_segments.py::CudaGraphTreeTests::test_graph_partition_log_message, test/inductor/test_cudagraph_trees_expandable_segments.py::CudaGraphTreeTests::test_graph_partition_multiple_devices_msg, test/inductor/test_cudagraph_trees_expandable_segments.py::CudaGraphTreeTests::test_graph_partition_reduce_overhead_mode_effectiveness, test/inductor/test_cudagraph_trees_expandable_segments.py::CudaGraphTreeTests::test_graph_partition_reorder_cpu_and_gpu, test/inductor/test_cudagraph_trees_expandable_segments.py::CudaGraphTreeTests::test_graph_partition_reorder_cpu_and_gpu_interleave, test/inductor/test_cudagraph_trees_expandable_segments.py::CudaGraphTreeTests::test_graph_partition_reorder_custom_op_with_no_dependency, test/inductor/test_cudagraph_trees_expandable_segments.py::CudaGraphTreeTests::test_graph_partition_reorder_custom_op_with_no_dependency1, test/inductor/test_cudagraph_trees_expandable_segments.py::CudaGraphTreeTests::test_graph_partition_simple, test/inductor/test_cudagraph_trees_expandable_segments.py::CudaGraphTreeTests::test_graph_partition_symint, test/inductor/test_cudagraph_trees_expandable_segments.py::CudaGraphTreeTests::test_graph_partition_symint_cat_backward, test/inductor/test_cudagraph_trees_expandable_segments.py::CudaGraphTreeTests::test_graph_partition_symint_from_mutation_index, test/inductor/test_cudagraph_trees_expandable_segments.py::CudaGraphTreeTests::test_graph_partition_symint_from_nested_indirect_indexing, test/inductor/test_cudagraph_trees_expandable_segments.py::CudaGraphTreeTests::test_graph_partition_unbacked_symint, test/inductor/test_cudagraph_trees_expandable_segments.py::CudaGraphTreeTests::test_graph_partition_unbacked_symint_multi_output_layout, test/inductor/test_cudagraph_trees_expandable_segments.py::CudaGraphTreeTests::test_graph_partition_user_defined_triton_kernel_reuse, test/inductor/test_cudagraph_trees_expandable_segments.py::CudaGraphTreeTests::test_graph_partition_view_fallback, test/inductor/test_cudagraph_trees_expandable_segments.py::CudaGraphTreeTests::test_graph_partition_with_memory_plan_reuse, test/inductor/test_cudagraph_trees_expandable_segments.py::CudaGraphTreeTests::test_incompatible_cudagraph_ops_item, test/inductor/test_cudagraph_trees_expandable_segments.py::CudaGraphTreeTests::test_incompatible_cudagraph_ops_nonzero, test/inductor/test_cudagraph_trees_expandable_segments.py::CudaGraphTreeTests::test_incompatible_cudagraph_ops_nonzero_backend, test/inductor/test_cudagraph_trees_expandable_segments.py::CudaGraphTreeTests::test_incompatible_cudagraph_ops_nonzero_graph_breaks, test/inductor/test_cudagraph_trees_expandable_segments.py::CudaGraphTreeTests::test_index_put, test/inductor/test_cudagraph_trees_expandable_segments.py::CudaGraphTreeTests::test_live_outputs_multiple_graphs, test/inductor/test_cudagraph_trees_expandable_segments.py::CudaGraphTreeTests::test_manager_per_device, test/inductor/test_cudagraph_trees_expandable_segments.py::CudaGraphTreeTests::test_mark_step, test/inductor/test_cudagraph_trees_expandable_segments.py::CudaGraphTreeTests::test_meta_tensor, test/inductor/test_cudagraph_trees_expandable_segments.py::CudaGraphTreeTests::test_multi_dispatch_child_node, test/inductor/test_cudagraph_trees_expandable_segments.py::CudaGraphTreeTests::test_multi_dispatch_custom_module, test/inductor/test_cudagraph_trees_expandable_segments.py::CudaGraphTreeTests::test_multi_dispatch_custom_module_buffer, test/inductor/test_cudagraph_trees_expandable_segments.py::CudaGraphTreeTests::test_multi_dispatch_parent_node, test/inductor/test_cudagraph_trees_expandable_segments.py::CudaGraphTreeTests::test_multi_dispatch_single_compile_builtin_module, test/inductor/test_cudagraph_trees_expandable_segments.py::CudaGraphTreeTests::test_multi_dispatch_single_compile_builtin_module_buffers, test/inductor/test_cudagraph_trees_expandable_segments.py::CudaGraphTreeTests::test_multi_dispatch_single_compile_param_inputs, test/inductor/test_cudagraph_trees_expandable_segments.py::CudaGraphTreeTests::test_multinomial, test/inductor/test_cudagraph_trees_expandable_segments.py::CudaGraphTreeTests::test_multiple_devices_msg_backend_cudagraphs, test/inductor/test_cudagraph_trees_expandable_segments.py::CudaGraphTreeTests::test_multiple_devices_msg_backend_inductor, test/inductor/test_cudagraph_trees_expandable_segments.py::CudaGraphTreeTests::test_multiple_insert_removal_caching, test/inductor/test_cudagraph_trees_expandable_segments.py::CudaGraphTreeTests::test_mutation_cudagraph_managed_tensor_warn_backend_cudagraphs, test/inductor/test_cudagraph_trees_expandable_segments.py::CudaGraphTreeTests::test_mutation_cudagraph_managed_tensor_warn_backend_inductor, test/inductor/test_cudagraph_trees_expandable_segments.py::CudaGraphTreeTests::test_mutation_cudagraph_managed_tensor_warn_only_once_backend_cudagraphs, test/inductor/test_cudagraph_trees_expandable_segments.py::CudaGraphTreeTests::test_mutation_cudagraph_managed_tensor_warn_only_once_backend_inductor, test/inductor/test_cudagraph_trees_expandable_segments.py::CudaGraphTreeTests::test_mutation_cudagraph_managed_tensors_backend_cudagraphs, test/inductor/test_cudagraph_trees_expandable_segments.py::CudaGraphTreeTests::test_mutation_cudagraph_managed_tensors_backend_inductor, test/inductor/test_cudagraph_trees_expandable_segments.py::CudaGraphTreeTests::test_mutation_cudagraph_managed_tensors_config_backend_cudagraphs, test/inductor/test_cudagraph_trees_expandable_segments.py::CudaGraphTreeTests::test_mutation_cudagraph_managed_tensors_config_backend_inductor, test/inductor/test_cudagraph_trees_expandable_segments.py::CudaGraphTreeTests::test_mutation_on_inp_backend_cudagraphs, test/inductor/test_cudagraph_trees_expandable_segments.py::CudaGraphTreeTests::test_mutation_on_inp_backend_inductor, test/inductor/test_cudagraph_trees_expandable_segments.py::CudaGraphTreeTests::test_mutation_reinplaced, test/inductor/test_cudagraph_trees_expandable_segments.py::CudaGraphTreeTests::test_no_rerecord_with_mark_static_address, test/inductor/test_cudagraph_trees_expandable_segments.py::CudaGraphTreeTests::test_not_fallback_to_eager_if_have_not_recompiling_too_many_times, test/inductor/test_cudagraph_trees_expandable_segments.py::CudaGraphTreeTests::test_output_alias, test/inductor/test_cudagraph_trees_expandable_segments.py::CudaGraphTreeTests::test_peristed_output_livenes, test/inductor/test_cudagraph_trees_expandable_segments.py::CudaGraphTreeTests::test_remove_hooks_on_cached_tensors, test/inductor/test_cudagraph_trees_expandable_segments.py::CudaGraphTreeTests::test_rerecord_if_static_input_address_changed, test/inductor/test_cudagraph_trees_expandable_segments.py::CudaGraphTreeTests::test_rng_non_trees, test/inductor/test_cudagraph_trees_expandable_segments.py::CudaGraphTreeTests::test_rng_trees, test/inductor/test_cudagraph_trees_expandable_segments.py::CudaGraphTreeTests::test_run_simple, test/inductor/test_cudagraph_trees_expandable_segments.py::CudaGraphTreeTests::test_separate_recordings, test/inductor/test_cudagraph_trees_expandable_segments.py::CudaGraphTreeTests::test_side_stream_memory_allocation, test/inductor/test_cudagraph_trees_expandable_segments.py::CudaGraphTreeTests::test_single_stream_use, test/inductor/test_cudagraph_trees_expandable_segments.py::CudaGraphTreeTests::test_skip_cpp_wrapper, test/inductor/test_cudagraph_trees_expandable_segments.py::CudaGraphTreeTests::test_skip_cudagraph_unsafe_ops, test/inductor/test_cudagraph_trees_expandable_segments.py::CudaGraphTreeTests::test_skip_if_dynamic_shape_limit_reached1, test/inductor/test_cudagraph_trees_expandable_segments.py::CudaGraphTreeTests::test_skip_if_dynamic_shape_limit_reached2, test/inductor/test_cudagraph_trees_expandable_segments.py::CudaGraphTreeTests::test_skip_symbolic, test/inductor/test_cudagraph_trees_expandable_segments.py::CudaGraphTreeTests::test_sparsity, test/inductor/test_cudagraph_trees_expandable_segments.py::CudaGraphTreeTests::test_static_inputs_address_mutation_log, test/inductor/test_cudagraph_trees_expandable_segments.py::CudaGraphTreeTests::test_storage_access_error, test/inductor/test_cudagraph_trees_expandable_segments.py::CudaGraphTreeTests::test_tensor_constant_mutation, test/inductor/test_cudagraph_trees_expandable_segments.py::CudaGraphTreeTests::test_tensor_dies_between_checkpoint, test/inductor/test_cudagraph_trees_expandable_segments.py::CudaGraphTreeTests::test_tensor_no_longer_in_pool, test/inductor/test_cudagraph_trees_expandable_segments.py::CudaGraphTreeTests::test_unaligned_static_input_no_cudagraphs, test/inductor/test_cudagraph_trees_expandable_segments.py::CudaGraphTreeTests::test_unaligned_static_input_non_trees, test/inductor/test_cudagraph_trees_expandable_segments.py::CudaGraphTreeTests::test_unaligned_static_input_trees, test/inductor/test_cudagraph_trees_expandable_segments.py::CudaGraphTreeTests::test_unaligned_static_parameter, test/inductor/test_cudagraph_trees_expandable_segments.py::CudaGraphTreeTests::test_unstable_ptr, test/inductor/test_cudagraph_trees_expandable_segments.py::CudaGraphTreeTests::test_warmup_stream_sync, test/inductor/test_cudagraph_trees_expandable_segments.py::CudaGraphTreeTests::test_warn_on_pending_backward, test/inductor/test_cudagraph_trees_expandable_segments.py::CudaGraphTreeTests::test_warn_once_if_dynamic_shape_limit_reached, test/inductor/test_cudagraph_trees_expandable_segments.py::CudaGraphTreeTests::test_workspace_allocation_error 2025-12-04T11:27:13.6916800Z 2025-12-04T11:27:13.6917620Z inductor/test_cudagraph_trees_expandable_segments.py::CudaGraphTreeTests::test_accumulate_grad <- test/inductor/test_cudagraph_trees.py PASSED [1.6406s] [ 0%] 2025-12-04T11:27:13.6919409Z inductor/test_cudagraph_trees_expandable_segments.py::CudaGraphTreeTests::test_accumulate_multiple_recordings <- test/inductor/test_cudagraph_trees.py PASSED [1.2814s] [ 1%] 2025-12-04T11:27:13.6921206Z inductor/test_cudagraph_trees_expandable_segments.py::CudaGraphTreeTests::test_alias_of_parameter <- test/inductor/test_cudagraph_trees.py PASSED [0.3391s] [ 1%] 2025-12-04T11:27:13.6922952Z inductor/test_cudagraph_trees_expandable_segments.py::CudaGraphTreeTests::test_aliased_output_checkpoint <- test/inductor/test_cudagraph_trees.py PASSED [0.1542s] [ 2%] 2025-12-04T11:27:13.6924772Z inductor/test_cudagraph_trees_expandable_segments.py::CudaGraphTreeTests::test_aliased_static_parameter <- test/inductor/test_cudagraph_trees.py PASSED [0.1486s] [ 3%] 2025-12-04T11:27:13.6927154Z inductor/test_cudagraph_trees_expandable_segments.py::CudaGraphTreeTests::test_aliased_storage_single_weakref <- test/inductor/test_cudagraph_trees.py W1204 11:25:07.052000 109768 site-packages/torch/_dynamo/variables/tensor.py:1073] [1/0] Graph break from `Tensor.item()`, consider setting: 2025-12-04T11:27:13.6929310Z W1204 11:25:07.052000 109768 site-packages/torch/_dynamo/variables/tensor.py:1073] [1/0] torch._dynamo.config.capture_scalar_outputs = True 2025-12-04T11:27:13.6930431Z W1204 11:25:07.052000 109768 site-packages/torch/_dynamo/variables/tensor.py:1073] [1/0] or: 2025-12-04T11:27:13.6931507Z W1204 11:25:07.052000 109768 site-packages/torch/_dynamo/variables/tensor.py:1073] [1/0] env TORCHDYNAMO_CAPTURE_SCALAR_OUTPUTS=1 2025-12-04T11:27:13.6932818Z W1204 11:25:07.052000 109768 site-packages/torch/_dynamo/variables/tensor.py:1073] [1/0] to include these operations in the captured graph. 2025-12-04T11:27:13.6934054Z W1204 11:25:07.052000 109768 site-packages/torch/_dynamo/variables/tensor.py:1073] [1/0] 2025-12-04T11:27:13.6935066Z W1204 11:25:07.052000 109768 site-packages/torch/_dynamo/variables/tensor.py:1073] [1/0] Graph break: from user code at: 2025-12-04T11:27:13.6936729Z W1204 11:25:07.052000 109768 site-packages/torch/_dynamo/variables/tensor.py:1073] [1/0] File "/var/lib/jenkins/workspace/test/inductor/test_cudagraph_trees.py", line 1280, in torch_dynamo_resume_in_foo_at_1278 2025-12-04T11:27:13.6938322Z W1204 11:25:07.052000 109768 site-packages/torch/_dynamo/variables/tensor.py:1073] [1/0] x_alias2 = x[ind:] 2025-12-04T11:27:13.6939275Z W1204 11:25:07.052000 109768 site-packages/torch/_dynamo/variables/tensor.py:1073] [1/0] 2025-12-04T11:27:13.6940208Z W1204 11:25:07.052000 109768 site-packages/torch/_dynamo/variables/tensor.py:1073] [1/0] 2025-12-04T11:27:13.6940854Z PASSED [0.5799s] [ 3%] 2025-12-04T11:27:13.6942316Z inductor/test_cudagraph_trees_expandable_segments.py::CudaGraphTreeTests::test_aliasing_static_ref <- test/inductor/test_cudagraph_trees.py W1204 11:25:07.309000 109768 site-packages/torch/_inductor/utils.py:1703] [0/0] Not enough SMs to use max_autotune_gemm mode 2025-12-04T11:27:13.6944186Z PASSED [0.4940s] [ 4%] 2025-12-04T11:27:13.6945183Z inductor/test_cudagraph_trees_expandable_segments.py::CudaGraphTreeTests::test_amp_cache_disabled <- test/inductor/test_cudagraph_trees.py PASSED [0.7193s] [ 5%] 2025-12-04T11:27:13.6946972Z inductor/test_cudagraph_trees_expandable_segments.py::CudaGraphTreeTests::test_backward_gets_cached_cudagraphs <- test/inductor/test_cudagraph_trees.py PASSED [1.4243s] [ 5%] 2025-12-04T11:27:13.6948814Z inductor/test_cudagraph_trees_expandable_segments.py::CudaGraphTreeTests::test_cache_hit_forward_miss_backward <- test/inductor/test_cudagraph_trees.py PASSED [1.4306s] [ 6%] 2025-12-04T11:27:13.6950833Z inductor/test_cudagraph_trees_expandable_segments.py::CudaGraphTreeTests::test_cached_boxed_forward_device_index <- test/inductor/test_cudagraph_trees.py SKIPPED [0.0003s] (requires multiple cuda devices) [ 7%] 2025-12-04T11:27:13.6952836Z inductor/test_cudagraph_trees_expandable_segments.py::CudaGraphTreeTests::test_cached_forward_backward <- test/inductor/test_cudagraph_trees.py PASSED [1.0626s] [ 7%] 2025-12-04T11:27:13.6954741Z inductor/test_cudagraph_trees_expandable_segments.py::CudaGraphTreeTests::test_checkpoint_shared_output_storage_deallocation <- test/inductor/test_cudagraph_trees.py PASSED [0.1592s] [ 8%] 2025-12-04T11:27:13.6956706Z inductor/test_cudagraph_trees_expandable_segments.py::CudaGraphTreeTests::test_checkpointing_resets_persistent_refs <- test/inductor/test_cudagraph_trees.py PASSED [0.3407s] [ 9%] 2025-12-04T11:27:13.6958466Z inductor/test_cudagraph_trees_expandable_segments.py::CudaGraphTreeTests::test_cleanup <- test/inductor/test_cudagraph_trees.py PASSED [0.5484s] [ 9%] 2025-12-04T11:27:13.6960219Z inductor/test_cudagraph_trees_expandable_segments.py::CudaGraphTreeTests::test_compiled_autograd_static_input_params <- test/inductor/test_cudagraph_trees.py PASSED [0.8444s] [ 10%] 2025-12-04T11:27:13.6962033Z inductor/test_cudagraph_trees_expandable_segments.py::CudaGraphTreeTests::test_constant_output <- test/inductor/test_cudagraph_trees.py PASSED [0.6446s] [ 11%] 2025-12-04T11:27:13.6963716Z inductor/test_cudagraph_trees_expandable_segments.py::CudaGraphTreeTests::test_conv_benchmark <- test/inductor/test_cudagraph_trees.py PASSED [0.8013s] [ 11%] 2025-12-04T11:27:13.6965365Z inductor/test_cudagraph_trees_expandable_segments.py::CudaGraphTreeTests::test_cpp_wrapper <- test/inductor/test_cudagraph_trees.py PASSED [2.1390s] [ 12%] 2025-12-04T11:27:13.6967078Z inductor/test_cudagraph_trees_expandable_segments.py::CudaGraphTreeTests::test_cudagraph_capture_sizes <- test/inductor/test_cudagraph_trees.py PASSED [0.8166s] [ 13%] 2025-12-04T11:27:13.6968840Z inductor/test_cudagraph_trees_expandable_segments.py::CudaGraphTreeTests::test_cudagraph_capture_sizes1 <- test/inductor/test_cudagraph_trees.py PASSED [0.4345s] [ 13%] 2025-12-04T11:27:13.6970724Z inductor/test_cudagraph_trees_expandable_segments.py::CudaGraphTreeTests::test_cudagraph_capture_sizes2 <- test/inductor/test_cudagraph_trees.py PASSED [0.4381s] [ 14%] 2025-12-04T11:27:13.6972478Z inductor/test_cudagraph_trees_expandable_segments.py::CudaGraphTreeTests::test_cudagraph_or_error <- test/inductor/test_cudagraph_trees.py PASSED [0.3099s] [ 15%] 2025-12-04T11:27:13.6974163Z inductor/test_cudagraph_trees_expandable_segments.py::CudaGraphTreeTests::test_dynamic_backward <- test/inductor/test_cudagraph_trees.py PASSED [1.2394s] [ 15%] 2025-12-04T11:27:13.6975834Z inductor/test_cudagraph_trees_expandable_segments.py::CudaGraphTreeTests::test_dynamic_warmup <- test/inductor/test_cudagraph_trees.py PASSED [0.1825s] [ 16%] 2025-12-04T11:27:13.6977588Z inductor/test_cudagraph_trees_expandable_segments.py::CudaGraphTreeTests::test_empty_cpu_tensor <- test/inductor/test_cudagraph_trees.py PASSED [0.3375s] [ 17%] 2025-12-04T11:27:13.6979443Z inductor/test_cudagraph_trees_expandable_segments.py::CudaGraphTreeTests::test_empty_storage <- test/inductor/test_cudagraph_trees.py PASSED [0.5764s] [ 17%] 2025-12-04T11:27:13.6981359Z inductor/test_cudagraph_trees_expandable_segments.py::CudaGraphTreeTests::test_end_recording_early <- test/inductor/test_cudagraph_trees.py PASSED [0.6011s] [ 18%] 2025-12-04T11:27:13.6983088Z inductor/test_cudagraph_trees_expandable_segments.py::CudaGraphTreeTests::test_error_on_dealloc_use <- test/inductor/test_cudagraph_trees.py PASSED [0.3274s] [ 19%] 2025-12-04T11:27:13.6984880Z inductor/test_cudagraph_trees_expandable_segments.py::CudaGraphTreeTests::test_error_on_dealloc_use2 <- test/inductor/test_cudagraph_trees.py PASSED [0.3241s] [ 19%] 2025-12-04T11:27:13.6986621Z inductor/test_cudagraph_trees_expandable_segments.py::CudaGraphTreeTests::test_execution_into_recording <- test/inductor/test_cudagraph_trees.py PASSED [0.6104s] [ 20%] 2025-12-04T11:27:13.6988342Z inductor/test_cudagraph_trees_expandable_segments.py::CudaGraphTreeTests::test_expanded_inputs <- test/inductor/test_cudagraph_trees.py PASSED [0.3504s] [ 21%] 2025-12-04T11:27:13.6990190Z inductor/test_cudagraph_trees_expandable_segments.py::CudaGraphTreeTests::test_fallback_to_eager_if_recompiling_too_many_times <- test/inductor/test_cudagraph_trees.py PASSED [0.3933s] [ 21%] 2025-12-04T11:27:13.6992334Z inductor/test_cudagraph_trees_expandable_segments.py::CudaGraphTreeTests::test_fallback_to_eager_if_recompiling_too_many_times_due_to_cudagraph_managed_tensor <- test/inductor/test_cudagraph_trees.py PASSED [0.4642s] [ 22%] 2025-12-04T11:27:13.6994546Z inductor/test_cudagraph_trees_expandable_segments.py::CudaGraphTreeTests::test_fallback_to_eager_if_recompiling_too_many_times_warn_only_once <- test/inductor/test_cudagraph_trees.py PASSED [0.3985s] [ 23%] 2025-12-04T11:27:13.6996471Z inductor/test_cudagraph_trees_expandable_segments.py::CudaGraphTreeTests::test_forward_backward <- test/inductor/test_cudagraph_trees.py PASSED [0.6392s] [ 23%] 2025-12-04T11:27:13.6998318Z inductor/test_cudagraph_trees_expandable_segments.py::CudaGraphTreeTests::test_forward_backward_not_called_backend_cudagraphs <- test/inductor/test_cudagraph_trees.py PASSED [0.3396s] [ 24%] 2025-12-04T11:27:13.7000331Z inductor/test_cudagraph_trees_expandable_segments.py::CudaGraphTreeTests::test_forward_backward_not_called_backend_inductor <- test/inductor/test_cudagraph_trees.py PASSED [0.5326s] [ 25%] 2025-12-04T11:27:13.7002185Z inductor/test_cudagraph_trees_expandable_segments.py::CudaGraphTreeTests::test_forward_generation <- test/inductor/test_cudagraph_trees.py PASSED [0.7015s] [ 25%] 2025-12-04T11:27:13.7004016Z inductor/test_cudagraph_trees_expandable_segments.py::CudaGraphTreeTests::test_forward_with_skipped_cudagraphed_backward <- test/inductor/test_cudagraph_trees.py PASSED [0.4319s] [ 26%] 2025-12-04T11:27:13.7005813Z inductor/test_cudagraph_trees_expandable_segments.py::CudaGraphTreeTests::test_frozen_fn <- test/inductor/test_cudagraph_trees.py PASSED [0.2789s] [ 26%] 2025-12-04T11:27:13.7007693Z inductor/test_cudagraph_trees_expandable_segments.py::CudaGraphTreeTests::test_function_compiled_multiple_times <- test/inductor/test_cudagraph_trees.py PASSED [0.5558s] [ 27%] 2025-12-04T11:27:13.7009879Z inductor/test_cudagraph_trees_expandable_segments.py::CudaGraphTreeTests::test_graph_partition <- test/inductor/test_cudagraph_trees.py W1204 11:25:29.249000 109768 site-packages/torch/_inductor/utils.py:2565] [0/0] DeviceCopy in input program 2025-12-04T11:27:13.7011627Z W1204 11:25:29.251000 109768 site-packages/torch/_inductor/utils.py:2565] [0/0] DeviceCopy in input program 2025-12-04T11:27:13.7012337Z PASSED [0.8053s] [ 28%] 2025-12-04T11:27:13.7013913Z inductor/test_cudagraph_trees_expandable_segments.py::CudaGraphTreeTests::test_graph_partition_buffer_reuse <- test/inductor/test_cudagraph_trees.py W1204 11:25:30.070000 109768 site-packages/torch/_inductor/utils.py:2565] [0/0] DeviceCopy in input program 2025-12-04T11:27:13.7015734Z W1204 11:25:30.071000 109768 site-packages/torch/_inductor/utils.py:2565] [0/0] DeviceCopy in input program 2025-12-04T11:27:13.7016849Z W1204 11:25:30.074000 109768 site-packages/torch/_inductor/utils.py:2565] [0/0] DeviceCopy in input program 2025-12-04T11:27:13.7017943Z W1204 11:25:30.075000 109768 site-packages/torch/_inductor/utils.py:2565] [0/0] DeviceCopy in input program 2025-12-04T11:27:13.7018659Z PASSED [0.8478s] [ 28%] 2025-12-04T11:27:13.7019707Z inductor/test_cudagraph_trees_expandable_segments.py::CudaGraphTreeTests::test_graph_partition_condition_op <- test/inductor/test_cudagraph_trees.py PASSED [0.8205s] [ 29%] 2025-12-04T11:27:13.7021480Z inductor/test_cudagraph_trees_expandable_segments.py::CudaGraphTreeTests::test_graph_partition_cpu_only <- test/inductor/test_cudagraph_trees.py PASSED [1.3312s] [ 30%] 2025-12-04T11:27:13.7023825Z inductor/test_cudagraph_trees_expandable_segments.py::CudaGraphTreeTests::test_graph_partition_cpu_op_and_dynamic_shapes <- test/inductor/test_cudagraph_trees.py W1204 11:25:33.057000 109768 site-packages/torch/_inductor/utils.py:2565] [0/0] DeviceCopy in input program 2025-12-04T11:27:13.7025712Z W1204 11:25:33.058000 109768 site-packages/torch/_inductor/utils.py:2565] [0/0] DeviceCopy in input program 2025-12-04T11:27:13.7026739Z W1204 11:25:33.787000 109768 site-packages/torch/_inductor/utils.py:2565] [0/1] DeviceCopy in input program 2025-12-04T11:27:13.7027767Z W1204 11:25:33.789000 109768 site-packages/torch/_inductor/utils.py:2565] [0/1] DeviceCopy in input program 2025-12-04T11:27:13.7028464Z PASSED [1.6716s] [ 30%] 2025-12-04T11:27:13.7029833Z inductor/test_cudagraph_trees_expandable_segments.py::CudaGraphTreeTests::test_graph_partition_cpu_scalar1 <- test/inductor/test_cudagraph_trees.py W1204 11:25:34.706000 109768 site-packages/torch/_inductor/utils.py:2565] [0/0] DeviceCopy in input program 2025-12-04T11:27:13.7031268Z PASSED [0.7537s] [ 31%] 2025-12-04T11:27:13.7032630Z inductor/test_cudagraph_trees_expandable_segments.py::CudaGraphTreeTests::test_graph_partition_cpu_scalar2 <- test/inductor/test_cudagraph_trees.py W1204 11:25:35.464000 109768 site-packages/torch/_inductor/utils.py:2565] [0/0] DeviceCopy in input program 2025-12-04T11:27:13.7034034Z PASSED [0.7622s] [ 32%] 2025-12-04T11:27:13.7035378Z inductor/test_cudagraph_trees_expandable_segments.py::CudaGraphTreeTests::test_graph_partition_cpu_scalar3 <- test/inductor/test_cudagraph_trees.py W1204 11:25:36.231000 109768 site-packages/torch/_inductor/utils.py:2565] [0/0] DeviceCopy in input program 2025-12-04T11:27:13.7036808Z PASSED [0.7593s] [ 32%] 2025-12-04T11:27:13.7038137Z inductor/test_cudagraph_trees_expandable_segments.py::CudaGraphTreeTests::test_graph_partition_cpu_scalar4 <- test/inductor/test_cudagraph_trees.py W1204 11:25:36.990000 109768 site-packages/torch/_inductor/utils.py:2565] [0/0] DeviceCopy in input program 2025-12-04T11:27:13.7039578Z PASSED [0.7519s] [ 33%] 2025-12-04T11:27:13.7041039Z inductor/test_cudagraph_trees_expandable_segments.py::CudaGraphTreeTests::test_graph_partition_cpu_scalar_device_put <- test/inductor/test_cudagraph_trees.py W1204 11:25:37.739000 109768 site-packages/torch/_inductor/utils.py:2565] [0/0] DeviceCopy in input program 2025-12-04T11:27:13.7042998Z W1204 11:25:37.741000 109768 site-packages/torch/_inductor/utils.py:2565] [0/0] DeviceCopy in input program 2025-12-04T11:27:13.7043698Z PASSED [0.7196s] [ 34%] 2025-12-04T11:27:13.7045134Z inductor/test_cudagraph_trees_expandable_segments.py::CudaGraphTreeTests::test_graph_partition_cpu_scalar_multiple <- test/inductor/test_cudagraph_trees.py W1204 11:25:38.467000 109768 site-packages/torch/_inductor/utils.py:2565] [0/0] DeviceCopy in input program 2025-12-04T11:27:13.7046652Z PASSED [0.7721s] [ 34%] 2025-12-04T11:27:13.7048174Z inductor/test_cudagraph_trees_expandable_segments.py::CudaGraphTreeTests::test_graph_partition_cpu_scalar_mutation <- test/inductor/test_cudagraph_trees.py W1204 11:25:39.233000 109768 site-packages/torch/_inductor/utils.py:2565] [0/0] DeviceCopy in input program 2025-12-04T11:27:13.7050003Z W1204 11:25:39.235000 109768 site-packages/torch/_inductor/utils.py:2565] [0/0] DeviceCopy in input program 2025-12-04T11:27:13.7050769Z PASSED [0.7427s] [ 35%] 2025-12-04T11:27:13.7051857Z inductor/test_cudagraph_trees_expandable_segments.py::CudaGraphTreeTests::test_graph_partition_cpu_tensor_symints <- test/inductor/test_cudagraph_trees.py PASSED [1.5645s] [ 36%] 2025-12-04T11:27:13.7053613Z inductor/test_cudagraph_trees_expandable_segments.py::CudaGraphTreeTests::test_graph_partition_custom_op <- test/inductor/test_cudagraph_trees.py PASSED [0.4862s] [ 36%] 2025-12-04T11:27:13.7055413Z inductor/test_cudagraph_trees_expandable_segments.py::CudaGraphTreeTests::test_graph_partition_custom_op_dynamoc_shapes <- test/inductor/test_cudagraph_trees.py PASSED [0.6905s] [ 37%] 2025-12-04T11:27:13.7057252Z inductor/test_cudagraph_trees_expandable_segments.py::CudaGraphTreeTests::test_graph_partition_custom_op_mutation <- test/inductor/test_cudagraph_trees.py PASSED [0.3874s] [ 38%] 2025-12-04T11:27:13.7059107Z inductor/test_cudagraph_trees_expandable_segments.py::CudaGraphTreeTests::test_graph_partition_custom_op_mutation_late_free <- test/inductor/test_cudagraph_trees.py PASSED [0.4822s] [ 38%] 2025-12-04T11:27:13.7060948Z inductor/test_cudagraph_trees_expandable_segments.py::CudaGraphTreeTests::test_graph_partition_custom_op_no_split <- test/inductor/test_cudagraph_trees.py PASSED [0.6180s] [ 39%] 2025-12-04T11:27:13.7062662Z inductor/test_cudagraph_trees_expandable_segments.py::CudaGraphTreeTests::test_graph_partition_custom_rule <- test/inductor/test_cudagraph_trees.py PASSED [0.7221s] [ 40%] 2025-12-04T11:27:13.7064838Z inductor/test_cudagraph_trees_expandable_segments.py::CudaGraphTreeTests::test_graph_partition_dynamic_scalar_inputs <- test/inductor/test_cudagraph_trees.py W1204 11:25:45.034000 109768 site-packages/torch/_inductor/utils.py:2565] [0/0] DeviceCopy in input program 2025-12-04T11:27:13.7066520Z W1204 11:25:45.035000 109768 site-packages/torch/_inductor/utils.py:2565] [0/0] DeviceCopy in input program 2025-12-04T11:27:13.7067416Z W1204 11:25:45.677000 109768 site-packages/torch/_inductor/utils.py:2565] [0/1] DeviceCopy in input program 2025-12-04T11:27:13.7068320Z W1204 11:25:45.679000 109768 site-packages/torch/_inductor/utils.py:2565] [0/1] DeviceCopy in input program 2025-12-04T11:27:13.7068929Z PASSED [1.1273s] [ 40%] 2025-12-04T11:27:13.7069855Z inductor/test_cudagraph_trees_expandable_segments.py::CudaGraphTreeTests::test_graph_partition_dynamic_shapes <- test/inductor/test_cudagraph_trees.py PASSED [0.4988s] [ 41%] 2025-12-04T11:27:13.7071499Z inductor/test_cudagraph_trees_expandable_segments.py::CudaGraphTreeTests::test_graph_partition_foreach_op <- test/inductor/test_cudagraph_trees.py PASSED [0.3884s] [ 42%] 2025-12-04T11:27:13.7073492Z inductor/test_cudagraph_trees_expandable_segments.py::CudaGraphTreeTests::test_graph_partition_forward_backward <- test/inductor/test_cudagraph_trees.py W1204 11:25:47.022000 109768 site-packages/torch/_inductor/utils.py:2565] [0/0] DeviceCopy in input program 2025-12-04T11:27:13.7075119Z W1204 11:25:47.025000 109768 site-packages/torch/_inductor/utils.py:2565] [0/0] DeviceCopy in input program 2025-12-04T11:27:13.7075816Z PASSED [1.1289s] [ 42%] 2025-12-04T11:27:13.7076864Z inductor/test_cudagraph_trees_expandable_segments.py::CudaGraphTreeTests::test_graph_partition_forward_backward_not_called <- test/inductor/test_cudagraph_trees.py PASSED [0.5591s] [ 43%] 2025-12-04T11:27:13.7079024Z inductor/test_cudagraph_trees_expandable_segments.py::CudaGraphTreeTests::test_graph_partition_forward_with_skipped_cudagraphed_backward <- test/inductor/test_cudagraph_trees.py PASSED [0.4577s] [ 44%] 2025-12-04T11:27:13.7080857Z inductor/test_cudagraph_trees_expandable_segments.py::CudaGraphTreeTests::test_graph_partition_fused_scheduler_node <- test/inductor/test_cudagraph_trees.py PASSED [0.3802s] [ 44%] 2025-12-04T11:27:13.7082641Z inductor/test_cudagraph_trees_expandable_segments.py::CudaGraphTreeTests::test_graph_partition_gc <- test/inductor/test_cudagraph_trees.py PASSED [0.5492s] [ 45%] 2025-12-04T11:27:13.7084299Z inductor/test_cudagraph_trees_expandable_segments.py::CudaGraphTreeTests::test_graph_partition_item <- test/inductor/test_cudagraph_trees.py PASSED [0.3589s] [ 46%] 2025-12-04T11:27:13.7086639Z inductor/test_cudagraph_trees_expandable_segments.py::CudaGraphTreeTests::test_graph_partition_kernel_reuse_autotune_at_compile_time_False <- test/inductor/test_cudagraph_trees.py W1204 11:25:50.424000 109768 site-packages/torch/_inductor/utils.py:2565] [0/0] DeviceCopy in input program 2025-12-04T11:27:13.7088424Z W1204 11:25:50.426000 109768 site-packages/torch/_inductor/utils.py:2565] [0/0] DeviceCopy in input program 2025-12-04T11:27:13.7089334Z W1204 11:25:50.426000 109768 site-packages/torch/_inductor/utils.py:2565] [0/0] DeviceCopy in input program 2025-12-04T11:27:13.7089984Z PASSED [0.8204s] [ 46%] 2025-12-04T11:27:13.7091466Z inductor/test_cudagraph_trees_expandable_segments.py::CudaGraphTreeTests::test_graph_partition_kernel_reuse_autotune_at_compile_time_True <- test/inductor/test_cudagraph_trees.py W1204 11:25:51.243000 109768 site-packages/torch/_inductor/utils.py:2565] [0/0] DeviceCopy in input program 2025-12-04T11:27:13.7093370Z W1204 11:25:51.244000 109768 site-packages/torch/_inductor/utils.py:2565] [0/0] DeviceCopy in input program 2025-12-04T11:27:13.7094365Z W1204 11:25:51.245000 109768 site-packages/torch/_inductor/utils.py:2565] [0/0] DeviceCopy in input program 2025-12-04T11:27:13.7095018Z PASSED [0.8364s] [ 47%] 2025-12-04T11:27:13.7095977Z inductor/test_cudagraph_trees_expandable_segments.py::CudaGraphTreeTests::test_graph_partition_log_message <- test/inductor/test_cudagraph_trees.py PASSED [0.7716s] [ 48%] 2025-12-04T11:27:13.7097995Z inductor/test_cudagraph_trees_expandable_segments.py::CudaGraphTreeTests::test_graph_partition_multiple_devices_msg <- test/inductor/test_cudagraph_trees.py SKIPPED [0.0002s] (requires multiple cuda devices) [ 48%] 2025-12-04T11:27:13.7100145Z inductor/test_cudagraph_trees_expandable_segments.py::CudaGraphTreeTests::test_graph_partition_reduce_overhead_mode_effectiveness <- test/inductor/test_cudagraph_trees.py W1204 11:25:52.856000 109768 site-packages/torch/_inductor/utils.py:2565] [0/0] DeviceCopy in input program 2025-12-04T11:27:13.7101884Z W1204 11:25:52.857000 109768 site-packages/torch/_inductor/utils.py:2565] [0/0] DeviceCopy in input program 2025-12-04T11:27:13.7102529Z PASSED [0.7941s] [ 49%] 2025-12-04T11:27:13.7103185Z inductor/test_cudagraph_trees_expandable_segments.py::CudaGraphTreeTests::test_graph_partition_reorder_cpu_and_gpu <- test/inductor/test_cudagraph_trees.py PASSED [0.8436s] [ 50%] 2025-12-04T11:27:13.7104516Z inductor/test_cudagraph_trees_expandable_segments.py::CudaGraphTreeTests::test_graph_partition_reorder_cpu_and_gpu_interleave <- test/inductor/test_cudagraph_trees.py W1204 11:25:54.542000 109768 site-packages/torch/_inductor/utils.py:2565] [0/0] DeviceCopy in input program 2025-12-04T11:27:13.7105366Z PASSED [0.9109s] [ 50%] 2025-12-04T11:27:13.7105986Z inductor/test_cudagraph_trees_expandable_segments.py::CudaGraphTreeTests::test_graph_partition_reorder_custom_op_with_no_dependency <- test/inductor/test_cudagraph_trees.py PASSED [0.6615s] [ 51%] 2025-12-04T11:27:13.7107192Z inductor/test_cudagraph_trees_expandable_segments.py::CudaGraphTreeTests::test_graph_partition_reorder_custom_op_with_no_dependency1 <- test/inductor/test_cudagraph_trees.py PASSED [0.7130s] [ 51%] 2025-12-04T11:27:13.7108412Z inductor/test_cudagraph_trees_expandable_segments.py::CudaGraphTreeTests::test_graph_partition_simple <- test/inductor/test_cudagraph_trees.py W1204 11:25:56.791000 109768 site-packages/torch/_inductor/utils.py:2565] [0/0] DeviceCopy in input program 2025-12-04T11:27:13.7109361Z W1204 11:25:56.792000 109768 site-packages/torch/_inductor/utils.py:2565] [0/0] DeviceCopy in input program 2025-12-04T11:27:13.7109737Z PASSED [0.8794s] [ 52%] 2025-12-04T11:27:13.7110527Z inductor/test_cudagraph_trees_expandable_segments.py::CudaGraphTreeTests::test_graph_partition_symint <- test/inductor/test_cudagraph_trees.py W1204 11:25:57.669000 109768 site-packages/torch/_inductor/utils.py:2565] [0/0] DeviceCopy in input program 2025-12-04T11:27:13.7111508Z W1204 11:25:57.670000 109768 site-packages/torch/_inductor/utils.py:2565] [0/0] DeviceCopy in input program 2025-12-04T11:27:13.7112320Z W1204 11:25:58.357000 109768 site-packages/torch/_inductor/utils.py:2565] [0/1] DeviceCopy in input program 2025-12-04T11:27:13.7113286Z W1204 11:25:58.359000 109768 site-packages/torch/_inductor/utils.py:2565] [0/1] DeviceCopy in input program 2025-12-04T11:27:13.7113948Z PASSED [1.5579s] [ 53%] 2025-12-04T11:27:13.7114855Z inductor/test_cudagraph_trees_expandable_segments.py::CudaGraphTreeTests::test_graph_partition_symint_cat_backward <- test/inductor/test_cudagraph_trees.py PASSED [0.9866s] [ 53%] 2025-12-04T11:27:13.7115899Z inductor/test_cudagraph_trees_expandable_segments.py::CudaGraphTreeTests::test_graph_partition_symint_from_mutation_index <- test/inductor/test_cudagraph_trees.py PASSED [0.6706s] [ 54%] 2025-12-04T11:27:13.7116962Z inductor/test_cudagraph_trees_expandable_segments.py::CudaGraphTreeTests::test_graph_partition_symint_from_nested_indirect_indexing <- test/inductor/test_cudagraph_trees.py PASSED [0.5365s] [ 55%] 2025-12-04T11:27:13.7118232Z inductor/test_cudagraph_trees_expandable_segments.py::CudaGraphTreeTests::test_graph_partition_unbacked_symint <- test/inductor/test_cudagraph_trees.py W1204 11:26:01.472000 109768 site-packages/torch/_inductor/utils.py:2565] [0/0] DeviceCopy in input program 2025-12-04T11:27:13.7119221Z W1204 11:26:01.473000 109768 site-packages/torch/_inductor/utils.py:2565] [0/0] DeviceCopy in input program 2025-12-04T11:27:13.7119597Z PASSED [0.8916s] [ 55%] 2025-12-04T11:27:13.7120214Z inductor/test_cudagraph_trees_expandable_segments.py::CudaGraphTreeTests::test_graph_partition_unbacked_symint_multi_output_layout <- test/inductor/test_cudagraph_trees.py PASSED [0.7837s] [ 56%] 2025-12-04T11:27:13.7121514Z inductor/test_cudagraph_trees_expandable_segments.py::CudaGraphTreeTests::test_graph_partition_user_defined_triton_kernel_reuse <- test/inductor/test_cudagraph_trees.py W1204 11:26:03.373000 109768 site-packages/torch/_inductor/utils.py:2565] [0/0] DeviceCopy in input program 2025-12-04T11:27:13.7122529Z W1204 11:26:03.374000 109768 site-packages/torch/_inductor/utils.py:2565] [0/0] DeviceCopy in input program 2025-12-04T11:27:13.7122922Z ('RERUN', {'yellow': True}) [1.0940s] [ 57%] 2025-12-04T11:27:13.7123777Z inductor/test_cudagraph_trees_expandable_segments.py::CudaGraphTreeTests::test_graph_partition_user_defined_triton_kernel_reuse <- test/inductor/test_cudagraph_trees.py W1204 11:26:04.244000 109768 site-packages/torch/_inductor/utils.py:2565] [0/0] DeviceCopy in input program 2025-12-04T11:27:13.7124780Z W1204 11:26:04.245000 109768 site-packages/torch/_inductor/utils.py:2565] [0/0] DeviceCopy in input program 2025-12-04T11:27:13.7125166Z ('RERUN', {'yellow': True}) [1.0170s] [ 57%] 2025-12-04T11:27:13.7126470Z inductor/test_cudagraph_trees_expandable_segments.py::CudaGraphTreeTests::test_graph_partition_user_defined_triton_kernel_reuse <- test/inductor/test_cudagraph_trees.py W1204 11:26:05.262000 109768 site-packages/torch/_inductor/utils.py:2565] [0/0] DeviceCopy in input program 2025-12-04T11:27:13.7128357Z W1204 11:26:05.263000 109768 site-packages/torch/_inductor/utils.py:2565] [0/0] DeviceCopy in input program 2025-12-04T11:27:13.7129035Z FAILED [1.0242s] [ 57%] 2025-12-04T11:27:13.7130136Z inductor/test_cudagraph_trees_expandable_segments.py::CudaGraphTreeTests::test_graph_partition_user_defined_triton_kernel_reuse <- test/inductor/test_cudagraph_trees.py ERROR [0.0001s] [ 57%] 2025-12-04T11:27:13.7131186Z 2025-12-04T11:27:13.7131352Z ==================================== RERUNS ==================================== 2025-12-04T11:27:13.7131988Z ___ CudaGraphTreeTests.test_graph_partition_user_defined_triton_kernel_reuse ___ 2025-12-04T11:27:13.7132599Z Traceback (most recent call last): 2025-12-04T11:27:13.7133549Z File "/var/lib/jenkins/workspace/test/inductor/test_cudagraph_trees.py", line 4171, in test_graph_partition_user_defined_triton_kernel_reuse 2025-12-04T11:27:13.7134500Z self.assertEqual(eager_out, compiled_out) 2025-12-04T11:27:13.7135348Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/test_case.py", line 113, in assertEqual 2025-12-04T11:27:13.7136220Z return super().assertEqual(x, y, *args, **kwargs) 2025-12-04T11:27:13.7137113Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 4284, in assertEqual 2025-12-04T11:27:13.7138050Z raise error_metas.pop()[0].to_error( # type: ignore[index] 2025-12-04T11:27:13.7138576Z AssertionError: Tensor-likes are not close! 2025-12-04T11:27:13.7138869Z 2025-12-04T11:27:13.7139000Z Mismatched elements: 64 / 128 (50.0%) 2025-12-04T11:27:13.7139583Z Greatest absolute difference: 2.7803521156311035 at index (65,) (up to 1e-05 allowed) 2025-12-04T11:27:13.7140346Z Greatest relative difference: inf at index (64,) (up to 1.3e-06 allowed) 2025-12-04T11:27:13.7140780Z 2025-12-04T11:27:13.7140916Z The failure occurred for item [0] 2025-12-04T11:27:13.7141162Z 2025-12-04T11:27:13.7141407Z To execute this test, run the following from the base repo dir: 2025-12-04T11:27:13.7142502Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cudagraph_trees.py CudaGraphTreeTests.test_graph_partition_user_defined_triton_kernel_reuse 2025-12-04T11:27:13.7143363Z 2025-12-04T11:27:13.7143759Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:27:13.7144438Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:27:13.7144946Z frames [('total', 1), ('ok', 1)] 2025-12-04T11:27:13.7145339Z stats [('calls_captured', 7), ('unique_graphs', 1)] 2025-12-04T11:27:13.7145988Z aot_autograd [('total', 1), ('autograd_cache_miss', 1), ('autograd_cache_saved', 1), ('ok', 1)] 2025-12-04T11:27:13.7147184Z inductor [('triton_bundler_save_kernel', 8), ('extern_calls', 4), ('pattern_matcher_count', 2), ('pattern_matcher_nodes', 2), ('fxgraph_cache_miss', 1), ('async_compile_cache_miss', 1)] 2025-12-04T11:27:13.7148148Z graph_break [] 2025-12-04T11:27:13.7148545Z ----------------------------- Captured stderr call ----------------------------- 2025-12-04T11:27:13.7149132Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T11:27:13.7149863Z File "/var/lib/jenkins/workspace/test/inductor/test_cudagraph_trees.py", line 4159, in foo 2025-12-04T11:27:13.7150541Z output1_cpu = output1.cpu() + 1 2025-12-04T11:27:13.7150794Z 2025-12-04T11:27:13.7150938Z cudagraph partition due to non gpu ops 2025-12-04T11:27:13.7151428Z cudagraph partition due to DeviceCopy ops. Found from : 2025-12-04T11:27:13.7152174Z File "/var/lib/jenkins/workspace/test/inductor/test_cudagraph_trees.py", line 4161, in foo 2025-12-04T11:27:13.7152829Z x2 = output1_cpu.to("cuda") 2025-12-04T11:27:13.7153068Z 2025-12-04T11:27:13.7153219Z cudagraph partition into 3 partitions 2025-12-04T11:27:13.7153824Z ___ CudaGraphTreeTests.test_graph_partition_user_defined_triton_kernel_reuse ___ 2025-12-04T11:27:13.7154525Z Traceback (most recent call last): 2025-12-04T11:27:13.7155432Z File "/var/lib/jenkins/workspace/test/inductor/test_cudagraph_trees.py", line 4171, in test_graph_partition_user_defined_triton_kernel_reuse 2025-12-04T11:27:13.7156397Z self.assertEqual(eager_out, compiled_out) 2025-12-04T11:27:13.7157178Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/test_case.py", line 113, in assertEqual 2025-12-04T11:27:13.7157977Z return super().assertEqual(x, y, *args, **kwargs) 2025-12-04T11:27:13.7158856Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 4284, in assertEqual 2025-12-04T11:27:13.7159797Z raise error_metas.pop()[0].to_error( # type: ignore[index] 2025-12-04T11:27:13.7160320Z AssertionError: Tensor-likes are not close! 2025-12-04T11:27:13.7160688Z 2025-12-04T11:27:13.7160828Z Mismatched elements: 64 / 128 (50.0%) 2025-12-04T11:27:13.7161429Z Greatest absolute difference: 2.7356221675872803 at index (90,) (up to 1e-05 allowed) 2025-12-04T11:27:13.7162253Z Greatest relative difference: inf at index (64,) (up to 1.3e-06 allowed) 2025-12-04T11:27:13.7162675Z 2025-12-04T11:27:13.7162877Z The failure occurred for item [0] 2025-12-04T11:27:13.7163131Z 2025-12-04T11:27:13.7163361Z To execute this test, run the following from the base repo dir: 2025-12-04T11:27:13.7164457Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cudagraph_trees.py CudaGraphTreeTests.test_graph_partition_user_defined_triton_kernel_reuse 2025-12-04T11:27:13.7165314Z 2025-12-04T11:27:13.7165609Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:27:13.7166279Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:27:13.7166778Z frames [('total', 1), ('ok', 1)] 2025-12-04T11:27:13.7167187Z stats [('calls_captured', 7), ('unique_graphs', 1)] 2025-12-04T11:27:13.7167818Z aot_autograd [('total', 1), ('autograd_cache_miss', 1), ('autograd_cache_saved', 1), ('ok', 1)] 2025-12-04T11:27:13.7169012Z inductor [('triton_bundler_save_kernel', 8), ('extern_calls', 4), ('pattern_matcher_count', 2), ('pattern_matcher_nodes', 2), ('fxgraph_cache_miss', 1), ('async_compile_cache_miss', 1)] 2025-12-04T11:27:13.7169980Z graph_break [] 2025-12-04T11:27:13.7170389Z ----------------------------- Captured stderr call ----------------------------- 2025-12-04T11:27:13.7170971Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T11:27:13.7171706Z File "/var/lib/jenkins/workspace/test/inductor/test_cudagraph_trees.py", line 4159, in foo 2025-12-04T11:27:13.7172394Z output1_cpu = output1.cpu() + 1 2025-12-04T11:27:13.7172648Z 2025-12-04T11:27:13.7172800Z cudagraph partition due to non gpu ops 2025-12-04T11:27:13.7173276Z cudagraph partition due to DeviceCopy ops. Found from : 2025-12-04T11:27:13.7174027Z File "/var/lib/jenkins/workspace/test/inductor/test_cudagraph_trees.py", line 4161, in foo 2025-12-04T11:27:13.7174701Z x2 = output1_cpu.to("cuda") 2025-12-04T11:27:13.7174931Z 2025-12-04T11:27:13.7175086Z cudagraph partition into 3 partitions 2025-12-04T11:27:13.7175576Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:27:13.7176080Z frames [('total', 1), ('ok', 1)] 2025-12-04T11:27:13.7176481Z stats [('calls_captured', 7), ('unique_graphs', 1)] 2025-12-04T11:27:13.7177114Z aot_autograd [('total', 1), ('autograd_cache_miss', 1), ('autograd_cache_saved', 1), ('ok', 1)] 2025-12-04T11:27:13.7178414Z inductor [('triton_bundler_save_kernel', 8), ('extern_calls', 4), ('pattern_matcher_count', 2), ('pattern_matcher_nodes', 2), ('fxgraph_cache_miss', 1), ('async_compile_cache_miss', 1)] 2025-12-04T11:27:13.7178960Z graph_break [] 2025-12-04T11:27:13.7179199Z ----------------------------- Captured stderr call ----------------------------- 2025-12-04T11:27:13.7179523Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T11:27:13.7179935Z File "/var/lib/jenkins/workspace/test/inductor/test_cudagraph_trees.py", line 4159, in foo 2025-12-04T11:27:13.7180447Z output1_cpu = output1.cpu() + 1 2025-12-04T11:27:13.7180592Z 2025-12-04T11:27:13.7180676Z cudagraph partition due to non gpu ops 2025-12-04T11:27:13.7180956Z cudagraph partition due to DeviceCopy ops. Found from : 2025-12-04T11:27:13.7181372Z File "/var/lib/jenkins/workspace/test/inductor/test_cudagraph_trees.py", line 4161, in foo 2025-12-04T11:27:13.7181744Z x2 = output1_cpu.to("cuda") 2025-12-04T11:27:13.7181876Z 2025-12-04T11:27:13.7181958Z cudagraph partition into 3 partitions 2025-12-04T11:27:13.7182215Z ==================================== ERRORS ==================================== 2025-12-04T11:27:13.7182626Z _ ERROR at teardown of CudaGraphTreeTests.test_graph_partition_user_defined_triton_kernel_reuse _ 2025-12-04T11:27:13.7183070Z Traceback (most recent call last): 2025-12-04T11:27:13.7183463Z File "/var/lib/jenkins/workspace/test/inductor/test_cudagraph_trees.py", line 174, in tearDown 2025-12-04T11:27:13.7183971Z self.assertEqual(all_live_block_count(), 0) 2025-12-04T11:27:13.7184486Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/test_case.py", line 113, in assertEqual 2025-12-04T11:27:13.7185001Z return super().assertEqual(x, y, *args, **kwargs) 2025-12-04T11:27:13.7185504Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 4284, in assertEqual 2025-12-04T11:27:13.7186029Z raise error_metas.pop()[0].to_error( # type: ignore[index] 2025-12-04T11:27:13.7186315Z AssertionError: Scalars are not equal! 2025-12-04T11:27:13.7186475Z 2025-12-04T11:27:13.7186544Z Expected 0 but got 2. 2025-12-04T11:27:13.7186724Z Absolute difference: 2 2025-12-04T11:27:13.7186906Z Relative difference: inf 2025-12-04T11:27:13.7187157Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:27:13.7187457Z frames [('total', 1), ('ok', 1)] 2025-12-04T11:27:13.7187692Z stats [('calls_captured', 7), ('unique_graphs', 1)] 2025-12-04T11:27:13.7188047Z aot_autograd [('total', 1), ('autograd_cache_miss', 1), ('autograd_cache_saved', 1), ('ok', 1)] 2025-12-04T11:27:13.7188710Z inductor [('triton_bundler_save_kernel', 8), ('extern_calls', 4), ('pattern_matcher_count', 2), ('pattern_matcher_nodes', 2), ('fxgraph_cache_miss', 1), ('async_compile_cache_miss', 1)] 2025-12-04T11:27:13.7189231Z graph_break [] 2025-12-04T11:27:13.7189458Z ----------------------------- Captured stderr call ----------------------------- 2025-12-04T11:27:13.7189779Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T11:27:13.7190191Z File "/var/lib/jenkins/workspace/test/inductor/test_cudagraph_trees.py", line 4159, in foo 2025-12-04T11:27:13.7190569Z output1_cpu = output1.cpu() + 1 2025-12-04T11:27:13.7190710Z 2025-12-04T11:27:13.7190797Z cudagraph partition due to non gpu ops 2025-12-04T11:27:13.7191080Z cudagraph partition due to DeviceCopy ops. Found from : 2025-12-04T11:27:13.7191492Z File "/var/lib/jenkins/workspace/test/inductor/test_cudagraph_trees.py", line 4161, in foo 2025-12-04T11:27:13.7191863Z x2 = output1_cpu.to("cuda") 2025-12-04T11:27:13.7191990Z 2025-12-04T11:27:13.7192070Z cudagraph partition into 3 partitions 2025-12-04T11:27:13.7192349Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:27:13.7192624Z frames [('total', 1), ('ok', 1)] 2025-12-04T11:27:13.7192840Z stats [('calls_captured', 7), ('unique_graphs', 1)] 2025-12-04T11:27:13.7193194Z aot_autograd [('total', 1), ('autograd_cache_miss', 1), ('autograd_cache_saved', 1), ('ok', 1)] 2025-12-04T11:27:13.7193839Z inductor [('triton_bundler_save_kernel', 8), ('extern_calls', 4), ('pattern_matcher_count', 2), ('pattern_matcher_nodes', 2), ('fxgraph_cache_miss', 1), ('async_compile_cache_miss', 1)] 2025-12-04T11:27:13.7194360Z graph_break [] 2025-12-04T11:27:13.7194585Z ----------------------------- Captured stderr call ----------------------------- 2025-12-04T11:27:13.7194956Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T11:27:13.7195359Z File "/var/lib/jenkins/workspace/test/inductor/test_cudagraph_trees.py", line 4159, in foo 2025-12-04T11:27:13.7195725Z output1_cpu = output1.cpu() + 1 2025-12-04T11:27:13.7195873Z 2025-12-04T11:27:13.7195954Z cudagraph partition due to non gpu ops 2025-12-04T11:27:13.7196221Z cudagraph partition due to DeviceCopy ops. Found from : 2025-12-04T11:27:13.7196630Z File "/var/lib/jenkins/workspace/test/inductor/test_cudagraph_trees.py", line 4161, in foo 2025-12-04T11:27:13.7196988Z x2 = output1_cpu.to("cuda") 2025-12-04T11:27:13.7197120Z 2025-12-04T11:27:13.7197200Z cudagraph partition into 3 partitions 2025-12-04T11:27:13.7197474Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:27:13.7197745Z frames [('total', 1), ('ok', 1)] 2025-12-04T11:27:13.7198013Z stats [('calls_captured', 7), ('unique_graphs', 1)] 2025-12-04T11:27:13.7198367Z aot_autograd [('total', 1), ('autograd_cache_miss', 1), ('autograd_cache_saved', 1), ('ok', 1)] 2025-12-04T11:27:13.7199116Z inductor [('triton_bundler_save_kernel', 8), ('extern_calls', 4), ('pattern_matcher_count', 2), ('pattern_matcher_nodes', 2), ('fxgraph_cache_miss', 1), ('async_compile_cache_miss', 1)] 2025-12-04T11:27:13.7199628Z graph_break [] 2025-12-04T11:27:13.7199854Z ----------------------------- Captured stderr call ----------------------------- 2025-12-04T11:27:13.7200176Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T11:27:13.7200571Z File "/var/lib/jenkins/workspace/test/inductor/test_cudagraph_trees.py", line 4159, in foo 2025-12-04T11:27:13.7200951Z output1_cpu = output1.cpu() + 1 2025-12-04T11:27:13.7201094Z 2025-12-04T11:27:13.7201175Z cudagraph partition due to non gpu ops 2025-12-04T11:27:13.7201458Z cudagraph partition due to DeviceCopy ops. Found from : 2025-12-04T11:27:13.7201864Z File "/var/lib/jenkins/workspace/test/inductor/test_cudagraph_trees.py", line 4161, in foo 2025-12-04T11:27:13.7202232Z x2 = output1_cpu.to("cuda") 2025-12-04T11:27:13.7202355Z 2025-12-04T11:27:13.7202447Z cudagraph partition into 3 partitions 2025-12-04T11:27:13.7202679Z =================================== FAILURES =================================== 2025-12-04T11:27:13.7203046Z ___ CudaGraphTreeTests.test_graph_partition_user_defined_triton_kernel_reuse ___ 2025-12-04T11:27:13.7203384Z Traceback (most recent call last): 2025-12-04T11:27:13.7203890Z File "/var/lib/jenkins/workspace/test/inductor/test_cudagraph_trees.py", line 4171, in test_graph_partition_user_defined_triton_kernel_reuse 2025-12-04T11:27:13.7204411Z self.assertEqual(eager_out, compiled_out) 2025-12-04T11:27:13.7204849Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/test_case.py", line 113, in assertEqual 2025-12-04T11:27:13.7205299Z return super().assertEqual(x, y, *args, **kwargs) 2025-12-04T11:27:13.7205788Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 4284, in assertEqual 2025-12-04T11:27:13.7206295Z raise error_metas.pop()[0].to_error( # type: ignore[index] 2025-12-04T11:27:13.7206588Z AssertionError: Tensor-likes are not close! 2025-12-04T11:27:13.7206747Z 2025-12-04T11:27:13.7206832Z Mismatched elements: 64 / 128 (50.0%) 2025-12-04T11:27:13.7207163Z Greatest absolute difference: 2.709859848022461 at index (126,) (up to 1e-05 allowed) 2025-12-04T11:27:13.7207590Z Greatest relative difference: inf at index (64,) (up to 1.3e-06 allowed) 2025-12-04T11:27:13.7207824Z 2025-12-04T11:27:13.7207901Z The failure occurred for item [0] 2025-12-04T11:27:13.7208035Z 2025-12-04T11:27:13.7208168Z To execute this test, run the following from the base repo dir: 2025-12-04T11:27:13.7208759Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cudagraph_trees.py CudaGraphTreeTests.test_graph_partition_user_defined_triton_kernel_reuse 2025-12-04T11:27:13.7209224Z 2025-12-04T11:27:13.7209382Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:27:13.7209811Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:27:13.7210095Z frames [('total', 1), ('ok', 1)] 2025-12-04T11:27:13.7210319Z stats [('calls_captured', 7), ('unique_graphs', 1)] 2025-12-04T11:27:13.7210679Z aot_autograd [('total', 1), ('autograd_cache_miss', 1), ('autograd_cache_saved', 1), ('ok', 1)] 2025-12-04T11:27:13.7211326Z inductor [('triton_bundler_save_kernel', 8), ('extern_calls', 4), ('pattern_matcher_count', 2), ('pattern_matcher_nodes', 2), ('fxgraph_cache_miss', 1), ('async_compile_cache_miss', 1)] 2025-12-04T11:27:13.7211855Z graph_break [] 2025-12-04T11:27:13.7212079Z ----------------------------- Captured stderr call ----------------------------- 2025-12-04T11:27:13.7212405Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T11:27:13.7212854Z File "/var/lib/jenkins/workspace/test/inductor/test_cudagraph_trees.py", line 4159, in foo 2025-12-04T11:27:13.7213235Z output1_cpu = output1.cpu() + 1 2025-12-04T11:27:13.7213382Z 2025-12-04T11:27:13.7213499Z cudagraph partition due to non gpu ops 2025-12-04T11:27:13.7213773Z cudagraph partition due to DeviceCopy ops. Found from : 2025-12-04T11:27:13.7214226Z File "/var/lib/jenkins/workspace/test/inductor/test_cudagraph_trees.py", line 4161, in foo 2025-12-04T11:27:13.7214597Z x2 = output1_cpu.to("cuda") 2025-12-04T11:27:13.7214730Z 2025-12-04T11:27:13.7214811Z cudagraph partition into 3 partitions 2025-12-04T11:27:13.7215089Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:27:13.7215361Z frames [('total', 1), ('ok', 1)] 2025-12-04T11:27:13.7215588Z stats [('calls_captured', 7), ('unique_graphs', 1)] 2025-12-04T11:27:13.7215944Z aot_autograd [('total', 1), ('autograd_cache_miss', 1), ('autograd_cache_saved', 1), ('ok', 1)] 2025-12-04T11:27:13.7216595Z inductor [('triton_bundler_save_kernel', 8), ('extern_calls', 4), ('pattern_matcher_count', 2), ('pattern_matcher_nodes', 2), ('fxgraph_cache_miss', 1), ('async_compile_cache_miss', 1)] 2025-12-04T11:27:13.7217124Z graph_break [] 2025-12-04T11:27:13.7217355Z ----------------------------- Captured stderr call ----------------------------- 2025-12-04T11:27:13.7217680Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T11:27:13.7218074Z File "/var/lib/jenkins/workspace/test/inductor/test_cudagraph_trees.py", line 4159, in foo 2025-12-04T11:27:13.7218449Z output1_cpu = output1.cpu() + 1 2025-12-04T11:27:13.7218589Z 2025-12-04T11:27:13.7218689Z cudagraph partition due to non gpu ops 2025-12-04T11:27:13.7218961Z cudagraph partition due to DeviceCopy ops. Found from : 2025-12-04T11:27:13.7219366Z File "/var/lib/jenkins/workspace/test/inductor/test_cudagraph_trees.py", line 4161, in foo 2025-12-04T11:27:13.7219732Z x2 = output1_cpu.to("cuda") 2025-12-04T11:27:13.7219859Z 2025-12-04T11:27:13.7219949Z cudagraph partition into 3 partitions 2025-12-04T11:27:13.7220219Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T11:27:13.7220499Z frames [('total', 1), ('ok', 1)] 2025-12-04T11:27:13.7220728Z stats [('calls_captured', 7), ('unique_graphs', 1)] 2025-12-04T11:27:13.7221078Z aot_autograd [('total', 1), ('autograd_cache_miss', 1), ('autograd_cache_saved', 1), ('ok', 1)] 2025-12-04T11:27:13.7221723Z inductor [('triton_bundler_save_kernel', 8), ('extern_calls', 4), ('pattern_matcher_count', 2), ('pattern_matcher_nodes', 2), ('fxgraph_cache_miss', 1), ('async_compile_cache_miss', 1)] 2025-12-04T11:27:13.7222244Z graph_break [] 2025-12-04T11:27:13.7222469Z ----------------------------- Captured stderr call ----------------------------- 2025-12-04T11:27:13.7222779Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T11:27:13.7223180Z File "/var/lib/jenkins/workspace/test/inductor/test_cudagraph_trees.py", line 4159, in foo 2025-12-04T11:27:13.7223626Z output1_cpu = output1.cpu() + 1 2025-12-04T11:27:13.7223766Z 2025-12-04T11:27:13.7223849Z cudagraph partition due to non gpu ops 2025-12-04T11:27:13.7224175Z cudagraph partition due to DeviceCopy ops. Found from : 2025-12-04T11:27:13.7224588Z File "/var/lib/jenkins/workspace/test/inductor/test_cudagraph_trees.py", line 4161, in foo 2025-12-04T11:27:13.7224952Z x2 = output1_cpu.to("cuda") 2025-12-04T11:27:13.7225076Z 2025-12-04T11:27:13.7225155Z cudagraph partition into 3 partitions 2025-12-04T11:27:13.7225876Z - generated xml file: /var/lib/jenkins/workspace/test/test-reports/python-pytest/inductor.test_cudagraph_trees_expandable_segments/inductor.test_cudagraph_trees_expandable_segments-e8ff2a368a15c33d.xml - 2025-12-04T11:27:13.7226627Z =========================== short test summary info ============================ 2025-12-04T11:27:13.7227343Z FAILED [1.0242s] inductor/test_cudagraph_trees_expandable_segments.py::CudaGraphTreeTests::test_graph_partition_user_defined_triton_kernel_reuse - AssertionError: Tensor-likes are not close! 2025-12-04T11:27:13.7227899Z 2025-12-04T11:27:13.7227974Z Mismatched elements: 64 / 128 (50.0%) 2025-12-04T11:27:13.7228298Z Greatest absolute difference: 2.709859848022461 at index (126,) (up to 1e-05 allowed) 2025-12-04T11:27:13.7228769Z Greatest relative difference: inf at index (64,) (up to 1.3e-06 allowed) 2025-12-04T11:27:13.7229035Z 2025-12-04T11:27:13.7229118Z The failure occurred for item [0] 2025-12-04T11:27:13.7229254Z 2025-12-04T11:27:13.7229381Z To execute this test, run the following from the base repo dir: 2025-12-04T11:27:13.7229980Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cudagraph_trees.py CudaGraphTreeTests.test_graph_partition_user_defined_triton_kernel_reuse 2025-12-04T11:27:13.7230443Z 2025-12-04T11:27:13.7230601Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T11:27:13.7231306Z ERROR [0.0001s] inductor/test_cudagraph_trees_expandable_segments.py::CudaGraphTreeTests::test_graph_partition_user_defined_triton_kernel_reuse - AssertionError: Scalars are not equal! 2025-12-04T11:27:13.7231847Z 2025-12-04T11:27:13.7231915Z Expected 0 but got 2. 2025-12-04T11:27:13.7232091Z Absolute difference: 2 2025-12-04T11:27:13.7232276Z Relative difference: inf 2025-12-04T11:27:13.7232504Z !!!!!!!!!!!!!!!!!!!!!!!!!! stopping after 2 failures !!!!!!!!!!!!!!!!!!!!!!!!!!! 2025-12-04T11:27:13.7232842Z ===== 1 failed, 84 passed, 2 skipped, 1 error, 2 rerun in 63.13s (0:01:03) ===== 2025-12-04T11:27:13.7233120Z Got exit code 1 2025-12-04T11:27:13.7233285Z Retrying single test... 2025-12-04T11:27:13.7233959Z /var/lib/jenkins/workspace/test/inductor/test_cudagraph_trees_expandable_segments.py:43: FutureWarning: torch.cuda._set_allocator_settings is deprecated. Use torch._C._accelerator_setAllocatorSettings instead. 2025-12-04T11:27:13.7234747Z torch.cuda.memory._set_allocator_settings("expandable_segments:True") 2025-12-04T11:27:13.7235299Z Downloading https://ossci-metrics.s3.amazonaws.com/disabled-tests-condensed.json to ./.pytorch-disabled-tests.json 2025-12-04T11:27:13.7236174Z Test results will be stored in test-reports/python-pytest/inductor.test_cudagraph_trees_expandable_segments/inductor.test_cudagraph_trees_expandable_segments-99f752af3a8e76fa.xml 2025-12-04T11:27:13.7236831Z ============================= test session starts ============================== 2025-12-04T11:27:13.7237230Z platform linux -- Python 3.10.14, pytest-7.3.2, pluggy-1.6.0 -- /opt/conda/envs/py_3.10/bin/python 2025-12-04T11:27:13.7237585Z cachedir: .pytest_cache 2025-12-04T11:27:13.7238012Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow] 2025-12-04T11:27:13.7238465Z rootdir: /var/lib/jenkins/workspace 2025-12-04T11:27:13.7238756Z configfile: pytest.ini 2025-12-04T11:27:13.7239483Z plugins: hypothesis-6.56.4, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-14.0, subtests-0.13.1, xdist-3.3.1, xdoctest-1.3.0, anyio-4.12.0, typeguard-4.3.0 2025-12-04T11:27:13.7248310Z collecting ... collected 152 items / 151 deselected / 1 selected 2025-12-04T11:27:13.7249133Z stepcurrent: skipping 86 already run items. Running only test/inductor/test_cudagraph_trees_expandable_segments.py::CudaGraphTreeTests::test_graph_partition_user_defined_triton_kernel_reuse 2025-12-04T11:27:13.7249886Z Running 1 items in this shard 2025-12-04T11:27:13.7250031Z 2025-12-04T11:27:13.7250767Z inductor/test_cudagraph_trees_expandable_segments.py::CudaGraphTreeTests::test_graph_partition_user_defined_triton_kernel_reuse <- test/inductor/test_cudagraph_trees.py W1204 11:26:14.573000 112196 site-packages/torch/_inductor/utils.py:2565] [0/0] DeviceCopy in input program 2025-12-04T11:27:13.7251825Z W1204 11:26:14.575000 112196 site-packages/torch/_inductor/utils.py:2565] [0/0] DeviceCopy in input program 2025-12-04T11:27:13.7252212Z PASSED [2.6137s] [100%] 2025-12-04T11:27:13.7252340Z 2025-12-04T11:27:13.7252993Z - generated xml file: /var/lib/jenkins/workspace/test/test-reports/python-pytest/inductor.test_cudagraph_trees_expandable_segments/inductor.test_cudagraph_trees_expandable_segments-99f752af3a8e76fa.xml - 2025-12-04T11:27:13.7253783Z ====================== 1 passed, 151 deselected in 2.64s ======================= 2025-12-04T11:27:13.7254092Z Got exit code 0 2025-12-04T11:27:13.7254340Z Test succeeded in new process, continuing with the rest of the tests 2025-12-04T11:27:13.7255185Z /var/lib/jenkins/workspace/test/inductor/test_cudagraph_trees_expandable_segments.py:43: FutureWarning: torch.cuda._set_allocator_settings is deprecated. Use torch._C._accelerator_setAllocatorSettings instead. 2025-12-04T11:27:13.7255989Z torch.cuda.memory._set_allocator_settings("expandable_segments:True") 2025-12-04T11:27:13.7256533Z Downloading https://ossci-metrics.s3.amazonaws.com/disabled-tests-condensed.json to ./.pytorch-disabled-tests.json 2025-12-04T11:27:13.7257417Z Test results will be stored in test-reports/python-pytest/inductor.test_cudagraph_trees_expandable_segments/inductor.test_cudagraph_trees_expandable_segments-3b9cbca8ed6c6a9f.xml 2025-12-04T11:27:13.7258091Z ============================= test session starts ============================== 2025-12-04T11:27:13.7258511Z platform linux -- Python 3.10.14, pytest-7.3.2, pluggy-1.6.0 -- /opt/conda/envs/py_3.10/bin/python 2025-12-04T11:27:13.7258873Z cachedir: .pytest_cache 2025-12-04T11:27:13.7259303Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow] 2025-12-04T11:27:13.7259771Z rootdir: /var/lib/jenkins/workspace 2025-12-04T11:27:13.7259987Z configfile: pytest.ini 2025-12-04T11:27:13.7260439Z plugins: hypothesis-6.56.4, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-14.0, subtests-0.13.1, xdist-3.3.1, xdoctest-1.3.0, anyio-4.12.0, typeguard-4.3.0 2025-12-04T11:27:13.7260994Z collecting ... collected 152 items / 87 deselected / 65 selected 2025-12-04T11:27:13.7261296Z stepcurrent: skipping 87 already run items. 2025-12-04T11:27:13.7261536Z Running 65 items in this shard 2025-12-04T11:27:13.7261669Z 2025-12-04T11:27:13.7262345Z inductor/test_cudagraph_trees_expandable_segments.py::CudaGraphTreeTests::test_graph_partition_view_fallback <- test/inductor/test_cudagraph_trees.py W1204 11:26:24.572000 112464 site-packages/torch/_inductor/utils.py:2565] [0/0] DeviceCopy in input program 2025-12-04T11:27:13.7263317Z W1204 11:26:24.573000 112464 site-packages/torch/_inductor/utils.py:2565] [0/0] DeviceCopy in input program 2025-12-04T11:27:13.7263806Z PASSED [1.7335s] [ 1%] 2025-12-04T11:27:13.7264634Z inductor/test_cudagraph_trees_expandable_segments.py::CudaGraphTreeTests::test_graph_partition_with_memory_plan_reuse <- test/inductor/test_cudagraph_trees.py W1204 11:26:25.567000 112464 site-packages/torch/_inductor/utils.py:1703] [0/0] Not enough SMs to use max_autotune_gemm mode 2025-12-04T11:27:13.7265489Z PASSED [1.4051s] [ 3%] 2025-12-04T11:27:13.7266041Z inductor/test_cudagraph_trees_expandable_segments.py::CudaGraphTreeTests::test_incompatible_cudagraph_ops_item <- test/inductor/test_cudagraph_trees.py PASSED [0.2461s] [ 4%] 2025-12-04T11:27:13.7267023Z inductor/test_cudagraph_trees_expandable_segments.py::CudaGraphTreeTests::test_incompatible_cudagraph_ops_nonzero <- test/inductor/test_cudagraph_trees.py PASSED [0.2886s] [ 6%] 2025-12-04T11:27:13.7268093Z inductor/test_cudagraph_trees_expandable_segments.py::CudaGraphTreeTests::test_incompatible_cudagraph_ops_nonzero_backend <- test/inductor/test_cudagraph_trees.py PASSED [0.2218s] [ 7%] 2025-12-04T11:27:13.7269138Z inductor/test_cudagraph_trees_expandable_segments.py::CudaGraphTreeTests::test_incompatible_cudagraph_ops_nonzero_graph_breaks <- test/inductor/test_cudagraph_trees.py PASSED [0.4984s] [ 9%] 2025-12-04T11:27:13.7270092Z inductor/test_cudagraph_trees_expandable_segments.py::CudaGraphTreeTests::test_index_put <- test/inductor/test_cudagraph_trees.py PASSED [0.6564s] [ 10%] 2025-12-04T11:27:13.7271033Z inductor/test_cudagraph_trees_expandable_segments.py::CudaGraphTreeTests::test_live_outputs_multiple_graphs <- test/inductor/test_cudagraph_trees.py PASSED [0.9603s] [ 12%] 2025-12-04T11:27:13.7272056Z inductor/test_cudagraph_trees_expandable_segments.py::CudaGraphTreeTests::test_manager_per_device <- test/inductor/test_cudagraph_trees.py SKIPPED [0.0003s] (requires multiple cuda devices) [ 13%] 2025-12-04T11:27:13.7273105Z inductor/test_cudagraph_trees_expandable_segments.py::CudaGraphTreeTests::test_mark_step <- test/inductor/test_cudagraph_trees.py PASSED [0.5584s] [ 15%] 2025-12-04T11:27:13.7273967Z inductor/test_cudagraph_trees_expandable_segments.py::CudaGraphTreeTests::test_meta_tensor <- test/inductor/test_cudagraph_trees.py PASSED [0.5524s] [ 16%] 2025-12-04T11:27:13.7274872Z inductor/test_cudagraph_trees_expandable_segments.py::CudaGraphTreeTests::test_multi_dispatch_child_node <- test/inductor/test_cudagraph_trees.py PASSED [0.9209s] [ 18%] 2025-12-04T11:27:13.7275813Z inductor/test_cudagraph_trees_expandable_segments.py::CudaGraphTreeTests::test_multi_dispatch_custom_module <- test/inductor/test_cudagraph_trees.py PASSED [0.6802s] [ 20%] 2025-12-04T11:27:13.7276781Z inductor/test_cudagraph_trees_expandable_segments.py::CudaGraphTreeTests::test_multi_dispatch_custom_module_buffer <- test/inductor/test_cudagraph_trees.py PASSED [0.9153s] [ 21%] 2025-12-04T11:27:13.7277744Z inductor/test_cudagraph_trees_expandable_segments.py::CudaGraphTreeTests::test_multi_dispatch_parent_node <- test/inductor/test_cudagraph_trees.py PASSED [0.9551s] [ 23%] 2025-12-04T11:27:13.7279020Z inductor/test_cudagraph_trees_expandable_segments.py::CudaGraphTreeTests::test_multi_dispatch_single_compile_builtin_module <- test/inductor/test_cudagraph_trees.py PASSED [0.4538s] [ 24%] 2025-12-04T11:27:13.7280077Z inductor/test_cudagraph_trees_expandable_segments.py::CudaGraphTreeTests::test_multi_dispatch_single_compile_builtin_module_buffers <- test/inductor/test_cudagraph_trees.py PASSED [0.7306s] [ 26%] 2025-12-04T11:27:13.7281125Z inductor/test_cudagraph_trees_expandable_segments.py::CudaGraphTreeTests::test_multi_dispatch_single_compile_param_inputs <- test/inductor/test_cudagraph_trees.py PASSED [0.3975s] [ 27%] 2025-12-04T11:27:13.7282851Z inductor/test_cudagraph_trees_expandable_segments.py::CudaGraphTreeTests::test_multinomial <- test/inductor/test_cudagraph_trees.py SKIPPED [0.0007s] (Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/166682 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests.) [ 29%] 2025-12-04T11:27:13.7284651Z inductor/test_cudagraph_trees_expandable_segments.py::CudaGraphTreeTests::test_multiple_devices_msg_backend_cudagraphs <- test/inductor/test_cudagraph_trees.py SKIPPED [0.0002s] (requires multiple cuda devices) [ 30%] 2025-12-04T11:27:13.7285848Z inductor/test_cudagraph_trees_expandable_segments.py::CudaGraphTreeTests::test_multiple_devices_msg_backend_inductor <- test/inductor/test_cudagraph_trees.py SKIPPED [0.0002s] (requires multiple cuda devices) [ 32%] 2025-12-04T11:27:13.7286927Z inductor/test_cudagraph_trees_expandable_segments.py::CudaGraphTreeTests::test_multiple_insert_removal_caching <- test/inductor/test_cudagraph_trees.py PASSED [0.1541s] [ 33%] 2025-12-04T11:27:13.7288080Z inductor/test_cudagraph_trees_expandable_segments.py::CudaGraphTreeTests::test_mutation_cudagraph_managed_tensor_warn_backend_cudagraphs <- test/inductor/test_cudagraph_trees.py PASSED [0.2595s] [ 35%] 2025-12-04T11:27:13.7289180Z inductor/test_cudagraph_trees_expandable_segments.py::CudaGraphTreeTests::test_mutation_cudagraph_managed_tensor_warn_backend_inductor <- test/inductor/test_cudagraph_trees.py PASSED [0.4535s] [ 36%] 2025-12-04T11:27:13.7290305Z inductor/test_cudagraph_trees_expandable_segments.py::CudaGraphTreeTests::test_mutation_cudagraph_managed_tensor_warn_only_once_backend_cudagraphs <- test/inductor/test_cudagraph_trees.py PASSED [0.2585s] [ 38%] 2025-12-04T11:27:13.7291527Z inductor/test_cudagraph_trees_expandable_segments.py::CudaGraphTreeTests::test_mutation_cudagraph_managed_tensor_warn_only_once_backend_inductor <- test/inductor/test_cudagraph_trees.py PASSED [0.4580s] [ 40%] 2025-12-04T11:27:13.7292665Z inductor/test_cudagraph_trees_expandable_segments.py::CudaGraphTreeTests::test_mutation_cudagraph_managed_tensors_backend_cudagraphs <- test/inductor/test_cudagraph_trees.py PASSED [0.2664s] [ 41%] 2025-12-04T11:27:13.7293849Z inductor/test_cudagraph_trees_expandable_segments.py::CudaGraphTreeTests::test_mutation_cudagraph_managed_tensors_backend_inductor <- test/inductor/test_cudagraph_trees.py PASSED [0.4421s] [ 43%] 2025-12-04T11:27:13.7294949Z inductor/test_cudagraph_trees_expandable_segments.py::CudaGraphTreeTests::test_mutation_cudagraph_managed_tensors_config_backend_cudagraphs <- test/inductor/test_cudagraph_trees.py PASSED [0.2589s] [ 44%] 2025-12-04T11:27:13.7296072Z inductor/test_cudagraph_trees_expandable_segments.py::CudaGraphTreeTests::test_mutation_cudagraph_managed_tensors_config_backend_inductor <- test/inductor/test_cudagraph_trees.py PASSED [0.4320s] [ 46%] 2025-12-04T11:27:13.7297122Z inductor/test_cudagraph_trees_expandable_segments.py::CudaGraphTreeTests::test_mutation_on_inp_backend_cudagraphs <- test/inductor/test_cudagraph_trees.py PASSED [0.2678s] [ 47%] 2025-12-04T11:27:13.7298098Z inductor/test_cudagraph_trees_expandable_segments.py::CudaGraphTreeTests::test_mutation_on_inp_backend_inductor <- test/inductor/test_cudagraph_trees.py PASSED [0.4841s] [ 49%] 2025-12-04T11:27:13.7299037Z inductor/test_cudagraph_trees_expandable_segments.py::CudaGraphTreeTests::test_mutation_reinplaced <- test/inductor/test_cudagraph_trees.py PASSED [0.3587s] [ 50%] 2025-12-04T11:27:13.7299984Z inductor/test_cudagraph_trees_expandable_segments.py::CudaGraphTreeTests::test_no_rerecord_with_mark_static_address <- test/inductor/test_cudagraph_trees.py PASSED [0.5920s] [ 52%] 2025-12-04T11:27:13.7301017Z inductor/test_cudagraph_trees_expandable_segments.py::CudaGraphTreeTests::test_not_fallback_to_eager_if_have_not_recompiling_too_many_times <- test/inductor/test_cudagraph_trees.py PASSED [0.4020s] [ 53%] 2025-12-04T11:27:13.7301992Z inductor/test_cudagraph_trees_expandable_segments.py::CudaGraphTreeTests::test_output_alias <- test/inductor/test_cudagraph_trees.py PASSED [0.1670s] [ 55%] 2025-12-04T11:27:13.7302891Z inductor/test_cudagraph_trees_expandable_segments.py::CudaGraphTreeTests::test_peristed_output_livenes <- test/inductor/test_cudagraph_trees.py PASSED [0.3059s] [ 56%] 2025-12-04T11:27:13.7303906Z inductor/test_cudagraph_trees_expandable_segments.py::CudaGraphTreeTests::test_remove_hooks_on_cached_tensors <- test/inductor/test_cudagraph_trees.py PASSED [0.3402s] [ 58%] 2025-12-04T11:27:13.7304878Z inductor/test_cudagraph_trees_expandable_segments.py::CudaGraphTreeTests::test_rerecord_if_static_input_address_changed <- test/inductor/test_cudagraph_trees.py PASSED [0.4027s] [ 60%] 2025-12-04T11:27:13.7305814Z inductor/test_cudagraph_trees_expandable_segments.py::CudaGraphTreeTests::test_rng_non_trees <- test/inductor/test_cudagraph_trees.py PASSED [0.2427s] [ 61%] 2025-12-04T11:27:13.7306659Z inductor/test_cudagraph_trees_expandable_segments.py::CudaGraphTreeTests::test_rng_trees <- test/inductor/test_cudagraph_trees.py PASSED [0.2416s] [ 63%] 2025-12-04T11:27:13.7307556Z inductor/test_cudagraph_trees_expandable_segments.py::CudaGraphTreeTests::test_run_simple <- test/inductor/test_cudagraph_trees.py PASSED [0.6178s] [ 64%] 2025-12-04T11:27:13.7308422Z inductor/test_cudagraph_trees_expandable_segments.py::CudaGraphTreeTests::test_separate_recordings <- test/inductor/test_cudagraph_trees.py PASSED [0.5721s] [ 66%] 2025-12-04T11:27:13.7309346Z inductor/test_cudagraph_trees_expandable_segments.py::CudaGraphTreeTests::test_side_stream_memory_allocation <- test/inductor/test_cudagraph_trees.py PASSED [0.1812s] [ 67%] 2025-12-04T11:27:13.7310256Z inductor/test_cudagraph_trees_expandable_segments.py::CudaGraphTreeTests::test_single_stream_use <- test/inductor/test_cudagraph_trees.py PASSED [0.4669s] [ 69%] 2025-12-04T11:27:13.7311179Z inductor/test_cudagraph_trees_expandable_segments.py::CudaGraphTreeTests::test_skip_cpp_wrapper <- test/inductor/test_cudagraph_trees.py PASSED [1.8561s] [ 70%] 2025-12-04T11:27:13.7312093Z inductor/test_cudagraph_trees_expandable_segments.py::CudaGraphTreeTests::test_skip_cudagraph_unsafe_ops <- test/inductor/test_cudagraph_trees.py PASSED [0.3536s] [ 72%] 2025-12-04T11:27:13.7313109Z inductor/test_cudagraph_trees_expandable_segments.py::CudaGraphTreeTests::test_skip_if_dynamic_shape_limit_reached1 <- test/inductor/test_cudagraph_trees.py PASSED [0.7871s] [ 73%] 2025-12-04T11:27:13.7314091Z inductor/test_cudagraph_trees_expandable_segments.py::CudaGraphTreeTests::test_skip_if_dynamic_shape_limit_reached2 <- test/inductor/test_cudagraph_trees.py PASSED [8.2304s] [ 75%] 2025-12-04T11:27:13.7315009Z inductor/test_cudagraph_trees_expandable_segments.py::CudaGraphTreeTests::test_skip_symbolic <- test/inductor/test_cudagraph_trees.py PASSED [0.3655s] [ 76%] 2025-12-04T11:27:13.7315862Z inductor/test_cudagraph_trees_expandable_segments.py::CudaGraphTreeTests::test_sparsity <- test/inductor/test_cudagraph_trees.py PASSED [0.2605s] [ 78%] 2025-12-04T11:27:13.7316770Z inductor/test_cudagraph_trees_expandable_segments.py::CudaGraphTreeTests::test_static_inputs_address_mutation_log <- test/inductor/test_cudagraph_trees.py PASSED [0.4483s] [ 80%] 2025-12-04T11:27:13.7317710Z inductor/test_cudagraph_trees_expandable_segments.py::CudaGraphTreeTests::test_storage_access_error <- test/inductor/test_cudagraph_trees.py PASSED [0.1965s] [ 81%] 2025-12-04T11:27:13.7318622Z inductor/test_cudagraph_trees_expandable_segments.py::CudaGraphTreeTests::test_tensor_constant_mutation <- test/inductor/test_cudagraph_trees.py PASSED [0.3822s] [ 83%] 2025-12-04T11:27:13.7319564Z inductor/test_cudagraph_trees_expandable_segments.py::CudaGraphTreeTests::test_tensor_dies_between_checkpoint <- test/inductor/test_cudagraph_trees.py PASSED [0.2053s] [ 84%] 2025-12-04T11:27:13.7320502Z inductor/test_cudagraph_trees_expandable_segments.py::CudaGraphTreeTests::test_tensor_no_longer_in_pool <- test/inductor/test_cudagraph_trees.py PASSED [0.2059s] [ 86%] 2025-12-04T11:27:13.7321448Z inductor/test_cudagraph_trees_expandable_segments.py::CudaGraphTreeTests::test_unaligned_static_input_no_cudagraphs <- test/inductor/test_cudagraph_trees.py PASSED [0.2897s] [ 87%] 2025-12-04T11:27:13.7322427Z inductor/test_cudagraph_trees_expandable_segments.py::CudaGraphTreeTests::test_unaligned_static_input_non_trees <- test/inductor/test_cudagraph_trees.py PASSED [0.2889s] [ 89%] 2025-12-04T11:27:13.7323392Z inductor/test_cudagraph_trees_expandable_segments.py::CudaGraphTreeTests::test_unaligned_static_input_trees <- test/inductor/test_cudagraph_trees.py PASSED [0.2912s] [ 90%] 2025-12-04T11:27:13.7324331Z inductor/test_cudagraph_trees_expandable_segments.py::CudaGraphTreeTests::test_unaligned_static_parameter <- test/inductor/test_cudagraph_trees.py PASSED [0.2087s] [ 92%] 2025-12-04T11:27:13.7325225Z inductor/test_cudagraph_trees_expandable_segments.py::CudaGraphTreeTests::test_unstable_ptr <- test/inductor/test_cudagraph_trees.py PASSED [0.3412s] [ 93%] 2025-12-04T11:27:13.7326091Z inductor/test_cudagraph_trees_expandable_segments.py::CudaGraphTreeTests::test_warmup_stream_sync <- test/inductor/test_cudagraph_trees.py PASSED [1.8800s] [ 95%] 2025-12-04T11:27:13.7327065Z inductor/test_cudagraph_trees_expandable_segments.py::CudaGraphTreeTests::test_warn_on_pending_backward <- test/inductor/test_cudagraph_trees.py PASSED [0.3616s] [ 96%] 2025-12-04T11:27:13.7328012Z inductor/test_cudagraph_trees_expandable_segments.py::CudaGraphTreeTests::test_warn_once_if_dynamic_shape_limit_reached <- test/inductor/test_cudagraph_trees.py PASSED [0.8556s] [ 98%] 2025-12-04T11:27:13.7329246Z inductor/test_cudagraph_trees_expandable_segments.py::CudaGraphTreeTests::test_workspace_allocation_error <- test/inductor/test_cudagraph_trees.py [W1204 11:27:02.124189315 unwind.cpp:219] Warning: Unsupported unwinding pattern: Address not in range (function unwinderFor) 2025-12-04T11:27:13.7330136Z PASSED [9.8199s] [100%] 2025-12-04T11:27:13.7330254Z 2025-12-04T11:27:13.7330832Z - generated xml file: /var/lib/jenkins/workspace/test/test-reports/python-pytest/inductor.test_cudagraph_trees_expandable_segments/inductor.test_cudagraph_trees_expandable_segments-3b9cbca8ed6c6a9f.xml - 2025-12-04T11:27:13.7331656Z ================ 61 passed, 4 skipped, 87 deselected in 48.55s ================= 2025-12-04T11:27:13.7332424Z The following tests failed and then succeeded when run in a new process['test/inductor/test_cudagraph_trees_expandable_segments.py::CudaGraphTreeTests::test_graph_partition_user_defined_triton_kernel_reuse'] 2025-12-04T11:27:13.7333020Z 2025-12-04T11:27:13.7333487Z FINISHED PRINTING LOG FILE of inductor/test_cudagraph_trees_expandable_segments 1/1 (test/test-reports/inductor.test_cudagraph_trees_expandable_segments_1.1_7984e53c82f556c8_.log) 2025-12-04T11:27:13.7334015Z 2025-12-04T11:27:13.7334300Z Finished inductor/test_cudagraph_trees_expandable_segments 1/1 ... [2025-12-04 11:27:13.671691][6481.613905257], took 2.26min 2025-12-04T11:27:13.7335247Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/inductor.test_cudagraph_trees_expandable_segments/inductor.test_cudagraph_trees_expandable_segments-e8ff2a368a15c33d.xml 2025-12-04T11:27:13.7926460Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/inductor.test_cudagraph_trees_expandable_segments/inductor.test_cudagraph_trees_expandable_segments-99f752af3a8e76fa.xml 2025-12-04T11:27:13.8223695Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/inductor.test_cudagraph_trees_expandable_segments/inductor.test_cudagraph_trees_expandable_segments-3b9cbca8ed6c6a9f.xml 2025-12-04T11:27:13.8544107Z Running test_autograd 1/1 ... [2025-12-04 11:27:13.854183][6481.796400888] 2025-12-04T11:27:13.8544531Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2025-12-04T11:27:13.8547588Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'test_autograd.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-12-04 11:27:13.854504] 2025-12-04T11:28:13.1124075Z 2025-12-04T11:28:13.1125016Z test_autograd 1/1 was successful, full logs can be found in artifacts with path test/test-reports/test_autograd_1.1_afae20be844dcfb4_.log 2025-12-04T11:28:13.1275690Z Running 659 items in this shard: test/test_autograd.py::TestAutograd::test_access_saved_tensor_twice_without_recomputation_works, test/test_autograd.py::TestAutograd::test_accumulate_grad, test/test_autograd.py::TestAutograd::test_accumulate_grad_posthooks_can_observe_tensor_prehook, test/test_autograd.py::TestAutograd::test_accumulate_grad_posthooks_should_not_execute, test/test_autograd.py::TestAutograd::test_accumulate_grad_tensor_reference, test/test_autograd.py::TestAutograd::test_accumulate_grad_with_zero_numel_grad, test/test_autograd.py::TestAutograd::test_anomaly_assign_parent_cleanup, test/test_autograd.py::TestAutograd::test_anomaly_detect_nan, test/test_autograd.py::TestAutograd::test_anomaly_grad_warnings, test/test_autograd.py::TestAutograd::test_anomaly_mode_no_check_nan, test/test_autograd.py::TestAutograd::test_attribute_deletion, test/test_autograd.py::TestAutograd::test_autograd_inplace_view_of_view, test/test_autograd.py::TestAutograd::test_autograd_inplace_views_creation_meta, test/test_autograd.py::TestAutograd::test_autograd_inplace_views_cross_dtype, test/test_autograd.py::TestAutograd::test_autograd_multiple_views_python, test/test_autograd.py::TestAutograd::test_autograd_node_isinstance, test/test_autograd.py::TestAutograd::test_autograd_print_tensor, test/test_autograd.py::TestAutograd::test_autograd_python_custom_function_inplace, test/test_autograd.py::TestAutograd::test_autograd_simple_views_python, test/test_autograd.py::TestAutograd::test_autograd_views_codegen, test/test_autograd.py::TestAutograd::test_backward, test/test_autograd.py::TestAutograd::test_backward_badcalls, test/test_autograd.py::TestAutograd::test_backward_copy, test/test_autograd.py::TestAutograd::test_backward_create_graph_warns, test/test_autograd.py::TestAutograd::test_backward_hook_relative_ordering, test/test_autograd.py::TestAutograd::test_backward_no_grad, test/test_autograd.py::TestAutograd::test_backward_to_node, test/test_autograd.py::TestAutograd::test_backward_twice_retained_graph_with_saved_values, test/test_autograd.py::TestAutograd::test_backward_twice_retained_graph_without_saved_values, test/test_autograd.py::TestAutograd::test_backward_twice_with_saved_values, test/test_autograd.py::TestAutograd::test_backward_twice_without_saved_values, test/test_autograd.py::TestAutograd::test_backward_with_inputs, test/test_autograd.py::TestAutograd::test_backward_with_nonleaf_inputs, test/test_autograd.py::TestAutograd::test_backward_with_scalar_input, test/test_autograd.py::TestAutograd::test_calculate_shape_util, test/test_autograd.py::TestAutograd::test_callback_adds_callback, test/test_autograd.py::TestAutograd::test_callback_propagates_errors_from_device_thread, test/test_autograd.py::TestAutograd::test_cant_create_saved_tensors, test/test_autograd.py::TestAutograd::test_checkpoint_detects_non_determinism, test/test_autograd.py::TestAutograd::test_checkpoint_graph_execution_group, test/test_autograd.py::TestAutograd::test_checkpoint_sequential_warns_if_use_reentrant_not_passed_explcitly, test/test_autograd.py::TestAutograd::test_checkpoint_valid_reset_on_error, test/test_autograd.py::TestAutograd::test_checkpoint_warns_if_use_reentrant_not_passed_explcitly, test/test_autograd.py::TestAutograd::test_checkpointing, test/test_autograd.py::TestAutograd::test_checkpointing_non_reentrant_autocast_cpu, test/test_autograd.py::TestAutograd::test_checkpointing_non_reentrant_autocast_gpu, test/test_autograd.py::TestAutograd::test_checkpointing_without_reentrant_arbitrary_input_output, test/test_autograd.py::TestAutograd::test_checkpointing_without_reentrant_correct_grad, test/test_autograd.py::TestAutograd::test_checkpointing_without_reentrant_custom_function_works, test/test_autograd.py::TestAutograd::test_checkpointing_without_reentrant_dataparallel, test/test_autograd.py::TestAutograd::test_checkpointing_without_reentrant_detached_tensor_use_reentrant_False, test/test_autograd.py::TestAutograd::test_checkpointing_without_reentrant_detached_tensor_use_reentrant_True, test/test_autograd.py::TestAutograd::test_checkpointing_without_reentrant_input_requires_grad_False, test/test_autograd.py::TestAutograd::test_checkpointing_without_reentrant_input_requires_grad_True, test/test_autograd.py::TestAutograd::test_checkpointing_without_reentrant_memory_savings, test/test_autograd.py::TestAutograd::test_checkpointing_without_reentrant_parameter_used_in_an_out, test/test_autograd.py::TestAutograd::test_checkpointing_without_reentrant_saved_object_identity, test/test_autograd.py::TestAutograd::test_checkpointing_without_reentrant_with_context_fn, test/test_autograd.py::TestAutograd::test_copy_slices_graph_task_updates, test/test_autograd.py::TestAutograd::test_create_graph_and_full_backward_hook_cycle, test/test_autograd.py::TestAutograd::test_current_graph_task_execution_order, test/test_autograd.py::TestAutograd::test_current_graph_task_id, test/test_autograd.py::TestAutograd::test_current_node, test/test_autograd.py::TestAutograd::test_custom_autograd_ac_early_stop, test/test_autograd.py::TestAutograd::test_custom_autograd_no_early_free, test/test_autograd.py::TestAutograd::test_custom_autograd_repeated_grad_grad, test/test_autograd.py::TestAutograd::test_custom_function_cycle, test/test_autograd.py::TestAutograd::test_custom_function_error, test/test_autograd.py::TestAutograd::test_custom_function_exception, test/test_autograd.py::TestAutograd::test_custom_function_forward_mode_forward_is_no_op, test/test_autograd.py::TestAutograd::test_custom_function_forward_mode_inplace_checks, test/test_autograd.py::TestAutograd::test_custom_function_forward_mode_non_differentiable, test/test_autograd.py::TestAutograd::test_custom_function_forward_mode_non_tensor_before_tensor_args, test/test_autograd.py::TestAutograd::test_custom_function_forward_mode_view_checks, test/test_autograd.py::TestAutograd::test_custom_function_forward_mode_wrong_formula, test/test_autograd.py::TestAutograd::test_custom_function_inplace_on_non_default_view, test/test_autograd.py::TestAutograd::test_custom_function_inplace_on_view_of_leaf, test/test_autograd.py::TestAutograd::test_custom_function_local_inplace, test/test_autograd.py::TestAutograd::test_custom_function_mark_dirty_not_differentiable, test/test_autograd.py::TestAutograd::test_custom_function_mark_output_view_of_intermediate, test/test_autograd.py::TestAutograd::test_custom_function_no_tensors, test/test_autograd.py::TestAutograd::test_custom_function_non_tensor_inputs_outputs, test/test_autograd.py::TestAutograd::test_custom_function_preserve_torch_function_when_return_as_is, test/test_autograd.py::TestAutograd::test_custom_function_return_view_in_nograd, test/test_autograd.py::TestAutograd::test_custom_function_save_for_forward, test/test_autograd.py::TestAutograd::test_custom_function_saved_tensors, test/test_autograd.py::TestAutograd::test_custom_function_saving_mutated_view_no_leak, test/test_autograd.py::TestAutograd::test_custom_function_setup_context_multi_input, test/test_autograd.py::TestAutograd::test_custom_function_setup_context_multi_output, test/test_autograd.py::TestAutograd::test_custom_function_setup_context_simple, test/test_autograd.py::TestAutograd::test_custom_function_vmap_defaults, test/test_autograd.py::TestAutograd::test_deep_reentrant, test/test_autograd.py::TestAutograd::test_default_saved_tensors_hooks_double_backward, test/test_autograd.py::TestAutograd::test_dep_nograd, test/test_autograd.py::TestAutograd::test_dependent_backward, test/test_autograd.py::TestAutograd::test_detach, test/test_autograd.py::TestAutograd::test_detach_base, test/test_autograd.py::TestAutograd::test_detach_then_inplace_raises_in_autograd, test/test_autograd.py::TestAutograd::test_diagonal_expanded_v, test/test_autograd.py::TestAutograd::test_dir, test/test_autograd.py::TestAutograd::test_disabling_saved_tensor_hooks, test/test_autograd.py::TestAutograd::test_disabling_saved_tensor_hooks_nested, test/test_autograd.py::TestAutograd::test_dont_materialize_grads, test/test_autograd.py::TestAutograd::test_duplicate_backward_root, test/test_autograd.py::TestAutograd::test_enable_grad_decorator_no_paren, test/test_autograd.py::TestAutograd::test_first_grad_fn_access_in_no_grad_mode, test/test_autograd.py::TestAutograd::test_free_deep_graph, test/test_autograd.py::TestAutograd::test_free_deep_graph_complicated, test/test_autograd.py::TestAutograd::test_free_deep_graph_pyfunction, test/test_autograd.py::TestAutograd::test_full_backward_hook_double_backward, test/test_autograd.py::TestAutograd::test_function, test/test_autograd.py::TestAutograd::test_function_returns_input, test/test_autograd.py::TestAutograd::test_function_returns_undefined_tensor, test/test_autograd.py::TestAutograd::test_gc_in_destructor, test/test_autograd.py::TestAutograd::test_get_data_and_hooks_from_raw_saved_variable, test/test_autograd.py::TestAutograd::test_grad, test/test_autograd.py::TestAutograd::test_grad_badcalls, test/test_autograd.py::TestAutograd::test_grad_batched_grad, test/test_autograd.py::TestAutograd::test_grad_dtype, test/test_autograd.py::TestAutograd::test_grad_empty_inputs, test/test_autograd.py::TestAutograd::test_grad_fn_attr_bindings, test/test_autograd.py::TestAutograd::test_grad_fn_badcalls, test/test_autograd.py::TestAutograd::test_grad_fn_input_metadata, test/test_autograd.py::TestAutograd::test_grad_fn_prehooks, test/test_autograd.py::TestAutograd::test_grad_fn_prehooks_multiple_outputs, test/test_autograd.py::TestAutograd::test_grad_fn_prehooks_remove_hooks, test/test_autograd.py::TestAutograd::test_grad_materialize_grads, test/test_autograd.py::TestAutograd::test_grad_mode_class_decoration, test/test_autograd.py::TestAutograd::test_grad_mode_restored_reentrant, test/test_autograd.py::TestAutograd::test_grad_nonleaf, test/test_autograd.py::TestAutograd::test_grad_nonleaf_many_outputs, test/test_autograd.py::TestAutograd::test_grad_nonleaf_register_hook, test/test_autograd.py::TestAutograd::test_grad_thread_safety, test/test_autograd.py::TestAutograd::test_grad_to_node, test/test_autograd.py::TestAutograd::test_grad_to_node_inplace, test/test_autograd.py::TestAutograd::test_grad_to_node_materialize, test/test_autograd.py::TestAutograd::test_grad_to_node_multi, test/test_autograd.py::TestAutograd::test_grad_to_node_set, test/test_autograd.py::TestAutograd::test_grad_unreachable, test/test_autograd.py::TestAutograd::test_grad_unreachable_discovery, test/test_autograd.py::TestAutograd::test_gradcheck_backward_mul_by_grad_output, test/test_autograd.py::TestAutograd::test_gradcheck_check_batched_grad, test/test_autograd.py::TestAutograd::test_gradcheck_check_forward_or_backward_only, test/test_autograd.py::TestAutograd::test_gradcheck_check_no_differentiable_outputs, test/test_autograd.py::TestAutograd::test_gradcheck_complex_non_complex_outputs, test/test_autograd.py::TestAutograd::test_gradcheck_custom_error, test/test_autograd.py::TestAutograd::test_gradcheck_default_device_placement_context, test/test_autograd.py::TestAutograd::test_gradcheck_dense_and_sparse_inputs, test/test_autograd.py::TestAutograd::test_gradcheck_forward_ad, test/test_autograd.py::TestAutograd::test_gradcheck_forward_ad_batched_grad, test/test_autograd.py::TestAutograd::test_gradcheck_forward_ad_respects_requires_grad, test/test_autograd.py::TestAutograd::test_gradcheck_forward_ad_runs_with_no_requires_grad, test/test_autograd.py::TestAutograd::test_gradcheck_get_analytical_jacobian, test/test_autograd.py::TestAutograd::test_gradcheck_get_numerical_jacobian, test/test_autograd.py::TestAutograd::test_gradcheck_input_layout0, test/test_autograd.py::TestAutograd::test_gradcheck_input_layout1, test/test_autograd.py::TestAutograd::test_gradcheck_input_layout2, test/test_autograd.py::TestAutograd::test_gradcheck_input_layout3, test/test_autograd.py::TestAutograd::test_gradcheck_input_layout4, test/test_autograd.py::TestAutograd::test_gradcheck_jacobian_mismatch, test/test_autograd.py::TestAutograd::test_gradcheck_multiple_mkldnn_inputs, test/test_autograd.py::TestAutograd::test_gradcheck_nondeterministic, test/test_autograd.py::TestAutograd::test_gradcheck_output_shape_or_dtype_depend_on_values, test/test_autograd.py::TestAutograd::test_gradcheck_single_input, test/test_autograd.py::TestAutograd::test_gradcheck_test_outputs, test/test_autograd.py::TestAutograd::test_gradcheck_undefined_grad, test/test_autograd.py::TestAutograd::test_gradcheck_validates_input_mkldnn, test/test_autograd.py::TestAutograd::test_gradcheck_validates_inputs, test/test_autograd.py::TestAutograd::test_gradient_edge_graph_ownership, test/test_autograd.py::TestAutograd::test_gradient_edge_output, test/test_autograd.py::TestAutograd::test_graph_save_on_cpu, test/test_autograd.py::TestAutograd::test_graph_save_on_cpu_cuda, test/test_autograd.py::TestAutograd::test_hessian_vector, test/test_autograd.py::TestAutograd::test_hook_closure_cycle_use_custom_function_False_use_tensor_hook_False, test/test_autograd.py::TestAutograd::test_hook_closure_cycle_use_custom_function_False_use_tensor_hook_True, test/test_autograd.py::TestAutograd::test_hook_closure_cycle_use_custom_function_True_use_tensor_hook_False, test/test_autograd.py::TestAutograd::test_hook_closure_cycle_use_custom_function_True_use_tensor_hook_True, test/test_autograd.py::TestAutograd::test_hook_edge_case_when_called_with_grad, test/test_autograd.py::TestAutograd::test_hook_none, test/test_autograd.py::TestAutograd::test_hook_with_no_name, test/test_autograd.py::TestAutograd::test_hooks, test/test_autograd.py::TestAutograd::test_hooks_cpp, test/test_autograd.py::TestAutograd::test_increment_version, test/test_autograd.py::TestAutograd::test_index_backward_does_not_save_tensor, test/test_autograd.py::TestAutograd::test_indexing, test/test_autograd.py::TestAutograd::test_indexing_duplicates, test/test_autograd.py::TestAutograd::test_inplace, test/test_autograd.py::TestAutograd::test_inplace_not_requires_grad, test/test_autograd.py::TestAutograd::test_inplace_on_view_backward, test/test_autograd.py::TestAutograd::test_inplace_on_view_leaf_errors, test/test_autograd.py::TestAutograd::test_inplace_on_view_saved_output, test/test_autograd.py::TestAutograd::test_inplace_on_view_weak_grad_fn, test/test_autograd.py::TestAutograd::test_input_buffer_accum, test/test_autograd.py::TestAutograd::test_integer_outputs, test/test_autograd.py::TestAutograd::test_invalid_gradients, test/test_autograd.py::TestAutograd::test_isolated_node, test/test_autograd.py::TestAutograd::test_leaf_assignment, test/test_autograd.py::TestAutograd::test_legacy_function_deprecation_exception, test/test_autograd.py::TestAutograd::test_lobpcg, test/test_autograd.py::TestAutograd::test_mark_non_differentiable, test/test_autograd.py::TestAutograd::test_mark_non_differentiable_mixed, test/test_autograd.py::TestAutograd::test_mark_non_differentiable_none, test/test_autograd.py::TestAutograd::test_materialize_grads, test/test_autograd.py::TestAutograd::test_multi_backward, test/test_autograd.py::TestAutograd::test_multi_backward_no_grad, test/test_autograd.py::TestAutograd::test_multi_grad_all_hooks, test/test_autograd.py::TestAutograd::test_multi_grad_any_hooks, test/test_autograd.py::TestAutograd::test_multi_grad_hooks_invalid_mode, test/test_autograd.py::TestAutograd::test_multiple_insert_removal_caching, test/test_autograd.py::TestAutograd::test_named_tensor_for_complex_views, test/test_autograd.py::TestAutograd::test_naughty_anomaly_access, test/test_autograd.py::TestAutograd::test_naughty_autograd_function_attribute_access, test/test_autograd.py::TestAutograd::test_naughty_autograd_function_stashing_ctx, test/test_autograd.py::TestAutograd::test_nested_anomaly_detect_nan, test/test_autograd.py::TestAutograd::test_nested_anomaly_printstack_cleanup, test/test_autograd.py::TestAutograd::test_next_functions, test/test_autograd.py::TestAutograd::test_no_grad, test/test_autograd.py::TestAutograd::test_no_grad_assignment, test/test_autograd.py::TestAutograd::test_no_grad_copy, test/test_autograd.py::TestAutograd::test_no_grad_copy_sparse, test/test_autograd.py::TestAutograd::test_no_grad_input, test/test_autograd.py::TestAutograd::test_no_grad_modifies_version, test/test_autograd.py::TestAutograd::test_no_grad_python_function, test/test_autograd.py::TestAutograd::test_no_requires_grad_inplace, test/test_autograd.py::TestAutograd::test_no_unnecessary_save, test/test_autograd.py::TestAutograd::test_no_unnecessary_unwrapping, test/test_autograd.py::TestAutograd::test_node_ordering_when_none_returned, test/test_autograd.py::TestAutograd::test_node_post_hook_registered_during_unpack_hook, test/test_autograd.py::TestAutograd::test_not_implemented_fwad, test/test_autograd.py::TestAutograd::test_not_implemented_grad, test/test_autograd.py::TestAutograd::test_numpy_requires_grad, test/test_autograd.py::TestAutograd::test_once_differentiable, test/test_autograd.py::TestAutograd::test_out_variant_raises_when_inputs_require_grad, test/test_autograd.py::TestAutograd::test_pack_hook_with_inplace_modification_should_fail, test/test_autograd.py::TestAutograd::test_pickle, test/test_autograd.py::TestAutograd::test_post_accumulate_grad_hook_e2e, test/test_autograd.py::TestAutograd::test_post_accumulate_grad_hook_gets_cleaned_up, test/test_autograd.py::TestAutograd::test_post_accumulate_grad_hook_multiple_hooks, test/test_autograd.py::TestAutograd::test_post_accumulate_grad_hook_multiple_tensors, test/test_autograd.py::TestAutograd::test_post_accumulate_grad_hook_on_non_leaf, test/test_autograd.py::TestAutograd::test_post_accumulate_grad_hook_ordering, test/test_autograd.py::TestAutograd::test_post_accumulate_grad_hook_returns_not_None, test/test_autograd.py::TestAutograd::test_pow_zero_tensor_gradient, test/test_autograd.py::TestAutograd::test_power_function, test/test_autograd.py::TestAutograd::test_prehook_ordering, test/test_autograd.py::TestAutograd::test_profiler, test/test_autograd.py::TestAutograd::test_profiler_aggregation_fake, test/test_autograd.py::TestAutograd::test_profiler_aggregation_lstm, test/test_autograd.py::TestAutograd::test_profiler_aggregation_table, test/test_autograd.py::TestAutograd::test_profiler_function_event_avg, test/test_autograd.py::TestAutograd::test_profiler_propagation, test/test_autograd.py::TestAutograd::test_profiler_seq_nr, test/test_autograd.py::TestAutograd::test_profiler_shapes, test/test_autograd.py::TestAutograd::test_profiler_unboxed_only, test/test_autograd.py::TestAutograd::test_pynode_destruction_deadlock, test/test_autograd.py::TestAutograd::test_record_function, test/test_autograd.py::TestAutograd::test_record_function_callbacks, test/test_autograd.py::TestAutograd::test_record_function_legacy, test/test_autograd.py::TestAutograd::test_record_function_multithreaded, test/test_autograd.py::TestAutograd::test_reentrant_child_error, test/test_autograd.py::TestAutograd::test_reentrant_priority, test/test_autograd.py::TestAutograd::test_reentrant_with_callbacks_both_depths, test/test_autograd.py::TestAutograd::test_reentrant_with_callbacks_depth_0, test/test_autograd.py::TestAutograd::test_reentrant_with_callbacks_depth_1, test/test_autograd.py::TestAutograd::test_reentrant_with_leaf_variable_hook, test/test_autograd.py::TestAutograd::test_reentrant_with_non_leaf_variable_hook, test/test_autograd.py::TestAutograd::test_requires_grad, test/test_autograd.py::TestAutograd::test_requires_grad_, test/test_autograd.py::TestAutograd::test_requires_grad_inplace, test/test_autograd.py::TestAutograd::test_retain_grad, test/test_autograd.py::TestAutograd::test_retain_grad_cycle, test/test_autograd.py::TestAutograd::test_retain_grad_inplace, test/test_autograd.py::TestAutograd::test_retain_grad_inplace_over_view, test/test_autograd.py::TestAutograd::test_retains_grad_can_always_observe_tensor_prehook, test/test_autograd.py::TestAutograd::test_retains_grad_inplace_multiple_outputs, test/test_autograd.py::TestAutograd::test_return_duplicate, test/test_autograd.py::TestAutograd::test_return_duplicate_inplace, test/test_autograd.py::TestAutograd::test_return_leaf, test/test_autograd.py::TestAutograd::test_return_leaf_inplace, test/test_autograd.py::TestAutograd::test_save_none_for_backward, test/test_autograd.py::TestAutograd::test_save_on_cpu_and_checkpoint, test/test_autograd.py::TestAutograd::test_save_output_nr, test/test_autograd.py::TestAutograd::test_saved_tensor_hooks_custom_error_propagation, test/test_autograd.py::TestAutograd::test_saved_tensor_hooks_custom_function_intermediates, test/test_autograd.py::TestAutograd::test_saved_tensor_hooks_extra_enter_during_bw_no_leak, test/test_autograd.py::TestAutograd::test_saved_tensor_hooks_extra_exit_during_bw_no_crash, test/test_autograd.py::TestAutograd::test_saved_tensors_hook_version_counter_not_shared, test/test_autograd.py::TestAutograd::test_saved_variable_packing_unpacking_did_not_save_original_with_default_hooks, test/test_autograd.py::TestAutograd::test_saved_variable_packing_unpacking_did_not_save_original_with_hooks, test/test_autograd.py::TestAutograd::test_saved_variable_packing_unpacking_saved_original_with_default_hooks, test/test_autograd.py::TestAutograd::test_saved_variable_packing_unpacking_saved_original_with_hooks, test/test_autograd.py::TestAutograd::test_saved_variable_saved_original_inplace_detach, test/test_autograd.py::TestAutograd::test_saved_variable_version_counter, test/test_autograd.py::TestAutograd::test_saved_variables_deprecated, test/test_autograd.py::TestAutograd::test_saving_variable_to_disk, test/test_autograd.py::TestAutograd::test_scalar_grad_mixed_device, test/test_autograd.py::TestAutograd::test_select_expanded_v, test/test_autograd.py::TestAutograd::test_select_sum, test/test_autograd.py::TestAutograd::test_set_data_preserve_pyobj, test/test_autograd.py::TestAutograd::test_set_data_self_requires_grad, test/test_autograd.py::TestAutograd::test_set_data_tensorimpl_type, test/test_autograd.py::TestAutograd::test_set_grad_coroutines, test/test_autograd.py::TestAutograd::test_set_grad_coroutines_benign_exceptions, test/test_autograd.py::TestAutograd::test_set_grad_coroutines_critical_exceptions, test/test_autograd.py::TestAutograd::test_set_grad_coroutines_exit, test/test_autograd.py::TestAutograd::test_set_grad_enabled, test/test_autograd.py::TestAutograd::test_set_grad_enabled_wraps, test/test_autograd.py::TestAutograd::test_set_grad_generator_functions, test/test_autograd.py::TestAutograd::test_set_grad_generator_functions_recursive, test/test_autograd.py::TestAutograd::test_set_materialize_non_diff_grads, test/test_autograd.py::TestAutograd::test_setitem, test/test_autograd.py::TestAutograd::test_setitem_mask, test/test_autograd.py::TestAutograd::test_setting_default_saved_variable_hooks_twice_should_not_fail, test/test_autograd.py::TestAutograd::test_setting_default_saved_variable_hooks_twice_should_use_inner, test/test_autograd.py::TestAutograd::test_setup_context_when_forward_has_default_args, test/test_autograd.py::TestAutograd::test_shape, test/test_autograd.py::TestAutograd::test_sharded_grad, test/test_autograd.py::TestAutograd::test_simple_reentrant, test/test_autograd.py::TestAutograd::test_slice_expanded_v, test/test_autograd.py::TestAutograd::test_sparse_gather_both_scalar, test/test_autograd.py::TestAutograd::test_sparse_gather_dim0, test/test_autograd.py::TestAutograd::test_sparse_gather_dim1, test/test_autograd.py::TestAutograd::test_sparse_gather_dim_neg, test/test_autograd.py::TestAutograd::test_sparse_gather_ind_scalar, test/test_autograd.py::TestAutograd::test_sparse_gather_x_scalar, test/test_autograd.py::TestAutograd::test_sparse_mm_backward, test/test_autograd.py::TestAutograd::test_tensor_grad_warnings, test/test_autograd.py::TestAutograd::test_tensor_hooks_inplace, test/test_autograd.py::TestAutograd::test_tensor_hooks_inplace_multiple_outputs, test/test_autograd.py::TestAutograd::test_tensor_hooks_inplace_over_view, test/test_autograd.py::TestAutograd::test_thread_shutdown, test/test_autograd.py::TestAutograd::test_to_sparse_backward, test/test_autograd.py::TestAutograd::test_too_many_grads, test/test_autograd.py::TestAutograd::test_type_conversions, test/test_autograd.py::TestAutograd::test_unpack_hooks_exec_count, test/test_autograd.py::TestAutograd::test_unrelated_inputs, test/test_autograd.py::TestAutograd::test_unsafe_set_version_counter, test/test_autograd.py::TestAutograd::test_unused_grad_requires_grad_with_materialize, test/test_autograd.py::TestAutograd::test_unused_output, test/test_autograd.py::TestAutograd::test_var_mean_differentiable, test/test_autograd.py::TestAutograd::test_variable_traverse, test/test_autograd.py::TestAutograd::test_version_counter, test/test_autograd.py::TestAutograd::test_view_func_replay, test/test_autograd.py::TestAutograd::test_view_func_replay_with_modified_state, test/test_autograd.py::TestAutograd::test_view_replay_enabled, test/test_autograd.py::TestAutograd::test_volatile_deprecated, test/test_autograd.py::TestAutograd::test_will_engine_execute_node, test/test_autograd.py::TestAutograd::test_wrapped_number_saved_tensors_hooks, test/test_autograd.py::TestAutogradForwardModeBatchedGrad::test_inplace_on_view_not_same_layout, test/test_autograd.py::TestAutogradForwardModeBatchedGrad::test_inplace_on_view_same_layout, test/test_autograd.py::TestAutogradForwardModeBatchedGrad::test_metadata_check_for_storage_numel_skipped, test/test_autograd.py::TestAutogradForwardModeBatchedGrad::test_out_of_place_basic, test/test_autograd.py::TestAutogradForwardModeBatchedGrad::test_out_of_place_not_same_layout, test/test_autograd.py::TestAutogradForwardMode::test_advanced_packing_unpacking, test/test_autograd.py::TestAutogradForwardMode::test_backward_graph_destruction, test/test_autograd.py::TestAutogradForwardMode::test_basic_packing_unpacking, test/test_autograd.py::TestAutogradForwardMode::test_codegen_ignores_undefined_outputs, test/test_autograd.py::TestAutogradForwardMode::test_create_new_zeros_with_same_meta, test/test_autograd.py::TestAutogradForwardMode::test_default_level, test/test_autograd.py::TestAutogradForwardMode::test_detach_view_tracking, test/test_autograd.py::TestAutogradForwardMode::test_forward_level_cleanup, test/test_autograd.py::TestAutogradForwardMode::test_fwd_grad_enabled, test/test_autograd.py::TestAutogradForwardMode::test_grad_cleanup, test/test_autograd.py::TestAutogradForwardMode::test_make_dual_forbid_integral_dtype, test/test_autograd.py::TestAutogradForwardMode::test_make_dual_inference_tensor_in_inference_mode, test/test_autograd.py::TestAutogradForwardMode::test_make_dual_torch_dispatch, test/test_autograd.py::TestAutogradForwardMode::test_metadata_check_check_conj, test/test_autograd.py::TestAutogradForwardMode::test_metadata_check_checks_ignores_size_zero, test/test_autograd.py::TestAutogradForwardMode::test_metadata_check_checks_storage_numel, test/test_autograd.py::TestAutogradForwardMode::test_metadata_check_ignore_storage_offset_for_zero_numel_tensor, test/test_autograd.py::TestAutogradForwardMode::test_metadata_check_when_primal_has_conj_bit, test/test_autograd.py::TestAutogradForwardMode::test_metadata_check_when_primal_has_neg_bit, test/test_autograd.py::TestAutogradForwardMode::test_nested_level, test/test_autograd.py::TestAutogradForwardMode::test_non_differentiable, test/test_autograd.py::TestAutogradForwardMode::test_out_variant, test/test_autograd.py::TestAutogradForwardMode::test_print, test/test_autograd.py::TestAutogradForwardMode::test_set_fw_grad_having_own_fw_grad_at_same_level, test/test_autograd.py::TestAutogradForwardMode::test_set_fwd_grad_enabled, test/test_autograd.py::TestAutogradForwardMode::test_size_check, test/test_autograd.py::TestAutogradForwardMode::test_view_inplace_always_creates_a_view, test/test_autograd.py::TestAutogradForwardMode::test_view_inplace_differentiable_views, test/test_autograd.py::TestAutogradForwardMode::test_view_inplace_non_differentiable_views, test/test_autograd.py::TestAllowMutationOnSaved::test_backward_out_of_context, test/test_autograd.py::TestAllowMutationOnSaved::test_basic, test/test_autograd.py::TestAllowMutationOnSaved::test_disallow_nesting, test/test_autograd.py::TestAllowMutationOnSaved::test_double_backward, test/test_autograd.py::TestAllowMutationOnSaved::test_inplace_foreach, test/test_autograd.py::TestAllowMutationOnSaved::test_save_base_and_modify_view, test/test_autograd.py::TestAllowMutationOnSaved::test_save_view_modify_base, test/test_autograd.py::TestAllowMutationOnSaved::test_saved_but_not_anymore, test/test_autograd.py::TestAllowMutationOnSaved::test_saved_same_tensor_different_versions, test/test_autograd.py::TestAllowMutationOnSaved::test_saved_same_tensor_many_times, test/test_autograd.py::TestAllowMutationOnSaved::test_views, test/test_autograd.py::TestAllowMutationOnSaved::test_with_math_views, test/test_autograd.py::TestAllowMutationOnSaved::test_with_out_variant, test/test_autograd.py::TestAutogradInferenceMode::test_inference_mode_context_manager, test/test_autograd.py::TestAutogradInferenceMode::test_inference_mode_decorator, test/test_autograd.py::TestAutogradInferenceMode::test_inference_mode_existing_autograd_session, test/test_autograd.py::TestAutogradInferenceMode::test_inference_mode_handle_direct_view_on_rebase, test/test_autograd.py::TestAutogradInferenceMode::test_inference_mode_handle_indirect_view_on_rebase, test/test_autograd.py::TestAutogradInferenceMode::test_inference_mode_inf_tensor_in_inf_mode_functional_op, test/test_autograd.py::TestAutogradInferenceMode::test_inference_mode_inf_tensor_in_inf_mode_inplace_op, test/test_autograd.py::TestAutogradInferenceMode::test_inference_mode_inf_tensor_in_inf_mode_view_op, test/test_autograd.py::TestAutogradInferenceMode::test_inference_mode_inf_tensor_in_normal_mode_functional_op, test/test_autograd.py::TestAutogradInferenceMode::test_inference_mode_inf_tensor_in_normal_mode_inplace_op, test/test_autograd.py::TestAutogradInferenceMode::test_inference_mode_inf_tensor_in_normal_mode_view_op, test/test_autograd.py::TestAutogradInferenceMode::test_inference_mode_tensor_creation, test/test_autograd.py::TestAutogradInferenceMode::test_mix_inference_and_normal_tensor_functional_op, test/test_autograd.py::TestAutogradInferenceMode::test_mix_inference_and_normal_tensor_inplace_op, test/test_autograd.py::TestAutogradInferenceMode::test_mix_inference_and_normal_tensor_view_op, test/test_autograd.py::TestAutogradInferenceMode::test_normal_tensor_inplace_output_in_inference_mode, test/test_autograd.py::TestAutogradInferenceMode::test_normal_tensor_inplace_output_in_normal_mode, test/test_autograd.py::TestAutogradInferenceMode::test_normal_tensor_view_output_in_inference_mode, test/test_autograd.py::TestAutogradInferenceMode::test_normal_tensor_view_output_in_normal_mode, test/test_autograd.py::TestMultithreadAutograd::test_cat_stack_r_to_c, test/test_autograd.py::TestMultithreadAutograd::test_custom_function_propagates_errors_from_device_thread, test/test_autograd.py::TestMultithreadAutograd::test_dataparallel_saved_tensors_hooks, test/test_autograd.py::TestMultithreadAutograd::test_fork_join_in_middle, test/test_autograd.py::TestMultithreadAutograd::test_multi_grad_all_hooks, test/test_autograd.py::TestMultithreadAutograd::test_multi_grad_any_hooks, test/test_autograd.py::TestMultithreadAutograd::test_multithreaded_exception_propagation, test/test_autograd.py::TestMultithreadAutograd::test_preserve_backtrace, test/test_autograd.py::TestMultithreadAutograd::test_python_thread_in_middle, test/test_autograd.py::TestMultithreadAutograd::test_set_multithreading_enabled_as_context_manager_and_function, test/test_autograd.py::TestMultithreadAutograd::test_simple_backward, test/test_autograd.py::TestMultithreadAutograd::test_simple_backward_same_input, test/test_autograd.py::TestNestedCheckpoint::test_nested_checkpoint_early_stop_False, test/test_autograd.py::TestNestedCheckpoint::test_nested_checkpoint_early_stop_True, test/test_autograd.py::TestNestedCheckpoint::test_nested_checkpoint_kwargs_early_stop_False, test/test_autograd.py::TestNestedCheckpoint::test_nested_checkpoint_kwargs_early_stop_True, test/test_autograd.py::TestNestedCheckpoint::test_nested_checkpoint_non_tensor_inputs_and_outputs_early_stop_False, test/test_autograd.py::TestNestedCheckpoint::test_nested_checkpoint_non_tensor_inputs_and_outputs_early_stop_True, test/test_autograd.py::TestNestedCheckpoint::test_nested_checkpoint_reentrant_backwards_early_stop_False, test/test_autograd.py::TestNestedCheckpoint::test_nested_checkpoint_reentrant_backwards_early_stop_True, test/test_autograd.py::TestNestedCheckpoint::test_nested_checkpoint_same_graph_early_stop_False, test/test_autograd.py::TestNestedCheckpoint::test_nested_checkpoint_same_graph_early_stop_True, test/test_autograd.py::TestNestedCheckpoint::test_nested_checkpoint_set_early_stop, test/test_autograd.py::TestNestedCheckpoint::test_nested_checkpoint_set_early_stop_no_recompution_needed, test/test_autograd.py::TestNestedCheckpoint::test_nested_checkpoint_two_children_early_stop_False, test/test_autograd.py::TestNestedCheckpoint::test_nested_checkpoint_two_children_early_stop_True, test/test_autograd.py::TestSelectiveActivationCheckpoint::test_bad_inputs, test/test_autograd.py::TestSelectiveActivationCheckpoint::test_can_only_trigger_recompute_once, test/test_autograd.py::TestSelectiveActivationCheckpoint::test_flops_and_mem, test/test_autograd.py::TestSelectiveActivationCheckpoint::test_function_with_more_than_one_output, test/test_autograd.py::TestSelectiveActivationCheckpoint::test_function_with_non_tensor_output, test/test_autograd.py::TestSelectiveActivationCheckpoint::test_output_already_has_autograd_meta, test/test_autograd.py::TestSelectiveActivationCheckpoint::test_policy_with_state, test/test_autograd.py::TestSelectiveActivationCheckpoint::test_storage_lifetime, test/test_autograd.py::TestSelectiveActivationCheckpoint::test_subclass_dispatching_sizes, test/test_autograd.py::TestSelectiveActivationCheckpoint::test_version_counter, test/test_autograd.py::TestAutogradComplex::test_view_func_for_complex_views, test/test_autograd.py::TestAutogradComplex::test_view_with_multi_output, test/test_autograd.py::TestAutogradFunctional::test_construct_standard_basis_for_base_tensor, test/test_autograd.py::TestAutogradFunctional::test_construct_standard_basis_for_cuda_base_tensor, test/test_autograd.py::TestAutogradFunctional::test_construct_standard_basis_for_cuda_logging_tensor, test/test_autograd.py::TestAutogradFunctional::test_construct_standard_basis_for_logging_tensor, test/test_autograd.py::TestAutogradFunctional::test_hessian_create_graph_vectorize_False_base_tensor, test/test_autograd.py::TestAutogradFunctional::test_hessian_create_graph_vectorize_False_logging_tensor, test/test_autograd.py::TestAutogradFunctional::test_hessian_create_graph_vectorize_True_base_tensor, test/test_autograd.py::TestAutogradFunctional::test_hessian_create_graph_vectorize_True_logging_tensor, test/test_autograd.py::TestAutogradFunctional::test_hessian_err_check_strict_base_tensor, test/test_autograd.py::TestAutogradFunctional::test_hessian_err_check_strict_logging_tensor, test/test_autograd.py::TestAutogradFunctional::test_hessian_err_check_strict_vectorize_base_tensor, test/test_autograd.py::TestAutogradFunctional::test_hessian_err_check_strict_vectorize_logging_tensor, test/test_autograd.py::TestAutogradFunctional::test_hessian_err_check_vectorize_False_base_tensor, test/test_autograd.py::TestAutogradFunctional::test_hessian_err_check_vectorize_False_logging_tensor, test/test_autograd.py::TestAutogradFunctional::test_hessian_err_check_vectorize_True_base_tensor, test/test_autograd.py::TestAutogradFunctional::test_hessian_err_check_vectorize_True_logging_tensor, test/test_autograd.py::TestAutogradFunctional::test_hessian_match_vhp_hvp_base_tensor, test/test_autograd.py::TestAutogradFunctional::test_hessian_match_vhp_hvp_logging_tensor, test/test_autograd.py::TestAutogradFunctional::test_hessian_no_grad_base_tensor, test/test_autograd.py::TestAutogradFunctional::test_hessian_no_grad_logging_tensor, test/test_autograd.py::TestAutogradFunctional::test_hessian_output_base_tensor, test/test_autograd.py::TestAutogradFunctional::test_hessian_output_logging_tensor, test/test_autograd.py::TestAutogradFunctional::test_hessian_output_vectorized_base_tensor, test/test_autograd.py::TestAutogradFunctional::test_hessian_output_vectorized_logging_tensor, test/test_autograd.py::TestAutogradFunctional::test_hessian_scalar_vectorize_False_base_tensor, test/test_autograd.py::TestAutogradFunctional::test_hessian_scalar_vectorize_False_logging_tensor, test/test_autograd.py::TestAutogradFunctional::test_hessian_scalar_vectorize_True_base_tensor, test/test_autograd.py::TestAutogradFunctional::test_hessian_scalar_vectorize_True_logging_tensor, test/test_autograd.py::TestAutogradFunctional::test_hessian_vectorize_correctness_multi_input_base_tensor, test/test_autograd.py::TestAutogradFunctional::test_hessian_vectorize_correctness_multi_input_logging_tensor, test/test_autograd.py::TestAutogradFunctional::test_hessian_vectorize_correctness_simple_base_tensor, test/test_autograd.py::TestAutogradFunctional::test_hessian_vectorize_correctness_simple_logging_tensor, test/test_autograd.py::TestAutogradFunctional::test_hessian_vectorize_correctness_unrelated_outputs_base_tensor, test/test_autograd.py::TestAutogradFunctional::test_hessian_vectorize_correctness_unrelated_outputs_logging_tensor, test/test_autograd.py::TestAutogradFunctional::test_hessian_vectorize_raises_no_warnings_base_tensor, test/test_autograd.py::TestAutogradFunctional::test_hessian_vectorize_raises_no_warnings_logging_tensor, test/test_autograd.py::TestAutogradFunctional::test_hvp_create_graph_base_tensor, test/test_autograd.py::TestAutogradFunctional::test_hvp_create_graph_logging_tensor, test/test_autograd.py::TestAutogradFunctional::test_hvp_err_check_base_tensor, test/test_autograd.py::TestAutogradFunctional::test_hvp_err_check_logging_tensor, test/test_autograd.py::TestAutogradFunctional::test_hvp_err_check_strict_base_tensor, test/test_autograd.py::TestAutogradFunctional::test_hvp_err_check_strict_logging_tensor, test/test_autograd.py::TestAutogradFunctional::test_hvp_no_grad_base_tensor, test/test_autograd.py::TestAutogradFunctional::test_hvp_no_grad_logging_tensor, test/test_autograd.py::TestAutogradFunctional::test_hvp_output_base_tensor, test/test_autograd.py::TestAutogradFunctional::test_hvp_output_logging_tensor, test/test_autograd.py::TestAutogradFunctional::test_hvp_scalar_base_tensor, test/test_autograd.py::TestAutogradFunctional::test_hvp_scalar_logging_tensor, test/test_autograd.py::TestAutogradFunctional::test_jacobian_create_graph_vectorize_False_base_tensor, test/test_autograd.py::TestAutogradFunctional::test_jacobian_create_graph_vectorize_False_logging_tensor, test/test_autograd.py::TestAutogradFunctional::test_jacobian_create_graph_vectorize_True_base_tensor, test/test_autograd.py::TestAutogradFunctional::test_jacobian_create_graph_vectorize_True_logging_tensor, test/test_autograd.py::TestAutogradFunctional::test_jacobian_err_check_strict_base_tensor, test/test_autograd.py::TestAutogradFunctional::test_jacobian_err_check_strict_logging_tensor, test/test_autograd.py::TestAutogradFunctional::test_jacobian_err_check_strict_vectorize_base_tensor, test/test_autograd.py::TestAutogradFunctional::test_jacobian_err_check_strict_vectorize_logging_tensor, test/test_autograd.py::TestAutogradFunctional::test_jacobian_err_check_vectorize_False_base_tensor, test/test_autograd.py::TestAutogradFunctional::test_jacobian_err_check_vectorize_False_logging_tensor, test/test_autograd.py::TestAutogradFunctional::test_jacobian_err_check_vectorize_True_base_tensor, test/test_autograd.py::TestAutogradFunctional::test_jacobian_err_check_vectorize_True_logging_tensor, test/test_autograd.py::TestAutogradFunctional::test_jacobian_match_vjp_jvp_base_tensor, test/test_autograd.py::TestAutogradFunctional::test_jacobian_match_vjp_jvp_logging_tensor, test/test_autograd.py::TestAutogradFunctional::test_jacobian_no_grad_base_tensor, test/test_autograd.py::TestAutogradFunctional::test_jacobian_no_grad_logging_tensor, test/test_autograd.py::TestAutogradFunctional::test_jacobian_output_base_tensor, test/test_autograd.py::TestAutogradFunctional::test_jacobian_output_logging_tensor, test/test_autograd.py::TestAutogradFunctional::test_jacobian_output_vectorized_base_tensor, test/test_autograd.py::TestAutogradFunctional::test_jacobian_output_vectorized_logging_tensor, test/test_autograd.py::TestAutogradFunctional::test_jacobian_scalar_base_tensor, test/test_autograd.py::TestAutogradFunctional::test_jacobian_scalar_logging_tensor, test/test_autograd.py::TestAutogradFunctional::test_jacobian_scalar_vectorized_base_tensor, test/test_autograd.py::TestAutogradFunctional::test_jacobian_scalar_vectorized_logging_tensor, test/test_autograd.py::TestAutogradFunctional::test_jacobian_vectorize_correctness_different_devices_base_tensor, test/test_autograd.py::TestAutogradFunctional::test_jacobian_vectorize_correctness_different_devices_logging_tensor, test/test_autograd.py::TestAutogradFunctional::test_jacobian_vectorize_correctness_different_dtype_base_tensor, test/test_autograd.py::TestAutogradFunctional::test_jacobian_vectorize_correctness_different_dtype_logging_tensor, test/test_autograd.py::TestAutogradFunctional::test_jacobian_vectorize_correctness_multi_input_base_tensor, test/test_autograd.py::TestAutogradFunctional::test_jacobian_vectorize_correctness_multi_input_logging_tensor, test/test_autograd.py::TestAutogradFunctional::test_jacobian_vectorize_correctness_multi_input_multi_output_base_tensor, test/test_autograd.py::TestAutogradFunctional::test_jacobian_vectorize_correctness_multi_input_multi_output_logging_tensor, test/test_autograd.py::TestAutogradFunctional::test_jacobian_vectorize_correctness_simple_base_tensor, test/test_autograd.py::TestAutogradFunctional::test_jacobian_vectorize_correctness_simple_logging_tensor, test/test_autograd.py::TestAutogradFunctional::test_jacobian_vectorize_correctness_unrelated_outputs_base_tensor, test/test_autograd.py::TestAutogradFunctional::test_jacobian_vectorize_correctness_unrelated_outputs_logging_tensor, test/test_autograd.py::TestAutogradFunctional::test_jacobian_vectorize_correctness_zero_dim_base_tensor, test/test_autograd.py::TestAutogradFunctional::test_jacobian_vectorize_correctness_zero_dim_logging_tensor, test/test_autograd.py::TestAutogradFunctional::test_jacobian_vectorize_raises_no_warnings_base_tensor, test/test_autograd.py::TestAutogradFunctional::test_jacobian_vectorize_raises_no_warnings_logging_tensor, test/test_autograd.py::TestAutogradFunctional::test_jvp_create_graph_base_tensor, test/test_autograd.py::TestAutogradFunctional::test_jvp_create_graph_logging_tensor, test/test_autograd.py::TestAutogradFunctional::test_jvp_err_check_base_tensor, test/test_autograd.py::TestAutogradFunctional::test_jvp_err_check_logging_tensor, test/test_autograd.py::TestAutogradFunctional::test_jvp_err_check_strict_base_tensor, test/test_autograd.py::TestAutogradFunctional::test_jvp_err_check_strict_logging_tensor, test/test_autograd.py::TestAutogradFunctional::test_jvp_no_grad_base_tensor, test/test_autograd.py::TestAutogradFunctional::test_jvp_no_grad_logging_tensor, test/test_autograd.py::TestAutogradFunctional::test_jvp_output_base_tensor, test/test_autograd.py::TestAutogradFunctional::test_jvp_output_logging_tensor, test/test_autograd.py::TestAutogradFunctional::test_jvp_scalar_base_tensor, test/test_autograd.py::TestAutogradFunctional::test_jvp_scalar_logging_tensor, test/test_autograd.py::TestAutogradFunctional::test_vhp_create_graph_base_tensor, test/test_autograd.py::TestAutogradFunctional::test_vhp_create_graph_logging_tensor, test/test_autograd.py::TestAutogradFunctional::test_vhp_err_check_base_tensor, test/test_autograd.py::TestAutogradFunctional::test_vhp_err_check_logging_tensor, test/test_autograd.py::TestAutogradFunctional::test_vhp_err_check_strict_base_tensor, test/test_autograd.py::TestAutogradFunctional::test_vhp_err_check_strict_logging_tensor, test/test_autograd.py::TestAutogradFunctional::test_vhp_no_grad_base_tensor, test/test_autograd.py::TestAutogradFunctional::test_vhp_no_grad_logging_tensor, test/test_autograd.py::TestAutogradFunctional::test_vhp_output_base_tensor, test/test_autograd.py::TestAutogradFunctional::test_vhp_output_logging_tensor, test/test_autograd.py::TestAutogradFunctional::test_vhp_scalar_base_tensor, test/test_autograd.py::TestAutogradFunctional::test_vhp_scalar_logging_tensor, test/test_autograd.py::TestAutogradFunctional::test_vjp_create_graph_base_tensor, test/test_autograd.py::TestAutogradFunctional::test_vjp_create_graph_logging_tensor, test/test_autograd.py::TestAutogradFunctional::test_vjp_err_check_base_tensor, test/test_autograd.py::TestAutogradFunctional::test_vjp_err_check_logging_tensor, test/test_autograd.py::TestAutogradFunctional::test_vjp_err_check_strict_base_tensor, test/test_autograd.py::TestAutogradFunctional::test_vjp_err_check_strict_logging_tensor, test/test_autograd.py::TestAutogradFunctional::test_vjp_no_grad_base_tensor, test/test_autograd.py::TestAutogradFunctional::test_vjp_no_grad_logging_tensor, test/test_autograd.py::TestAutogradFunctional::test_vjp_output_base_tensor, test/test_autograd.py::TestAutogradFunctional::test_vjp_output_logging_tensor, test/test_autograd.py::TestAutogradFunctional::test_vjp_scalar_base_tensor, test/test_autograd.py::TestAutogradFunctional::test_vjp_scalar_logging_tensor, test/test_autograd.py::TestAutogradLogging::test_logging, test/test_autograd.py::TestAutogradDeviceTypeCUDA::test_advanced_indexing_backwards_large_cuda, test/test_autograd.py::TestAutogradDeviceTypeCUDA::test_advanced_indexing_backwards_memory_format_cuda, test/test_autograd.py::TestAutogradDeviceTypeCUDA::test_backward_device_cuda, test/test_autograd.py::TestAutogradDeviceTypeCUDA::test_complex_scalar_backward_cuda, test/test_autograd.py::TestAutogradDeviceTypeCUDA::test_copy__cuda, test/test_autograd.py::TestAutogradDeviceTypeCUDA::test_copy_forward_ad_broadcasting_cuda, test/test_autograd.py::TestAutogradDeviceTypeCUDA::test_copy_forward_ad_same_layout_copies_grad_cuda, test/test_autograd.py::TestAutogradDeviceTypeCUDA::test_copy_r_to_c_cuda, test/test_autograd.py::TestAutogradDeviceTypeCUDA::test_cross_device_reentrant_autograd_cuda, test/test_autograd.py::TestAutogradDeviceTypeCUDA::test_free_unneeded_tensor_cuda, test/test_autograd.py::TestAutogradDeviceTypeCUDA::test_grad_assignment_cuda, test/test_autograd.py::TestAutogradDeviceTypeCUDA::test_gradcheck_input_output_different_device_cuda, test/test_autograd.py::TestAutogradDeviceTypeCUDA::test_inplace_multiple_output_view_of_view_cuda, test/test_autograd.py::TestAutogradDeviceTypeCUDA::test_inplace_on_view_backprop_base_cuda, test/test_autograd.py::TestAutogradDeviceTypeCUDA::test_inplace_on_view_backprop_view_cuda, test/test_autograd.py::TestAutogradDeviceTypeCUDA::test_inplace_on_view_backprop_view_of_view_cuda, test/test_autograd.py::TestAutogradDeviceTypeCUDA::test_inplace_on_view_gradcheck_cuda, test/test_autograd.py::TestAutogradDeviceTypeCUDA::test_inplace_on_view_makes_base_require_grad_cuda, test/test_autograd.py::TestAutogradDeviceTypeCUDA::test_inplace_on_view_modify_base_cuda, test/test_autograd.py::TestAutogradDeviceTypeCUDA::test_inplace_on_view_multi_output_safe_cuda, test/test_autograd.py::TestAutogradDeviceTypeCUDA::test_inplace_on_view_multi_output_unsafe_cuda, test/test_autograd.py::TestAutogradDeviceTypeCUDA::test_inplace_on_view_multiple_outputs_cuda, test/test_autograd.py::TestAutogradDeviceTypeCUDA::test_inplace_on_view_non_contig_cuda, test/test_autograd.py::TestAutogradDeviceTypeCUDA::test_inplace_on_view_of_multiple_output_view_cuda, test/test_autograd.py::TestAutogradDeviceTypeCUDA::test_inplace_on_view_of_view_cuda, test/test_autograd.py::TestAutogradDeviceTypeCUDA::test_inplace_on_view_python_cuda, test/test_autograd.py::TestAutogradDeviceTypeCUDA::test_inplace_on_view_then_no_grad_cuda, test/test_autograd.py::TestAutogradDeviceTypeCUDA::test_inplace_on_view_undefined_grad_output_cuda, test/test_autograd.py::TestAutogradDeviceTypeCUDA::test_inputbuffer_add_multidevice_cuda, test/test_autograd.py::TestAutogradDeviceTypeCUDA::test_min_max_median_backprops_to_all_values_cuda, test/test_autograd.py::TestAutogradDeviceTypeCUDA::test_mv_grad_stride_0_cuda, test/test_autograd.py::TestAutogradDeviceTypeCUDA::test_non_differentiable_ops_cuda, test/test_autograd.py::TestAutogradDeviceTypeCUDA::test_parameter_resize_cuda, test/test_autograd.py::TestAutogradDeviceTypeCUDA::test_pin_memory_cuda, test/test_autograd.py::TestAutogradDeviceTypeCUDA::test_pow_real_negative_base_complex_exponent_cuda, test/test_autograd.py::TestAutogradDeviceTypeCUDA::test_profiler_emit_itt_cuda, test/test_autograd.py::TestAutogradDeviceTypeCUDA::test_profiler_emit_nvtx_cuda, test/test_autograd.py::TestAutogradDeviceTypeCUDA::test_pyscalar_conversions_cuda, test/test_autograd.py::TestAutogradDeviceTypeCUDA::test_reentrant_parent_error_on_cpu_cuda, test/test_autograd.py::TestAutogradDeviceTypeCUDA::test_requires_grad_factory_cuda_float32, test/test_autograd.py::TestAutogradDeviceTypeCUDA::test_requires_grad_factory_cuda_float64, test/test_autograd.py::TestAutogradDeviceTypeCUDA::test_resize_version_bump_cuda, test/test_autograd.py::TestAutogradDeviceTypeCUDA::test_rnn_backward_to_input_but_not_parameters_cuda, test/test_autograd.py::TestAutogradDeviceTypeCUDA::test_scatter_index_reduce_amin_amax_backprops_to_all_values_cuda, test/test_autograd.py::TestAutogradDeviceTypeCUDA::test_scatter_index_reduce_prod_gradgrad_error_cuda, test/test_autograd.py::TestAutogradDeviceTypeCUDA::test_set_requires_grad_only_for_floats_cuda_float16, test/test_autograd.py::TestAutogradDeviceTypeCUDA::test_set_requires_grad_only_for_floats_cuda_float32, test/test_autograd.py::TestAutogradDeviceTypeCUDA::test_set_requires_grad_only_for_floats_cuda_float64, test/test_autograd.py::TestAutogradDeviceTypeCUDA::test_set_requires_grad_only_for_floats_cuda_int16, test/test_autograd.py::TestAutogradDeviceTypeCUDA::test_set_requires_grad_only_for_floats_cuda_int32, test/test_autograd.py::TestAutogradDeviceTypeCUDA::test_set_requires_grad_only_for_floats_cuda_int64, test/test_autograd.py::TestAutogradDeviceTypeCUDA::test_set_requires_grad_only_for_floats_cuda_int8, test/test_autograd.py::TestAutogradDeviceTypeCUDA::test_simple_reentrant_cross_device_cuda, test/test_autograd.py::TestAutogradDeviceTypeCUDA::test_sparse_backward_cuda_complex128, test/test_autograd.py::TestAutogradDeviceTypeCUDA::test_sparse_backward_cuda_float64, test/test_autograd.py::TestAutogradDeviceTypeCUDA::test_sparse_ctor_getter_backward_cuda_complex128, test/test_autograd.py::TestAutogradDeviceTypeCUDA::test_sparse_ctor_getter_backward_cuda_float64, test/test_autograd.py::TestAutogradDeviceTypeCUDA::test_sparse_mask_autograd_cuda, test/test_autograd.py::TestAutogradDeviceTypeCUDA::test_strided_leaf_grad_layout_cuda, test/test_autograd.py::TestAutogradDeviceTypeCUDA::test_to_r_to_c_cuda, test/test_autograd.py::TestAutogradDeviceTypeCUDA::test_unused_output_device_cuda, test/test_autograd.py::TestAutogradDeviceTypeCUDA::test_warning_in_backward_cuda, test/test_autograd.py::TestAutogradDeviceTypeCUDA::test_where_functional_cuda, test/test_autograd.py::TestAutogradDeviceTypeCUDA::test_where_scalar_cuda, test/test_autograd.py::TestAutogradDeviceTypeCUDA::test_zero_dim_param_mixed_device_grad_cuda, test/test_autograd.py::TestAutogradMultipleDispatchCUDA::test_atan2_zero_gradient_cuda, test/test_autograd.py::TestAutogradMultipleDispatchCUDA::test_autograd_composite_implicit_and_dispatch_registration_cuda, test/test_autograd.py::TestAutogradMultipleDispatchCUDA::test_autograd_multiple_dispatch_registrations_cuda, test/test_autograd.py::TestAutogradMultipleDispatchCUDA::test_backward_single_threaded_cuda, test/test_autograd.py::TestAutogradMultipleDispatchCUDA::test_backward_tls_stash_cuda, test/test_autograd.py::TestAutogradMultipleDispatchCUDA::test_foward_mode_AD_cuda, test/test_autograd.py::TestAutogradMultipleDispatchCUDA::test_is_retain_graph_cuda, test/test_autograd.py::TestAutogradMultipleDispatchCUDA::test_per_dispatch_key_input_saving_cuda, test/test_autograd.py::TestAutogradMultipleDispatchCUDA::test_set_sequence_nr_cuda, test/test_autograd.py::TestAutogradMultipleDispatchCUDA::test_view_copy_cuda, test/test_autograd.py::TestAutogradStreamSynchronizationCUDA::test_consumer_to_multi_producer_case_4_correctness_cuda, test/test_autograd.py::TestAutogradStreamSynchronizationCUDA::test_consumer_to_single_producer_case_2_correctness_cuda, test/test_autograd.py::TestAutogradStreamSynchronizationCUDA::test_consumer_to_single_producer_case_3_correctness_cuda, test/test_autograd.py::TestAutogradStreamSynchronizationCUDA::test_consumer_to_single_producer_case_3_correctness_non_default_ambient_stream_cuda, test/test_autograd.py::TestAutogradStreamSynchronizationCUDA::test_consumer_to_single_producer_case_4_correctness_cuda, test/test_autograd.py::TestAutogradStreamSynchronizationCUDA::test_side_stream_backward_overlap_cuda, test/test_autograd.py::TestAutogradStreamSynchronizationCUDA::test_warn_on_accumulate_grad_stream_mismatch_flag_cuda 2025-12-04T11:28:13.1420475Z 2025-12-04T11:28:13.1420681Z Finished test_autograd 1/1 ... [2025-12-04 11:28:13.112979][6541.055195789], took 0.99min 2025-12-04T11:28:13.1421325Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/test_autograd/test_autograd-428700f7d6c2123c.xml 2025-12-04T11:28:13.2274640Z Running test_dataloader 2/2 ... [2025-12-04 11:28:13.227243][6541.169460828] 2025-12-04T11:28:13.2275066Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2025-12-04T11:28:13.2278246Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'test_dataloader.py', '--shard-id=2', '--num-shards=2', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-12-04 11:28:13.227554] 2025-12-04T11:33:59.9899128Z 2025-12-04T11:33:59.9899924Z test_dataloader 2/2 was successful, full logs can be found in artifacts with path test/test-reports/test_dataloader_2.2_c5a35c6a3d01e804_.log 2025-12-04T11:33:59.9922505Z Running 92 items in this shard: test/test_dataloader.py::TestDatasetRandomSplit::test_lengths_must_equal_dataset_size, test/test_dataloader.py::TestDatasetRandomSplit::test_slicing_of_subset_of_dataset, test/test_dataloader.py::TestDatasetRandomSplit::test_splits_generator, test/test_dataloader.py::TestDatasetRandomSplit::test_splits_have_correct_size, test/test_dataloader.py::TestDatasetRandomSplit::test_splits_reproducibility, test/test_dataloader.py::TestTensorDataset::test_getitem_1d, test/test_dataloader.py::TestTensorDataset::test_single_tensor, test/test_dataloader.py::TestStackDataset::test_empty, test/test_dataloader.py::TestStackDataset::test_len, test/test_dataloader.py::TestConcatDataset::test_add_dataset, test/test_dataloader.py::TestConcatDataset::test_concat_raises_index_error, test/test_dataloader.py::TestConcatDataset::test_concat_two_singletons, test/test_dataloader.py::TestDataLoader::test_batch_sampler, test/test_dataloader.py::TestDataLoader::test_bulk_loading_nobatch, test/test_dataloader.py::TestDataLoader::test_chain_iterable_style_dataset, test/test_dataloader.py::TestDataLoader::test_default_collate_bad_sequence_type, test/test_dataloader.py::TestDataLoader::test_default_collate_dtype, test/test_dataloader.py::TestDataLoader::test_default_collate_numpy_memmap, test/test_dataloader.py::TestDataLoader::test_default_convert_mapping_keep_type, test/test_dataloader.py::TestDataLoader::test_default_convert_sequence_keep_type, test/test_dataloader.py::TestDataLoader::test_distributed_sampler_invalid_rank, test/test_dataloader.py::TestDataLoader::test_error_workers, test/test_dataloader.py::TestDataLoader::test_growing_dataset, test/test_dataloader.py::TestDataLoader::test_invalid_ctor_args_combinations, test/test_dataloader.py::TestDataLoader::test_large_sampler_indices, test/test_dataloader.py::TestDataLoader::test_multiple_dataloaders, test/test_dataloader.py::TestDataLoader::test_multiprocessing_iterdatapipe, test/test_dataloader.py::TestDataLoader::test_numpy_gen_state, test/test_dataloader.py::TestDataLoader::test_numpy_scalars, test/test_dataloader.py::TestDataLoader::test_proper_exit, test/test_dataloader.py::TestDataLoader::test_segfault, test/test_dataloader.py::TestDataLoader::test_sequential_batch, test/test_dataloader.py::TestDataLoader::test_sequential_nonbatch, test/test_dataloader.py::TestDataLoader::test_sequential_pin_memory, test/test_dataloader.py::TestDataLoader::test_sequential_workers, test/test_dataloader.py::TestDataLoader::test_shuffle_batch_none, test/test_dataloader.py::TestDataLoader::test_shuffle_batch_workers, test/test_dataloader.py::TestDataLoader::test_shuffle_batch_workers_prefetch, test/test_dataloader.py::TestDataLoader::test_shuffle_pin_memory, test/test_dataloader.py::TestDataLoader::test_shuffle_reproducibility, test/test_dataloader.py::TestDataLoader::test_shuffle_workers, test/test_dataloader.py::TestDataLoader::test_timeout, test/test_dataloader.py::TestDataLoader::test_typing, test/test_dataloader.py::TestDataLoader::test_worker_init_fn_forkserver, test/test_dataloader.py::TestDataLoader::test_worker_seed, test/test_dataloader.py::TestDataLoader::test_worker_seed_reproducibility, test/test_dataloader.py::IntegrationTestDataLoaderDataPipe::test_shuffler_iterdatapipe, test/test_dataloader.py::TestStringDataLoader::test_shuffle_pin_memory, test/test_dataloader.py::TestDictDataLoader::test_pin_memory, test/test_dataloader.py::TestDictDataLoader::test_pin_memory_device, test/test_dataloader.py::TestDictDataLoader::test_sequential_batch, test/test_dataloader.py::TestDataLoaderPersistentWorkers::test_builtin_collection_conversion, test/test_dataloader.py::TestDataLoaderPersistentWorkers::test_bulk_loading_nobatch, test/test_dataloader.py::TestDataLoaderPersistentWorkers::test_chain_iterable_style_dataset, test/test_dataloader.py::TestDataLoaderPersistentWorkers::test_default_collate_bad_sequence_type, test/test_dataloader.py::TestDataLoaderPersistentWorkers::test_default_collate_shared_tensor, test/test_dataloader.py::TestDataLoaderPersistentWorkers::test_default_convert_mapping_keep_type, test/test_dataloader.py::TestDataLoaderPersistentWorkers::test_distributed_sampler_invalid_rank, test/test_dataloader.py::TestDataLoaderPersistentWorkers::test_early_exit, test/test_dataloader.py::TestDataLoaderPersistentWorkers::test_error_in_init, test/test_dataloader.py::TestDataLoaderPersistentWorkers::test_error_workers, test/test_dataloader.py::TestDataLoaderPersistentWorkers::test_excessive_thread_creation_warning, test/test_dataloader.py::TestDataLoaderPersistentWorkers::test_fd_limit_exceeded, test/test_dataloader.py::TestDataLoaderPersistentWorkers::test_growing_dataset, test/test_dataloader.py::TestDataLoaderPersistentWorkers::test_invalid_assign_after_init, test/test_dataloader.py::TestDataLoaderPersistentWorkers::test_len, test/test_dataloader.py::TestDataLoaderPersistentWorkers::test_multi_epochs_reproducibility, test/test_dataloader.py::TestDataLoaderPersistentWorkers::test_multiple_dataloaders, test/test_dataloader.py::TestDataLoaderPersistentWorkers::test_multiprocessing_contexts, test/test_dataloader.py::TestDataLoaderPersistentWorkers::test_multiprocessing_iterdatapipe_with_dill, test/test_dataloader.py::TestDataLoaderPersistentWorkers::test_no_segfault, test/test_dataloader.py::TestDataLoaderPersistentWorkers::test_numpy, test/test_dataloader.py::TestDataLoaderPersistentWorkers::test_numpy_gen_state, test/test_dataloader.py::TestDataLoaderPersistentWorkers::test_proper_exit, test/test_dataloader.py::TestDataLoaderPersistentWorkers::test_random_sampler_len_with_replacement, test/test_dataloader.py::TestDataLoaderPersistentWorkers::test_random_sampler_len_without_replacement, test/test_dataloader.py::TestDataLoaderPersistentWorkers::test_sampler, test/test_dataloader.py::TestDataLoaderPersistentWorkers::test_sampler_reproducibility, test/test_dataloader.py::TestDataLoaderPersistentWorkers::test_sequential_batch, test/test_dataloader.py::TestDataLoaderPersistentWorkers::test_sequential_nonbatch, test/test_dataloader.py::TestDataLoaderPersistentWorkers::test_sequential_pin_memory, test/test_dataloader.py::TestDataLoaderPersistentWorkers::test_shuffle_batch, test/test_dataloader.py::TestDataLoaderPersistentWorkers::test_shuffle_reproducibility, test/test_dataloader.py::TestDataLoaderPersistentWorkers::test_typing, test/test_dataloader.py::TestDataLoaderPersistentWorkers::test_worker_init_fn_forkserver, test/test_dataloader.py::TestDataLoaderPersistentWorkers::test_worker_seed, test/test_dataloader.py::TestCustomPinFn::test_custom_batch_pin_worker, test/test_dataloader.py::TestIndividualWorkerQueue::test_ind_worker_queue, test/test_dataloader.py::TestSetAffinity::test_set_affinity_in_worker_init, test/test_dataloader.py::TestOutOfOrderDataLoader::test_in_order_iterable_ds, test/test_dataloader.py::TestDataLoaderDeviceTypeCUDA::test_nested_tensor_multiprocessing_context_forkserver_cuda, test/test_dataloader.py::TestDataLoaderDeviceTypeCUDA::test_sparse_tensor_multiprocessing_context_forkserver_cuda 2025-12-04T11:33:59.9942685Z 2025-12-04T11:33:59.9942882Z Finished test_dataloader 2/2 ... [2025-12-04 11:33:59.989035][6887.931251336], took 5.78min 2025-12-04T11:34:00.0152546Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/test_dataloader/test_dataloader-67cc6525e23b4ee5.xml 2025-12-04T11:34:00.4853371Z Uploading artifacts took 0.39 seconds 2025-12-04T11:34:00.4855708Z Running test_decomp 5/17 ... [2025-12-04 11:34:00.485344][6888.427559333] 2025-12-04T11:34:00.4856114Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2025-12-04T11:34:00.4859060Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'test_decomp.py', '--shard-id=5', '--num-shards=17', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-12-04 11:34:00.485662] 2025-12-04T11:45:03.6529603Z 2025-12-04T11:45:03.6530430Z test_decomp 5/17 was successful, full logs can be found in artifacts with path test/test-reports/test_decomp_5.17_bbc96279acf4332b_.log 2025-12-04T11:45:03.6646188Z Running 538 items in this shard: test/test_decomp.py::TestDecompCUDA::test_comprehensive_H_cuda_bfloat16, test/test_decomp.py::TestDecompCUDA::test_comprehensive_H_cuda_float32, test/test_decomp.py::TestDecompCUDA::test_comprehensive_T_cuda_uint8, test/test_decomp.py::TestDecompCUDA::test_comprehensive___getitem___cuda_complex128, test/test_decomp.py::TestDecompCUDA::test_comprehensive___getitem___cuda_float16, test/test_decomp.py::TestDecompCUDA::test_comprehensive___rmatmul___cuda_bfloat16, test/test_decomp.py::TestDecompCUDA::test_comprehensive___rmod___cuda_bfloat16, test/test_decomp.py::TestDecompCUDA::test_comprehensive___rmul___cuda_int8, test/test_decomp.py::TestDecompCUDA::test_comprehensive___rpow___cuda_float32, test/test_decomp.py::TestDecompCUDA::test_comprehensive__chunk_cat_cuda_float64, test/test_decomp.py::TestDecompCUDA::test_comprehensive__softmax_backward_data_cuda_float32, test/test_decomp.py::TestDecompCUDA::test_comprehensive_abs_cuda_complex64, test/test_decomp.py::TestDecompCUDA::test_comprehensive_acos_cuda_uint8, test/test_decomp.py::TestDecompCUDA::test_comprehensive_acosh_cuda_uint8, test/test_decomp.py::TestDecompCUDA::test_comprehensive_add_cuda_uint8, test/test_decomp.py::TestDecompCUDA::test_comprehensive_addbmm_cuda_bfloat16, test/test_decomp.py::TestDecompCUDA::test_comprehensive_addcdiv_cuda_bfloat16, test/test_decomp.py::TestDecompCUDA::test_comprehensive_addcdiv_cuda_complex64, test/test_decomp.py::TestDecompCUDA::test_comprehensive_addmm_decomposed_cuda_complex128, test/test_decomp.py::TestDecompCUDA::test_comprehensive_addmv_cuda_float32, test/test_decomp.py::TestDecompCUDA::test_comprehensive_addr_cuda_int16, test/test_decomp.py::TestDecompCUDA::test_comprehensive_alias_copy_cuda_bfloat16, test/test_decomp.py::TestDecompCUDA::test_comprehensive_allclose_cuda_complex128, test/test_decomp.py::TestDecompCUDA::test_comprehensive_any_cuda_float64, test/test_decomp.py::TestDecompCUDA::test_comprehensive_argsort_cuda_bool, test/test_decomp.py::TestDecompCUDA::test_comprehensive_argsort_cuda_int32, test/test_decomp.py::TestDecompCUDA::test_comprehensive_argsort_cuda_int64, test/test_decomp.py::TestDecompCUDA::test_comprehensive_argsort_cuda_int8, test/test_decomp.py::TestDecompCUDA::test_comprehensive_argsort_cuda_uint8, test/test_decomp.py::TestDecompCUDA::test_comprehensive_as_strided_copy_cuda_complex128, test/test_decomp.py::TestDecompCUDA::test_comprehensive_as_strided_scatter_cuda_bool, test/test_decomp.py::TestDecompCUDA::test_comprehensive_as_strided_scatter_cuda_complex128, test/test_decomp.py::TestDecompCUDA::test_comprehensive_asin_cuda_int64, test/test_decomp.py::TestDecompCUDA::test_comprehensive_atan_cuda_float16, test/test_decomp.py::TestDecompCUDA::test_comprehensive_atanh_cuda_complex32, test/test_decomp.py::TestDecompCUDA::test_comprehensive_atanh_cuda_complex64, test/test_decomp.py::TestDecompCUDA::test_comprehensive_atleast_1d_cuda_complex64, test/test_decomp.py::TestDecompCUDA::test_comprehensive_atleast_2d_cuda_float32, test/test_decomp.py::TestDecompCUDA::test_comprehensive_atleast_3d_cuda_complex32, test/test_decomp.py::TestDecompCUDA::test_comprehensive_atleast_3d_cuda_float32, test/test_decomp.py::TestDecompCUDA::test_comprehensive_baddbmm_cuda_float16, test/test_decomp.py::TestDecompCUDA::test_comprehensive_bernoulli_cuda_float16, test/test_decomp.py::TestDecompCUDA::test_comprehensive_block_diag_cuda_bool, test/test_decomp.py::TestDecompCUDA::test_comprehensive_block_diag_cuda_float64, test/test_decomp.py::TestDecompCUDA::test_comprehensive_broadcast_tensors_cuda_complex128, test/test_decomp.py::TestDecompCUDA::test_comprehensive_broadcast_tensors_cuda_float16, test/test_decomp.py::TestDecompCUDA::test_comprehensive_bucketize_cuda_uint8, test/test_decomp.py::TestDecompCUDA::test_comprehensive_byte_cuda_complex64, test/test_decomp.py::TestDecompCUDA::test_comprehensive_byte_cuda_float64, test/test_decomp.py::TestDecompCUDA::test_comprehensive_byte_cuda_int16, test/test_decomp.py::TestDecompCUDA::test_comprehensive_cdouble_cuda_int32, test/test_decomp.py::TestDecompCUDA::test_comprehensive_ceil_cuda_float64, test/test_decomp.py::TestDecompCUDA::test_comprehensive_cfloat_cuda_complex32, test/test_decomp.py::TestDecompCUDA::test_comprehensive_char_cuda_uint8, test/test_decomp.py::TestDecompCUDA::test_comprehensive_cholesky_solve_cuda_float32, test/test_decomp.py::TestDecompCUDA::test_comprehensive_clamp_cuda_float32, test/test_decomp.py::TestDecompCUDA::test_comprehensive_clamp_cuda_int8, test/test_decomp.py::TestDecompCUDA::test_comprehensive_clamp_min_cuda_bfloat16, test/test_decomp.py::TestDecompCUDA::test_comprehensive_clamp_min_cuda_float64, test/test_decomp.py::TestDecompCUDA::test_comprehensive_combinations_cuda_int64, test/test_decomp.py::TestDecompCUDA::test_comprehensive_constant_pad_nd_cuda_int8, test/test_decomp.py::TestDecompCUDA::test_comprehensive_contiguous_cuda_int32, test/test_decomp.py::TestDecompCUDA::test_comprehensive_cummin_cuda_bfloat16, test/test_decomp.py::TestDecompCUDA::test_comprehensive_deg2rad_cuda_int16, test/test_decomp.py::TestDecompCUDA::test_comprehensive_deg2rad_cuda_int64, test/test_decomp.py::TestDecompCUDA::test_comprehensive_diag_cuda_float64, test/test_decomp.py::TestDecompCUDA::test_comprehensive_diagflat_cuda_complex128, test/test_decomp.py::TestDecompCUDA::test_comprehensive_diagonal_scatter_cuda_bfloat16, test/test_decomp.py::TestDecompCUDA::test_comprehensive_div_no_rounding_mode_cuda_float64, test/test_decomp.py::TestDecompCUDA::test_comprehensive_double_cuda_bfloat16, test/test_decomp.py::TestDecompCUDA::test_comprehensive_double_cuda_uint8, test/test_decomp.py::TestDecompCUDA::test_comprehensive_empty_cuda_complex128, test/test_decomp.py::TestDecompCUDA::test_comprehensive_empty_cuda_float32, test/test_decomp.py::TestDecompCUDA::test_comprehensive_eq_cuda_bool, test/test_decomp.py::TestDecompCUDA::test_comprehensive_eq_cuda_complex128, test/test_decomp.py::TestDecompCUDA::test_comprehensive_eq_cuda_int32, test/test_decomp.py::TestDecompCUDA::test_comprehensive_erfinv_cuda_int32, test/test_decomp.py::TestDecompCUDA::test_comprehensive_exp_cuda_int64, test/test_decomp.py::TestDecompCUDA::test_comprehensive_expand_cuda_bfloat16, test/test_decomp.py::TestDecompCUDA::test_comprehensive_expm1_cuda_float32, test/test_decomp.py::TestDecompCUDA::test_comprehensive_expm1_cuda_int64, test/test_decomp.py::TestDecompCUDA::test_comprehensive_exponential_cuda_bfloat16, test/test_decomp.py::TestDecompCUDA::test_comprehensive_fft_fftshift_cuda_float32, test/test_decomp.py::TestDecompCUDA::test_comprehensive_fft_hfft2_cuda_int8, test/test_decomp.py::TestDecompCUDA::test_comprehensive_fft_ifft2_cuda_uint8, test/test_decomp.py::TestDecompCUDA::test_comprehensive_fft_ifft_cuda_int16, test/test_decomp.py::TestDecompCUDA::test_comprehensive_fft_ihfft2_cuda_float64, test/test_decomp.py::TestDecompCUDA::test_comprehensive_fft_ihfft_cuda_int16, test/test_decomp.py::TestDecompCUDA::test_comprehensive_fft_ihfftn_cuda_float64, test/test_decomp.py::TestDecompCUDA::test_comprehensive_fft_irfft2_cuda_bool, test/test_decomp.py::TestDecompCUDA::test_comprehensive_fft_irfft2_cuda_float32, test/test_decomp.py::TestDecompCUDA::test_comprehensive_fft_irfft_cuda_complex32, test/test_decomp.py::TestDecompCUDA::test_comprehensive_fft_irfft_cuda_complex64, test/test_decomp.py::TestDecompCUDA::test_comprehensive_fft_irfftn_cuda_float64, test/test_decomp.py::TestDecompCUDA::test_comprehensive_fft_rfftn_cuda_float16, test/test_decomp.py::TestDecompCUDA::test_comprehensive_fill_cuda_int16, test/test_decomp.py::TestDecompCUDA::test_comprehensive_flatten_cuda_int64, test/test_decomp.py::TestDecompCUDA::test_comprehensive_fliplr_cuda_int32, test/test_decomp.py::TestDecompCUDA::test_comprehensive_float_power_cuda_complex64, test/test_decomp.py::TestDecompCUDA::test_comprehensive_float_power_cuda_float16, test/test_decomp.py::TestDecompCUDA::test_comprehensive_floor_divide_cuda_bfloat16, test/test_decomp.py::TestDecompCUDA::test_comprehensive_fmax_cuda_uint8, test/test_decomp.py::TestDecompCUDA::test_comprehensive_fmin_cuda_bfloat16, test/test_decomp.py::TestDecompCUDA::test_comprehensive_full_like_cuda_float16, test/test_decomp.py::TestDecompCUDA::test_comprehensive_gcd_cuda_int16, test/test_decomp.py::TestDecompCUDA::test_comprehensive_gcd_cuda_uint8, test/test_decomp.py::TestDecompCUDA::test_comprehensive_geometric_cuda_uint8, test/test_decomp.py::TestDecompCUDA::test_comprehensive_geqrf_cuda_float64, test/test_decomp.py::TestDecompCUDA::test_comprehensive_gradient_cuda_float16, test/test_decomp.py::TestDecompCUDA::test_comprehensive_gradient_cuda_int8, test/test_decomp.py::TestDecompCUDA::test_comprehensive_half_cuda_float32, test/test_decomp.py::TestDecompCUDA::test_comprehensive_hash_tensor_cuda_int8, test/test_decomp.py::TestDecompCUDA::test_comprehensive_hsplit_cuda_complex64, test/test_decomp.py::TestDecompCUDA::test_comprehensive_hypot_cuda_bfloat16, test/test_decomp.py::TestDecompCUDA::test_comprehensive_i0_cuda_int16, test/test_decomp.py::TestDecompCUDA::test_comprehensive_index_copy_cuda_float16, test/test_decomp.py::TestDecompCUDA::test_comprehensive_index_fill_cuda_int16, test/test_decomp.py::TestDecompCUDA::test_comprehensive_index_reduce_mean_cuda_bfloat16, test/test_decomp.py::TestDecompCUDA::test_comprehensive_int_cuda_int16, test/test_decomp.py::TestDecompCUDA::test_comprehensive_isfinite_cuda_bool, test/test_decomp.py::TestDecompCUDA::test_comprehensive_isfinite_cuda_int32, test/test_decomp.py::TestDecompCUDA::test_comprehensive_isnan_cuda_uint8, test/test_decomp.py::TestDecompCUDA::test_comprehensive_isposinf_cuda_int64, test/test_decomp.py::TestDecompCUDA::test_comprehensive_isreal_cuda_float32, test/test_decomp.py::TestDecompCUDA::test_comprehensive_isreal_cuda_int32, test/test_decomp.py::TestDecompCUDA::test_comprehensive_item_cuda_bool, test/test_decomp.py::TestDecompCUDA::test_comprehensive_item_cuda_int64, test/test_decomp.py::TestDecompCUDA::test_comprehensive_jiterator_2inputs_2outputs_cuda_uint8, test/test_decomp.py::TestDecompCUDA::test_comprehensive_jiterator_4inputs_with_extra_args_cuda_bool, test/test_decomp.py::TestDecompCUDA::test_comprehensive_jiterator_binary_return_by_ref_cuda_float64, test/test_decomp.py::TestDecompCUDA::test_comprehensive_jiterator_binary_return_by_ref_cuda_int64, test/test_decomp.py::TestDecompCUDA::test_comprehensive_kron_cuda_float16, test/test_decomp.py::TestDecompCUDA::test_comprehensive_ldexp_cuda_int8, test/test_decomp.py::TestDecompCUDA::test_comprehensive_le_cuda_float32, test/test_decomp.py::TestDecompCUDA::test_comprehensive_linalg_cross_cuda_bfloat16, test/test_decomp.py::TestDecompCUDA::test_comprehensive_linalg_eigh_cuda_complex128, test/test_decomp.py::TestDecompCUDA::test_comprehensive_linalg_eigvalsh_cuda_float32, test/test_decomp.py::TestDecompCUDA::test_comprehensive_linalg_inv_cuda_float32, test/test_decomp.py::TestDecompCUDA::test_comprehensive_linalg_inv_ex_cuda_float32, test/test_decomp.py::TestDecompCUDA::test_comprehensive_linalg_lu_cuda_complex128, test/test_decomp.py::TestDecompCUDA::test_comprehensive_linalg_matrix_rank_cuda_float64, test/test_decomp.py::TestDecompCUDA::test_comprehensive_linalg_norm_cuda_bfloat16, test/test_decomp.py::TestDecompCUDA::test_comprehensive_linalg_pinv_singular_cuda_complex128, test/test_decomp.py::TestDecompCUDA::test_comprehensive_linalg_pinv_singular_cuda_complex64, test/test_decomp.py::TestDecompCUDA::test_comprehensive_linalg_vander_cuda_int8, test/test_decomp.py::TestDecompCUDA::test_comprehensive_linalg_vecdot_cuda_float16, test/test_decomp.py::TestDecompCUDA::test_comprehensive_linspace_tensor_overload_cuda_complex128, test/test_decomp.py::TestDecompCUDA::test_comprehensive_linspace_tensor_overload_cuda_int16, test/test_decomp.py::TestDecompCUDA::test_comprehensive_log10_cuda_bfloat16, test/test_decomp.py::TestDecompCUDA::test_comprehensive_log1p_cuda_float32, test/test_decomp.py::TestDecompCUDA::test_comprehensive_log_softmax_cuda_bfloat16, test/test_decomp.py::TestDecompCUDA::test_comprehensive_logaddexp2_cuda_float32, test/test_decomp.py::TestDecompCUDA::test_comprehensive_logical_not_cuda_float32, test/test_decomp.py::TestDecompCUDA::test_comprehensive_logical_not_cuda_int64, test/test_decomp.py::TestDecompCUDA::test_comprehensive_logical_not_cuda_uint8, test/test_decomp.py::TestDecompCUDA::test_comprehensive_logical_or_cuda_int8, test/test_decomp.py::TestDecompCUDA::test_comprehensive_logical_xor_cuda_int32, test/test_decomp.py::TestDecompCUDA::test_comprehensive_logit_cuda_int64, test/test_decomp.py::TestDecompCUDA::test_comprehensive_logspace_cuda_int64, test/test_decomp.py::TestDecompCUDA::test_comprehensive_logspace_tensor_overload_cuda_float32, test/test_decomp.py::TestDecompCUDA::test_comprehensive_logsumexp_cuda_bfloat16, test/test_decomp.py::TestDecompCUDA::test_comprehensive_lt_cuda_int16, test/test_decomp.py::TestDecompCUDA::test_comprehensive_lu_cuda_float32, test/test_decomp.py::TestDecompCUDA::test_comprehensive_mH_cuda_bfloat16, test/test_decomp.py::TestDecompCUDA::test_comprehensive_mH_cuda_int64, test/test_decomp.py::TestDecompCUDA::test_comprehensive_mT_cuda_bool, test/test_decomp.py::TestDecompCUDA::test_comprehensive_mT_cuda_complex64, test/test_decomp.py::TestDecompCUDA::test_comprehensive_masked_cumprod_cuda_complex128, test/test_decomp.py::TestDecompCUDA::test_comprehensive_masked_cumprod_cuda_float32, test/test_decomp.py::TestDecompCUDA::test_comprehensive_masked_fill_cuda_int16, test/test_decomp.py::TestDecompCUDA::test_comprehensive_masked_median_cuda_bfloat16, test/test_decomp.py::TestDecompCUDA::test_comprehensive_masked_normalize_cuda_bfloat16, test/test_decomp.py::TestDecompCUDA::test_comprehensive_masked_select_cuda_bool, test/test_decomp.py::TestDecompCUDA::test_comprehensive_masked_select_cuda_int32, test/test_decomp.py::TestDecompCUDA::test_comprehensive_masked_std_cuda_int8, test/test_decomp.py::TestDecompCUDA::test_comprehensive_masked_sum_cuda_bfloat16, test/test_decomp.py::TestDecompCUDA::test_comprehensive_masked_sum_cuda_uint8, test/test_decomp.py::TestDecompCUDA::test_comprehensive_masked_var_cuda_float16, test/test_decomp.py::TestDecompCUDA::test_comprehensive_meshgrid_variadic_tensors_cuda_complex128, test/test_decomp.py::TestDecompCUDA::test_comprehensive_min_binary_cuda_int16, test/test_decomp.py::TestDecompCUDA::test_comprehensive_min_reduction_with_dim_cuda_int8, test/test_decomp.py::TestDecompCUDA::test_comprehensive_min_reduction_with_dim_cuda_uint8, test/test_decomp.py::TestDecompCUDA::test_comprehensive_minimum_cuda_float64, test/test_decomp.py::TestDecompCUDA::test_comprehensive_mode_cuda_int32, test/test_decomp.py::TestDecompCUDA::test_comprehensive_multinomial_cuda_bfloat16, test/test_decomp.py::TestDecompCUDA::test_comprehensive_mv_cuda_complex128, test/test_decomp.py::TestDecompCUDA::test_comprehensive_mvlgamma_mvlgamma_p_1_cuda_float32, test/test_decomp.py::TestDecompCUDA::test_comprehensive_nansum_cuda_complex128, test/test_decomp.py::TestDecompCUDA::test_comprehensive_nansum_cuda_int64, test/test_decomp.py::TestDecompCUDA::test_comprehensive_nansum_cuda_uint8, test/test_decomp.py::TestDecompCUDA::test_comprehensive_narrow_copy_cuda_complex128, test/test_decomp.py::TestDecompCUDA::test_comprehensive_narrow_cuda_uint8, test/test_decomp.py::TestDecompCUDA::test_comprehensive_native_batch_norm_cuda_float16, test/test_decomp.py::TestDecompCUDA::test_comprehensive_new_full_cuda_complex128, test/test_decomp.py::TestDecompCUDA::test_comprehensive_new_ones_cuda_complex128, test/test_decomp.py::TestDecompCUDA::test_comprehensive_new_ones_cuda_complex32, test/test_decomp.py::TestDecompCUDA::test_comprehensive_nn_functional_adaptive_max_pool1d_cuda_bfloat16, test/test_decomp.py::TestDecompCUDA::test_comprehensive_nn_functional_batch_norm_cuda_float64, test/test_decomp.py::TestDecompCUDA::test_comprehensive_nn_functional_celu_cuda_float16, test/test_decomp.py::TestDecompCUDA::test_comprehensive_nn_functional_conv1d_cuda_float32, test/test_decomp.py::TestDecompCUDA::test_comprehensive_nn_functional_conv_transpose1d_cuda_float32, test/test_decomp.py::TestDecompCUDA::test_comprehensive_nn_functional_cross_entropy_cuda_float64, test/test_decomp.py::TestDecompCUDA::test_comprehensive_nn_functional_gelu_cuda_float16, test/test_decomp.py::TestDecompCUDA::test_comprehensive_nn_functional_hardtanh_cuda_int16, test/test_decomp.py::TestDecompCUDA::test_comprehensive_nn_functional_interpolate_area_cuda_float16, test/test_decomp.py::TestDecompCUDA::test_comprehensive_nn_functional_interpolate_trilinear_cuda_float16, test/test_decomp.py::TestDecompCUDA::test_comprehensive_nn_functional_layer_norm_cuda_bfloat16, test/test_decomp.py::TestDecompCUDA::test_comprehensive_nn_functional_leaky_relu_cuda_float64, test/test_decomp.py::TestDecompCUDA::test_comprehensive_nn_functional_margin_ranking_loss_cuda_float32, test/test_decomp.py::TestDecompCUDA::test_comprehensive_nn_functional_max_pool1d_cuda_float16, test/test_decomp.py::TestDecompCUDA::test_comprehensive_nn_functional_max_pool3d_cuda_float32, test/test_decomp.py::TestDecompCUDA::test_comprehensive_nn_functional_max_unpool1d_grad_cuda_float32, test/test_decomp.py::TestDecompCUDA::test_comprehensive_nn_functional_max_unpool2d_cuda_float64, test/test_decomp.py::TestDecompCUDA::test_comprehensive_nn_functional_max_unpool3d_cuda_bfloat16, test/test_decomp.py::TestDecompCUDA::test_comprehensive_nn_functional_multilabel_margin_loss_cuda_float32, test/test_decomp.py::TestDecompCUDA::test_comprehensive_nn_functional_multilabel_margin_loss_cuda_float64, test/test_decomp.py::TestDecompCUDA::test_comprehensive_nn_functional_multilabel_soft_margin_loss_cuda_bfloat16, test/test_decomp.py::TestDecompCUDA::test_comprehensive_nn_functional_pad_circular_cuda_float64, test/test_decomp.py::TestDecompCUDA::test_comprehensive_nn_functional_pad_replicate_cuda_int64, test/test_decomp.py::TestDecompCUDA::test_comprehensive_nn_functional_pad_replicate_negative_cuda_bfloat16, test/test_decomp.py::TestDecompCUDA::test_comprehensive_nn_functional_pixel_shuffle_cuda_bool, test/test_decomp.py::TestDecompCUDA::test_comprehensive_nn_functional_pixel_unshuffle_cuda_complex128, test/test_decomp.py::TestDecompCUDA::test_comprehensive_nn_functional_relu6_cuda_int8, test/test_decomp.py::TestDecompCUDA::test_comprehensive_nn_functional_softshrink_cuda_float32, test/test_decomp.py::TestDecompCUDA::test_comprehensive_nn_functional_softsign_cuda_int16, test/test_decomp.py::TestDecompCUDA::test_comprehensive_nn_functional_threshold_cuda_float32, test/test_decomp.py::TestDecompCUDA::test_comprehensive_nn_functional_triplet_margin_loss_cuda_uint8, test/test_decomp.py::TestDecompCUDA::test_comprehensive_nn_functional_triplet_margin_with_distance_loss_cuda_bfloat16, test/test_decomp.py::TestDecompCUDA::test_comprehensive_nn_functional_triplet_margin_with_distance_loss_cuda_float64, test/test_decomp.py::TestDecompCUDA::test_comprehensive_nn_functional_unfold_cuda_bool, test/test_decomp.py::TestDecompCUDA::test_comprehensive_nn_functional_unfold_cuda_complex128, test/test_decomp.py::TestDecompCUDA::test_comprehensive_nn_functional_upsample_bilinear_cuda_float16, test/test_decomp.py::TestDecompCUDA::test_comprehensive_nonzero_cuda_int16, test/test_decomp.py::TestDecompCUDA::test_comprehensive_nonzero_cuda_int8, test/test_decomp.py::TestDecompCUDA::test_comprehensive_normal_number_mean_cuda_float32, test/test_decomp.py::TestDecompCUDA::test_comprehensive_ones_like_cuda_bool, test/test_decomp.py::TestDecompCUDA::test_comprehensive_ones_like_cuda_complex32, test/test_decomp.py::TestDecompCUDA::test_comprehensive_ones_like_cuda_int16, test/test_decomp.py::TestDecompCUDA::test_comprehensive_ormqr_cuda_complex128, test/test_decomp.py::TestDecompCUDA::test_comprehensive_outer_cuda_float32, test/test_decomp.py::TestDecompCUDA::test_comprehensive_permute_cuda_complex128, test/test_decomp.py::TestDecompCUDA::test_comprehensive_permute_cuda_float16, test/test_decomp.py::TestDecompCUDA::test_comprehensive_pinverse_cuda_complex128, test/test_decomp.py::TestDecompCUDA::test_comprehensive_polar_cuda_float64, test/test_decomp.py::TestDecompCUDA::test_comprehensive_polygamma_polygamma_n_0_cuda_bfloat16, test/test_decomp.py::TestDecompCUDA::test_comprehensive_polygamma_polygamma_n_2_cuda_float32, test/test_decomp.py::TestDecompCUDA::test_comprehensive_polygamma_polygamma_n_3_cuda_bfloat16, test/test_decomp.py::TestDecompCUDA::test_comprehensive_positive_cuda_int8, test/test_decomp.py::TestDecompCUDA::test_comprehensive_pow_cuda_complex128, test/test_decomp.py::TestDecompCUDA::test_comprehensive_pow_cuda_complex64, test/test_decomp.py::TestDecompCUDA::test_comprehensive_pow_cuda_float64, test/test_decomp.py::TestDecompCUDA::test_comprehensive_prod_cuda_bfloat16, test/test_decomp.py::TestDecompCUDA::test_comprehensive_prod_cuda_int16, test/test_decomp.py::TestDecompCUDA::test_comprehensive_randint_cuda_bfloat16, test/test_decomp.py::TestDecompCUDA::test_comprehensive_repeat_interleave_cuda_complex32, test/test_decomp.py::TestDecompCUDA::test_comprehensive_reshape_as_cuda_complex64, test/test_decomp.py::TestDecompCUDA::test_comprehensive_resize_as__cuda_int32, test/test_decomp.py::TestDecompCUDA::test_comprehensive_resize_as__cuda_int64, test/test_decomp.py::TestDecompCUDA::test_comprehensive_resolve_conj_cuda_bfloat16, test/test_decomp.py::TestDecompCUDA::test_comprehensive_resolve_conj_cuda_complex128, test/test_decomp.py::TestDecompCUDA::test_comprehensive_resolve_conj_cuda_int64, test/test_decomp.py::TestDecompCUDA::test_comprehensive_resolve_conj_cuda_int8, test/test_decomp.py::TestDecompCUDA::test_comprehensive_resolve_neg_cuda_bfloat16, test/test_decomp.py::TestDecompCUDA::test_comprehensive_resolve_neg_cuda_complex128, test/test_decomp.py::TestDecompCUDA::test_comprehensive_rot90_cuda_complex64, test/test_decomp.py::TestDecompCUDA::test_comprehensive_scatter_add_cuda_float64, test/test_decomp.py::TestDecompCUDA::test_comprehensive_scatter_add_cuda_int64, test/test_decomp.py::TestDecompCUDA::test_comprehensive_scatter_cuda_complex128, test/test_decomp.py::TestDecompCUDA::test_comprehensive_scatter_reduce_amin_cuda_float32, test/test_decomp.py::TestDecompCUDA::test_comprehensive_scatter_reduce_amin_cuda_float64, test/test_decomp.py::TestDecompCUDA::test_comprehensive_scatter_reduce_prod_cuda_int8, test/test_decomp.py::TestDecompCUDA::test_comprehensive_searchsorted_cuda_float64, test/test_decomp.py::TestDecompCUDA::test_comprehensive_select_scatter_cuda_float16, test/test_decomp.py::TestDecompCUDA::test_comprehensive_select_scatter_cuda_int64, test/test_decomp.py::TestDecompCUDA::test_comprehensive_sgn_cuda_bfloat16, test/test_decomp.py::TestDecompCUDA::test_comprehensive_sgn_cuda_int64, test/test_decomp.py::TestDecompCUDA::test_comprehensive_sigmoid_cuda_bool, test/test_decomp.py::TestDecompCUDA::test_comprehensive_sigmoid_cuda_int8, test/test_decomp.py::TestDecompCUDA::test_comprehensive_signal_windows_bartlett_cuda_float64, test/test_decomp.py::TestDecompCUDA::test_comprehensive_signal_windows_cosine_cuda_float32, test/test_decomp.py::TestDecompCUDA::test_comprehensive_signal_windows_hamming_cuda_float64, test/test_decomp.py::TestDecompCUDA::test_comprehensive_sin_cuda_float64, test/test_decomp.py::TestDecompCUDA::test_comprehensive_sin_cuda_int32, test/test_decomp.py::TestDecompCUDA::test_comprehensive_sin_cuda_int64, test/test_decomp.py::TestDecompCUDA::test_comprehensive_sinc_cuda_complex64, test/test_decomp.py::TestDecompCUDA::test_comprehensive_sinh_cuda_bfloat16, test/test_decomp.py::TestDecompCUDA::test_comprehensive_slice_scatter_cuda_float32, test/test_decomp.py::TestDecompCUDA::test_comprehensive_softmax_with_dtype_cuda_float16, test/test_decomp.py::TestDecompCUDA::test_comprehensive_sort_cuda_uint8, test/test_decomp.py::TestDecompCUDA::test_comprehensive_special_bessel_j0_cuda_int64, test/test_decomp.py::TestDecompCUDA::test_comprehensive_special_chebyshev_polynomial_t_cuda_bool, test/test_decomp.py::TestDecompCUDA::test_comprehensive_special_chebyshev_polynomial_t_cuda_uint8, test/test_decomp.py::TestDecompCUDA::test_comprehensive_special_i0e_cuda_float32, test/test_decomp.py::TestDecompCUDA::test_comprehensive_special_i0e_cuda_float64, test/test_decomp.py::TestDecompCUDA::test_comprehensive_special_i1_cuda_float32, test/test_decomp.py::TestDecompCUDA::test_comprehensive_special_i1_cuda_float64, test/test_decomp.py::TestDecompCUDA::test_comprehensive_special_i1_cuda_int16, test/test_decomp.py::TestDecompCUDA::test_comprehensive_special_i1e_cuda_float16, test/test_decomp.py::TestDecompCUDA::test_comprehensive_special_modified_bessel_i1_cuda_int16, test/test_decomp.py::TestDecompCUDA::test_comprehensive_special_modified_bessel_k0_cuda_int16, test/test_decomp.py::TestDecompCUDA::test_comprehensive_special_ndtr_cuda_bfloat16, test/test_decomp.py::TestDecompCUDA::test_comprehensive_special_ndtri_cuda_int32, test/test_decomp.py::TestDecompCUDA::test_comprehensive_special_spherical_bessel_j0_cuda_int32, test/test_decomp.py::TestDecompCUDA::test_comprehensive_split_cuda_complex64, test/test_decomp.py::TestDecompCUDA::test_comprehensive_split_list_args_cuda_bool, test/test_decomp.py::TestDecompCUDA::test_comprehensive_split_with_sizes_copy_cuda_complex64, test/test_decomp.py::TestDecompCUDA::test_comprehensive_squeeze_copy_cuda_float16, test/test_decomp.py::TestDecompCUDA::test_comprehensive_squeeze_copy_cuda_float64, test/test_decomp.py::TestDecompCUDA::test_comprehensive_squeeze_cuda_complex128, test/test_decomp.py::TestDecompCUDA::test_comprehensive_squeeze_multiple_cuda_bfloat16, test/test_decomp.py::TestDecompCUDA::test_comprehensive_squeeze_multiple_cuda_int16, test/test_decomp.py::TestDecompCUDA::test_comprehensive_stack_cuda_int16, test/test_decomp.py::TestDecompCUDA::test_comprehensive_std_unbiased_cuda_bfloat16, test/test_decomp.py::TestDecompCUDA::test_comprehensive_std_unbiased_cuda_float32, test/test_decomp.py::TestDecompCUDA::test_comprehensive_sub_cuda_float16, test/test_decomp.py::TestDecompCUDA::test_comprehensive_sum_to_size_cuda_float16, test/test_decomp.py::TestDecompCUDA::test_comprehensive_t_copy_cuda_int8, test/test_decomp.py::TestDecompCUDA::test_comprehensive_t_cuda_bool, test/test_decomp.py::TestDecompCUDA::test_comprehensive_t_cuda_int32, test/test_decomp.py::TestDecompCUDA::test_comprehensive_take_cuda_bool, test/test_decomp.py::TestDecompCUDA::test_comprehensive_take_cuda_float32, test/test_decomp.py::TestDecompCUDA::test_comprehensive_tan_cuda_uint8, test/test_decomp.py::TestDecompCUDA::test_comprehensive_tile_cuda_int16, test/test_decomp.py::TestDecompCUDA::test_comprehensive_tile_cuda_uint8, test/test_decomp.py::TestDecompCUDA::test_comprehensive_to_sparse_cuda_int16, test/test_decomp.py::TestDecompCUDA::test_comprehensive_topk_cuda_float16, test/test_decomp.py::TestDecompCUDA::test_comprehensive_topk_cuda_float64, test/test_decomp.py::TestDecompCUDA::test_comprehensive_torch_ops_aten__safe_softmax_default_cuda_int64, test/test_decomp.py::TestDecompCUDA::test_comprehensive_trapezoid_cuda_float16, test/test_decomp.py::TestDecompCUDA::test_comprehensive_trapezoid_cuda_int64, test/test_decomp.py::TestDecompCUDA::test_comprehensive_trapz_cuda_bfloat16, test/test_decomp.py::TestDecompCUDA::test_comprehensive_trapz_cuda_int32, test/test_decomp.py::TestDecompCUDA::test_comprehensive_triu_cuda_bool, test/test_decomp.py::TestDecompCUDA::test_comprehensive_triu_indices_cuda_int64, test/test_decomp.py::TestDecompCUDA::test_comprehensive_true_divide_cuda_uint8, test/test_decomp.py::TestDecompCUDA::test_comprehensive_unbind_copy_cuda_int16, test/test_decomp.py::TestDecompCUDA::test_comprehensive_unbind_cuda_bfloat16, test/test_decomp.py::TestDecompCUDA::test_comprehensive_unbind_cuda_bool, test/test_decomp.py::TestDecompCUDA::test_comprehensive_unique_cuda_float32, test/test_decomp.py::TestDecompCUDA::test_comprehensive_unsafe_chunk_cuda_int64, test/test_decomp.py::TestDecompCUDA::test_comprehensive_unsqueeze_cuda_complex128, test/test_decomp.py::TestDecompCUDA::test_comprehensive_var_cuda_float32, test/test_decomp.py::TestDecompCUDA::test_comprehensive_var_mean_unbiased_cuda_float16, test/test_decomp.py::TestDecompCUDA::test_comprehensive_var_unbiased_cuda_float16, test/test_decomp.py::TestDecompCUDA::test_comprehensive_vdot_cuda_float16, test/test_decomp.py::TestDecompCUDA::test_comprehensive_view_as_cuda_uint8, test/test_decomp.py::TestDecompCUDA::test_comprehensive_vsplit_cuda_complex64, test/test_decomp.py::TestDecompCUDA::test_comprehensive_where_cuda_int16, test/test_decomp.py::TestDecompCUDA::test_comprehensive_zero__cuda_int32, test/test_decomp.py::TestDecompCUDA::test_comprehensive_zeros_cuda_int16, test/test_decomp.py::TestDecompCUDA::test_quick__batch_norm_with_update_cuda_float16, test/test_decomp.py::TestDecompCUDA::test_quick__chunk_cat_cuda_float32, test/test_decomp.py::TestDecompCUDA::test_quick_abs_cuda_float64, test/test_decomp.py::TestDecompCUDA::test_quick_acos_cuda_float16, test/test_decomp.py::TestDecompCUDA::test_quick_acosh_cuda_bfloat16, test/test_decomp.py::TestDecompCUDA::test_quick_acosh_cuda_int8, test/test_decomp.py::TestDecompCUDA::test_quick_add_cuda_complex32, test/test_decomp.py::TestDecompCUDA::test_quick_add_cuda_float16, test/test_decomp.py::TestDecompCUDA::test_quick_add_cuda_int16, test/test_decomp.py::TestDecompCUDA::test_quick_addmv_cuda_complex64, test/test_decomp.py::TestDecompCUDA::test_quick_addr_cuda_bfloat16, test/test_decomp.py::TestDecompCUDA::test_quick_addr_cuda_int32, test/test_decomp.py::TestDecompCUDA::test_quick_all_cuda_float32, test/test_decomp.py::TestDecompCUDA::test_quick_all_cuda_float64, test/test_decomp.py::TestDecompCUDA::test_quick_amin_cuda_float16, test/test_decomp.py::TestDecompCUDA::test_quick_amin_cuda_int32, test/test_decomp.py::TestDecompCUDA::test_quick_any_cuda_bfloat16, test/test_decomp.py::TestDecompCUDA::test_quick_as_strided_scatter_cuda_complex64, test/test_decomp.py::TestDecompCUDA::test_quick_as_strided_scatter_cuda_int32, test/test_decomp.py::TestDecompCUDA::test_quick_asin_cuda_int32, test/test_decomp.py::TestDecompCUDA::test_quick_atan_cuda_int8, test/test_decomp.py::TestDecompCUDA::test_quick_atanh_cuda_float16, test/test_decomp.py::TestDecompCUDA::test_quick_atanh_cuda_uint8, test/test_decomp.py::TestDecompCUDA::test_quick_bitwise_not_cuda_int8, test/test_decomp.py::TestDecompCUDA::test_quick_bitwise_right_shift_cuda_uint8, test/test_decomp.py::TestDecompCUDA::test_quick_cat_cuda_complex64, test/test_decomp.py::TestDecompCUDA::test_quick_cat_cuda_int16, test/test_decomp.py::TestDecompCUDA::test_quick_cauchy_cuda_float32, test/test_decomp.py::TestDecompCUDA::test_quick_ceil_cuda_bfloat16, test/test_decomp.py::TestDecompCUDA::test_quick_ceil_cuda_int64, test/test_decomp.py::TestDecompCUDA::test_quick_clamp_cuda_float32, test/test_decomp.py::TestDecompCUDA::test_quick_clamp_cuda_float64, test/test_decomp.py::TestDecompCUDA::test_quick_clamp_max_cuda_int16, test/test_decomp.py::TestDecompCUDA::test_quick_clone_cuda_bool, test/test_decomp.py::TestDecompCUDA::test_quick_conj_physical_cuda_bool, test/test_decomp.py::TestDecompCUDA::test_quick_constant_pad_nd_cuda_bool, test/test_decomp.py::TestDecompCUDA::test_quick_constant_pad_nd_cuda_int64, test/test_decomp.py::TestDecompCUDA::test_quick_copysign_cuda_int32, test/test_decomp.py::TestDecompCUDA::test_quick_core_backward_logsumexp_cuda_float64, test/test_decomp.py::TestDecompCUDA::test_quick_core_backward_norm_cuda_float64, test/test_decomp.py::TestDecompCUDA::test_quick_core_backward_renorm_cuda_float64, test/test_decomp.py::TestDecompCUDA::test_quick_core_backward_special_entr_cuda_float64, test/test_decomp.py::TestDecompCUDA::test_quick_core_backward_t_cuda_float64, test/test_decomp.py::TestDecompCUDA::test_quick_cos_cuda_bfloat16, test/test_decomp.py::TestDecompCUDA::test_quick_cos_cuda_complex32, test/test_decomp.py::TestDecompCUDA::test_quick_cos_cuda_float64, test/test_decomp.py::TestDecompCUDA::test_quick_count_nonzero_cuda_float32, test/test_decomp.py::TestDecompCUDA::test_quick_cumprod_cuda_complex128, test/test_decomp.py::TestDecompCUDA::test_quick_cumsum_cuda_float16, test/test_decomp.py::TestDecompCUDA::test_quick_cumsum_cuda_float32, test/test_decomp.py::TestDecompCUDA::test_quick_deg2rad_cuda_float32, test/test_decomp.py::TestDecompCUDA::test_quick_diag_cuda_float16, test/test_decomp.py::TestDecompCUDA::test_quick_diagonal_copy_cuda_bool, test/test_decomp.py::TestDecompCUDA::test_quick_diagonal_copy_cuda_int8, test/test_decomp.py::TestDecompCUDA::test_quick_diagonal_scatter_cuda_complex64, test/test_decomp.py::TestDecompCUDA::test_quick_diagonal_scatter_cuda_float64, test/test_decomp.py::TestDecompCUDA::test_quick_digamma_cuda_bfloat16, test/test_decomp.py::TestDecompCUDA::test_quick_div_floor_rounding_cuda_float16, test/test_decomp.py::TestDecompCUDA::test_quick_div_floor_rounding_cuda_int8, test/test_decomp.py::TestDecompCUDA::test_quick_empty_like_cuda_float32, test/test_decomp.py::TestDecompCUDA::test_quick_erfc_cuda_float32, test/test_decomp.py::TestDecompCUDA::test_quick_erfinv_cuda_float64, test/test_decomp.py::TestDecompCUDA::test_quick_exp2_cuda_complex128, test/test_decomp.py::TestDecompCUDA::test_quick_expand_copy_cuda_float16, test/test_decomp.py::TestDecompCUDA::test_quick_expand_copy_cuda_float64, test/test_decomp.py::TestDecompCUDA::test_quick_expand_cuda_bfloat16, test/test_decomp.py::TestDecompCUDA::test_quick_expand_cuda_int8, test/test_decomp.py::TestDecompCUDA::test_quick_expand_cuda_uint8, test/test_decomp.py::TestDecompCUDA::test_quick_expm1_cuda_int32, test/test_decomp.py::TestDecompCUDA::test_quick_eye_cuda_float16, test/test_decomp.py::TestDecompCUDA::test_quick_eye_cuda_float8_e4m3fnuz, test/test_decomp.py::TestDecompCUDA::test_quick_fft_fft2_cuda_float32, test/test_decomp.py::TestDecompCUDA::test_quick_fft_fft2_cuda_float64, test/test_decomp.py::TestDecompCUDA::test_quick_fft_fft_cuda_int32, test/test_decomp.py::TestDecompCUDA::test_quick_fft_fftn_cuda_int16, test/test_decomp.py::TestDecompCUDA::test_quick_fft_fftn_cuda_int32, test/test_decomp.py::TestDecompCUDA::test_quick_fft_hfftn_cuda_int16, test/test_decomp.py::TestDecompCUDA::test_quick_fft_ifft2_cuda_complex32, test/test_decomp.py::TestDecompCUDA::test_quick_fft_ifft_cuda_int16, test/test_decomp.py::TestDecompCUDA::test_quick_fft_ihfft2_cuda_float32, test/test_decomp.py::TestDecompCUDA::test_quick_fft_ihfft2_cuda_int8, test/test_decomp.py::TestDecompCUDA::test_quick_fft_ihfft2_cuda_uint8, test/test_decomp.py::TestDecompCUDA::test_quick_fft_ihfft_cuda_float16, test/test_decomp.py::TestDecompCUDA::test_quick_fft_irfft2_cuda_int16, test/test_decomp.py::TestDecompCUDA::test_quick_floor_cuda_int64, test/test_decomp.py::TestDecompCUDA::test_quick_fmax_cuda_float32, test/test_decomp.py::TestDecompCUDA::test_quick_fmin_cuda_bfloat16, test/test_decomp.py::TestDecompCUDA::test_quick_fmod_cuda_uint8, test/test_decomp.py::TestDecompCUDA::test_quick_i0_cuda_float32, test/test_decomp.py::TestDecompCUDA::test_quick_i0_cuda_int16, test/test_decomp.py::TestDecompCUDA::test_quick_index_copy_cuda_int64, test/test_decomp.py::TestDecompCUDA::test_quick_index_copy_cuda_int8, test/test_decomp.py::TestDecompCUDA::test_quick_index_fill_cuda_bool, test/test_decomp.py::TestDecompCUDA::test_quick_index_select_cuda_float16, test/test_decomp.py::TestDecompCUDA::test_quick_isinf_cuda_float32, test/test_decomp.py::TestDecompCUDA::test_quick_item_cuda_float64, test/test_decomp.py::TestDecompCUDA::test_quick_item_cuda_int16, test/test_decomp.py::TestDecompCUDA::test_quick_item_cuda_int8, test/test_decomp.py::TestDecompCUDA::test_quick_le_cuda_int64, test/test_decomp.py::TestDecompCUDA::test_quick_linalg_diagonal_cuda_int8, test/test_decomp.py::TestDecompCUDA::test_quick_linspace_cuda_int8, test/test_decomp.py::TestDecompCUDA::test_quick_log1p_cuda_float16, test/test_decomp.py::TestDecompCUDA::test_quick_log2_cuda_float16, test/test_decomp.py::TestDecompCUDA::test_quick_log_cuda_int32, test/test_decomp.py::TestDecompCUDA::test_quick_log_softmax_cuda_float64, test/test_decomp.py::TestDecompCUDA::test_quick_logaddexp_cuda_float64, test/test_decomp.py::TestDecompCUDA::test_quick_logical_and_cuda_int64, test/test_decomp.py::TestDecompCUDA::test_quick_logical_not_cuda_uint8, test/test_decomp.py::TestDecompCUDA::test_quick_logical_or_cuda_complex64, test/test_decomp.py::TestDecompCUDA::test_quick_logspace_cuda_int64, test/test_decomp.py::TestDecompCUDA::test_quick_logsumexp_cuda_bool, test/test_decomp.py::TestDecompCUDA::test_quick_logsumexp_cuda_float32, test/test_decomp.py::TestDecompCUDA::test_quick_logsumexp_cuda_int16, test/test_decomp.py::TestDecompCUDA::test_quick_lt_cuda_int16, test/test_decomp.py::TestDecompCUDA::test_quick_native_dropout_backward_cuda_float64, test/test_decomp.py::TestDecompCUDA::test_quick_ne_cuda_float32, test/test_decomp.py::TestDecompCUDA::test_quick_neg_cuda_int8, test/test_decomp.py::TestDecompCUDA::test_quick_new_empty_cuda_complex64, test/test_decomp.py::TestDecompCUDA::test_quick_new_empty_strided_cuda_bfloat16, test/test_decomp.py::TestDecompCUDA::test_quick_new_empty_strided_cuda_complex64, test/test_decomp.py::TestDecompCUDA::test_quick_new_empty_strided_cuda_int64, test/test_decomp.py::TestDecompCUDA::test_quick_new_ones_cuda_complex32, test/test_decomp.py::TestDecompCUDA::test_quick_new_ones_cuda_complex64, test/test_decomp.py::TestDecompCUDA::test_quick_new_zeros_cuda_bool, test/test_decomp.py::TestDecompCUDA::test_quick_new_zeros_cuda_int32, test/test_decomp.py::TestDecompCUDA::test_quick_nn_functional_binary_cross_entropy_with_logits_cuda_float16, test/test_decomp.py::TestDecompCUDA::test_quick_nn_functional_hardtanh_cuda_float16, test/test_decomp.py::TestDecompCUDA::test_quick_nn_functional_hardtanh_cuda_float32, test/test_decomp.py::TestDecompCUDA::test_quick_nn_functional_huber_loss_cuda_float16, test/test_decomp.py::TestDecompCUDA::test_quick_nn_functional_logsigmoid_cuda_float32, test/test_decomp.py::TestDecompCUDA::test_quick_nn_functional_max_unpool3d_cuda_float64, test/test_decomp.py::TestDecompCUDA::test_quick_nn_functional_mse_loss_cuda_float64, test/test_decomp.py::TestDecompCUDA::test_quick_nn_functional_pad_constant_cuda_float16, test/test_decomp.py::TestDecompCUDA::test_quick_nn_functional_relu_cuda_int16, test/test_decomp.py::TestDecompCUDA::test_quick_nn_functional_softshrink_cuda_bfloat16, test/test_decomp.py::TestDecompCUDA::test_quick_norm_nuc_cuda_float32, test/test_decomp.py::TestDecompCUDA::test_quick_ones_cuda_complex32, test/test_decomp.py::TestDecompCUDA::test_quick_ones_cuda_float16, test/test_decomp.py::TestDecompCUDA::test_quick_permute_cuda_bool, test/test_decomp.py::TestDecompCUDA::test_quick_polar_cuda_float32, test/test_decomp.py::TestDecompCUDA::test_quick_rad2deg_cuda_uint8, test/test_decomp.py::TestDecompCUDA::test_quick_renorm_cuda_float16, test/test_decomp.py::TestDecompCUDA::test_quick_repeat_cuda_complex128, test/test_decomp.py::TestDecompCUDA::test_quick_repeat_cuda_int16, test/test_decomp.py::TestDecompCUDA::test_quick_repeat_cuda_int32, test/test_decomp.py::TestDecompCUDA::test_quick_repeat_cuda_int64, test/test_decomp.py::TestDecompCUDA::test_quick_roll_cuda_int64, test/test_decomp.py::TestDecompCUDA::test_quick_round_cuda_int32, test/test_decomp.py::TestDecompCUDA::test_quick_rsqrt_cuda_complex32, test/test_decomp.py::TestDecompCUDA::test_quick_rsub_cuda_complex128, test/test_decomp.py::TestDecompCUDA::test_quick_rsub_cuda_float32, test/test_decomp.py::TestDecompCUDA::test_quick_select_scatter_cuda_float32, test/test_decomp.py::TestDecompCUDA::test_quick_sign_cuda_float64, test/test_decomp.py::TestDecompCUDA::test_quick_signbit_cuda_bfloat16, test/test_decomp.py::TestDecompCUDA::test_quick_sinc_cuda_float64, test/test_decomp.py::TestDecompCUDA::test_quick_slice_cuda_float16, test/test_decomp.py::TestDecompCUDA::test_quick_softmax_cuda_float16, test/test_decomp.py::TestDecompCUDA::test_quick_special_erfcx_cuda_int8, test/test_decomp.py::TestDecompCUDA::test_quick_special_i0e_cuda_bool, test/test_decomp.py::TestDecompCUDA::test_quick_special_log_ndtr_cuda_int32, test/test_decomp.py::TestDecompCUDA::test_quick_special_zeta_cuda_int8, test/test_decomp.py::TestDecompCUDA::test_quick_special_zeta_cuda_uint8, test/test_decomp.py::TestDecompCUDA::test_quick_split_with_sizes_copy_cuda_float16, test/test_decomp.py::TestDecompCUDA::test_quick_sqrt_cuda_uint8, test/test_decomp.py::TestDecompCUDA::test_quick_squeeze_copy_cuda_bool, test/test_decomp.py::TestDecompCUDA::test_quick_squeeze_copy_cuda_float64, test/test_decomp.py::TestDecompCUDA::test_quick_squeeze_cuda_bfloat16, test/test_decomp.py::TestDecompCUDA::test_quick_squeeze_multiple_cuda_float64, test/test_decomp.py::TestDecompCUDA::test_quick_squeeze_multiple_cuda_uint8, test/test_decomp.py::TestDecompCUDA::test_quick_sub_cuda_int16, test/test_decomp.py::TestDecompCUDA::test_quick_t_cuda_complex64, test/test_decomp.py::TestDecompCUDA::test_quick_t_cuda_int8, test/test_decomp.py::TestDecompCUDA::test_quick_tanh_cuda_complex128, test/test_decomp.py::TestDecompCUDA::test_quick_tanh_cuda_float16, test/test_decomp.py::TestDecompCUDA::test_quick_trace_cuda_uint8, test/test_decomp.py::TestDecompCUDA::test_quick_tril_cuda_int8, test/test_decomp.py::TestDecompCUDA::test_quick_unbind_copy_cuda_float32, test/test_decomp.py::TestDecompCUDA::test_quick_unfold_copy_cuda_float64, test/test_decomp.py::TestDecompCUDA::test_quick_unfold_cuda_complex64, test/test_decomp.py::TestDecompCUDA::test_quick_unfold_cuda_float32, test/test_decomp.py::TestDecompCUDA::test_quick_unfold_cuda_float64, test/test_decomp.py::TestDecompCUDA::test_quick_uniform_cuda_float64, test/test_decomp.py::TestDecompCUDA::test_quick_unsafe_split_cuda_float32, test/test_decomp.py::TestDecompCUDA::test_quick_unsqueeze_cuda_float64, test/test_decomp.py::TestDecompCUDA::test_quick_var_unbiased_cuda_float16, test/test_decomp.py::TestDecompCUDA::test_quick_view_cuda_float64, test/test_decomp.py::TestDecompCUDA::test_rnn_decomp_module_nn_GRU_train_mode_cuda_float64, test/test_decomp.py::DecompOneOffTestsCUDA::test_rms_norm_decomp_cuda_cuda 2025-12-04T11:45:03.6760499Z 2025-12-04T11:45:03.6760721Z Finished test_decomp 5/17 ... [2025-12-04 11:45:03.653418][7551.595634481], took 11.05min 2025-12-04T11:45:03.6795530Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/test_decomp/test_decomp-0baa94e9decc4066.xml 2025-12-04T11:45:03.7770124Z Running test_decomp 12/17 ... [2025-12-04 11:45:03.776779][7551.718996539] 2025-12-04T11:45:03.7770560Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2025-12-04T11:45:03.7773218Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'test_decomp.py', '--shard-id=12', '--num-shards=17', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-12-04 11:45:03.777068] 2025-12-04T11:54:19.0824149Z 2025-12-04T11:54:19.0824994Z test_decomp 12/17 was successful, full logs can be found in artifacts with path test/test-reports/test_decomp_12.17_25a251c7912fccac_.log 2025-12-04T11:54:19.0943550Z Running 527 items in this shard: test/test_decomp.py::TestDecompCUDA::test_comprehensive___radd___cuda_uint8, test/test_decomp.py::TestDecompCUDA::test_comprehensive___rdiv___cuda_complex128, test/test_decomp.py::TestDecompCUDA::test_comprehensive___rmatmul___cuda_float32, test/test_decomp.py::TestDecompCUDA::test_comprehensive___rmul___cuda_bfloat16, test/test_decomp.py::TestDecompCUDA::test_comprehensive___rmul___cuda_float16, test/test_decomp.py::TestDecompCUDA::test_comprehensive___rmul___cuda_float64, test/test_decomp.py::TestDecompCUDA::test_comprehensive___rsub___cuda_complex128, test/test_decomp.py::TestDecompCUDA::test_comprehensive__native_batch_norm_legit_cuda_bfloat16, test/test_decomp.py::TestDecompCUDA::test_comprehensive__native_batch_norm_legit_cuda_float32, test/test_decomp.py::TestDecompCUDA::test_comprehensive__segment_reduce_offsets_cuda_bfloat16, test/test_decomp.py::TestDecompCUDA::test_comprehensive_acosh_cuda_int64, test/test_decomp.py::TestDecompCUDA::test_comprehensive_add_cuda_int32, test/test_decomp.py::TestDecompCUDA::test_comprehensive_addbmm_cuda_complex64, test/test_decomp.py::TestDecompCUDA::test_comprehensive_addbmm_cuda_float16, test/test_decomp.py::TestDecompCUDA::test_comprehensive_addmm_decomposed_cuda_bfloat16, test/test_decomp.py::TestDecompCUDA::test_comprehensive_addr_cuda_complex128, test/test_decomp.py::TestDecompCUDA::test_comprehensive_addr_cuda_float16, test/test_decomp.py::TestDecompCUDA::test_comprehensive_alias_copy_cuda_complex64, test/test_decomp.py::TestDecompCUDA::test_comprehensive_amax_cuda_int16, test/test_decomp.py::TestDecompCUDA::test_comprehensive_amin_cuda_float64, test/test_decomp.py::TestDecompCUDA::test_comprehensive_amin_cuda_int8, test/test_decomp.py::TestDecompCUDA::test_comprehensive_aminmax_cuda_float16, test/test_decomp.py::TestDecompCUDA::test_comprehensive_aminmax_cuda_int8, test/test_decomp.py::TestDecompCUDA::test_comprehensive_aminmax_cuda_uint8, test/test_decomp.py::TestDecompCUDA::test_comprehensive_argmin_cuda_int8, test/test_decomp.py::TestDecompCUDA::test_comprehensive_argwhere_cuda_float16, test/test_decomp.py::TestDecompCUDA::test_comprehensive_as_strided_copy_cuda_float64, test/test_decomp.py::TestDecompCUDA::test_comprehensive_as_strided_cuda_complex32, test/test_decomp.py::TestDecompCUDA::test_comprehensive_as_strided_cuda_int16, test/test_decomp.py::TestDecompCUDA::test_comprehensive_as_strided_partial_views_cuda_float16, test/test_decomp.py::TestDecompCUDA::test_comprehensive_asinh_cuda_int64, test/test_decomp.py::TestDecompCUDA::test_comprehensive_atleast_1d_cuda_bfloat16, test/test_decomp.py::TestDecompCUDA::test_comprehensive_atleast_2d_cuda_bool, test/test_decomp.py::TestDecompCUDA::test_comprehensive_atleast_2d_cuda_complex32, test/test_decomp.py::TestDecompCUDA::test_comprehensive_atleast_3d_cuda_int32, test/test_decomp.py::TestDecompCUDA::test_comprehensive_bincount_cuda_int16, test/test_decomp.py::TestDecompCUDA::test_comprehensive_block_diag_cuda_int16, test/test_decomp.py::TestDecompCUDA::test_comprehensive_bool_cuda_int32, test/test_decomp.py::TestDecompCUDA::test_comprehensive_broadcast_tensors_cuda_complex64, test/test_decomp.py::TestDecompCUDA::test_comprehensive_broadcast_to_cuda_float32, test/test_decomp.py::TestDecompCUDA::test_comprehensive_bucketize_cuda_float16, test/test_decomp.py::TestDecompCUDA::test_comprehensive_byte_cuda_uint8, test/test_decomp.py::TestDecompCUDA::test_comprehensive_cartesian_prod_cuda_float16, test/test_decomp.py::TestDecompCUDA::test_comprehensive_cartesian_prod_cuda_int16, test/test_decomp.py::TestDecompCUDA::test_comprehensive_cat_cuda_bool, test/test_decomp.py::TestDecompCUDA::test_comprehensive_char_cuda_int64, test/test_decomp.py::TestDecompCUDA::test_comprehensive_cholesky_solve_cuda_complex128, test/test_decomp.py::TestDecompCUDA::test_comprehensive_combinations_cuda_float32, test/test_decomp.py::TestDecompCUDA::test_comprehensive_combinations_cuda_int16, test/test_decomp.py::TestDecompCUDA::test_comprehensive_conj_physical_cuda_int64, test/test_decomp.py::TestDecompCUDA::test_comprehensive_constant_pad_nd_cuda_complex64, test/test_decomp.py::TestDecompCUDA::test_comprehensive_constant_pad_nd_cuda_int16, test/test_decomp.py::TestDecompCUDA::test_comprehensive_corrcoef_cuda_complex128, test/test_decomp.py::TestDecompCUDA::test_comprehensive_corrcoef_cuda_int64, test/test_decomp.py::TestDecompCUDA::test_comprehensive_cos_cuda_complex64, test/test_decomp.py::TestDecompCUDA::test_comprehensive_cos_cuda_int32, test/test_decomp.py::TestDecompCUDA::test_comprehensive_cosh_cuda_complex32, test/test_decomp.py::TestDecompCUDA::test_comprehensive_count_nonzero_cuda_uint8, test/test_decomp.py::TestDecompCUDA::test_comprehensive_cummin_cuda_int32, test/test_decomp.py::TestDecompCUDA::test_comprehensive_cumprod_cuda_bfloat16, test/test_decomp.py::TestDecompCUDA::test_comprehensive_cumprod_cuda_int16, test/test_decomp.py::TestDecompCUDA::test_comprehensive_cumsum_cuda_bfloat16, test/test_decomp.py::TestDecompCUDA::test_comprehensive_diagonal_copy_cuda_bfloat16, test/test_decomp.py::TestDecompCUDA::test_comprehensive_diagonal_scatter_cuda_float64, test/test_decomp.py::TestDecompCUDA::test_comprehensive_diff_cuda_bool, test/test_decomp.py::TestDecompCUDA::test_comprehensive_dist_cuda_complex64, test/test_decomp.py::TestDecompCUDA::test_comprehensive_div_no_rounding_mode_cuda_complex128, test/test_decomp.py::TestDecompCUDA::test_comprehensive_dsplit_cuda_complex64, test/test_decomp.py::TestDecompCUDA::test_comprehensive_dsplit_cuda_float16, test/test_decomp.py::TestDecompCUDA::test_comprehensive_dsplit_cuda_float32, test/test_decomp.py::TestDecompCUDA::test_comprehensive_dstack_cuda_bfloat16, test/test_decomp.py::TestDecompCUDA::test_comprehensive_dstack_cuda_complex32, test/test_decomp.py::TestDecompCUDA::test_comprehensive_dstack_cuda_float16, test/test_decomp.py::TestDecompCUDA::test_comprehensive_erf_cuda_int32, test/test_decomp.py::TestDecompCUDA::test_comprehensive_erfc_cuda_uint8, test/test_decomp.py::TestDecompCUDA::test_comprehensive_erfinv_cuda_float32, test/test_decomp.py::TestDecompCUDA::test_comprehensive_erfinv_cuda_uint8, test/test_decomp.py::TestDecompCUDA::test_comprehensive_expand_copy_cuda_complex64, test/test_decomp.py::TestDecompCUDA::test_comprehensive_expand_cuda_uint8, test/test_decomp.py::TestDecompCUDA::test_comprehensive_fft_fft_cuda_complex64, test/test_decomp.py::TestDecompCUDA::test_comprehensive_fft_fftn_cuda_float16, test/test_decomp.py::TestDecompCUDA::test_comprehensive_fft_ifftn_cuda_complex128, test/test_decomp.py::TestDecompCUDA::test_comprehensive_fft_ihfft2_cuda_int64, test/test_decomp.py::TestDecompCUDA::test_comprehensive_fft_rfft_cuda_float32, test/test_decomp.py::TestDecompCUDA::test_comprehensive_fft_rfftn_cuda_int16, test/test_decomp.py::TestDecompCUDA::test_comprehensive_fill_cuda_complex32, test/test_decomp.py::TestDecompCUDA::test_comprehensive_flatten_cuda_uint8, test/test_decomp.py::TestDecompCUDA::test_comprehensive_flip_cuda_float32, test/test_decomp.py::TestDecompCUDA::test_comprehensive_flip_cuda_uint8, test/test_decomp.py::TestDecompCUDA::test_comprehensive_flipud_cuda_bool, test/test_decomp.py::TestDecompCUDA::test_comprehensive_flipud_cuda_float32, test/test_decomp.py::TestDecompCUDA::test_comprehensive_flipud_cuda_uint8, test/test_decomp.py::TestDecompCUDA::test_comprehensive_float_cuda_float32, test/test_decomp.py::TestDecompCUDA::test_comprehensive_float_cuda_int16, test/test_decomp.py::TestDecompCUDA::test_comprehensive_fmin_cuda_int16, test/test_decomp.py::TestDecompCUDA::test_comprehensive_full_cuda_complex128, test/test_decomp.py::TestDecompCUDA::test_comprehensive_full_cuda_complex32, test/test_decomp.py::TestDecompCUDA::test_comprehensive_full_like_cuda_float32, test/test_decomp.py::TestDecompCUDA::test_comprehensive_gather_cuda_int64, test/test_decomp.py::TestDecompCUDA::test_comprehensive_ge_cuda_float32, test/test_decomp.py::TestDecompCUDA::test_comprehensive_gradient_cuda_bfloat16, test/test_decomp.py::TestDecompCUDA::test_comprehensive_grid_sampler_2d_cuda_bfloat16, test/test_decomp.py::TestDecompCUDA::test_comprehensive_grid_sampler_2d_cuda_float64, test/test_decomp.py::TestDecompCUDA::test_comprehensive_half_cuda_bool, test/test_decomp.py::TestDecompCUDA::test_comprehensive_half_cuda_complex64, test/test_decomp.py::TestDecompCUDA::test_comprehensive_hstack_cuda_float16, test/test_decomp.py::TestDecompCUDA::test_comprehensive_i0_cuda_int8, test/test_decomp.py::TestDecompCUDA::test_comprehensive_index_add_cuda_int16, test/test_decomp.py::TestDecompCUDA::test_comprehensive_index_copy_cuda_int64, test/test_decomp.py::TestDecompCUDA::test_comprehensive_index_fill_cuda_float32, test/test_decomp.py::TestDecompCUDA::test_comprehensive_index_reduce_amax_cuda_int8, test/test_decomp.py::TestDecompCUDA::test_comprehensive_index_reduce_amin_cuda_bfloat16, test/test_decomp.py::TestDecompCUDA::test_comprehensive_index_reduce_amin_cuda_int64, test/test_decomp.py::TestDecompCUDA::test_comprehensive_index_reduce_amin_cuda_int8, test/test_decomp.py::TestDecompCUDA::test_comprehensive_index_reduce_mean_cuda_int8, test/test_decomp.py::TestDecompCUDA::test_comprehensive_index_reduce_prod_cuda_float64, test/test_decomp.py::TestDecompCUDA::test_comprehensive_index_select_cuda_complex128, test/test_decomp.py::TestDecompCUDA::test_comprehensive_isfinite_cuda_float32, test/test_decomp.py::TestDecompCUDA::test_comprehensive_isfinite_cuda_float64, test/test_decomp.py::TestDecompCUDA::test_comprehensive_isneginf_cuda_bfloat16, test/test_decomp.py::TestDecompCUDA::test_comprehensive_isneginf_cuda_int8, test/test_decomp.py::TestDecompCUDA::test_comprehensive_isposinf_cuda_int16, test/test_decomp.py::TestDecompCUDA::test_comprehensive_isreal_cuda_complex32, test/test_decomp.py::TestDecompCUDA::test_comprehensive_jiterator_4inputs_with_extra_args_cuda_complex64, test/test_decomp.py::TestDecompCUDA::test_comprehensive_jiterator_binary_return_by_ref_cuda_int16, test/test_decomp.py::TestDecompCUDA::test_comprehensive_lcm_cuda_int16, test/test_decomp.py::TestDecompCUDA::test_comprehensive_ldexp_cuda_bfloat16, test/test_decomp.py::TestDecompCUDA::test_comprehensive_linalg_cholesky_cuda_complex64, test/test_decomp.py::TestDecompCUDA::test_comprehensive_linalg_cond_cuda_complex128, test/test_decomp.py::TestDecompCUDA::test_comprehensive_linalg_cond_cuda_complex64, test/test_decomp.py::TestDecompCUDA::test_comprehensive_linalg_cross_cuda_float64, test/test_decomp.py::TestDecompCUDA::test_comprehensive_linalg_diagonal_cuda_complex128, test/test_decomp.py::TestDecompCUDA::test_comprehensive_linalg_eig_cuda_float64, test/test_decomp.py::TestDecompCUDA::test_comprehensive_linalg_ldl_factor_cuda_complex128, test/test_decomp.py::TestDecompCUDA::test_comprehensive_linalg_lu_factor_ex_cuda_float64, test/test_decomp.py::TestDecompCUDA::test_comprehensive_linalg_matrix_norm_cuda_bfloat16, test/test_decomp.py::TestDecompCUDA::test_comprehensive_linalg_norm_subgradients_at_zero_cuda_bfloat16, test/test_decomp.py::TestDecompCUDA::test_comprehensive_linalg_pinv_hermitian_cuda_float32, test/test_decomp.py::TestDecompCUDA::test_comprehensive_linalg_svd_cuda_complex64, test/test_decomp.py::TestDecompCUDA::test_comprehensive_linalg_vecdot_cuda_bfloat16, test/test_decomp.py::TestDecompCUDA::test_comprehensive_linalg_vecdot_cuda_float64, test/test_decomp.py::TestDecompCUDA::test_comprehensive_linspace_tensor_overload_cuda_float16, test/test_decomp.py::TestDecompCUDA::test_comprehensive_linspace_tensor_overload_cuda_float64, test/test_decomp.py::TestDecompCUDA::test_comprehensive_log10_cuda_float32, test/test_decomp.py::TestDecompCUDA::test_comprehensive_log10_cuda_uint8, test/test_decomp.py::TestDecompCUDA::test_comprehensive_log1p_cuda_uint8, test/test_decomp.py::TestDecompCUDA::test_comprehensive_log_cuda_complex32, test/test_decomp.py::TestDecompCUDA::test_comprehensive_log_softmax_with_dtype_cuda_int16, test/test_decomp.py::TestDecompCUDA::test_comprehensive_logical_and_cuda_complex128, test/test_decomp.py::TestDecompCUDA::test_comprehensive_logical_and_cuda_int64, test/test_decomp.py::TestDecompCUDA::test_comprehensive_logspace_cuda_uint8, test/test_decomp.py::TestDecompCUDA::test_comprehensive_logsumexp_cuda_int32, test/test_decomp.py::TestDecompCUDA::test_comprehensive_long_cuda_complex32, test/test_decomp.py::TestDecompCUDA::test_comprehensive_long_cuda_int32, test/test_decomp.py::TestDecompCUDA::test_comprehensive_lt_cuda_float64, test/test_decomp.py::TestDecompCUDA::test_comprehensive_mH_cuda_int16, test/test_decomp.py::TestDecompCUDA::test_comprehensive_mT_cuda_complex128, test/test_decomp.py::TestDecompCUDA::test_comprehensive_mT_cuda_int32, test/test_decomp.py::TestDecompCUDA::test_comprehensive_masked_amax_cuda_float32, test/test_decomp.py::TestDecompCUDA::test_comprehensive_masked_amax_cuda_int32, test/test_decomp.py::TestDecompCUDA::test_comprehensive_masked_argmax_cuda_bfloat16, test/test_decomp.py::TestDecompCUDA::test_comprehensive_masked_argmin_cuda_float64, test/test_decomp.py::TestDecompCUDA::test_comprehensive_masked_fill_cuda_complex64, test/test_decomp.py::TestDecompCUDA::test_comprehensive_masked_fill_cuda_float32, test/test_decomp.py::TestDecompCUDA::test_comprehensive_masked_logaddexp_cuda_float64, test/test_decomp.py::TestDecompCUDA::test_comprehensive_masked_logsumexp_cuda_bfloat16, test/test_decomp.py::TestDecompCUDA::test_comprehensive_masked_logsumexp_cuda_complex128, test/test_decomp.py::TestDecompCUDA::test_comprehensive_masked_logsumexp_cuda_float16, test/test_decomp.py::TestDecompCUDA::test_comprehensive_masked_logsumexp_cuda_float32, test/test_decomp.py::TestDecompCUDA::test_comprehensive_masked_mean_cuda_complex128, test/test_decomp.py::TestDecompCUDA::test_comprehensive_masked_select_cuda_float64, test/test_decomp.py::TestDecompCUDA::test_comprehensive_masked_std_cuda_float64, test/test_decomp.py::TestDecompCUDA::test_comprehensive_masked_sum_cuda_float16, test/test_decomp.py::TestDecompCUDA::test_comprehensive_masked_var_cuda_bfloat16, test/test_decomp.py::TestDecompCUDA::test_comprehensive_masked_var_cuda_int32, test/test_decomp.py::TestDecompCUDA::test_comprehensive_matrix_exp_cuda_bfloat16, test/test_decomp.py::TestDecompCUDA::test_comprehensive_max_binary_cuda_float64, test/test_decomp.py::TestDecompCUDA::test_comprehensive_max_reduction_with_dim_cuda_int16, test/test_decomp.py::TestDecompCUDA::test_comprehensive_maximum_cuda_float16, test/test_decomp.py::TestDecompCUDA::test_comprehensive_meshgrid_list_of_tensors_cuda_bool, test/test_decomp.py::TestDecompCUDA::test_comprehensive_meshgrid_list_of_tensors_cuda_complex128, test/test_decomp.py::TestDecompCUDA::test_comprehensive_meshgrid_variadic_tensors_cuda_int32, test/test_decomp.py::TestDecompCUDA::test_comprehensive_minimum_cuda_float16, test/test_decomp.py::TestDecompCUDA::test_comprehensive_minimum_cuda_float32, test/test_decomp.py::TestDecompCUDA::test_comprehensive_mm_cuda_bfloat16, test/test_decomp.py::TestDecompCUDA::test_comprehensive_movedim_cuda_float64, test/test_decomp.py::TestDecompCUDA::test_comprehensive_msort_cuda_bool, test/test_decomp.py::TestDecompCUDA::test_comprehensive_multinomial_cuda_float64, test/test_decomp.py::TestDecompCUDA::test_comprehensive_mv_cuda_complex64, test/test_decomp.py::TestDecompCUDA::test_comprehensive_mv_cuda_float16, test/test_decomp.py::TestDecompCUDA::test_comprehensive_mvlgamma_mvlgamma_p_3_cuda_int16, test/test_decomp.py::TestDecompCUDA::test_comprehensive_nanmean_cuda_complex128, test/test_decomp.py::TestDecompCUDA::test_comprehensive_nanmedian_cuda_int32, test/test_decomp.py::TestDecompCUDA::test_comprehensive_nansum_cuda_int8, test/test_decomp.py::TestDecompCUDA::test_comprehensive_narrow_cuda_float32, test/test_decomp.py::TestDecompCUDA::test_comprehensive_narrow_cuda_int16, test/test_decomp.py::TestDecompCUDA::test_comprehensive_narrow_cuda_int8, test/test_decomp.py::TestDecompCUDA::test_comprehensive_ne_cuda_float64, test/test_decomp.py::TestDecompCUDA::test_comprehensive_neg_cuda_complex32, test/test_decomp.py::TestDecompCUDA::test_comprehensive_neg_cuda_float64, test/test_decomp.py::TestDecompCUDA::test_comprehensive_new_empty_cuda_bool, test/test_decomp.py::TestDecompCUDA::test_comprehensive_new_full_cuda_int16, test/test_decomp.py::TestDecompCUDA::test_comprehensive_new_zeros_cuda_float32, test/test_decomp.py::TestDecompCUDA::test_comprehensive_new_zeros_cuda_int16, test/test_decomp.py::TestDecompCUDA::test_comprehensive_nn_functional_avg_pool2d_cuda_float64, test/test_decomp.py::TestDecompCUDA::test_comprehensive_nn_functional_avg_pool3d_cuda_bfloat16, test/test_decomp.py::TestDecompCUDA::test_comprehensive_nn_functional_avg_pool3d_cuda_float64, test/test_decomp.py::TestDecompCUDA::test_comprehensive_nn_functional_batch_norm_cuda_float32, test/test_decomp.py::TestDecompCUDA::test_comprehensive_nn_functional_binary_cross_entropy_with_logits_cuda_float64, test/test_decomp.py::TestDecompCUDA::test_comprehensive_nn_functional_celu_cuda_float32, test/test_decomp.py::TestDecompCUDA::test_comprehensive_nn_functional_celu_cuda_float64, test/test_decomp.py::TestDecompCUDA::test_comprehensive_nn_functional_channel_shuffle_cuda_float64, test/test_decomp.py::TestDecompCUDA::test_comprehensive_nn_functional_channel_shuffle_cuda_uint8, test/test_decomp.py::TestDecompCUDA::test_comprehensive_nn_functional_conv_transpose2d_cuda_complex64, test/test_decomp.py::TestDecompCUDA::test_comprehensive_nn_functional_cosine_similarity_cuda_float32, test/test_decomp.py::TestDecompCUDA::test_comprehensive_nn_functional_cross_entropy_cuda_bfloat16, test/test_decomp.py::TestDecompCUDA::test_comprehensive_nn_functional_dropout3d_cuda_float16, test/test_decomp.py::TestDecompCUDA::test_comprehensive_nn_functional_embedding_bag_cuda_float16, test/test_decomp.py::TestDecompCUDA::test_comprehensive_nn_functional_fractional_max_pool3d_cuda_float64, test/test_decomp.py::TestDecompCUDA::test_comprehensive_nn_functional_glu_cuda_float64, test/test_decomp.py::TestDecompCUDA::test_comprehensive_nn_functional_grid_sample_cuda_float64, test/test_decomp.py::TestDecompCUDA::test_comprehensive_nn_functional_hardtanh_cuda_bfloat16, test/test_decomp.py::TestDecompCUDA::test_comprehensive_nn_functional_hardtanh_cuda_int64, test/test_decomp.py::TestDecompCUDA::test_comprehensive_nn_functional_interpolate_area_cuda_float64, test/test_decomp.py::TestDecompCUDA::test_comprehensive_nn_functional_l1_loss_cuda_float32, test/test_decomp.py::TestDecompCUDA::test_comprehensive_nn_functional_linear_cuda_bfloat16, test/test_decomp.py::TestDecompCUDA::test_comprehensive_nn_functional_margin_ranking_loss_cuda_uint8, test/test_decomp.py::TestDecompCUDA::test_comprehensive_nn_functional_max_unpool1d_cuda_bfloat16, test/test_decomp.py::TestDecompCUDA::test_comprehensive_nn_functional_max_unpool1d_cuda_float64, test/test_decomp.py::TestDecompCUDA::test_comprehensive_nn_functional_normalize_cuda_bfloat16, test/test_decomp.py::TestDecompCUDA::test_comprehensive_nn_functional_pad_circular_cuda_uint8, test/test_decomp.py::TestDecompCUDA::test_comprehensive_nn_functional_pad_constant_cuda_float32, test/test_decomp.py::TestDecompCUDA::test_comprehensive_nn_functional_pad_constant_cuda_int32, test/test_decomp.py::TestDecompCUDA::test_comprehensive_nn_functional_pad_reflect_cuda_complex64, test/test_decomp.py::TestDecompCUDA::test_comprehensive_nn_functional_pad_reflect_cuda_int64, test/test_decomp.py::TestDecompCUDA::test_comprehensive_nn_functional_pad_replicate_cuda_complex128, test/test_decomp.py::TestDecompCUDA::test_comprehensive_nn_functional_pad_replicate_negative_cuda_int8, test/test_decomp.py::TestDecompCUDA::test_comprehensive_nn_functional_pixel_shuffle_cuda_uint8, test/test_decomp.py::TestDecompCUDA::test_comprehensive_nn_functional_relu6_cuda_uint8, test/test_decomp.py::TestDecompCUDA::test_comprehensive_nn_functional_rms_norm_cuda_complex128, test/test_decomp.py::TestDecompCUDA::test_comprehensive_nn_functional_rrelu_cuda_bfloat16, test/test_decomp.py::TestDecompCUDA::test_comprehensive_nn_functional_selu_cuda_float64, test/test_decomp.py::TestDecompCUDA::test_comprehensive_nn_functional_softplus_cuda_bfloat16, test/test_decomp.py::TestDecompCUDA::test_comprehensive_nn_functional_softsign_cuda_float64, test/test_decomp.py::TestDecompCUDA::test_comprehensive_nn_functional_tanhshrink_cuda_float32, test/test_decomp.py::TestDecompCUDA::test_comprehensive_nn_functional_triplet_margin_with_distance_loss_cuda_complex128, test/test_decomp.py::TestDecompCUDA::test_comprehensive_nonzero_cuda_complex32, test/test_decomp.py::TestDecompCUDA::test_comprehensive_nonzero_static_cuda_complex128, test/test_decomp.py::TestDecompCUDA::test_comprehensive_norm_cuda_complex64, test/test_decomp.py::TestDecompCUDA::test_comprehensive_norm_fro_cuda_float16, test/test_decomp.py::TestDecompCUDA::test_comprehensive_norm_nuc_cuda_complex64, test/test_decomp.py::TestDecompCUDA::test_comprehensive_normal_cuda_bfloat16, test/test_decomp.py::TestDecompCUDA::test_comprehensive_normal_number_mean_cuda_bfloat16, test/test_decomp.py::TestDecompCUDA::test_comprehensive_outer_cuda_bfloat16, test/test_decomp.py::TestDecompCUDA::test_comprehensive_outer_cuda_complex64, test/test_decomp.py::TestDecompCUDA::test_comprehensive_permute_copy_cuda_int8, test/test_decomp.py::TestDecompCUDA::test_comprehensive_polar_cuda_float32, test/test_decomp.py::TestDecompCUDA::test_comprehensive_polygamma_polygamma_n_1_cuda_uint8, test/test_decomp.py::TestDecompCUDA::test_comprehensive_polygamma_polygamma_n_2_cuda_uint8, test/test_decomp.py::TestDecompCUDA::test_comprehensive_polygamma_polygamma_n_4_cuda_uint8, test/test_decomp.py::TestDecompCUDA::test_comprehensive_positive_cuda_bfloat16, test/test_decomp.py::TestDecompCUDA::test_comprehensive_positive_cuda_float64, test/test_decomp.py::TestDecompCUDA::test_comprehensive_pow_cuda_complex32, test/test_decomp.py::TestDecompCUDA::test_comprehensive_put_cuda_int64, test/test_decomp.py::TestDecompCUDA::test_comprehensive_quantile_cuda_float32, test/test_decomp.py::TestDecompCUDA::test_comprehensive_rad2deg_cuda_float16, test/test_decomp.py::TestDecompCUDA::test_comprehensive_randint_cuda_int32, test/test_decomp.py::TestDecompCUDA::test_comprehensive_ravel_cuda_float16, test/test_decomp.py::TestDecompCUDA::test_comprehensive_ravel_cuda_float32, test/test_decomp.py::TestDecompCUDA::test_comprehensive_real_cuda_complex64, test/test_decomp.py::TestDecompCUDA::test_comprehensive_real_cuda_int16, test/test_decomp.py::TestDecompCUDA::test_comprehensive_repeat_cuda_complex64, test/test_decomp.py::TestDecompCUDA::test_comprehensive_repeat_cuda_int64, test/test_decomp.py::TestDecompCUDA::test_comprehensive_repeat_interleave_cuda_int64, test/test_decomp.py::TestDecompCUDA::test_comprehensive_reshape_cuda_complex128, test/test_decomp.py::TestDecompCUDA::test_comprehensive_resize__cuda_int64, test/test_decomp.py::TestDecompCUDA::test_comprehensive_resize__cuda_int8, test/test_decomp.py::TestDecompCUDA::test_comprehensive_resize_as__cuda_float32, test/test_decomp.py::TestDecompCUDA::test_comprehensive_resolve_neg_cuda_int8, test/test_decomp.py::TestDecompCUDA::test_comprehensive_roll_cuda_bool, test/test_decomp.py::TestDecompCUDA::test_comprehensive_rot90_cuda_int16, test/test_decomp.py::TestDecompCUDA::test_comprehensive_round_decimals_0_cuda_float16, test/test_decomp.py::TestDecompCUDA::test_comprehensive_round_decimals_0_cuda_float64, test/test_decomp.py::TestDecompCUDA::test_comprehensive_round_decimals_3_cuda_bfloat16, test/test_decomp.py::TestDecompCUDA::test_comprehensive_rsub_cuda_complex64, test/test_decomp.py::TestDecompCUDA::test_comprehensive_scalar_tensor_cuda_complex128, test/test_decomp.py::TestDecompCUDA::test_comprehensive_scatter_cuda_complex64, test/test_decomp.py::TestDecompCUDA::test_comprehensive_scatter_reduce_amax_cuda_int8, test/test_decomp.py::TestDecompCUDA::test_comprehensive_scatter_reduce_amin_cuda_float16, test/test_decomp.py::TestDecompCUDA::test_comprehensive_scatter_reduce_prod_cuda_float32, test/test_decomp.py::TestDecompCUDA::test_comprehensive_sign_cuda_float64, test/test_decomp.py::TestDecompCUDA::test_comprehensive_sign_cuda_int16, test/test_decomp.py::TestDecompCUDA::test_comprehensive_signal_windows_general_cosine_cuda_float64, test/test_decomp.py::TestDecompCUDA::test_comprehensive_signal_windows_nuttall_cuda_float32, test/test_decomp.py::TestDecompCUDA::test_comprehensive_sinc_cuda_int8, test/test_decomp.py::TestDecompCUDA::test_comprehensive_slice_cuda_complex32, test/test_decomp.py::TestDecompCUDA::test_comprehensive_slice_cuda_float16, test/test_decomp.py::TestDecompCUDA::test_comprehensive_slice_scatter_cuda_bfloat16, test/test_decomp.py::TestDecompCUDA::test_comprehensive_slice_scatter_cuda_float64, test/test_decomp.py::TestDecompCUDA::test_comprehensive_slice_scatter_cuda_int8, test/test_decomp.py::TestDecompCUDA::test_comprehensive_special_airy_ai_cuda_float32, test/test_decomp.py::TestDecompCUDA::test_comprehensive_special_bessel_j0_cuda_bool, test/test_decomp.py::TestDecompCUDA::test_comprehensive_special_chebyshev_polynomial_t_cuda_int64, test/test_decomp.py::TestDecompCUDA::test_comprehensive_special_chebyshev_polynomial_t_cuda_int8, test/test_decomp.py::TestDecompCUDA::test_comprehensive_special_chebyshev_polynomial_u_cuda_float32, test/test_decomp.py::TestDecompCUDA::test_comprehensive_special_chebyshev_polynomial_u_cuda_int8, test/test_decomp.py::TestDecompCUDA::test_comprehensive_special_erfcx_cuda_int64, test/test_decomp.py::TestDecompCUDA::test_comprehensive_special_hermite_polynomial_h_cuda_int8, test/test_decomp.py::TestDecompCUDA::test_comprehensive_special_hermite_polynomial_he_cuda_float64, test/test_decomp.py::TestDecompCUDA::test_comprehensive_special_i0e_cuda_bfloat16, test/test_decomp.py::TestDecompCUDA::test_comprehensive_special_i1e_cuda_int16, test/test_decomp.py::TestDecompCUDA::test_comprehensive_special_laguerre_polynomial_l_cuda_int32, test/test_decomp.py::TestDecompCUDA::test_comprehensive_special_legendre_polynomial_p_cuda_int64, test/test_decomp.py::TestDecompCUDA::test_comprehensive_special_log_ndtr_cuda_int16, test/test_decomp.py::TestDecompCUDA::test_comprehensive_special_modified_bessel_i0_cuda_int8, test/test_decomp.py::TestDecompCUDA::test_comprehensive_special_modified_bessel_i1_cuda_float64, test/test_decomp.py::TestDecompCUDA::test_comprehensive_special_modified_bessel_k1_cuda_int8, test/test_decomp.py::TestDecompCUDA::test_comprehensive_special_polygamma_special_polygamma_n_0_cuda_uint8, test/test_decomp.py::TestDecompCUDA::test_comprehensive_special_scaled_modified_bessel_k1_cuda_int64, test/test_decomp.py::TestDecompCUDA::test_comprehensive_special_shifted_chebyshev_polynomial_v_cuda_float64, test/test_decomp.py::TestDecompCUDA::test_comprehensive_special_xlog1py_cuda_int32, test/test_decomp.py::TestDecompCUDA::test_comprehensive_special_xlog1py_cuda_int8, test/test_decomp.py::TestDecompCUDA::test_comprehensive_split_cuda_int32, test/test_decomp.py::TestDecompCUDA::test_comprehensive_split_with_sizes_copy_cuda_complex128, test/test_decomp.py::TestDecompCUDA::test_comprehensive_split_with_sizes_cuda_bfloat16, test/test_decomp.py::TestDecompCUDA::test_comprehensive_split_with_sizes_cuda_int32, test/test_decomp.py::TestDecompCUDA::test_comprehensive_sqrt_cuda_float32, test/test_decomp.py::TestDecompCUDA::test_comprehensive_squeeze_copy_cuda_complex64, test/test_decomp.py::TestDecompCUDA::test_comprehensive_squeeze_copy_cuda_float32, test/test_decomp.py::TestDecompCUDA::test_comprehensive_squeeze_cuda_int16, test/test_decomp.py::TestDecompCUDA::test_comprehensive_stack_cuda_float32, test/test_decomp.py::TestDecompCUDA::test_comprehensive_std_mean_cuda_complex128, test/test_decomp.py::TestDecompCUDA::test_comprehensive_sub_cuda_int64, test/test_decomp.py::TestDecompCUDA::test_comprehensive_svd_cuda_complex128, test/test_decomp.py::TestDecompCUDA::test_comprehensive_svd_cuda_float64, test/test_decomp.py::TestDecompCUDA::test_comprehensive_t_copy_cuda_int32, test/test_decomp.py::TestDecompCUDA::test_comprehensive_t_cuda_bfloat16, test/test_decomp.py::TestDecompCUDA::test_comprehensive_tile_cuda_complex128, test/test_decomp.py::TestDecompCUDA::test_comprehensive_torch__scaled_mm_v2_cuda_float8_e4m3fn, test/test_decomp.py::TestDecompCUDA::test_comprehensive_torch_ops_aten__flash_attention_forward_cuda_bfloat16, test/test_decomp.py::TestDecompCUDA::test_comprehensive_torch_ops_aten__safe_softmax_default_cuda_bfloat16, test/test_decomp.py::TestDecompCUDA::test_comprehensive_torch_ops_aten__safe_softmax_default_cuda_float32, test/test_decomp.py::TestDecompCUDA::test_comprehensive_transpose_copy_cuda_int8, test/test_decomp.py::TestDecompCUDA::test_comprehensive_transpose_cuda_complex64, test/test_decomp.py::TestDecompCUDA::test_comprehensive_triangular_solve_cuda_complex128, test/test_decomp.py::TestDecompCUDA::test_comprehensive_triangular_solve_cuda_float64, test/test_decomp.py::TestDecompCUDA::test_comprehensive_tril_cuda_complex32, test/test_decomp.py::TestDecompCUDA::test_comprehensive_triu_cuda_float32, test/test_decomp.py::TestDecompCUDA::test_comprehensive_trunc_cuda_bfloat16, test/test_decomp.py::TestDecompCUDA::test_comprehensive_unbind_copy_cuda_complex64, test/test_decomp.py::TestDecompCUDA::test_comprehensive_unflatten_cuda_float64, test/test_decomp.py::TestDecompCUDA::test_comprehensive_unsqueeze_copy_cuda_float64, test/test_decomp.py::TestDecompCUDA::test_comprehensive_unsqueeze_cuda_float16, test/test_decomp.py::TestDecompCUDA::test_comprehensive_view_copy_cuda_bool, test/test_decomp.py::TestDecompCUDA::test_comprehensive_vstack_cuda_complex32, test/test_decomp.py::TestDecompCUDA::test_comprehensive_where_cuda_complex128, test/test_decomp.py::TestDecompCUDA::test_comprehensive_where_cuda_uint8, test/test_decomp.py::TestDecompCUDA::test_comprehensive_xlogy_cuda_float32, test/test_decomp.py::TestDecompCUDA::test_comprehensive_zeros_cuda_complex64, test/test_decomp.py::TestDecompCUDA::test_comprehensive_zeros_like_cuda_bool, test/test_decomp.py::TestDecompCUDA::test_comprehensive_zeros_like_cuda_int16, test/test_decomp.py::TestDecompCUDA::test_comprehensive_zeros_like_cuda_int8, test/test_decomp.py::TestDecompCUDA::test_quick__chunk_cat_cuda_int32, test/test_decomp.py::TestDecompCUDA::test_quick__unsafe_masked_index_put_accumulate_cuda_float64, test/test_decomp.py::TestDecompCUDA::test_quick_acos_cuda_bool, test/test_decomp.py::TestDecompCUDA::test_quick_acos_cuda_int16, test/test_decomp.py::TestDecompCUDA::test_quick_addmm_cuda_float16, test/test_decomp.py::TestDecompCUDA::test_quick_addmm_decomposed_cuda_bfloat16, test/test_decomp.py::TestDecompCUDA::test_quick_addmv_cuda_float32, test/test_decomp.py::TestDecompCUDA::test_quick_addr_cuda_bool, test/test_decomp.py::TestDecompCUDA::test_quick_addr_cuda_uint8, test/test_decomp.py::TestDecompCUDA::test_quick_amax_cuda_float16, test/test_decomp.py::TestDecompCUDA::test_quick_any_cuda_float16, test/test_decomp.py::TestDecompCUDA::test_quick_any_cuda_float32, test/test_decomp.py::TestDecompCUDA::test_quick_as_strided_copy_cuda_float16, test/test_decomp.py::TestDecompCUDA::test_quick_as_strided_copy_cuda_int8, test/test_decomp.py::TestDecompCUDA::test_quick_asinh_cuda_bfloat16, test/test_decomp.py::TestDecompCUDA::test_quick_asinh_cuda_complex128, test/test_decomp.py::TestDecompCUDA::test_quick_atan2_cuda_int64, test/test_decomp.py::TestDecompCUDA::test_quick_atan2_cuda_int8, test/test_decomp.py::TestDecompCUDA::test_quick_atanh_cuda_bfloat16, test/test_decomp.py::TestDecompCUDA::test_quick_atanh_cuda_complex128, test/test_decomp.py::TestDecompCUDA::test_quick_bitwise_or_cuda_int16, test/test_decomp.py::TestDecompCUDA::test_quick_bitwise_right_shift_cuda_int8, test/test_decomp.py::TestDecompCUDA::test_quick_block_diag_cuda_int8, test/test_decomp.py::TestDecompCUDA::test_quick_cat_cuda_bool, test/test_decomp.py::TestDecompCUDA::test_quick_ceil_cuda_float16, test/test_decomp.py::TestDecompCUDA::test_quick_clone_cuda_float64, test/test_decomp.py::TestDecompCUDA::test_quick_constant_pad_nd_cuda_int16, test/test_decomp.py::TestDecompCUDA::test_quick_copysign_cuda_bfloat16, test/test_decomp.py::TestDecompCUDA::test_quick_core_backward_diag_embed_cuda_float64, test/test_decomp.py::TestDecompCUDA::test_quick_core_backward_nn_functional_binary_cross_entropy_with_logits_cuda_float64, test/test_decomp.py::TestDecompCUDA::test_quick_cos_cuda_float32, test/test_decomp.py::TestDecompCUDA::test_quick_cosh_cuda_int32, test/test_decomp.py::TestDecompCUDA::test_quick_cosh_cuda_int8, test/test_decomp.py::TestDecompCUDA::test_quick_count_nonzero_cuda_float16, test/test_decomp.py::TestDecompCUDA::test_quick_diag_cuda_complex64, test/test_decomp.py::TestDecompCUDA::test_quick_diag_cuda_int16, test/test_decomp.py::TestDecompCUDA::test_quick_diagonal_copy_cuda_float16, test/test_decomp.py::TestDecompCUDA::test_quick_diagonal_copy_cuda_int64, test/test_decomp.py::TestDecompCUDA::test_quick_diagonal_cuda_float16, test/test_decomp.py::TestDecompCUDA::test_quick_diagonal_cuda_int8, test/test_decomp.py::TestDecompCUDA::test_quick_diagonal_scatter_cuda_bfloat16, test/test_decomp.py::TestDecompCUDA::test_quick_digamma_cuda_float64, test/test_decomp.py::TestDecompCUDA::test_quick_div_floor_rounding_cuda_int16, test/test_decomp.py::TestDecompCUDA::test_quick_div_no_rounding_mode_cuda_bfloat16, test/test_decomp.py::TestDecompCUDA::test_quick_empty_strided_cuda_int32, test/test_decomp.py::TestDecompCUDA::test_quick_empty_strided_cuda_uint8, test/test_decomp.py::TestDecompCUDA::test_quick_erf_cuda_int8, test/test_decomp.py::TestDecompCUDA::test_quick_exp_cuda_uint8, test/test_decomp.py::TestDecompCUDA::test_quick_expand_cuda_complex64, test/test_decomp.py::TestDecompCUDA::test_quick_fft_fft_cuda_complex128, test/test_decomp.py::TestDecompCUDA::test_quick_fft_fft_cuda_uint8, test/test_decomp.py::TestDecompCUDA::test_quick_fft_fftn_cuda_complex64, test/test_decomp.py::TestDecompCUDA::test_quick_fft_fftn_cuda_int8, test/test_decomp.py::TestDecompCUDA::test_quick_fft_ihfftn_cuda_float32, test/test_decomp.py::TestDecompCUDA::test_quick_fft_ihfftn_cuda_int64, test/test_decomp.py::TestDecompCUDA::test_quick_fft_irfft_cuda_int64, test/test_decomp.py::TestDecompCUDA::test_quick_fft_rfftn_cuda_uint8, test/test_decomp.py::TestDecompCUDA::test_quick_floor_cuda_float32, test/test_decomp.py::TestDecompCUDA::test_quick_floor_divide_cuda_float32, test/test_decomp.py::TestDecompCUDA::test_quick_fmax_cuda_int16, test/test_decomp.py::TestDecompCUDA::test_quick_full_cuda_bool, test/test_decomp.py::TestDecompCUDA::test_quick_gcd_cuda_int32, test/test_decomp.py::TestDecompCUDA::test_quick_ge_cuda_int32, test/test_decomp.py::TestDecompCUDA::test_quick_geometric_cuda_float16, test/test_decomp.py::TestDecompCUDA::test_quick_geometric_cuda_int32, test/test_decomp.py::TestDecompCUDA::test_quick_index_add_cuda_int8, test/test_decomp.py::TestDecompCUDA::test_quick_index_select_cuda_int64, test/test_decomp.py::TestDecompCUDA::test_quick_isin_cuda_float32, test/test_decomp.py::TestDecompCUDA::test_quick_isinf_cuda_int64, test/test_decomp.py::TestDecompCUDA::test_quick_isnan_cuda_int64, test/test_decomp.py::TestDecompCUDA::test_quick_isneginf_cuda_int16, test/test_decomp.py::TestDecompCUDA::test_quick_lgamma_cuda_int16, test/test_decomp.py::TestDecompCUDA::test_quick_log10_cuda_complex128, test/test_decomp.py::TestDecompCUDA::test_quick_log1p_cuda_uint8, test/test_decomp.py::TestDecompCUDA::test_quick_log2_cuda_bool, test/test_decomp.py::TestDecompCUDA::test_quick_log_cuda_bfloat16, test/test_decomp.py::TestDecompCUDA::test_quick_log_softmax_cuda_bfloat16, test/test_decomp.py::TestDecompCUDA::test_quick_logaddexp2_cuda_float64, test/test_decomp.py::TestDecompCUDA::test_quick_logit_cuda_bfloat16, test/test_decomp.py::TestDecompCUDA::test_quick_logspace_cuda_complex64, test/test_decomp.py::TestDecompCUDA::test_quick_minimum_cuda_int64, test/test_decomp.py::TestDecompCUDA::test_quick_mvlgamma_mvlgamma_p_1_cuda_int64, test/test_decomp.py::TestDecompCUDA::test_quick_mvlgamma_mvlgamma_p_3_cuda_int64, test/test_decomp.py::TestDecompCUDA::test_quick_nan_to_num_cuda_float32, test/test_decomp.py::TestDecompCUDA::test_quick_nan_to_num_cuda_float64, test/test_decomp.py::TestDecompCUDA::test_quick_nansum_cuda_float16, test/test_decomp.py::TestDecompCUDA::test_quick_narrow_copy_cuda_complex64, test/test_decomp.py::TestDecompCUDA::test_quick_narrow_copy_cuda_float16, test/test_decomp.py::TestDecompCUDA::test_quick_narrow_copy_cuda_float32, test/test_decomp.py::TestDecompCUDA::test_quick_ne_cuda_uint8, test/test_decomp.py::TestDecompCUDA::test_quick_neg_cuda_int64, test/test_decomp.py::TestDecompCUDA::test_quick_neg_cuda_uint8, test/test_decomp.py::TestDecompCUDA::test_quick_new_empty_cuda_complex128, test/test_decomp.py::TestDecompCUDA::test_quick_new_empty_cuda_int16, test/test_decomp.py::TestDecompCUDA::test_quick_new_empty_cuda_uint8, test/test_decomp.py::TestDecompCUDA::test_quick_new_empty_strided_cuda_float32, test/test_decomp.py::TestDecompCUDA::test_quick_new_ones_cuda_float64, test/test_decomp.py::TestDecompCUDA::test_quick_nn_functional_hardswish_cuda_float64, test/test_decomp.py::TestDecompCUDA::test_quick_nn_functional_hardtanh_cuda_bfloat16, test/test_decomp.py::TestDecompCUDA::test_quick_nn_functional_mish_cuda_bfloat16, test/test_decomp.py::TestDecompCUDA::test_quick_nn_functional_relu_cuda_uint8, test/test_decomp.py::TestDecompCUDA::test_quick_nn_functional_softshrink_cuda_float32, test/test_decomp.py::TestDecompCUDA::test_quick_norm_fro_cuda_bfloat16, test/test_decomp.py::TestDecompCUDA::test_quick_normal_in_place_cuda_bfloat16, test/test_decomp.py::TestDecompCUDA::test_quick_ones_cuda_bool, test/test_decomp.py::TestDecompCUDA::test_quick_ones_cuda_float32, test/test_decomp.py::TestDecompCUDA::test_quick_ones_like_cuda_bool, test/test_decomp.py::TestDecompCUDA::test_quick_ones_like_cuda_float16, test/test_decomp.py::TestDecompCUDA::test_quick_permute_copy_cuda_bool, test/test_decomp.py::TestDecompCUDA::test_quick_pow_cuda_complex128, test/test_decomp.py::TestDecompCUDA::test_quick_prod_cuda_float64, test/test_decomp.py::TestDecompCUDA::test_quick_prod_cuda_int32, test/test_decomp.py::TestDecompCUDA::test_quick_randn_cuda_complex128, test/test_decomp.py::TestDecompCUDA::test_quick_reciprocal_cuda_int16, test/test_decomp.py::TestDecompCUDA::test_quick_renorm_cuda_complex64, test/test_decomp.py::TestDecompCUDA::test_quick_rot90_cuda_complex128, test/test_decomp.py::TestDecompCUDA::test_quick_rot90_cuda_float16, test/test_decomp.py::TestDecompCUDA::test_quick_rot90_cuda_float64, test/test_decomp.py::TestDecompCUDA::test_quick_rsqrt_cuda_float64, test/test_decomp.py::TestDecompCUDA::test_quick_select_cuda_int64, test/test_decomp.py::TestDecompCUDA::test_quick_sgn_cuda_int16, test/test_decomp.py::TestDecompCUDA::test_quick_sigmoid_cuda_int8, test/test_decomp.py::TestDecompCUDA::test_quick_sinh_cuda_uint8, test/test_decomp.py::TestDecompCUDA::test_quick_slice_cuda_complex128, test/test_decomp.py::TestDecompCUDA::test_quick_special_i1_cuda_bfloat16, test/test_decomp.py::TestDecompCUDA::test_quick_special_i1_cuda_float64, test/test_decomp.py::TestDecompCUDA::test_quick_special_log_ndtr_cuda_float32, test/test_decomp.py::TestDecompCUDA::test_quick_special_ndtri_cuda_int32, test/test_decomp.py::TestDecompCUDA::test_quick_special_xlog1py_cuda_float32, test/test_decomp.py::TestDecompCUDA::test_quick_special_xlog1py_cuda_int64, test/test_decomp.py::TestDecompCUDA::test_quick_split_cuda_complex128, test/test_decomp.py::TestDecompCUDA::test_quick_split_cuda_int8, test/test_decomp.py::TestDecompCUDA::test_quick_split_list_args_cuda_int32, test/test_decomp.py::TestDecompCUDA::test_quick_split_with_sizes_cuda_int32, test/test_decomp.py::TestDecompCUDA::test_quick_squeeze_cuda_float32, test/test_decomp.py::TestDecompCUDA::test_quick_std_mean_cuda_float64, test/test_decomp.py::TestDecompCUDA::test_quick_sub_cuda_float64, test/test_decomp.py::TestDecompCUDA::test_quick_sum_cuda_complex32, test/test_decomp.py::TestDecompCUDA::test_quick_t_copy_cuda_complex128, test/test_decomp.py::TestDecompCUDA::test_quick_t_cuda_bfloat16, test/test_decomp.py::TestDecompCUDA::test_quick_t_cuda_float16, test/test_decomp.py::TestDecompCUDA::test_quick_tan_cuda_int32, test/test_decomp.py::TestDecompCUDA::test_quick_transpose_copy_cuda_int16, test/test_decomp.py::TestDecompCUDA::test_quick_triu_cuda_int8, test/test_decomp.py::TestDecompCUDA::test_quick_trunc_cuda_int16, test/test_decomp.py::TestDecompCUDA::test_quick_unbind_cuda_bfloat16, test/test_decomp.py::TestDecompCUDA::test_quick_unbind_cuda_float16, test/test_decomp.py::TestDecompCUDA::test_quick_unbind_cuda_float32, test/test_decomp.py::TestDecompCUDA::test_quick_unfold_copy_cuda_complex32, test/test_decomp.py::TestDecompCUDA::test_quick_unsafe_split_cuda_int64, test/test_decomp.py::TestDecompCUDA::test_quick_unsqueeze_cuda_bool, test/test_decomp.py::TestDecompCUDA::test_quick_unsqueeze_cuda_float16, test/test_decomp.py::TestDecompCUDA::test_quick_var_unbiased_cuda_complex128, test/test_decomp.py::TestDecompCUDA::test_quick_var_unbiased_cuda_complex64, test/test_decomp.py::TestDecompCUDA::test_quick_var_unbiased_cuda_float32, test/test_decomp.py::TestDecompCUDA::test_quick_vdot_cuda_float32, test/test_decomp.py::TestDecompCUDA::test_quick_view_copy_cuda_int16, test/test_decomp.py::TestDecompCUDA::test_quick_xlogy_cuda_int16, test/test_decomp.py::TestDecompCUDA::test_quick_xlogy_cuda_uint8, test/test_decomp.py::TestDecompCUDA::test_quick_zero__cuda_bfloat16, test/test_decomp.py::TestDecompCUDA::test_quick_zero__cuda_float32, test/test_decomp.py::TestDecompCUDA::test_quick_zeros_cuda_int16, test/test_decomp.py::TestDecompCUDA::test_quick_zeros_like_cuda_int32, test/test_decomp.py::DecompOneOffTestsCUDA::test_elu_backward_cuda, test/test_decomp.py::HasDecompTest::test_mm_decompose_mm_dde 2025-12-04T11:54:19.1056391Z 2025-12-04T11:54:19.1056591Z Finished test_decomp 12/17 ... [2025-12-04 11:54:19.083073][8107.025290109], took 9.26min 2025-12-04T11:54:19.1097539Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/test_decomp/test_decomp-febede57690ded63.xml 2025-12-04T11:54:19.6420581Z Uploading artifacts took 0.45 seconds 2025-12-04T11:54:19.6424676Z Running test_ops_fwd_gradients 2/2 ... [2025-12-04 11:54:19.642224][8107.584439429] 2025-12-04T11:54:19.6425142Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2025-12-04T11:54:19.6428712Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'test_ops_fwd_gradients.py', '--shard-id=2', '--num-shards=2', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-12-04 11:54:19.642598] 2025-12-04T12:02:38.2622523Z 2025-12-04T12:02:38.2624225Z test_ops_fwd_gradients 2/2 was successful, full logs can be found in artifacts with path test/test-reports/test_ops_fwd_gradients_2.2_14b4262c9eb464e0_.log 2025-12-04T12:02:38.3090355Z Running 1579 items in this shard: test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_fn_fwgrad_bwgrad_H_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_fn_fwgrad_bwgrad_T_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_fn_fwgrad_bwgrad___getitem___cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_fn_fwgrad_bwgrad___rdiv___cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_fn_fwgrad_bwgrad___rdiv___cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_fn_fwgrad_bwgrad___rmatmul___cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_fn_fwgrad_bwgrad___rmul___cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_fn_fwgrad_bwgrad___rpow___cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_fn_fwgrad_bwgrad___rsub___cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_fn_fwgrad_bwgrad___rsub___cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_fn_fwgrad_bwgrad__batch_norm_with_update_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_fn_fwgrad_bwgrad__chunk_cat_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_fn_fwgrad_bwgrad__chunk_cat_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_fn_fwgrad_bwgrad__native_batch_norm_legit_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_fn_fwgrad_bwgrad__softmax_backward_data_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_fn_fwgrad_bwgrad__unsafe_masked_index_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_fn_fwgrad_bwgrad__unsafe_masked_index_put_accumulate_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_fn_fwgrad_bwgrad__upsample_bilinear2d_aa_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_fn_fwgrad_bwgrad_abs_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_fn_fwgrad_bwgrad_acos_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_fn_fwgrad_bwgrad_acosh_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_fn_fwgrad_bwgrad_add_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_fn_fwgrad_bwgrad_addbmm_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_fn_fwgrad_bwgrad_addcdiv_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_fn_fwgrad_bwgrad_addcmul_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_fn_fwgrad_bwgrad_addcmul_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_fn_fwgrad_bwgrad_addmm_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_fn_fwgrad_bwgrad_addmm_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_fn_fwgrad_bwgrad_addmm_decomposed_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_fn_fwgrad_bwgrad_addmv_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_fn_fwgrad_bwgrad_addr_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_fn_fwgrad_bwgrad_all_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_fn_fwgrad_bwgrad_allclose_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_fn_fwgrad_bwgrad_amax_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_fn_fwgrad_bwgrad_any_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_fn_fwgrad_bwgrad_arange_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_fn_fwgrad_bwgrad_argmax_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_fn_fwgrad_bwgrad_argwhere_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_fn_fwgrad_bwgrad_as_strided_copy_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_fn_fwgrad_bwgrad_as_strided_partial_views_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_fn_fwgrad_bwgrad_as_strided_scatter_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_fn_fwgrad_bwgrad_asin_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_fn_fwgrad_bwgrad_atanh_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_fn_fwgrad_bwgrad_atleast_1d_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_fn_fwgrad_bwgrad_atleast_2d_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_fn_fwgrad_bwgrad_atleast_3d_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_fn_fwgrad_bwgrad_atleast_3d_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_fn_fwgrad_bwgrad_baddbmm_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_fn_fwgrad_bwgrad_baddbmm_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_fn_fwgrad_bwgrad_bernoulli_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_fn_fwgrad_bwgrad_bfloat16_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_fn_fwgrad_bwgrad_block_diag_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_fn_fwgrad_bwgrad_bmm_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_fn_fwgrad_bwgrad_broadcast_tensors_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_fn_fwgrad_bwgrad_broadcast_tensors_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_fn_fwgrad_bwgrad_bucketize_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_fn_fwgrad_bwgrad_byte_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_fn_fwgrad_bwgrad_cartesian_prod_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_fn_fwgrad_bwgrad_cdist_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_fn_fwgrad_bwgrad_cdouble_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_fn_fwgrad_bwgrad_ceil_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_fn_fwgrad_bwgrad_cfloat_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_fn_fwgrad_bwgrad_chalf_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_fn_fwgrad_bwgrad_char_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_fn_fwgrad_bwgrad_char_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_fn_fwgrad_bwgrad_cholesky_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_fn_fwgrad_bwgrad_cholesky_inverse_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_fn_fwgrad_bwgrad_cholesky_solve_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_fn_fwgrad_bwgrad_chunk_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_fn_fwgrad_bwgrad_chunk_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_fn_fwgrad_bwgrad_combinations_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_fn_fwgrad_bwgrad_complex_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_fn_fwgrad_bwgrad_conj_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_fn_fwgrad_bwgrad_conj_physical_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_fn_fwgrad_bwgrad_conj_physical_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_fn_fwgrad_bwgrad_constant_pad_nd_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_fn_fwgrad_bwgrad_contiguous_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_fn_fwgrad_bwgrad_contiguous_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_fn_fwgrad_bwgrad_corrcoef_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_fn_fwgrad_bwgrad_cos_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_fn_fwgrad_bwgrad_cos_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_fn_fwgrad_bwgrad_cosh_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_fn_fwgrad_bwgrad_cosh_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_fn_fwgrad_bwgrad_count_nonzero_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_fn_fwgrad_bwgrad_count_nonzero_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_fn_fwgrad_bwgrad_cummin_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_fn_fwgrad_bwgrad_cumsum_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_fn_fwgrad_bwgrad_cumulative_trapezoid_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_fn_fwgrad_bwgrad_cumulative_trapezoid_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_fn_fwgrad_bwgrad_deg2rad_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_fn_fwgrad_bwgrad_diag_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_fn_fwgrad_bwgrad_diag_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_fn_fwgrad_bwgrad_diagflat_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_fn_fwgrad_bwgrad_diagflat_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_fn_fwgrad_bwgrad_diagonal_copy_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_fn_fwgrad_bwgrad_diagonal_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_fn_fwgrad_bwgrad_diagonal_scatter_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_fn_fwgrad_bwgrad_diff_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_fn_fwgrad_bwgrad_digamma_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_fn_fwgrad_bwgrad_dist_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_fn_fwgrad_bwgrad_div_floor_rounding_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_fn_fwgrad_bwgrad_div_no_rounding_mode_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_fn_fwgrad_bwgrad_div_trunc_rounding_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_fn_fwgrad_bwgrad_dot_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_fn_fwgrad_bwgrad_dot_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_fn_fwgrad_bwgrad_dstack_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_fn_fwgrad_bwgrad_dstack_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_fn_fwgrad_bwgrad_einsum_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_fn_fwgrad_bwgrad_einsum_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_fn_fwgrad_bwgrad_empty_permuted_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_fn_fwgrad_bwgrad_empty_strided_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_fn_fwgrad_bwgrad_eq_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_fn_fwgrad_bwgrad_equal_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_fn_fwgrad_bwgrad_erf_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_fn_fwgrad_bwgrad_erfinv_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_fn_fwgrad_bwgrad_exp2_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_fn_fwgrad_bwgrad_exp_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_fn_fwgrad_bwgrad_expand_as_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_fn_fwgrad_bwgrad_expand_copy_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_fn_fwgrad_bwgrad_expand_copy_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_fn_fwgrad_bwgrad_expand_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_fn_fwgrad_bwgrad_expm1_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_fn_fwgrad_bwgrad_fft_fft_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_fn_fwgrad_bwgrad_fft_fftn_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_fn_fwgrad_bwgrad_fft_hfft2_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_fn_fwgrad_bwgrad_fft_hfft_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_fn_fwgrad_bwgrad_fft_hfft_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_fn_fwgrad_bwgrad_fft_ifft2_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_fn_fwgrad_bwgrad_fft_ifft2_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_fn_fwgrad_bwgrad_fft_ifft_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_fn_fwgrad_bwgrad_fft_ifft_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_fn_fwgrad_bwgrad_fft_ifftshift_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_fn_fwgrad_bwgrad_fft_ihfft2_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_fn_fwgrad_bwgrad_fft_ihfft_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_fn_fwgrad_bwgrad_fft_irfft_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_fn_fwgrad_bwgrad_fft_irfftn_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_fn_fwgrad_bwgrad_fft_irfftn_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_fn_fwgrad_bwgrad_fft_rfftn_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_fn_fwgrad_bwgrad_fill_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_fn_fwgrad_bwgrad_flatten_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_fn_fwgrad_bwgrad_flatten_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_fn_fwgrad_bwgrad_flip_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_fn_fwgrad_bwgrad_fliplr_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_fn_fwgrad_bwgrad_flipud_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_fn_fwgrad_bwgrad_float_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_fn_fwgrad_bwgrad_float_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_fn_fwgrad_bwgrad_float_power_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_fn_fwgrad_bwgrad_float_power_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_fn_fwgrad_bwgrad_floor_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_fn_fwgrad_bwgrad_floor_divide_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_fn_fwgrad_bwgrad_fmin_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_fn_fwgrad_bwgrad_fmod_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_fn_fwgrad_bwgrad_frac_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_fn_fwgrad_bwgrad_full_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_fn_fwgrad_bwgrad_ge_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_fn_fwgrad_bwgrad_geometric_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_fn_fwgrad_bwgrad_geqrf_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_fn_fwgrad_bwgrad_geqrf_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_fn_fwgrad_bwgrad_gradient_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_fn_fwgrad_bwgrad_gradient_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_fn_fwgrad_bwgrad_grid_sampler_2d_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_fn_fwgrad_bwgrad_half_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_fn_fwgrad_bwgrad_hash_tensor_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_fn_fwgrad_bwgrad_heaviside_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_fn_fwgrad_bwgrad_hsplit_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_fn_fwgrad_bwgrad_i0_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_fn_fwgrad_bwgrad_imag_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_fn_fwgrad_bwgrad_index_add_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_fn_fwgrad_bwgrad_index_add_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_fn_fwgrad_bwgrad_index_select_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_fn_fwgrad_bwgrad_isfinite_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_fn_fwgrad_bwgrad_isnan_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_fn_fwgrad_bwgrad_isreal_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_fn_fwgrad_bwgrad_isreal_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_fn_fwgrad_bwgrad_item_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_fn_fwgrad_bwgrad_item_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_fn_fwgrad_bwgrad_jiterator_2inputs_2outputs_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_fn_fwgrad_bwgrad_jiterator_4inputs_with_extra_args_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_fn_fwgrad_bwgrad_jiterator_binary_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_fn_fwgrad_bwgrad_jiterator_binary_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_fn_fwgrad_bwgrad_jiterator_unary_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_fn_fwgrad_bwgrad_kron_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_fn_fwgrad_bwgrad_kron_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_fn_fwgrad_bwgrad_ldexp_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_fn_fwgrad_bwgrad_le_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_fn_fwgrad_bwgrad_lerp_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_fn_fwgrad_bwgrad_linalg_cholesky_ex_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_fn_fwgrad_bwgrad_linalg_cond_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_fn_fwgrad_bwgrad_linalg_cross_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_fn_fwgrad_bwgrad_linalg_cross_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_fn_fwgrad_bwgrad_linalg_det_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_fn_fwgrad_bwgrad_linalg_diagonal_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_fn_fwgrad_bwgrad_linalg_diagonal_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_fn_fwgrad_bwgrad_linalg_eig_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_fn_fwgrad_bwgrad_linalg_eigvals_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_fn_fwgrad_bwgrad_linalg_eigvalsh_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_fn_fwgrad_bwgrad_linalg_eigvalsh_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_fn_fwgrad_bwgrad_linalg_householder_product_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_fn_fwgrad_bwgrad_linalg_inv_ex_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_fn_fwgrad_bwgrad_linalg_ldl_factor_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_fn_fwgrad_bwgrad_linalg_ldl_solve_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_fn_fwgrad_bwgrad_linalg_lstsq_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_fn_fwgrad_bwgrad_linalg_lu_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_fn_fwgrad_bwgrad_linalg_lu_factor_ex_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_fn_fwgrad_bwgrad_linalg_matrix_norm_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_fn_fwgrad_bwgrad_linalg_matrix_power_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_fn_fwgrad_bwgrad_linalg_matrix_rank_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_fn_fwgrad_bwgrad_linalg_matrix_rank_hermitian_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_fn_fwgrad_bwgrad_linalg_norm_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_fn_fwgrad_bwgrad_linalg_norm_subgradients_at_zero_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_fn_fwgrad_bwgrad_linalg_pinv_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_fn_fwgrad_bwgrad_linalg_pinv_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_fn_fwgrad_bwgrad_linalg_pinv_singular_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_fn_fwgrad_bwgrad_linalg_slogdet_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_fn_fwgrad_bwgrad_linalg_solve_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_fn_fwgrad_bwgrad_linalg_solve_ex_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_fn_fwgrad_bwgrad_linalg_solve_ex_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_fn_fwgrad_bwgrad_linalg_solve_triangular_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_fn_fwgrad_bwgrad_linalg_tensorinv_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_fn_fwgrad_bwgrad_linalg_tensorsolve_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_fn_fwgrad_bwgrad_linalg_tensorsolve_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_fn_fwgrad_bwgrad_linalg_vander_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_fn_fwgrad_bwgrad_linalg_vander_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_fn_fwgrad_bwgrad_linalg_vector_norm_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_fn_fwgrad_bwgrad_linspace_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_fn_fwgrad_bwgrad_linspace_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_fn_fwgrad_bwgrad_linspace_tensor_overload_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_fn_fwgrad_bwgrad_linspace_tensor_overload_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_fn_fwgrad_bwgrad_log1p_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_fn_fwgrad_bwgrad_log2_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_fn_fwgrad_bwgrad_log_normal_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_fn_fwgrad_bwgrad_log_softmax_with_dtype_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_fn_fwgrad_bwgrad_log_softmax_with_dtype_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_fn_fwgrad_bwgrad_logaddexp_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_fn_fwgrad_bwgrad_logdet_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_fn_fwgrad_bwgrad_logical_and_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_fn_fwgrad_bwgrad_logical_not_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_fn_fwgrad_bwgrad_logical_or_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_fn_fwgrad_bwgrad_logical_or_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_fn_fwgrad_bwgrad_logical_xor_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_fn_fwgrad_bwgrad_logit_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_fn_fwgrad_bwgrad_logspace_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_fn_fwgrad_bwgrad_long_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_fn_fwgrad_bwgrad_lu_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_fn_fwgrad_bwgrad_lu_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_fn_fwgrad_bwgrad_lu_solve_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_fn_fwgrad_bwgrad_lu_solve_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_fn_fwgrad_bwgrad_lu_unpack_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_fn_fwgrad_bwgrad_mH_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_fn_fwgrad_bwgrad_mT_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_fn_fwgrad_bwgrad_masked_amin_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_fn_fwgrad_bwgrad_masked_argmax_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_fn_fwgrad_bwgrad_masked_cumprod_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_fn_fwgrad_bwgrad_masked_cumsum_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_fn_fwgrad_bwgrad_masked_fill_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_fn_fwgrad_bwgrad_masked_normalize_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_fn_fwgrad_bwgrad_masked_normalize_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_fn_fwgrad_bwgrad_masked_prod_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_fn_fwgrad_bwgrad_masked_softmin_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_fn_fwgrad_bwgrad_masked_sum_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_fn_fwgrad_bwgrad_matrix_exp_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_fn_fwgrad_bwgrad_max_binary_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_fn_fwgrad_bwgrad_maximum_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_fn_fwgrad_bwgrad_mean_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_fn_fwgrad_bwgrad_meshgrid_list_of_tensors_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_fn_fwgrad_bwgrad_meshgrid_list_of_tensors_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_fn_fwgrad_bwgrad_min_reduction_with_dim_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_fn_fwgrad_bwgrad_mm_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_fn_fwgrad_bwgrad_mode_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_fn_fwgrad_bwgrad_movedim_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_fn_fwgrad_bwgrad_mul_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_fn_fwgrad_bwgrad_mv_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_fn_fwgrad_bwgrad_mv_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_fn_fwgrad_bwgrad_mvlgamma_mvlgamma_p_3_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_fn_fwgrad_bwgrad_nan_to_num_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_fn_fwgrad_bwgrad_nanmean_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_fn_fwgrad_bwgrad_nanmean_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_fn_fwgrad_bwgrad_nanmedian_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_fn_fwgrad_bwgrad_narrow_copy_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_fn_fwgrad_bwgrad_narrow_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_fn_fwgrad_bwgrad_native_batch_norm_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_fn_fwgrad_bwgrad_native_layer_norm_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_fn_fwgrad_bwgrad_ne_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_fn_fwgrad_bwgrad_neg_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_fn_fwgrad_bwgrad_new_empty_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_fn_fwgrad_bwgrad_new_empty_strided_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_fn_fwgrad_bwgrad_new_full_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_fn_fwgrad_bwgrad_new_full_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_fn_fwgrad_bwgrad_new_ones_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_fn_fwgrad_bwgrad_new_ones_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_fn_fwgrad_bwgrad_new_zeros_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_fn_fwgrad_bwgrad_new_zeros_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_fn_fwgrad_bwgrad_nn_functional_adaptive_avg_pool2d_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_fn_fwgrad_bwgrad_nn_functional_adaptive_max_pool2d_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_fn_fwgrad_bwgrad_nn_functional_adaptive_max_pool3d_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_fn_fwgrad_bwgrad_nn_functional_alpha_dropout_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_fn_fwgrad_bwgrad_nn_functional_batch_norm_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_fn_fwgrad_bwgrad_nn_functional_batch_norm_without_cudnn_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_fn_fwgrad_bwgrad_nn_functional_bilinear_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_fn_fwgrad_bwgrad_nn_functional_binary_cross_entropy_with_logits_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_fn_fwgrad_bwgrad_nn_functional_conv2d_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_fn_fwgrad_bwgrad_nn_functional_conv2d_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_fn_fwgrad_bwgrad_nn_functional_conv_transpose2d_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_fn_fwgrad_bwgrad_nn_functional_conv_transpose3d_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_fn_fwgrad_bwgrad_nn_functional_cosine_embedding_loss_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_fn_fwgrad_bwgrad_nn_functional_cosine_similarity_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_fn_fwgrad_bwgrad_nn_functional_dropout3d_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_fn_fwgrad_bwgrad_nn_functional_embedding_bag_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_fn_fwgrad_bwgrad_nn_functional_fractional_max_pool2d_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_fn_fwgrad_bwgrad_nn_functional_glu_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_fn_fwgrad_bwgrad_nn_functional_grid_sample_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_fn_fwgrad_bwgrad_nn_functional_group_norm_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_fn_fwgrad_bwgrad_nn_functional_hardshrink_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_fn_fwgrad_bwgrad_nn_functional_hardsigmoid_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_fn_fwgrad_bwgrad_nn_functional_hardswish_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_fn_fwgrad_bwgrad_nn_functional_huber_loss_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_fn_fwgrad_bwgrad_nn_functional_instance_norm_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_fn_fwgrad_bwgrad_nn_functional_interpolate_area_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_fn_fwgrad_bwgrad_nn_functional_interpolate_nearest_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_fn_fwgrad_bwgrad_nn_functional_interpolate_trilinear_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_fn_fwgrad_bwgrad_nn_functional_kl_div_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_fn_fwgrad_bwgrad_nn_functional_l1_loss_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_fn_fwgrad_bwgrad_nn_functional_layer_norm_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_fn_fwgrad_bwgrad_nn_functional_linear_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_fn_fwgrad_bwgrad_nn_functional_logsigmoid_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_fn_fwgrad_bwgrad_nn_functional_max_pool1d_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_fn_fwgrad_bwgrad_nn_functional_max_pool2d_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_fn_fwgrad_bwgrad_nn_functional_max_pool3d_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_fn_fwgrad_bwgrad_nn_functional_max_unpool1d_grad_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_fn_fwgrad_bwgrad_nn_functional_max_unpool2d_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_fn_fwgrad_bwgrad_nn_functional_max_unpool3d_grad_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_fn_fwgrad_bwgrad_nn_functional_mish_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_fn_fwgrad_bwgrad_nn_functional_multilabel_soft_margin_loss_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_fn_fwgrad_bwgrad_nn_functional_normalize_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_fn_fwgrad_bwgrad_nn_functional_normalize_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_fn_fwgrad_bwgrad_nn_functional_pad_circular_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_fn_fwgrad_bwgrad_nn_functional_pad_constant_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_fn_fwgrad_bwgrad_nn_functional_pad_reflect_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_fn_fwgrad_bwgrad_nn_functional_pad_replicate_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_fn_fwgrad_bwgrad_nn_functional_pad_replicate_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_fn_fwgrad_bwgrad_nn_functional_pad_replicate_negative_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_fn_fwgrad_bwgrad_nn_functional_pixel_shuffle_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_fn_fwgrad_bwgrad_nn_functional_pixel_unshuffle_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_fn_fwgrad_bwgrad_nn_functional_relu_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_fn_fwgrad_bwgrad_nn_functional_rms_norm_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_fn_fwgrad_bwgrad_nn_functional_scaled_dot_product_attention_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_fn_fwgrad_bwgrad_nn_functional_silu_complex_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_fn_fwgrad_bwgrad_nn_functional_silu_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_fn_fwgrad_bwgrad_nn_functional_soft_margin_loss_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_fn_fwgrad_bwgrad_nn_functional_softmin_with_dtype_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_fn_fwgrad_bwgrad_nn_functional_softmin_with_dtype_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_fn_fwgrad_bwgrad_nn_functional_softplus_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_fn_fwgrad_bwgrad_nn_functional_softsign_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_fn_fwgrad_bwgrad_nn_functional_tanhshrink_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_fn_fwgrad_bwgrad_nn_functional_threshold_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_fn_fwgrad_bwgrad_nn_functional_triplet_margin_loss_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_fn_fwgrad_bwgrad_nn_functional_triplet_margin_with_distance_loss_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_fn_fwgrad_bwgrad_nn_functional_upsample_bilinear_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_fn_fwgrad_bwgrad_nonzero_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_fn_fwgrad_bwgrad_nonzero_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_fn_fwgrad_bwgrad_norm_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_fn_fwgrad_bwgrad_norm_fro_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_fn_fwgrad_bwgrad_norm_inf_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_fn_fwgrad_bwgrad_norm_nuc_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_fn_fwgrad_bwgrad_normal_number_mean_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_fn_fwgrad_bwgrad_ones_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_fn_fwgrad_bwgrad_ormqr_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_fn_fwgrad_bwgrad_permute_copy_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_fn_fwgrad_bwgrad_permute_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_fn_fwgrad_bwgrad_pinverse_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_fn_fwgrad_bwgrad_pinverse_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_fn_fwgrad_bwgrad_polar_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_fn_fwgrad_bwgrad_polygamma_polygamma_n_0_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_fn_fwgrad_bwgrad_polygamma_polygamma_n_1_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_fn_fwgrad_bwgrad_polygamma_polygamma_n_2_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_fn_fwgrad_bwgrad_polygamma_polygamma_n_3_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_fn_fwgrad_bwgrad_polygamma_polygamma_n_4_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_fn_fwgrad_bwgrad_positive_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_fn_fwgrad_bwgrad_pow_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_fn_fwgrad_bwgrad_pow_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_fn_fwgrad_bwgrad_prod_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_fn_fwgrad_bwgrad_put_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_fn_fwgrad_bwgrad_qr_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_fn_fwgrad_bwgrad_quantile_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_fn_fwgrad_bwgrad_randn_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_fn_fwgrad_bwgrad_randn_like_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_fn_fwgrad_bwgrad_ravel_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_fn_fwgrad_bwgrad_real_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_fn_fwgrad_bwgrad_reciprocal_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_fn_fwgrad_bwgrad_remainder_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_fn_fwgrad_bwgrad_repeat_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_fn_fwgrad_bwgrad_repeat_interleave_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_fn_fwgrad_bwgrad_resize__cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_fn_fwgrad_bwgrad_resize_as__cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_fn_fwgrad_bwgrad_resize_as__cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_fn_fwgrad_bwgrad_resolve_conj_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_fn_fwgrad_bwgrad_resolve_conj_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_fn_fwgrad_bwgrad_roll_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_fn_fwgrad_bwgrad_rot90_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_fn_fwgrad_bwgrad_round_decimals_0_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_fn_fwgrad_bwgrad_round_decimals_neg_3_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_fn_fwgrad_bwgrad_rsqrt_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_fn_fwgrad_bwgrad_rsqrt_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_fn_fwgrad_bwgrad_rsub_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_fn_fwgrad_bwgrad_scalar_tensor_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_fn_fwgrad_bwgrad_scatter_add_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_fn_fwgrad_bwgrad_scatter_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_fn_fwgrad_bwgrad_scatter_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_fn_fwgrad_bwgrad_scatter_reduce_amax_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_fn_fwgrad_bwgrad_scatter_reduce_prod_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_fn_fwgrad_bwgrad_searchsorted_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_fn_fwgrad_bwgrad_select_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_fn_fwgrad_bwgrad_sgn_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_fn_fwgrad_bwgrad_short_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_fn_fwgrad_bwgrad_short_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_fn_fwgrad_bwgrad_sigmoid_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_fn_fwgrad_bwgrad_sign_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_fn_fwgrad_bwgrad_signal_windows_cosine_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_fn_fwgrad_bwgrad_signal_windows_gaussian_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_fn_fwgrad_bwgrad_signal_windows_general_hamming_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_fn_fwgrad_bwgrad_signal_windows_hamming_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_fn_fwgrad_bwgrad_signal_windows_kaiser_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_fn_fwgrad_bwgrad_sin_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_fn_fwgrad_bwgrad_sin_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_fn_fwgrad_bwgrad_sinc_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_fn_fwgrad_bwgrad_slice_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_fn_fwgrad_bwgrad_slice_scatter_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_fn_fwgrad_bwgrad_softmax_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_fn_fwgrad_bwgrad_sparse_mm_reduce_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_fn_fwgrad_bwgrad_sparse_sampled_addmm_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_fn_fwgrad_bwgrad_special_bessel_j0_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_fn_fwgrad_bwgrad_special_chebyshev_polynomial_u_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_fn_fwgrad_bwgrad_special_chebyshev_polynomial_w_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_fn_fwgrad_bwgrad_special_erfcx_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_fn_fwgrad_bwgrad_special_hermite_polynomial_h_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_fn_fwgrad_bwgrad_special_hermite_polynomial_he_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_fn_fwgrad_bwgrad_special_i1_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_fn_fwgrad_bwgrad_special_laguerre_polynomial_l_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_fn_fwgrad_bwgrad_special_log_ndtr_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_fn_fwgrad_bwgrad_special_modified_bessel_i0_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_fn_fwgrad_bwgrad_special_modified_bessel_i1_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_fn_fwgrad_bwgrad_special_polygamma_special_polygamma_n_0_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_fn_fwgrad_bwgrad_special_shifted_chebyshev_polynomial_u_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_fn_fwgrad_bwgrad_special_spherical_bessel_j0_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_fn_fwgrad_bwgrad_split_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_fn_fwgrad_bwgrad_split_list_args_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_fn_fwgrad_bwgrad_split_list_args_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_fn_fwgrad_bwgrad_split_with_sizes_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_fn_fwgrad_bwgrad_sqrt_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_fn_fwgrad_bwgrad_square_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_fn_fwgrad_bwgrad_squeeze_copy_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_fn_fwgrad_bwgrad_squeeze_copy_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_fn_fwgrad_bwgrad_squeeze_multiple_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_fn_fwgrad_bwgrad_squeeze_multiple_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_fn_fwgrad_bwgrad_stack_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_fn_fwgrad_bwgrad_std_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_fn_fwgrad_bwgrad_std_mean_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_fn_fwgrad_bwgrad_std_mean_unbiased_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_fn_fwgrad_bwgrad_sub_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_fn_fwgrad_bwgrad_sum_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_fn_fwgrad_bwgrad_sum_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_fn_fwgrad_bwgrad_sum_to_size_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_fn_fwgrad_bwgrad_sum_to_size_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_fn_fwgrad_bwgrad_svd_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_fn_fwgrad_bwgrad_svd_lowrank_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_fn_fwgrad_bwgrad_t_copy_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_fn_fwgrad_bwgrad_t_copy_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_fn_fwgrad_bwgrad_take_along_dim_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_fn_fwgrad_bwgrad_take_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_fn_fwgrad_bwgrad_tanh_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_fn_fwgrad_bwgrad_tensor_split_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_fn_fwgrad_bwgrad_tensor_split_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_fn_fwgrad_bwgrad_tensordot_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_fn_fwgrad_bwgrad_tensordot_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_fn_fwgrad_bwgrad_tile_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_fn_fwgrad_bwgrad_to_sparse_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_fn_fwgrad_bwgrad_topk_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_fn_fwgrad_bwgrad_trace_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_fn_fwgrad_bwgrad_transpose_copy_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_fn_fwgrad_bwgrad_transpose_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_fn_fwgrad_bwgrad_transpose_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_fn_fwgrad_bwgrad_trapezoid_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_fn_fwgrad_bwgrad_triangular_solve_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_fn_fwgrad_bwgrad_triangular_solve_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_fn_fwgrad_bwgrad_triu_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_fn_fwgrad_bwgrad_trunc_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_fn_fwgrad_bwgrad_unbind_copy_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_fn_fwgrad_bwgrad_unbind_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_fn_fwgrad_bwgrad_unflatten_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_fn_fwgrad_bwgrad_unfold_copy_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_fn_fwgrad_bwgrad_unfold_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_fn_fwgrad_bwgrad_uniform_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_fn_fwgrad_bwgrad_uniform_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_fn_fwgrad_bwgrad_unique_consecutive_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_fn_fwgrad_bwgrad_unique_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_fn_fwgrad_bwgrad_unsafe_chunk_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_fn_fwgrad_bwgrad_unsafe_chunk_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_fn_fwgrad_bwgrad_unsqueeze_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_fn_fwgrad_bwgrad_var_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_fn_fwgrad_bwgrad_var_mean_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_fn_fwgrad_bwgrad_var_mean_unbiased_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_fn_fwgrad_bwgrad_var_unbiased_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_fn_fwgrad_bwgrad_vdot_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_fn_fwgrad_bwgrad_view_as_complex_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_fn_fwgrad_bwgrad_view_copy_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_fn_fwgrad_bwgrad_view_copy_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_fn_fwgrad_bwgrad_view_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_fn_fwgrad_bwgrad_vstack_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_fn_fwgrad_bwgrad_where_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_fn_fwgrad_bwgrad_xlogy_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_fn_fwgrad_bwgrad_zeros_like_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_forward_mode_AD_H_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_forward_mode_AD_T_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_forward_mode_AD___getitem___cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_forward_mode_AD___getitem___cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_forward_mode_AD___rdiv___cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_forward_mode_AD___rdiv___cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_forward_mode_AD___rmod___cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_forward_mode_AD___rmul___cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_forward_mode_AD___rsub___cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_forward_mode_AD__batch_norm_with_update_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_forward_mode_AD__chunk_cat_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_forward_mode_AD__chunk_cat_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_forward_mode_AD__native_batch_norm_legit_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_forward_mode_AD__softmax_backward_data_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_forward_mode_AD__unsafe_masked_index_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_forward_mode_AD__unsafe_masked_index_put_accumulate_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_forward_mode_AD__unsafe_masked_index_put_accumulate_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_forward_mode_AD__upsample_bilinear2d_aa_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_forward_mode_AD_abs_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_forward_mode_AD_acos_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_forward_mode_AD_acosh_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_forward_mode_AD_acosh_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_forward_mode_AD_add_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_forward_mode_AD_addbmm_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_forward_mode_AD_addcdiv_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_forward_mode_AD_addmv_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_forward_mode_AD_addmv_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_forward_mode_AD_addr_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_forward_mode_AD_all_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_forward_mode_AD_all_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_forward_mode_AD_allclose_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_forward_mode_AD_amin_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_forward_mode_AD_aminmax_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_forward_mode_AD_any_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_forward_mode_AD_as_strided_partial_views_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_forward_mode_AD_as_strided_scatter_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_forward_mode_AD_asin_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_forward_mode_AD_asinh_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_forward_mode_AD_atan2_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_forward_mode_AD_atan_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_forward_mode_AD_atan_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_forward_mode_AD_atanh_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_forward_mode_AD_atanh_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_forward_mode_AD_atleast_1d_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_forward_mode_AD_atleast_1d_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_forward_mode_AD_atleast_3d_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_forward_mode_AD_atleast_3d_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_forward_mode_AD_baddbmm_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_forward_mode_AD_bernoulli_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_forward_mode_AD_bfloat16_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_forward_mode_AD_block_diag_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_forward_mode_AD_bmm_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_forward_mode_AD_bool_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_forward_mode_AD_broadcast_tensors_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_forward_mode_AD_broadcast_tensors_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_forward_mode_AD_broadcast_to_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_forward_mode_AD_broadcast_to_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_forward_mode_AD_byte_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_forward_mode_AD_cartesian_prod_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_forward_mode_AD_cartesian_prod_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_forward_mode_AD_cat_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_forward_mode_AD_cauchy_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_forward_mode_AD_ceil_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_forward_mode_AD_chalf_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_forward_mode_AD_char_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_forward_mode_AD_char_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_forward_mode_AD_cholesky_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_forward_mode_AD_cholesky_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_forward_mode_AD_cholesky_solve_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_forward_mode_AD_chunk_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_forward_mode_AD_clamp_max_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_forward_mode_AD_clamp_min_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_forward_mode_AD_clone_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_forward_mode_AD_column_stack_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_forward_mode_AD_combinations_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_forward_mode_AD_conj_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_forward_mode_AD_constant_pad_nd_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_forward_mode_AD_constant_pad_nd_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_forward_mode_AD_contiguous_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_forward_mode_AD_contiguous_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_forward_mode_AD_corrcoef_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_forward_mode_AD_cos_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_forward_mode_AD_cosh_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_forward_mode_AD_count_nonzero_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_forward_mode_AD_count_nonzero_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_forward_mode_AD_cross_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_forward_mode_AD_cumprod_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_forward_mode_AD_cumprod_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_forward_mode_AD_cumsum_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_forward_mode_AD_cumsum_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_forward_mode_AD_deg2rad_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_forward_mode_AD_diag_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_forward_mode_AD_diag_embed_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_forward_mode_AD_diagonal_copy_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_forward_mode_AD_diagonal_scatter_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_forward_mode_AD_diagonal_scatter_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_forward_mode_AD_diff_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_forward_mode_AD_dist_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_forward_mode_AD_div_floor_rounding_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_forward_mode_AD_div_no_rounding_mode_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_forward_mode_AD_div_no_rounding_mode_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_forward_mode_AD_div_trunc_rounding_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_forward_mode_AD_dot_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_forward_mode_AD_dsplit_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_forward_mode_AD_dstack_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_forward_mode_AD_einsum_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_forward_mode_AD_einsum_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_forward_mode_AD_empty_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_forward_mode_AD_empty_like_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_forward_mode_AD_empty_permuted_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_forward_mode_AD_empty_permuted_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_forward_mode_AD_empty_strided_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_forward_mode_AD_eq_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_forward_mode_AD_equal_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_forward_mode_AD_erf_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_forward_mode_AD_erfinv_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_forward_mode_AD_exp2_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_forward_mode_AD_exp2_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_forward_mode_AD_expand_as_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_forward_mode_AD_expand_copy_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_forward_mode_AD_expand_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_forward_mode_AD_expm1_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_forward_mode_AD_expm1_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_forward_mode_AD_eye_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_forward_mode_AD_eye_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_forward_mode_AD_fft_fft_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_forward_mode_AD_fft_fftshift_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_forward_mode_AD_fft_hfft2_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_forward_mode_AD_fft_hfft2_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_forward_mode_AD_fft_hfft_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_forward_mode_AD_fft_ifft2_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_forward_mode_AD_fft_ifftn_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_forward_mode_AD_fft_ifftn_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_forward_mode_AD_fft_ifftshift_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_forward_mode_AD_fft_ihfft2_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_forward_mode_AD_fft_ihfft_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_forward_mode_AD_fft_irfft2_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_forward_mode_AD_fft_irfft_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_forward_mode_AD_fft_irfftn_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_forward_mode_AD_fft_rfft2_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_forward_mode_AD_fft_rfftn_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_forward_mode_AD_fill_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_forward_mode_AD_flatten_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_forward_mode_AD_flatten_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_forward_mode_AD_flip_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_forward_mode_AD_flip_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_forward_mode_AD_fliplr_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_forward_mode_AD_flipud_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_forward_mode_AD_float_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_forward_mode_AD_float_power_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_forward_mode_AD_floor_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_forward_mode_AD_fmax_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_forward_mode_AD_fmod_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_forward_mode_AD_frexp_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_forward_mode_AD_full_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_forward_mode_AD_full_like_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_forward_mode_AD_gather_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_forward_mode_AD_gather_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_forward_mode_AD_ge_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_forward_mode_AD_geometric_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_forward_mode_AD_geqrf_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_forward_mode_AD_geqrf_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_forward_mode_AD_gradient_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_forward_mode_AD_gt_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_forward_mode_AD_half_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_forward_mode_AD_half_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_forward_mode_AD_hash_tensor_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_forward_mode_AD_heaviside_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_forward_mode_AD_hsplit_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_forward_mode_AD_i0_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_forward_mode_AD_igamma_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_forward_mode_AD_imag_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_forward_mode_AD_index_copy_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_forward_mode_AD_index_fill_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_forward_mode_AD_index_fill_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_forward_mode_AD_index_put_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_forward_mode_AD_index_put_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_forward_mode_AD_index_reduce_amin_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_forward_mode_AD_index_select_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_forward_mode_AD_inner_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_forward_mode_AD_int_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_forward_mode_AD_isfinite_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_forward_mode_AD_isin_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_forward_mode_AD_isinf_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_forward_mode_AD_isposinf_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_forward_mode_AD_isreal_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_forward_mode_AD_isreal_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_forward_mode_AD_istft_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_forward_mode_AD_item_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_forward_mode_AD_item_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_forward_mode_AD_jiterator_2inputs_2outputs_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_forward_mode_AD_jiterator_2inputs_2outputs_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_forward_mode_AD_jiterator_4inputs_with_extra_args_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_forward_mode_AD_jiterator_binary_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_forward_mode_AD_jiterator_unary_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_forward_mode_AD_kron_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_forward_mode_AD_ldexp_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_forward_mode_AD_le_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_forward_mode_AD_lerp_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_forward_mode_AD_linalg_cholesky_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_forward_mode_AD_linalg_cholesky_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_forward_mode_AD_linalg_cond_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_forward_mode_AD_linalg_cond_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_forward_mode_AD_linalg_diagonal_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_forward_mode_AD_linalg_diagonal_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_forward_mode_AD_linalg_eig_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_forward_mode_AD_linalg_eigh_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_forward_mode_AD_linalg_eigvals_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_forward_mode_AD_linalg_inv_ex_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_forward_mode_AD_linalg_ldl_factor_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_forward_mode_AD_linalg_ldl_factor_ex_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_forward_mode_AD_linalg_lstsq_grad_oriented_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_forward_mode_AD_linalg_lu_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_forward_mode_AD_linalg_lu_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_forward_mode_AD_linalg_lu_factor_ex_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_forward_mode_AD_linalg_matrix_norm_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_forward_mode_AD_linalg_matrix_rank_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_forward_mode_AD_linalg_multi_dot_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_forward_mode_AD_linalg_norm_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_forward_mode_AD_linalg_qr_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_forward_mode_AD_linalg_slogdet_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_forward_mode_AD_linalg_solve_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_forward_mode_AD_linalg_solve_ex_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_forward_mode_AD_linalg_solve_ex_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_forward_mode_AD_linalg_solve_triangular_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_forward_mode_AD_linalg_solve_triangular_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_forward_mode_AD_linalg_svdvals_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_forward_mode_AD_linalg_svdvals_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_forward_mode_AD_linalg_tensorinv_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_forward_mode_AD_linalg_tensorsolve_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_forward_mode_AD_linalg_vander_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_forward_mode_AD_log2_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_forward_mode_AD_log_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_forward_mode_AD_log_normal_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_forward_mode_AD_log_softmax_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_forward_mode_AD_log_softmax_with_dtype_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_forward_mode_AD_log_softmax_with_dtype_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_forward_mode_AD_logaddexp2_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_forward_mode_AD_logaddexp_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_forward_mode_AD_logcumsumexp_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_forward_mode_AD_logdet_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_forward_mode_AD_logdet_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_forward_mode_AD_logical_and_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_forward_mode_AD_logical_not_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_forward_mode_AD_logical_xor_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_forward_mode_AD_logit_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_forward_mode_AD_logspace_tensor_overload_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_forward_mode_AD_logspace_tensor_overload_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_forward_mode_AD_logsumexp_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_forward_mode_AD_long_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_forward_mode_AD_long_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_forward_mode_AD_lt_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_forward_mode_AD_lu_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_forward_mode_AD_lu_solve_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_forward_mode_AD_lu_unpack_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_forward_mode_AD_mH_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_forward_mode_AD_masked_argmin_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_forward_mode_AD_masked_cumprod_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_forward_mode_AD_masked_cumsum_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_forward_mode_AD_masked_fill_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_forward_mode_AD_masked_log_softmax_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_forward_mode_AD_masked_logaddexp_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_forward_mode_AD_masked_logsumexp_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_forward_mode_AD_masked_logsumexp_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_forward_mode_AD_masked_mean_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_forward_mode_AD_masked_norm_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_forward_mode_AD_masked_normalize_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_forward_mode_AD_masked_normalize_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_forward_mode_AD_masked_prod_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_forward_mode_AD_masked_prod_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_forward_mode_AD_masked_scatter_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_forward_mode_AD_masked_std_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_forward_mode_AD_masked_std_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_forward_mode_AD_masked_sum_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_forward_mode_AD_masked_var_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_forward_mode_AD_matmul_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_forward_mode_AD_matmul_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_forward_mode_AD_max_pool2d_with_indices_backward_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_forward_mode_AD_max_reduction_no_dim_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_forward_mode_AD_mean_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_forward_mode_AD_mean_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_forward_mode_AD_median_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_forward_mode_AD_meshgrid_list_of_tensors_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_forward_mode_AD_meshgrid_variadic_tensors_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_forward_mode_AD_meshgrid_variadic_tensors_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_forward_mode_AD_min_reduction_no_dim_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_forward_mode_AD_mm_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_forward_mode_AD_mm_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_forward_mode_AD_movedim_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_forward_mode_AD_mul_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_forward_mode_AD_mv_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_forward_mode_AD_mvlgamma_mvlgamma_p_3_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_forward_mode_AD_nan_to_num_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_forward_mode_AD_nanmean_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_forward_mode_AD_nanquantile_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_forward_mode_AD_nansum_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_forward_mode_AD_nansum_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_forward_mode_AD_narrow_copy_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_forward_mode_AD_narrow_copy_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_forward_mode_AD_native_batch_norm_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_forward_mode_AD_ne_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_forward_mode_AD_neg_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_forward_mode_AD_new_empty_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_forward_mode_AD_new_empty_strided_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_forward_mode_AD_new_full_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_forward_mode_AD_new_full_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_forward_mode_AD_new_ones_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_forward_mode_AD_nextafter_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_forward_mode_AD_nn_functional_adaptive_avg_pool1d_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_forward_mode_AD_nn_functional_adaptive_avg_pool3d_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_forward_mode_AD_nn_functional_avg_pool3d_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_forward_mode_AD_nn_functional_batch_norm_without_cudnn_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_forward_mode_AD_nn_functional_bilinear_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_forward_mode_AD_nn_functional_binary_cross_entropy_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_forward_mode_AD_nn_functional_binary_cross_entropy_with_logits_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_forward_mode_AD_nn_functional_channel_shuffle_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_forward_mode_AD_nn_functional_conv2d_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_forward_mode_AD_nn_functional_conv2d_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_forward_mode_AD_nn_functional_conv_transpose1d_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_forward_mode_AD_nn_functional_conv_transpose1d_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_forward_mode_AD_nn_functional_conv_transpose2d_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_forward_mode_AD_nn_functional_conv_transpose2d_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_forward_mode_AD_nn_functional_cross_entropy_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_forward_mode_AD_nn_functional_ctc_loss_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_forward_mode_AD_nn_functional_elu_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_forward_mode_AD_nn_functional_embedding_bag_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_forward_mode_AD_nn_functional_embedding_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_forward_mode_AD_nn_functional_feature_alpha_dropout_with_train_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_forward_mode_AD_nn_functional_feature_alpha_dropout_without_train_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_forward_mode_AD_nn_functional_fractional_max_pool2d_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_forward_mode_AD_nn_functional_gelu_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_forward_mode_AD_nn_functional_glu_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_forward_mode_AD_nn_functional_grid_sample_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_forward_mode_AD_nn_functional_group_norm_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_forward_mode_AD_nn_functional_hardshrink_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_forward_mode_AD_nn_functional_hardswish_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_forward_mode_AD_nn_functional_hinge_embedding_loss_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_forward_mode_AD_nn_functional_huber_loss_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_forward_mode_AD_nn_functional_interpolate_area_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_forward_mode_AD_nn_functional_interpolate_bicubic_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_forward_mode_AD_nn_functional_interpolate_bilinear_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_forward_mode_AD_nn_functional_interpolate_linear_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_forward_mode_AD_nn_functional_interpolate_trilinear_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_forward_mode_AD_nn_functional_kl_div_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_forward_mode_AD_nn_functional_l1_loss_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_forward_mode_AD_nn_functional_layer_norm_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_forward_mode_AD_nn_functional_leaky_relu_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_forward_mode_AD_nn_functional_logsigmoid_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_forward_mode_AD_nn_functional_margin_ranking_loss_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_forward_mode_AD_nn_functional_max_pool1d_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_forward_mode_AD_nn_functional_max_pool2d_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_forward_mode_AD_nn_functional_max_unpool1d_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_forward_mode_AD_nn_functional_max_unpool2d_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_forward_mode_AD_nn_functional_max_unpool2d_grad_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_forward_mode_AD_nn_functional_max_unpool3d_grad_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_forward_mode_AD_nn_functional_mse_loss_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_forward_mode_AD_nn_functional_multi_head_attention_forward_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_forward_mode_AD_nn_functional_multilabel_margin_loss_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_forward_mode_AD_nn_functional_normalize_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_forward_mode_AD_nn_functional_pad_circular_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_forward_mode_AD_nn_functional_pad_constant_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_forward_mode_AD_nn_functional_pad_replicate_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_forward_mode_AD_nn_functional_pad_replicate_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_forward_mode_AD_nn_functional_pad_replicate_negative_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_forward_mode_AD_nn_functional_pad_replicate_negative_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_forward_mode_AD_nn_functional_pairwise_distance_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_forward_mode_AD_nn_functional_pdist_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_forward_mode_AD_nn_functional_pixel_unshuffle_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_forward_mode_AD_nn_functional_relu6_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_forward_mode_AD_nn_functional_rms_norm_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_forward_mode_AD_nn_functional_scaled_dot_product_attention_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_forward_mode_AD_nn_functional_silu_complex_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_forward_mode_AD_nn_functional_softmin_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_forward_mode_AD_nn_functional_softshrink_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_forward_mode_AD_nn_functional_softsign_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_forward_mode_AD_nn_functional_softsign_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_forward_mode_AD_nn_functional_threshold_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_forward_mode_AD_nn_functional_triplet_margin_with_distance_loss_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_forward_mode_AD_nn_functional_triplet_margin_with_distance_loss_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_forward_mode_AD_nn_functional_upsample_nearest_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_forward_mode_AD_nonzero_static_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_forward_mode_AD_norm_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_forward_mode_AD_norm_fro_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_forward_mode_AD_norm_inf_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_forward_mode_AD_norm_nuc_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_forward_mode_AD_normal_in_place_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_forward_mode_AD_normal_number_mean_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_forward_mode_AD_ones_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_forward_mode_AD_outer_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_forward_mode_AD_outer_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_forward_mode_AD_pca_lowrank_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_forward_mode_AD_permute_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_forward_mode_AD_permute_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_forward_mode_AD_pinverse_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_forward_mode_AD_pinverse_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_forward_mode_AD_polygamma_polygamma_n_0_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_forward_mode_AD_polygamma_polygamma_n_2_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_forward_mode_AD_polygamma_polygamma_n_3_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_forward_mode_AD_polygamma_polygamma_n_4_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_forward_mode_AD_positive_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_forward_mode_AD_pow_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_forward_mode_AD_prod_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_forward_mode_AD_prod_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_forward_mode_AD_put_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_forward_mode_AD_qr_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_forward_mode_AD_rand_like_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_forward_mode_AD_randint_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_forward_mode_AD_randint_like_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_forward_mode_AD_randn_like_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_forward_mode_AD_ravel_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_forward_mode_AD_ravel_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_forward_mode_AD_reciprocal_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_forward_mode_AD_renorm_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_forward_mode_AD_repeat_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_forward_mode_AD_reshape_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_forward_mode_AD_reshape_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_forward_mode_AD_resize__cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_forward_mode_AD_resize__cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_forward_mode_AD_resize_as__cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_forward_mode_AD_resolve_conj_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_forward_mode_AD_roll_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_forward_mode_AD_roll_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_forward_mode_AD_rot90_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_forward_mode_AD_round_decimals_0_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_forward_mode_AD_round_decimals_neg_3_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_forward_mode_AD_rsqrt_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_forward_mode_AD_rsub_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_forward_mode_AD_rsub_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_forward_mode_AD_scalar_tensor_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_forward_mode_AD_scalar_tensor_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_forward_mode_AD_scatter_add_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_forward_mode_AD_scatter_add_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_forward_mode_AD_scatter_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_forward_mode_AD_searchsorted_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_forward_mode_AD_select_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_forward_mode_AD_select_scatter_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_forward_mode_AD_sgn_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_forward_mode_AD_sigmoid_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_forward_mode_AD_signal_windows_blackman_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_forward_mode_AD_signal_windows_cosine_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_forward_mode_AD_signal_windows_general_hamming_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_forward_mode_AD_signal_windows_hamming_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_forward_mode_AD_signbit_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_forward_mode_AD_sin_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_forward_mode_AD_slice_scatter_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_forward_mode_AD_softmax_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_forward_mode_AD_softmax_with_dtype_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_forward_mode_AD_sort_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_forward_mode_AD_sparse_mm_reduce_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_forward_mode_AD_sparse_sampled_addmm_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_forward_mode_AD_sparse_sampled_addmm_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_forward_mode_AD_special_bessel_j0_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_forward_mode_AD_special_chebyshev_polynomial_v_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_forward_mode_AD_special_erfcx_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_forward_mode_AD_special_hermite_polynomial_he_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_forward_mode_AD_special_i1_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_forward_mode_AD_special_i1e_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_forward_mode_AD_special_legendre_polynomial_p_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_forward_mode_AD_special_modified_bessel_i0_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_forward_mode_AD_special_modified_bessel_k1_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_forward_mode_AD_special_ndtr_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_forward_mode_AD_special_polygamma_special_polygamma_n_0_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_forward_mode_AD_special_scaled_modified_bessel_k0_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_forward_mode_AD_special_shifted_chebyshev_polynomial_t_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_forward_mode_AD_special_shifted_chebyshev_polynomial_w_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_forward_mode_AD_special_xlog1py_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_forward_mode_AD_special_zeta_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_forward_mode_AD_split_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_forward_mode_AD_split_list_args_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_forward_mode_AD_split_with_sizes_copy_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_forward_mode_AD_split_with_sizes_copy_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_forward_mode_AD_split_with_sizes_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_forward_mode_AD_sqrt_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_forward_mode_AD_square_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_forward_mode_AD_squeeze_copy_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_forward_mode_AD_squeeze_copy_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_forward_mode_AD_squeeze_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_forward_mode_AD_squeeze_multiple_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_forward_mode_AD_stack_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_forward_mode_AD_stack_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_forward_mode_AD_std_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_forward_mode_AD_std_mean_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_forward_mode_AD_std_mean_unbiased_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_forward_mode_AD_std_unbiased_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_forward_mode_AD_stft_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_forward_mode_AD_sub_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_forward_mode_AD_sum_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_forward_mode_AD_sum_to_size_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_forward_mode_AD_sum_to_size_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_forward_mode_AD_svd_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_forward_mode_AD_svd_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_forward_mode_AD_svd_lowrank_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_forward_mode_AD_svd_lowrank_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_forward_mode_AD_t_copy_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_forward_mode_AD_tan_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_forward_mode_AD_tanh_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_forward_mode_AD_tensor_split_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_forward_mode_AD_tensordot_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_forward_mode_AD_tensordot_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_forward_mode_AD_tile_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_forward_mode_AD_to_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_forward_mode_AD_torch_ops_aten__safe_softmax_default_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_forward_mode_AD_transpose_copy_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_forward_mode_AD_transpose_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_forward_mode_AD_trapezoid_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_forward_mode_AD_trapz_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_forward_mode_AD_triangular_solve_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_forward_mode_AD_tril_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_forward_mode_AD_true_divide_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_forward_mode_AD_true_divide_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_forward_mode_AD_unbind_copy_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_forward_mode_AD_unbind_copy_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_forward_mode_AD_unbind_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_forward_mode_AD_unflatten_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_forward_mode_AD_unfold_copy_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_forward_mode_AD_uniform_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_forward_mode_AD_unique_consecutive_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_forward_mode_AD_unsafe_chunk_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_forward_mode_AD_unsafe_chunk_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_forward_mode_AD_unsafe_split_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_forward_mode_AD_unsqueeze_copy_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_forward_mode_AD_unsqueeze_copy_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_forward_mode_AD_unsqueeze_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_forward_mode_AD_var_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_forward_mode_AD_var_mean_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_forward_mode_AD_var_mean_unbiased_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_forward_mode_AD_vdot_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_forward_mode_AD_vdot_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_forward_mode_AD_view_as_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_forward_mode_AD_view_copy_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_forward_mode_AD_view_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_forward_mode_AD_zero__cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_forward_mode_AD_zeros_like_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_inplace_forward_mode_AD___getitem___cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_inplace_forward_mode_AD___radd___cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_inplace_forward_mode_AD___rdiv___cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_inplace_forward_mode_AD___rmod___cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_inplace_forward_mode_AD___rmul___cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_inplace_forward_mode_AD___rmul___cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_inplace_forward_mode_AD___rpow___cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_inplace_forward_mode_AD__batch_norm_with_update_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_inplace_forward_mode_AD__chunk_cat_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_inplace_forward_mode_AD__chunk_cat_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_inplace_forward_mode_AD__native_batch_norm_legit_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_inplace_forward_mode_AD__segment_reduce_offsets_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_inplace_forward_mode_AD__softmax_backward_data_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_inplace_forward_mode_AD__unsafe_masked_index_put_accumulate_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_inplace_forward_mode_AD__unsafe_masked_index_put_accumulate_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_inplace_forward_mode_AD__upsample_bilinear2d_aa_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_inplace_forward_mode_AD_acos_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_inplace_forward_mode_AD_add_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_inplace_forward_mode_AD_add_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_inplace_forward_mode_AD_addcdiv_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_inplace_forward_mode_AD_addcdiv_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_inplace_forward_mode_AD_addcmul_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_inplace_forward_mode_AD_addmm_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_inplace_forward_mode_AD_addmv_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_inplace_forward_mode_AD_addmv_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_inplace_forward_mode_AD_addr_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_inplace_forward_mode_AD_alias_copy_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_inplace_forward_mode_AD_alias_copy_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_inplace_forward_mode_AD_all_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_inplace_forward_mode_AD_all_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_inplace_forward_mode_AD_allclose_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_inplace_forward_mode_AD_amin_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_inplace_forward_mode_AD_angle_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_inplace_forward_mode_AD_any_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_inplace_forward_mode_AD_arange_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_inplace_forward_mode_AD_argwhere_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_inplace_forward_mode_AD_argwhere_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_inplace_forward_mode_AD_as_strided_partial_views_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_inplace_forward_mode_AD_as_strided_partial_views_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_inplace_forward_mode_AD_as_strided_scatter_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_inplace_forward_mode_AD_asin_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_inplace_forward_mode_AD_asin_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_inplace_forward_mode_AD_asinh_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_inplace_forward_mode_AD_atan2_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_inplace_forward_mode_AD_atan_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_inplace_forward_mode_AD_atan_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_inplace_forward_mode_AD_atanh_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_inplace_forward_mode_AD_atleast_1d_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_inplace_forward_mode_AD_atleast_1d_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_inplace_forward_mode_AD_atleast_3d_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_inplace_forward_mode_AD_baddbmm_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_inplace_forward_mode_AD_bfloat16_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_inplace_forward_mode_AD_bfloat16_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_inplace_forward_mode_AD_block_diag_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_inplace_forward_mode_AD_bool_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_inplace_forward_mode_AD_broadcast_tensors_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_inplace_forward_mode_AD_broadcast_tensors_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_inplace_forward_mode_AD_broadcast_to_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_inplace_forward_mode_AD_cartesian_prod_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_inplace_forward_mode_AD_cartesian_prod_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_inplace_forward_mode_AD_cdist_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_inplace_forward_mode_AD_cdouble_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_inplace_forward_mode_AD_ceil_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_inplace_forward_mode_AD_cfloat_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_inplace_forward_mode_AD_chalf_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_inplace_forward_mode_AD_cholesky_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_inplace_forward_mode_AD_cholesky_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_inplace_forward_mode_AD_cholesky_inverse_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_inplace_forward_mode_AD_cholesky_solve_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_inplace_forward_mode_AD_chunk_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_inplace_forward_mode_AD_clamp_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_inplace_forward_mode_AD_clamp_max_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_inplace_forward_mode_AD_clamp_min_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_inplace_forward_mode_AD_clone_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_inplace_forward_mode_AD_column_stack_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_inplace_forward_mode_AD_column_stack_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_inplace_forward_mode_AD_combinations_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_inplace_forward_mode_AD_constant_pad_nd_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_inplace_forward_mode_AD_constant_pad_nd_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_inplace_forward_mode_AD_contiguous_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_inplace_forward_mode_AD_corrcoef_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_inplace_forward_mode_AD_corrcoef_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_inplace_forward_mode_AD_cos_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_inplace_forward_mode_AD_cos_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_inplace_forward_mode_AD_cosh_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_inplace_forward_mode_AD_count_nonzero_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_inplace_forward_mode_AD_cov_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_inplace_forward_mode_AD_cross_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_inplace_forward_mode_AD_cummax_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_inplace_forward_mode_AD_cummin_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_inplace_forward_mode_AD_cumprod_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_inplace_forward_mode_AD_cumsum_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_inplace_forward_mode_AD_diag_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_inplace_forward_mode_AD_diag_embed_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_inplace_forward_mode_AD_diagflat_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_inplace_forward_mode_AD_diagonal_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_inplace_forward_mode_AD_diagonal_scatter_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_inplace_forward_mode_AD_diff_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_inplace_forward_mode_AD_diff_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_inplace_forward_mode_AD_dist_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_inplace_forward_mode_AD_div_floor_rounding_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_inplace_forward_mode_AD_dot_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_inplace_forward_mode_AD_dstack_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_inplace_forward_mode_AD_dstack_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_inplace_forward_mode_AD_empty_like_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_inplace_forward_mode_AD_empty_strided_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_inplace_forward_mode_AD_empty_strided_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_inplace_forward_mode_AD_eq_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_inplace_forward_mode_AD_equal_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_inplace_forward_mode_AD_equal_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_inplace_forward_mode_AD_erf_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_inplace_forward_mode_AD_erfc_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_inplace_forward_mode_AD_exp2_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_inplace_forward_mode_AD_exp_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_inplace_forward_mode_AD_expand_as_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_inplace_forward_mode_AD_expand_as_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_inplace_forward_mode_AD_expand_copy_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_inplace_forward_mode_AD_expand_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_inplace_forward_mode_AD_exponential_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_inplace_forward_mode_AD_fft_fft2_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_inplace_forward_mode_AD_fft_fft_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_inplace_forward_mode_AD_fft_fftn_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_inplace_forward_mode_AD_fft_fftshift_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_inplace_forward_mode_AD_fft_hfft2_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_inplace_forward_mode_AD_fft_hfft2_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_inplace_forward_mode_AD_fft_hfft_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_inplace_forward_mode_AD_fft_hfft_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_inplace_forward_mode_AD_fft_hfftn_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_inplace_forward_mode_AD_fft_ifft_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_inplace_forward_mode_AD_fft_ifftshift_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_inplace_forward_mode_AD_fft_ifftshift_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_inplace_forward_mode_AD_fft_ihfft2_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_inplace_forward_mode_AD_fft_ihfft_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_inplace_forward_mode_AD_fft_irfft2_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_inplace_forward_mode_AD_fft_irfft_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_inplace_forward_mode_AD_fft_irfftn_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_inplace_forward_mode_AD_fft_irfftn_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_inplace_forward_mode_AD_fft_rfft_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_inplace_forward_mode_AD_fill_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_inplace_forward_mode_AD_flatten_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_inplace_forward_mode_AD_flip_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_inplace_forward_mode_AD_fliplr_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_inplace_forward_mode_AD_flipud_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_inplace_forward_mode_AD_float_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_inplace_forward_mode_AD_float_power_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_inplace_forward_mode_AD_float_power_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_inplace_forward_mode_AD_floor_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_inplace_forward_mode_AD_floor_divide_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_inplace_forward_mode_AD_fmod_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_inplace_forward_mode_AD_frexp_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_inplace_forward_mode_AD_full_like_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_inplace_forward_mode_AD_full_like_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_inplace_forward_mode_AD_gather_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_inplace_forward_mode_AD_gather_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_inplace_forward_mode_AD_geometric_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_inplace_forward_mode_AD_geqrf_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_inplace_forward_mode_AD_gradient_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_inplace_forward_mode_AD_grid_sampler_2d_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_inplace_forward_mode_AD_grid_sampler_3d_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_inplace_forward_mode_AD_hsplit_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_inplace_forward_mode_AD_igamma_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_inplace_forward_mode_AD_imag_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_inplace_forward_mode_AD_index_add_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_inplace_forward_mode_AD_index_add_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_inplace_forward_mode_AD_index_reduce_amax_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_inplace_forward_mode_AD_index_reduce_mean_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_inplace_forward_mode_AD_index_select_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_inplace_forward_mode_AD_inner_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_inplace_forward_mode_AD_inner_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_inplace_forward_mode_AD_int_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_inplace_forward_mode_AD_int_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_inplace_forward_mode_AD_isclose_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_inplace_forward_mode_AD_isfinite_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_inplace_forward_mode_AD_isfinite_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_inplace_forward_mode_AD_isinf_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_inplace_forward_mode_AD_isnan_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_inplace_forward_mode_AD_isneginf_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_inplace_forward_mode_AD_isreal_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_inplace_forward_mode_AD_isreal_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_inplace_forward_mode_AD_istft_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_inplace_forward_mode_AD_item_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_inplace_forward_mode_AD_jiterator_2inputs_2outputs_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_inplace_forward_mode_AD_jiterator_2inputs_2outputs_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_inplace_forward_mode_AD_jiterator_4inputs_with_extra_args_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_inplace_forward_mode_AD_jiterator_binary_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_inplace_forward_mode_AD_jiterator_binary_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_inplace_forward_mode_AD_jiterator_binary_return_by_ref_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_inplace_forward_mode_AD_jiterator_unary_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_inplace_forward_mode_AD_jiterator_unary_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_inplace_forward_mode_AD_kron_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_inplace_forward_mode_AD_kron_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_inplace_forward_mode_AD_kthvalue_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_inplace_forward_mode_AD_ldexp_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_inplace_forward_mode_AD_linalg_cholesky_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_inplace_forward_mode_AD_linalg_cross_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_inplace_forward_mode_AD_linalg_det_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_inplace_forward_mode_AD_linalg_eig_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_inplace_forward_mode_AD_linalg_eig_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_inplace_forward_mode_AD_linalg_eigh_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_inplace_forward_mode_AD_linalg_eigh_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_inplace_forward_mode_AD_linalg_eigvals_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_inplace_forward_mode_AD_linalg_eigvalsh_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_inplace_forward_mode_AD_linalg_eigvalsh_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_inplace_forward_mode_AD_linalg_inv_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_inplace_forward_mode_AD_linalg_inv_ex_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_inplace_forward_mode_AD_linalg_ldl_factor_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_inplace_forward_mode_AD_linalg_ldl_solve_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_inplace_forward_mode_AD_linalg_lstsq_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_inplace_forward_mode_AD_linalg_lu_factor_ex_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_inplace_forward_mode_AD_linalg_lu_solve_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_inplace_forward_mode_AD_linalg_matrix_norm_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_inplace_forward_mode_AD_linalg_matrix_rank_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_inplace_forward_mode_AD_linalg_multi_dot_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_inplace_forward_mode_AD_linalg_norm_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_inplace_forward_mode_AD_linalg_pinv_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_inplace_forward_mode_AD_linalg_pinv_hermitian_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_inplace_forward_mode_AD_linalg_pinv_hermitian_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_inplace_forward_mode_AD_linalg_pinv_singular_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_inplace_forward_mode_AD_linalg_qr_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_inplace_forward_mode_AD_linalg_qr_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_inplace_forward_mode_AD_linalg_solve_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_inplace_forward_mode_AD_linalg_solve_ex_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_inplace_forward_mode_AD_linalg_solve_ex_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_inplace_forward_mode_AD_linalg_svdvals_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_inplace_forward_mode_AD_linalg_svdvals_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_inplace_forward_mode_AD_linalg_tensorsolve_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_inplace_forward_mode_AD_linalg_vander_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_inplace_forward_mode_AD_linalg_vecdot_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_inplace_forward_mode_AD_linalg_vector_norm_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_inplace_forward_mode_AD_linspace_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_inplace_forward_mode_AD_linspace_tensor_overload_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_inplace_forward_mode_AD_linspace_tensor_overload_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_inplace_forward_mode_AD_log10_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_inplace_forward_mode_AD_log1p_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_inplace_forward_mode_AD_log1p_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_inplace_forward_mode_AD_log2_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_inplace_forward_mode_AD_log_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_inplace_forward_mode_AD_log_softmax_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_inplace_forward_mode_AD_logaddexp2_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_inplace_forward_mode_AD_logaddexp_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_inplace_forward_mode_AD_logdet_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_inplace_forward_mode_AD_logdet_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_inplace_forward_mode_AD_logical_and_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_inplace_forward_mode_AD_logical_not_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_inplace_forward_mode_AD_logical_not_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_inplace_forward_mode_AD_logical_or_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_inplace_forward_mode_AD_logical_or_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_inplace_forward_mode_AD_logical_xor_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_inplace_forward_mode_AD_logical_xor_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_inplace_forward_mode_AD_logit_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_inplace_forward_mode_AD_long_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_inplace_forward_mode_AD_long_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_inplace_forward_mode_AD_lt_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_inplace_forward_mode_AD_lu_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_inplace_forward_mode_AD_lu_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_inplace_forward_mode_AD_lu_solve_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_inplace_forward_mode_AD_lu_solve_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_inplace_forward_mode_AD_lu_unpack_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_inplace_forward_mode_AD_mH_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_inplace_forward_mode_AD_mT_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_inplace_forward_mode_AD_masked_amin_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_inplace_forward_mode_AD_masked_argmax_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_inplace_forward_mode_AD_masked_cumprod_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_inplace_forward_mode_AD_masked_cumprod_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_inplace_forward_mode_AD_masked_fill_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_inplace_forward_mode_AD_masked_fill_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_inplace_forward_mode_AD_masked_log_softmax_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_inplace_forward_mode_AD_masked_logaddexp_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_inplace_forward_mode_AD_masked_logsumexp_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_inplace_forward_mode_AD_masked_mean_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_inplace_forward_mode_AD_masked_normalize_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_inplace_forward_mode_AD_masked_prod_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_inplace_forward_mode_AD_masked_scatter_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_inplace_forward_mode_AD_masked_scatter_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_inplace_forward_mode_AD_masked_select_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_inplace_forward_mode_AD_masked_select_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_inplace_forward_mode_AD_masked_softmax_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_inplace_forward_mode_AD_masked_std_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_inplace_forward_mode_AD_masked_sum_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_inplace_forward_mode_AD_masked_var_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_inplace_forward_mode_AD_matmul_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_inplace_forward_mode_AD_max_binary_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_inplace_forward_mode_AD_max_pool2d_with_indices_backward_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_inplace_forward_mode_AD_max_reduction_no_dim_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_inplace_forward_mode_AD_max_reduction_with_dim_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_inplace_forward_mode_AD_mean_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_inplace_forward_mode_AD_median_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_inplace_forward_mode_AD_meshgrid_list_of_tensors_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_inplace_forward_mode_AD_meshgrid_list_of_tensors_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_inplace_forward_mode_AD_meshgrid_variadic_tensors_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_inplace_forward_mode_AD_mode_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_inplace_forward_mode_AD_movedim_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_inplace_forward_mode_AD_mul_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_inplace_forward_mode_AD_multinomial_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_inplace_forward_mode_AD_mv_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_inplace_forward_mode_AD_mv_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_inplace_forward_mode_AD_nansum_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_inplace_forward_mode_AD_nansum_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_inplace_forward_mode_AD_narrow_copy_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_inplace_forward_mode_AD_narrow_copy_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_inplace_forward_mode_AD_narrow_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_inplace_forward_mode_AD_narrow_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_inplace_forward_mode_AD_native_dropout_backward_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_inplace_forward_mode_AD_ne_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_inplace_forward_mode_AD_neg_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_inplace_forward_mode_AD_neg_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_inplace_forward_mode_AD_new_empty_strided_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_inplace_forward_mode_AD_new_empty_strided_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_inplace_forward_mode_AD_new_full_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_inplace_forward_mode_AD_new_zeros_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_inplace_forward_mode_AD_nextafter_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_inplace_forward_mode_AD_nn_functional_adaptive_avg_pool1d_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_inplace_forward_mode_AD_nn_functional_adaptive_max_pool1d_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_inplace_forward_mode_AD_nn_functional_adaptive_max_pool3d_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_inplace_forward_mode_AD_nn_functional_avg_pool2d_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_inplace_forward_mode_AD_nn_functional_bilinear_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_inplace_forward_mode_AD_nn_functional_binary_cross_entropy_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_inplace_forward_mode_AD_nn_functional_binary_cross_entropy_with_logits_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_inplace_forward_mode_AD_nn_functional_celu_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_inplace_forward_mode_AD_nn_functional_conv1d_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_inplace_forward_mode_AD_nn_functional_conv1d_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_inplace_forward_mode_AD_nn_functional_conv2d_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_inplace_forward_mode_AD_nn_functional_conv_transpose1d_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_inplace_forward_mode_AD_nn_functional_conv_transpose3d_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_inplace_forward_mode_AD_nn_functional_cosine_embedding_loss_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_inplace_forward_mode_AD_nn_functional_cosine_similarity_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_inplace_forward_mode_AD_nn_functional_cross_entropy_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_inplace_forward_mode_AD_nn_functional_dropout3d_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_inplace_forward_mode_AD_nn_functional_embedding_bag_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_inplace_forward_mode_AD_nn_functional_feature_alpha_dropout_with_train_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_inplace_forward_mode_AD_nn_functional_feature_alpha_dropout_without_train_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_inplace_forward_mode_AD_nn_functional_fractional_max_pool2d_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_inplace_forward_mode_AD_nn_functional_gaussian_nll_loss_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_inplace_forward_mode_AD_nn_functional_hardshrink_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_inplace_forward_mode_AD_nn_functional_hardsigmoid_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_inplace_forward_mode_AD_nn_functional_hardswish_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_inplace_forward_mode_AD_nn_functional_hardtanh_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_inplace_forward_mode_AD_nn_functional_hinge_embedding_loss_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_inplace_forward_mode_AD_nn_functional_interpolate_area_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_inplace_forward_mode_AD_nn_functional_interpolate_bicubic_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_inplace_forward_mode_AD_nn_functional_interpolate_nearest-exact_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_inplace_forward_mode_AD_nn_functional_l1_loss_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_inplace_forward_mode_AD_nn_functional_layer_norm_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_inplace_forward_mode_AD_nn_functional_linear_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_inplace_forward_mode_AD_nn_functional_local_response_norm_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_inplace_forward_mode_AD_nn_functional_margin_ranking_loss_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_inplace_forward_mode_AD_nn_functional_max_pool1d_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_inplace_forward_mode_AD_nn_functional_max_pool3d_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_inplace_forward_mode_AD_nn_functional_max_unpool1d_grad_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_inplace_forward_mode_AD_nn_functional_max_unpool3d_grad_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_inplace_forward_mode_AD_nn_functional_mish_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_inplace_forward_mode_AD_nn_functional_mse_loss_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_inplace_forward_mode_AD_nn_functional_multi_head_attention_forward_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_inplace_forward_mode_AD_nn_functional_multi_margin_loss_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_inplace_forward_mode_AD_nn_functional_normalize_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_inplace_forward_mode_AD_nn_functional_pad_circular_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_inplace_forward_mode_AD_nn_functional_pad_circular_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_inplace_forward_mode_AD_nn_functional_pad_constant_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_inplace_forward_mode_AD_nn_functional_pad_reflect_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_inplace_forward_mode_AD_nn_functional_pad_reflect_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_inplace_forward_mode_AD_nn_functional_pad_replicate_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_inplace_forward_mode_AD_nn_functional_pad_replicate_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_inplace_forward_mode_AD_nn_functional_pairwise_distance_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_inplace_forward_mode_AD_nn_functional_pixel_shuffle_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_inplace_forward_mode_AD_nn_functional_pixel_unshuffle_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_inplace_forward_mode_AD_nn_functional_poisson_nll_loss_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_inplace_forward_mode_AD_nn_functional_rms_norm_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_inplace_forward_mode_AD_nn_functional_selu_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_inplace_forward_mode_AD_nn_functional_silu_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_inplace_forward_mode_AD_nn_functional_soft_margin_loss_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_inplace_forward_mode_AD_nn_functional_softmin_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_inplace_forward_mode_AD_nn_functional_softmin_with_dtype_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_inplace_forward_mode_AD_nn_functional_softplus_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_inplace_forward_mode_AD_nn_functional_softsign_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_inplace_forward_mode_AD_nn_functional_tanhshrink_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_inplace_forward_mode_AD_nn_functional_triplet_margin_loss_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_inplace_forward_mode_AD_nn_functional_triplet_margin_loss_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_inplace_forward_mode_AD_nn_functional_triplet_margin_with_distance_loss_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_inplace_forward_mode_AD_nn_functional_triplet_margin_with_distance_loss_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_inplace_forward_mode_AD_nn_functional_unfold_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_inplace_forward_mode_AD_nn_functional_upsample_bilinear_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_inplace_forward_mode_AD_nn_functional_upsample_nearest_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_inplace_forward_mode_AD_nonzero_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_inplace_forward_mode_AD_nonzero_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_inplace_forward_mode_AD_nonzero_static_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_inplace_forward_mode_AD_norm_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_inplace_forward_mode_AD_norm_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_inplace_forward_mode_AD_norm_fro_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_inplace_forward_mode_AD_norm_inf_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_inplace_forward_mode_AD_norm_nuc_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_inplace_forward_mode_AD_norm_nuc_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_inplace_forward_mode_AD_normal_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_inplace_forward_mode_AD_normal_in_place_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_inplace_forward_mode_AD_normal_in_place_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_inplace_forward_mode_AD_ones_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_inplace_forward_mode_AD_ones_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_inplace_forward_mode_AD_ormqr_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_inplace_forward_mode_AD_ormqr_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_inplace_forward_mode_AD_outer_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_inplace_forward_mode_AD_pca_lowrank_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_inplace_forward_mode_AD_permute_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_inplace_forward_mode_AD_pinverse_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_inplace_forward_mode_AD_polygamma_polygamma_n_1_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_inplace_forward_mode_AD_polygamma_polygamma_n_2_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_inplace_forward_mode_AD_polygamma_polygamma_n_4_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_inplace_forward_mode_AD_pow_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_inplace_forward_mode_AD_prod_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_inplace_forward_mode_AD_prod_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_inplace_forward_mode_AD_put_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_inplace_forward_mode_AD_qr_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_inplace_forward_mode_AD_rad2deg_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_inplace_forward_mode_AD_rand_like_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_inplace_forward_mode_AD_randn_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_inplace_forward_mode_AD_randn_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_inplace_forward_mode_AD_randn_like_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_inplace_forward_mode_AD_real_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_inplace_forward_mode_AD_real_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_inplace_forward_mode_AD_reciprocal_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_inplace_forward_mode_AD_reciprocal_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_inplace_forward_mode_AD_renorm_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_inplace_forward_mode_AD_repeat_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_inplace_forward_mode_AD_resize_as__cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_inplace_forward_mode_AD_round_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_inplace_forward_mode_AD_round_decimals_neg_3_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_inplace_forward_mode_AD_rsqrt_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_inplace_forward_mode_AD_rsub_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_inplace_forward_mode_AD_scalar_tensor_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_inplace_forward_mode_AD_scalar_tensor_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_inplace_forward_mode_AD_scatter_add_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_inplace_forward_mode_AD_scatter_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_inplace_forward_mode_AD_scatter_reduce_amax_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_inplace_forward_mode_AD_scatter_reduce_amin_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_inplace_forward_mode_AD_searchsorted_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_inplace_forward_mode_AD_select_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_inplace_forward_mode_AD_select_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_inplace_forward_mode_AD_select_scatter_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_inplace_forward_mode_AD_sgn_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_inplace_forward_mode_AD_sigmoid_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_inplace_forward_mode_AD_signal_windows_cosine_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_inplace_forward_mode_AD_signal_windows_exponential_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_inplace_forward_mode_AD_signal_windows_general_hamming_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_inplace_forward_mode_AD_signal_windows_hann_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_inplace_forward_mode_AD_signal_windows_kaiser_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_inplace_forward_mode_AD_signbit_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_inplace_forward_mode_AD_sinc_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_inplace_forward_mode_AD_sinh_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_inplace_forward_mode_AD_sinh_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_inplace_forward_mode_AD_softmax_with_dtype_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_inplace_forward_mode_AD_sort_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_inplace_forward_mode_AD_sparse_mm_reduce_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_inplace_forward_mode_AD_sparse_sampled_addmm_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_inplace_forward_mode_AD_sparse_sampled_addmm_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_inplace_forward_mode_AD_special_airy_ai_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_inplace_forward_mode_AD_special_bessel_j1_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_inplace_forward_mode_AD_special_bessel_y0_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_inplace_forward_mode_AD_special_chebyshev_polynomial_t_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_inplace_forward_mode_AD_special_chebyshev_polynomial_w_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_inplace_forward_mode_AD_special_hermite_polynomial_he_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_inplace_forward_mode_AD_special_i0e_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_inplace_forward_mode_AD_special_i1e_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_inplace_forward_mode_AD_special_laguerre_polynomial_l_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_inplace_forward_mode_AD_special_log_ndtr_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_inplace_forward_mode_AD_special_modified_bessel_i0_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_inplace_forward_mode_AD_special_modified_bessel_k1_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_inplace_forward_mode_AD_special_ndtr_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_inplace_forward_mode_AD_special_scaled_modified_bessel_k0_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_inplace_forward_mode_AD_special_shifted_chebyshev_polynomial_t_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_inplace_forward_mode_AD_special_shifted_chebyshev_polynomial_u_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_inplace_forward_mode_AD_special_shifted_chebyshev_polynomial_v_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_inplace_forward_mode_AD_special_shifted_chebyshev_polynomial_w_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_inplace_forward_mode_AD_special_spherical_bessel_j0_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_inplace_forward_mode_AD_special_xlog1py_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_inplace_forward_mode_AD_special_zeta_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_inplace_forward_mode_AD_split_with_sizes_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_inplace_forward_mode_AD_split_with_sizes_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_inplace_forward_mode_AD_sqrt_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_inplace_forward_mode_AD_square_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_inplace_forward_mode_AD_square_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_inplace_forward_mode_AD_squeeze_copy_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_inplace_forward_mode_AD_squeeze_multiple_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_inplace_forward_mode_AD_stack_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_inplace_forward_mode_AD_std_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_inplace_forward_mode_AD_std_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_inplace_forward_mode_AD_std_mean_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_inplace_forward_mode_AD_std_mean_unbiased_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_inplace_forward_mode_AD_std_mean_unbiased_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_inplace_forward_mode_AD_sub_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_inplace_forward_mode_AD_sum_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_inplace_forward_mode_AD_sum_to_size_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_inplace_forward_mode_AD_sum_to_size_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_inplace_forward_mode_AD_t_copy_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_inplace_forward_mode_AD_t_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_inplace_forward_mode_AD_take_along_dim_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_inplace_forward_mode_AD_take_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_inplace_forward_mode_AD_take_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_inplace_forward_mode_AD_tanh_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_inplace_forward_mode_AD_tanh_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_inplace_forward_mode_AD_tensor_split_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_inplace_forward_mode_AD_tensor_split_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_inplace_forward_mode_AD_tile_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_inplace_forward_mode_AD_to_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_inplace_forward_mode_AD_to_sparse_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_inplace_forward_mode_AD_trace_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_inplace_forward_mode_AD_trapezoid_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_inplace_forward_mode_AD_trapezoid_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_inplace_forward_mode_AD_trapz_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_inplace_forward_mode_AD_triangular_solve_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_inplace_forward_mode_AD_tril_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_inplace_forward_mode_AD_triu_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_inplace_forward_mode_AD_triu_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_inplace_forward_mode_AD_true_divide_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_inplace_forward_mode_AD_unbind_copy_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_inplace_forward_mode_AD_unbind_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_inplace_forward_mode_AD_unbind_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_inplace_forward_mode_AD_unflatten_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_inplace_forward_mode_AD_uniform_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_inplace_forward_mode_AD_uniform_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_inplace_forward_mode_AD_unique_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_inplace_forward_mode_AD_unsqueeze_copy_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_inplace_forward_mode_AD_unsqueeze_copy_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_inplace_forward_mode_AD_unsqueeze_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_inplace_forward_mode_AD_var_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_inplace_forward_mode_AD_var_mean_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_inplace_forward_mode_AD_var_unbiased_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_inplace_forward_mode_AD_var_unbiased_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_inplace_forward_mode_AD_vdot_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_inplace_forward_mode_AD_view_as_complex_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_inplace_forward_mode_AD_view_as_real_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_inplace_forward_mode_AD_view_copy_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_inplace_forward_mode_AD_view_copy_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_inplace_forward_mode_AD_view_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_inplace_forward_mode_AD_view_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_inplace_forward_mode_AD_vsplit_cuda_float64, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_inplace_forward_mode_AD_where_cuda_complex128, test/test_ops_fwd_gradients.py::TestFwdGradientsCUDA::test_inplace_forward_mode_AD_zeros_like_cuda_float64 2025-12-04T12:02:38.3539053Z 2025-12-04T12:02:38.3539293Z Finished test_ops_fwd_gradients 2/2 ... [2025-12-04 12:02:38.264407][8606.206623578], took 8.31min 2025-12-04T12:02:38.3540036Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/test_ops_fwd_gradients/test_ops_fwd_gradients-308b20ea781f4444.xml 2025-12-04T12:02:38.3964968Z Running test_ops_jit 2/2 ... [2025-12-04 12:02:38.396239][8606.338457358] 2025-12-04T12:02:38.3965375Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2025-12-04T12:02:38.3968059Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'test_ops_jit.py', '--shard-id=2', '--num-shards=2', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-12-04 12:02:38.396539] 2025-12-04T12:12:39.5376602Z 2025-12-04T12:12:39.5377449Z test_ops_jit 2/2 was successful, full logs can be found in artifacts with path test/test-reports/test_ops_jit_2.2_10156177962e62ad_.log 2025-12-04T12:12:39.5647163Z Running 594 items in this shard: test/test_ops_jit.py::TestJitCUDA::test_jit_alias_remapping_acos_cuda_float32, test/test_ops_jit.py::TestJitCUDA::test_jit_alias_remapping_asinh_cuda_float32, test/test_ops_jit.py::TestJitCUDA::test_jit_alias_remapping_atan_cuda_float32, test/test_ops_jit.py::TestJitCUDA::test_jit_alias_remapping_atanh_cuda_float32, test/test_ops_jit.py::TestJitCUDA::test_jit_alias_remapping_div_floor_rounding_cuda_float32, test/test_ops_jit.py::TestJitCUDA::test_jit_alias_remapping_erf_cuda_float32, test/test_ops_jit.py::TestJitCUDA::test_jit_alias_remapping_erfc_cuda_float32, test/test_ops_jit.py::TestJitCUDA::test_jit_alias_remapping_exp2_cuda_float32, test/test_ops_jit.py::TestJitCUDA::test_jit_alias_remapping_expm1_cuda_float32, test/test_ops_jit.py::TestJitCUDA::test_jit_alias_remapping_ge_cuda_float32, test/test_ops_jit.py::TestJitCUDA::test_jit_alias_remapping_gt_cuda_float32, test/test_ops_jit.py::TestJitCUDA::test_jit_alias_remapping_igammac_cuda_float32, test/test_ops_jit.py::TestJitCUDA::test_jit_alias_remapping_linalg_det_cuda_float32, test/test_ops_jit.py::TestJitCUDA::test_jit_alias_remapping_linalg_inv_cuda_float32, test/test_ops_jit.py::TestJitCUDA::test_jit_alias_remapping_log_softmax_with_dtype_cuda_float32, test/test_ops_jit.py::TestJitCUDA::test_jit_alias_remapping_logit_cuda_float32, test/test_ops_jit.py::TestJitCUDA::test_jit_alias_remapping_logsumexp_cuda_float32, test/test_ops_jit.py::TestJitCUDA::test_jit_alias_remapping_lt_cuda_float32, test/test_ops_jit.py::TestJitCUDA::test_jit_alias_remapping_mH_cuda_float32, test/test_ops_jit.py::TestJitCUDA::test_jit_alias_remapping_matrix_exp_cuda_float32, test/test_ops_jit.py::TestJitCUDA::test_jit_alias_remapping_max_binary_cuda_float32, test/test_ops_jit.py::TestJitCUDA::test_jit_alias_remapping_mvlgamma_mvlgamma_p_3_cuda_float32, test/test_ops_jit.py::TestJitCUDA::test_jit_alias_remapping_neg_cuda_float32, test/test_ops_jit.py::TestJitCUDA::test_jit_alias_remapping_nn_functional_conv2d_cuda_float32, test/test_ops_jit.py::TestJitCUDA::test_jit_alias_remapping_nn_functional_conv3d_cuda_float32, test/test_ops_jit.py::TestJitCUDA::test_jit_alias_remapping_outer_cuda_float32, test/test_ops_jit.py::TestJitCUDA::test_jit_alias_remapping_round_cuda_float32, test/test_ops_jit.py::TestJitCUDA::test_jit_alias_remapping_round_decimals_0_cuda_float32, test/test_ops_jit.py::TestJitCUDA::test_jit_alias_remapping_round_decimals_neg_3_cuda_float32, test/test_ops_jit.py::TestJitCUDA::test_jit_alias_remapping_transpose_cuda_float32, test/test_ops_jit.py::TestJitCUDA::test_jit_alias_remapping_trunc_cuda_float32, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit_H_cuda_float32, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit_T_cuda_complex64, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit_T_cuda_float32, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit___getitem___cuda_float32, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit___radd___cuda_complex64, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit___radd___cuda_float32, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit___rdiv___cuda_float32, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit___rmod___cuda_float32, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit___rmul___cuda_complex64, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit___rmul___cuda_float32, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit___rsub___cuda_complex64, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit___rsub___cuda_float32, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit__batch_norm_with_update_cuda_float32, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit__chunk_cat_cuda_complex64, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit__native_batch_norm_legit_cuda_float32, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit__segment_reduce_lengths_cuda_float32, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit__segment_reduce_offsets_cuda_float32, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit__softmax_backward_data_cuda_float32, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit__unsafe_masked_index_cuda_complex64, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit__unsafe_masked_index_cuda_float32, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit__unsafe_masked_index_put_accumulate_cuda_float32, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit__upsample_bilinear2d_aa_cuda_float32, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit_abs_cuda_complex64, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit_abs_cuda_float32, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit_acos_cuda_complex64, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit_acos_cuda_float32, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit_add_cuda_complex64, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit_addbmm_cuda_float32, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit_addcmul_cuda_float32, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit_addmm_cuda_float32, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit_addmm_decomposed_cuda_complex64, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit_addmv_cuda_complex64, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit_addr_cuda_complex64, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit_addr_cuda_float32, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit_alias_copy_cuda_complex64, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit_alias_copy_cuda_float32, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit_all_cuda_complex64, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit_all_cuda_float32, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit_allclose_cuda_complex64, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit_allclose_cuda_float32, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit_amin_cuda_float32, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit_angle_cuda_complex64, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit_any_cuda_complex64, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit_any_cuda_float32, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit_arange_cuda_float32, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit_argmax_cuda_float32, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit_as_strided_copy_cuda_float32, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit_as_strided_cuda_complex64, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit_as_strided_cuda_float32, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit_as_strided_partial_views_cuda_complex64, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit_as_strided_partial_views_cuda_float32, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit_as_strided_scatter_cuda_complex64, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit_as_strided_scatter_cuda_float32, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit_asinh_cuda_complex64, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit_atan_cuda_complex64, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit_atanh_cuda_complex64, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit_atanh_cuda_float32, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit_atleast_1d_cuda_complex64, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit_atleast_1d_cuda_float32, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit_atleast_2d_cuda_float32, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit_bfloat16_cuda_complex64, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit_bfloat16_cuda_float32, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit_block_diag_cuda_float32, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit_bmm_cuda_complex64, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit_bmm_cuda_float32, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit_bool_cuda_float32, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit_broadcast_tensors_cuda_complex64, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit_broadcast_to_cuda_float32, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit_byte_cuda_float32, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit_cartesian_prod_cuda_complex64, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit_cdist_cuda_float32, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit_cdouble_cuda_complex64, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit_ceil_cuda_float32, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit_chalf_cuda_complex64, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit_chalf_cuda_float32, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit_char_cuda_complex64, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit_cholesky_cuda_float32, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit_cholesky_inverse_cuda_complex64, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit_cholesky_solve_cuda_complex64, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit_chunk_cuda_complex64, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit_chunk_cuda_float32, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit_clamp_cuda_float32, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit_clamp_min_cuda_float32, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit_column_stack_cuda_complex64, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit_combinations_cuda_float32, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit_complex_cuda_float32, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit_conj_cuda_complex64, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit_conj_cuda_float32, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit_conj_physical_cuda_complex64, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit_contiguous_cuda_float32, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit_cummin_cuda_float32, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit_cumprod_cuda_float32, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit_cumsum_cuda_float32, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit_cumulative_trapezoid_cuda_float32, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit_deg2rad_cuda_float32, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit_diag_cuda_complex64, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit_diag_cuda_float32, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit_diagflat_cuda_complex64, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit_diagonal_copy_cuda_complex64, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit_diagonal_copy_cuda_float32, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit_diagonal_cuda_float32, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit_diagonal_scatter_cuda_complex64, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit_diagonal_scatter_cuda_float32, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit_diff_cuda_complex64, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit_diff_cuda_float32, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit_dist_cuda_complex64, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit_dist_cuda_float32, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit_div_floor_rounding_cuda_float32, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit_div_trunc_rounding_cuda_float32, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit_double_cuda_float32, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit_dstack_cuda_complex64, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit_einsum_cuda_complex64, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit_einsum_cuda_float32, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit_empty_like_cuda_complex64, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit_empty_permuted_cuda_float32, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit_equal_cuda_complex64, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit_erfc_cuda_float32, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit_erfinv_cuda_float32, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit_exp2_cuda_complex64, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit_exp2_cuda_float32, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit_exp_cuda_float32, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit_expand_as_cuda_complex64, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit_expand_copy_cuda_float32, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit_expand_cuda_complex64, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit_expm1_cuda_float32, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit_eye_cuda_float32, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit_fft_fft2_cuda_complex64, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit_fft_fft_cuda_complex64, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit_fft_fft_cuda_float32, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit_fft_fftn_cuda_float32, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit_fft_hfft2_cuda_complex64, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit_fft_ifft_cuda_complex64, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit_fft_ifftn_cuda_complex64, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit_fft_ifftn_cuda_float32, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit_fft_ihfft2_cuda_float32, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit_fft_ihfft_cuda_float32, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit_fft_irfft2_cuda_float32, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit_fft_irfft_cuda_complex64, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit_fft_irfftn_cuda_complex64, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit_fft_rfft_cuda_float32, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit_fft_rfftn_cuda_float32, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit_fill_cuda_complex64, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit_fill_cuda_float32, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit_flatten_cuda_float32, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit_flip_cuda_float32, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit_fliplr_cuda_float32, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit_flipud_cuda_float32, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit_float_power_cuda_float32, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit_floor_cuda_float32, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit_floor_divide_cuda_float32, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit_fmax_cuda_float32, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit_frac_cuda_float32, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit_frexp_cuda_float32, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit_full_cuda_float32, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit_full_like_cuda_complex64, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit_full_like_cuda_float32, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit_gather_cuda_complex64, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit_gather_cuda_float32, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit_ge_cuda_float32, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit_geometric_cuda_float32, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit_geqrf_cuda_complex64, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit_geqrf_cuda_float32, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit_grid_sampler_3d_cuda_float32, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit_half_cuda_float32, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit_hash_tensor_cuda_float32, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit_heaviside_cuda_float32, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit_hstack_cuda_complex64, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit_hstack_cuda_float32, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit_hypot_cuda_float32, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit_igamma_cuda_float32, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit_index_add_cuda_complex64, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit_index_add_cuda_float32, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit_index_copy_cuda_complex64, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit_index_copy_cuda_float32, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit_index_put_cuda_complex64, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit_index_put_cuda_float32, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit_index_reduce_amax_cuda_float32, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit_index_reduce_amin_cuda_float32, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit_index_reduce_prod_cuda_float32, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit_index_select_cuda_complex64, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit_index_select_cuda_float32, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit_inner_cuda_complex64, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit_isclose_cuda_complex64, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit_isfinite_cuda_complex64, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit_isin_cuda_float32, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit_isinf_cuda_complex64, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit_isneginf_cuda_float32, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit_isposinf_cuda_float32, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit_isreal_cuda_complex64, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit_isreal_cuda_float32, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit_istft_cuda_complex64, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit_item_cuda_complex64, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit_item_cuda_float32, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit_jiterator_2inputs_2outputs_cuda_complex64, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit_jiterator_4inputs_with_extra_args_cuda_complex64, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit_jiterator_4inputs_with_extra_args_cuda_float32, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit_jiterator_binary_cuda_float32, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit_jiterator_unary_cuda_float32, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit_kron_cuda_complex64, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit_kron_cuda_float32, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit_ldexp_cuda_float32, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit_le_cuda_float32, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit_lerp_cuda_float32, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit_linalg_cholesky_cuda_float32, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit_linalg_cholesky_ex_cuda_complex64, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit_linalg_cholesky_ex_cuda_float32, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit_linalg_cond_cuda_float32, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit_linalg_cross_cuda_complex64, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit_linalg_cross_cuda_float32, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit_linalg_diagonal_cuda_complex64, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit_linalg_diagonal_cuda_float32, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit_linalg_eig_cuda_float32, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit_linalg_eigvals_cuda_float32, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit_linalg_eigvalsh_cuda_complex64, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit_linalg_eigvalsh_cuda_float32, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit_linalg_householder_product_cuda_float32, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit_linalg_inv_cuda_float32, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit_linalg_ldl_factor_cuda_complex64, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit_linalg_ldl_factor_cuda_float32, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit_linalg_ldl_solve_cuda_float32, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit_linalg_lstsq_cuda_complex64, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit_linalg_lstsq_grad_oriented_cuda_complex64, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit_linalg_lstsq_grad_oriented_cuda_float32, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit_linalg_lu_cuda_float32, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit_linalg_lu_factor_cuda_complex64, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit_linalg_lu_factor_ex_cuda_float32, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit_linalg_lu_solve_cuda_complex64, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit_linalg_lu_solve_cuda_float32, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit_linalg_matrix_norm_cuda_complex64, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit_linalg_matrix_norm_cuda_float32, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit_linalg_matrix_power_cuda_complex64, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit_linalg_matrix_rank_hermitian_cuda_complex64, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit_linalg_multi_dot_cuda_complex64, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit_linalg_multi_dot_cuda_float32, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit_linalg_norm_cuda_float32, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit_linalg_norm_subgradients_at_zero_cuda_complex64, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit_linalg_pinv_hermitian_cuda_float32, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit_linalg_pinv_singular_cuda_complex64, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit_linalg_pinv_singular_cuda_float32, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit_linalg_qr_cuda_float32, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit_linalg_solve_cuda_float32, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit_linalg_solve_ex_cuda_float32, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit_linalg_solve_triangular_cuda_complex64, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit_linalg_solve_triangular_cuda_float32, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit_linalg_svd_cuda_float32, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit_linalg_svdvals_cuda_float32, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit_linalg_tensorinv_cuda_complex64, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit_linalg_tensorinv_cuda_float32, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit_linalg_vander_cuda_complex64, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit_linalg_vander_cuda_float32, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit_linalg_vecdot_cuda_complex64, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit_linspace_cuda_complex64, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit_linspace_cuda_float32, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit_linspace_tensor_overload_cuda_complex64, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit_linspace_tensor_overload_cuda_float32, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit_log1p_cuda_complex64, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit_log_cuda_float32, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit_log_softmax_cuda_float32, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit_logaddexp2_cuda_float32, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit_logaddexp_cuda_complex64, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit_logcumsumexp_cuda_complex64, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit_logdet_cuda_complex64, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit_logical_and_cuda_complex64, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit_logical_not_cuda_float32, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit_logical_or_cuda_float32, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit_logical_xor_cuda_complex64, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit_logical_xor_cuda_float32, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit_logit_cuda_float32, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit_logspace_cuda_complex64, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit_logspace_tensor_overload_cuda_float32, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit_logsumexp_cuda_complex64, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit_long_cuda_complex64, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit_long_cuda_float32, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit_lu_cuda_float32, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit_lu_solve_cuda_complex64, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit_lu_solve_cuda_float32, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit_lu_unpack_cuda_complex64, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit_mH_cuda_complex64, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit_mH_cuda_float32, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit_mT_cuda_complex64, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit_mT_cuda_float32, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit_masked_amax_cuda_float32, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit_masked_argmin_cuda_float32, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit_masked_cumsum_cuda_complex64, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit_masked_cumsum_cuda_float32, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit_masked_fill_cuda_complex64, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit_masked_fill_cuda_float32, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit_masked_log_softmax_cuda_float32, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit_masked_logaddexp_cuda_float32, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit_masked_logsumexp_cuda_complex64, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit_masked_mean_cuda_complex64, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit_masked_mean_cuda_float32, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit_masked_prod_cuda_complex64, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit_masked_prod_cuda_float32, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit_masked_select_cuda_complex64, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit_masked_select_cuda_float32, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit_masked_softmax_cuda_float32, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit_masked_softmin_cuda_float32, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit_masked_std_cuda_complex64, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit_masked_std_cuda_float32, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit_masked_sum_cuda_float32, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit_masked_var_cuda_float32, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit_matmul_cuda_complex64, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit_matmul_cuda_float32, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit_matrix_exp_cuda_complex64, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit_matrix_exp_cuda_float32, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit_max_binary_cuda_float32, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit_maximum_cuda_float32, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit_median_cuda_float32, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit_meshgrid_list_of_tensors_cuda_complex64, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit_meshgrid_variadic_tensors_cuda_complex64, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit_min_reduction_no_dim_cuda_float32, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit_min_reduction_with_dim_cuda_float32, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit_mm_cuda_float32, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit_mode_cuda_float32, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit_multinomial_cuda_float32, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit_mvlgamma_mvlgamma_p_3_cuda_float32, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit_nanmean_cuda_complex64, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit_nanmedian_cuda_float32, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit_nansum_cuda_complex64, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit_narrow_copy_cuda_float32, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit_native_batch_norm_cuda_float32, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit_native_dropout_backward_cuda_float32, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit_new_empty_cuda_complex64, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit_new_ones_cuda_complex64, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit_new_ones_cuda_float32, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit_new_zeros_cuda_complex64, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit_new_zeros_cuda_float32, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit_nextafter_cuda_float32, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit_nn_functional_adaptive_avg_pool2d_cuda_float32, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit_nn_functional_adaptive_max_pool1d_cuda_float32, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit_nn_functional_adaptive_max_pool2d_cuda_float32, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit_nn_functional_adaptive_max_pool3d_cuda_float32, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit_nn_functional_avg_pool2d_cuda_float32, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit_nn_functional_batch_norm_cuda_float32, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit_nn_functional_batch_norm_without_cudnn_cuda_float32, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit_nn_functional_bilinear_cuda_float32, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit_nn_functional_binary_cross_entropy_cuda_float32, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit_nn_functional_binary_cross_entropy_with_logits_cuda_float32, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit_nn_functional_celu_cuda_float32, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit_nn_functional_channel_shuffle_cuda_float32, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit_nn_functional_conv1d_cuda_complex64, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit_nn_functional_conv1d_cuda_float32, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit_nn_functional_conv_transpose1d_cuda_complex64, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit_nn_functional_conv_transpose1d_cuda_float32, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit_nn_functional_conv_transpose2d_cuda_float32, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit_nn_functional_conv_transpose3d_cuda_complex64, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit_nn_functional_cosine_embedding_loss_cuda_float32, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit_nn_functional_ctc_loss_cuda_float32, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit_nn_functional_dropout2d_cuda_float32, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit_nn_functional_dropout3d_cuda_float32, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit_nn_functional_embedding_cuda_float32, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit_nn_functional_feature_alpha_dropout_with_train_cuda_float32, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit_nn_functional_feature_alpha_dropout_without_train_cuda_complex64, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit_nn_functional_fractional_max_pool2d_cuda_float32, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit_nn_functional_fractional_max_pool3d_cuda_float32, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit_nn_functional_gelu_cuda_float32, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit_nn_functional_glu_cuda_float32, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit_nn_functional_grid_sample_cuda_float32, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit_nn_functional_hardshrink_cuda_float32, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit_nn_functional_hinge_embedding_loss_cuda_float32, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit_nn_functional_interpolate_area_cuda_float32, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit_nn_functional_interpolate_bicubic_cuda_float32, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit_nn_functional_interpolate_linear_cuda_float32, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit_nn_functional_interpolate_nearest-exact_cuda_float32, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit_nn_functional_interpolate_nearest_cuda_float32, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit_nn_functional_interpolate_trilinear_cuda_float32, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit_nn_functional_kl_div_cuda_float32, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit_nn_functional_layer_norm_cuda_float32, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit_nn_functional_leaky_relu_cuda_float32, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit_nn_functional_linear_cuda_complex64, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit_nn_functional_logsigmoid_cuda_float32, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit_nn_functional_margin_ranking_loss_cuda_float32, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit_nn_functional_max_pool1d_cuda_float32, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit_nn_functional_max_pool2d_cuda_float32, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit_nn_functional_max_unpool1d_grad_cuda_float32, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit_nn_functional_max_unpool2d_grad_cuda_float32, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit_nn_functional_max_unpool3d_cuda_float32, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit_nn_functional_mish_cuda_float32, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit_nn_functional_mse_loss_cuda_float32, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit_nn_functional_multi_head_attention_forward_cuda_float32, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit_nn_functional_multi_margin_loss_cuda_float32, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit_nn_functional_multilabel_margin_loss_cuda_float32, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit_nn_functional_multilabel_soft_margin_loss_cuda_float32, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit_nn_functional_nll_loss_cuda_float32, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit_nn_functional_pad_circular_cuda_complex64, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit_nn_functional_pad_constant_cuda_complex64, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit_nn_functional_pad_replicate_cuda_complex64, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit_nn_functional_pairwise_distance_cuda_complex64, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit_nn_functional_pairwise_distance_cuda_float32, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit_nn_functional_pixel_unshuffle_cuda_complex64, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit_nn_functional_relu6_cuda_float32, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit_nn_functional_relu_cuda_float32, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit_nn_functional_rms_norm_cuda_complex64, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit_nn_functional_rms_norm_cuda_float32, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit_nn_functional_silu_cuda_float32, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit_nn_functional_soft_margin_loss_cuda_float32, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit_nn_functional_softsign_cuda_float32, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit_nn_functional_tanhshrink_cuda_float32, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit_nn_functional_unfold_cuda_float32, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit_nn_functional_upsample_bilinear_cuda_float32, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit_nonzero_cuda_complex64, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit_nonzero_cuda_float32, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit_nonzero_static_cuda_complex64, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit_norm_cuda_float32, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit_norm_fro_cuda_complex64, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit_norm_nuc_cuda_complex64, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit_norm_nuc_cuda_float32, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit_normal_cuda_float32, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit_normal_in_place_cuda_float32, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit_ones_cuda_complex64, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit_ones_cuda_float32, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit_ones_like_cuda_float32, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit_ormqr_cuda_complex64, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit_outer_cuda_complex64, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit_outer_cuda_float32, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit_pca_lowrank_cuda_complex64, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit_pca_lowrank_cuda_float32, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit_permute_copy_cuda_complex64, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit_permute_cuda_float32, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit_pinverse_cuda_complex64, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit_polygamma_polygamma_n_4_cuda_float32, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit_positive_cuda_complex64, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit_positive_cuda_float32, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit_pow_cuda_complex64, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit_prod_cuda_complex64, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit_qr_cuda_complex64, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit_quantile_cuda_float32, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit_randint_like_cuda_float32, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit_ravel_cuda_complex64, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit_ravel_cuda_float32, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit_real_cuda_float32, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit_reciprocal_cuda_float32, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit_remainder_cuda_float32, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit_renorm_cuda_complex64, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit_renorm_cuda_float32, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit_repeat_cuda_complex64, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit_repeat_cuda_float32, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit_repeat_interleave_cuda_complex64, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit_repeat_interleave_cuda_float32, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit_reshape_as_cuda_complex64, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit_reshape_as_cuda_float32, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit_reshape_cuda_float32, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit_resize_as__cuda_float32, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit_resolve_conj_cuda_complex64, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit_resolve_neg_cuda_complex64, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit_resolve_neg_cuda_float32, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit_roll_cuda_complex64, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit_rot90_cuda_complex64, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit_rot90_cuda_float32, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit_round_cuda_float32, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit_round_decimals_neg_3_cuda_float32, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit_rsqrt_cuda_float32, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit_scalar_tensor_cuda_complex64, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit_scatter_add_cuda_complex64, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit_scatter_cuda_float32, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit_scatter_reduce_amax_cuda_float32, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit_scatter_reduce_amin_cuda_float32, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit_scatter_reduce_mean_cuda_float32, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit_select_cuda_complex64, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit_select_cuda_float32, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit_sgn_cuda_complex64, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit_sgn_cuda_float32, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit_short_cuda_float32, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit_sigmoid_cuda_float32, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit_signal_windows_exponential_cuda_float32, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit_signal_windows_general_cosine_cuda_float32, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit_signal_windows_general_hamming_cuda_float32, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit_signal_windows_hamming_cuda_float32, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit_sin_cuda_complex64, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit_sin_cuda_float32, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit_sinc_cuda_complex64, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit_sinh_cuda_float32, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit_slice_cuda_complex64, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit_slice_scatter_cuda_float32, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit_softmax_with_dtype_cuda_float32, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit_sparse_sampled_addmm_cuda_complex64, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit_sparse_sampled_addmm_cuda_float32, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit_special_bessel_j1_cuda_float32, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit_special_bessel_y1_cuda_float32, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit_special_chebyshev_polynomial_t_cuda_float32, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit_special_chebyshev_polynomial_v_cuda_float32, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit_special_chebyshev_polynomial_w_cuda_float32, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit_special_hermite_polynomial_he_cuda_float32, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit_special_i1_cuda_float32, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit_special_laguerre_polynomial_l_cuda_float32, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit_special_log_ndtr_cuda_float32, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit_special_modified_bessel_i0_cuda_float32, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit_special_ndtri_cuda_float32, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit_special_polygamma_special_polygamma_n_0_cuda_float32, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit_special_scaled_modified_bessel_k0_cuda_float32, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit_special_shifted_chebyshev_polynomial_u_cuda_float32, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit_special_shifted_chebyshev_polynomial_v_cuda_float32, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit_special_shifted_chebyshev_polynomial_w_cuda_float32, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit_special_spherical_bessel_j0_cuda_float32, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit_special_zeta_cuda_float32, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit_split_cuda_complex64, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit_split_cuda_float32, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit_split_list_args_cuda_complex64, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit_split_list_args_cuda_float32, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit_split_with_sizes_copy_cuda_complex64, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit_split_with_sizes_copy_cuda_float32, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit_split_with_sizes_cuda_complex64, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit_sqrt_cuda_complex64, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit_square_cuda_float32, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit_stack_cuda_complex64, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit_stack_cuda_float32, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit_std_cuda_complex64, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit_std_mean_cuda_float32, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit_std_unbiased_cuda_complex64, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit_std_unbiased_cuda_float32, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit_sub_cuda_complex64, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit_sub_cuda_float32, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit_sum_cuda_complex64, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit_sum_to_size_cuda_float32, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit_svd_cuda_float32, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit_svd_lowrank_cuda_complex64, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit_svd_lowrank_cuda_float32, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit_t_cuda_float32, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit_take_cuda_complex64, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit_take_cuda_float32, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit_tanh_cuda_complex64, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit_tanh_cuda_float32, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit_tensor_split_cuda_float32, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit_topk_cuda_float32, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit_torch_ops_aten__efficient_attention_forward_cuda_float32, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit_torch_ops_aten__safe_softmax_default_cuda_float32, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit_trace_cuda_complex64, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit_trace_cuda_float32, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit_transpose_copy_cuda_float32, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit_transpose_cuda_complex64, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit_trapezoid_cuda_float32, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit_trapz_cuda_float32, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit_triangular_solve_cuda_float32, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit_tril_cuda_float32, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit_triu_cuda_float32, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit_true_divide_cuda_float32, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit_trunc_cuda_float32, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit_unbind_copy_cuda_complex64, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit_unbind_cuda_complex64, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit_unbind_cuda_float32, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit_unflatten_cuda_complex64, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit_uniform_cuda_complex64, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit_uniform_cuda_float32, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit_unique_consecutive_cuda_float32, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit_unique_cuda_float32, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit_unsafe_chunk_cuda_complex64, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit_unsafe_chunk_cuda_float32, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit_unsqueeze_cuda_complex64, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit_unsqueeze_cuda_float32, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit_var_cuda_float32, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit_var_unbiased_cuda_complex64, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit_view_as_complex_cuda_float32, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit_view_as_cuda_complex64, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit_view_as_cuda_float32, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit_view_as_real_cuda_complex64, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit_view_copy_cuda_complex64, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit_view_copy_cuda_float32, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit_view_cuda_complex64, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit_view_cuda_float32, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit_vsplit_cuda_float32, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit_where_cuda_complex64, test/test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit_zeros_cuda_complex64 2025-12-04T12:12:39.5900152Z 2025-12-04T12:12:39.5900367Z Finished test_ops_jit 2/2 ... [2025-12-04 12:12:39.538412][9207.480628061], took 10.02min 2025-12-04T12:12:39.5901034Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/test_ops_jit/test_ops_jit-8aecbb695376eca7.xml 2025-12-04T12:12:39.6709640Z Running test_nestedtensor 3/3 ... [2025-12-04 12:12:39.670687][9207.61290541] 2025-12-04T12:12:39.6710249Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2025-12-04T12:12:39.6713176Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'test_nestedtensor.py', '--shard-id=3', '--num-shards=3', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-12-04 12:12:39.671005] 2025-12-04T12:22:50.2509189Z 2025-12-04T12:22:50.2510259Z test_nestedtensor 3/3 was successful, full logs can be found in artifacts with path test/test-reports/test_nestedtensor_3.3_f869b12f9fd94276_.log 2025-12-04T12:22:50.2695002Z Running 564 items in this shard: test/test_nestedtensor.py::TestNestedTensor::test_2d_nested_tensor_batch_size_2_max_seq_len_3_vocab_size_20, test/test_nestedtensor.py::TestNestedTensor::test_2d_nested_tensor_batch_size_2_max_seq_len_5_vocab_size_20, test/test_nestedtensor.py::TestNestedTensor::test_2d_nested_tensor_batch_size_4_max_seq_len_3_vocab_size_20, test/test_nestedtensor.py::TestNestedTensor::test_2d_nested_tensor_batch_size_4_max_seq_len_5_vocab_size_20, test/test_nestedtensor.py::TestNestedTensor::test_3d_nested_tensor_batch_size_2_max_seq_len_3_vocab_size_10, test/test_nestedtensor.py::TestNestedTensor::test_3d_nested_tensor_batch_size_2_max_seq_len_5_vocab_size_10, test/test_nestedtensor.py::TestNestedTensor::test_3d_nested_tensor_batch_size_2_max_seq_len_5_vocab_size_20, test/test_nestedtensor.py::TestNestedTensor::test_3d_nested_tensor_batch_size_4_max_seq_len_3_vocab_size_10, test/test_nestedtensor.py::TestNestedTensor::test_3d_nested_tensor_batch_size_4_max_seq_len_5_vocab_size_10, test/test_nestedtensor.py::TestNestedTensor::test_3d_nested_tensor_batch_size_4_max_seq_len_5_vocab_size_20, test/test_nestedtensor.py::TestNestedTensor::test_3d_nested_tensor_float_batch_size_2_max_seq_len_3_vocab_size_20, test/test_nestedtensor.py::TestNestedTensor::test_3d_nested_tensor_float_batch_size_2_max_seq_len_5_vocab_size_10, test/test_nestedtensor.py::TestNestedTensor::test_3d_nested_tensor_float_batch_size_4_max_seq_len_3_vocab_size_10, test/test_nestedtensor.py::TestNestedTensor::test_3d_nested_tensor_float_batch_size_4_max_seq_len_3_vocab_size_20, test/test_nestedtensor.py::TestNestedTensor::test_dim, test/test_nestedtensor.py::TestNestedTensor::test_like_functions_ones_like, test/test_nestedtensor.py::TestNestedTensor::test_nested_tensor, test/test_nestedtensor.py::TestNestedTensor::test_nested_tensor_matching_dim, test/test_nestedtensor.py::TestNestedTensor::test_unbind_1, test/test_nestedtensor.py::TestNestedTensor::test_unbind_4, test/test_nestedtensor.py::TestNestedTensor::test_unbind_dim, test/test_nestedtensor.py::TestNestedInt::test_comparisons, test/test_nestedtensor.py::TestNestedInt::test_with_factor, test/test_nestedtensor.py::TestNestedTensorDeviceTypeCUDA::test_binary_ops_with_scalar_eq_cuda, test/test_nestedtensor.py::TestNestedTensorDeviceTypeCUDA::test_binary_ops_with_scalar_ge_cuda, test/test_nestedtensor.py::TestNestedTensorDeviceTypeCUDA::test_bmm_cuda_cuda_float32, test/test_nestedtensor.py::TestNestedTensorDeviceTypeCUDA::test_bmm_noncontiguous_cuda_float64, test/test_nestedtensor.py::TestNestedTensorDeviceTypeCUDA::test_clone_cuda_float32, test/test_nestedtensor.py::TestNestedTensorDeviceTypeCUDA::test_contiguous_cuda_float32, test/test_nestedtensor.py::TestNestedTensorDeviceTypeCUDA::test_detach_cuda_float16, test/test_nestedtensor.py::TestNestedTensorDeviceTypeCUDA::test_detach_cuda_float32, test/test_nestedtensor.py::TestNestedTensorDeviceTypeCUDA::test_detach_cuda_float64, test/test_nestedtensor.py::TestNestedTensorDeviceTypeCUDA::test_dropout_strided_cuda_float64, test/test_nestedtensor.py::TestNestedTensorDeviceTypeCUDA::test_embedding_strided_cuda, test/test_nestedtensor.py::TestNestedTensorDeviceTypeCUDA::test_empty_like_cuda_float32, test/test_nestedtensor.py::TestNestedTensorDeviceTypeCUDA::test_jagged_amax_dtypes_cuda_bfloat16, test/test_nestedtensor.py::TestNestedTensorDeviceTypeCUDA::test_jagged_amax_dtypes_cuda_uint8, test/test_nestedtensor.py::TestNestedTensorDeviceTypeCUDA::test_jagged_amin_dtypes_cuda_float16, test/test_nestedtensor.py::TestNestedTensorDeviceTypeCUDA::test_jagged_amin_dtypes_cuda_int64, test/test_nestedtensor.py::TestNestedTensorDeviceTypeCUDA::test_jagged_argmax_dtypes_cuda_float32, test/test_nestedtensor.py::TestNestedTensorDeviceTypeCUDA::test_jagged_argmax_dtypes_cuda_float64, test/test_nestedtensor.py::TestNestedTensorDeviceTypeCUDA::test_jagged_argmax_dtypes_cuda_int32, test/test_nestedtensor.py::TestNestedTensorDeviceTypeCUDA::test_jagged_argmin_dtypes_cuda_int64, test/test_nestedtensor.py::TestNestedTensorDeviceTypeCUDA::test_jagged_max_dtypes_cuda_float64, test/test_nestedtensor.py::TestNestedTensorDeviceTypeCUDA::test_jagged_max_dtypes_cuda_int16, test/test_nestedtensor.py::TestNestedTensorDeviceTypeCUDA::test_jagged_max_dtypes_cuda_int8, test/test_nestedtensor.py::TestNestedTensorDeviceTypeCUDA::test_jagged_min_dtypes_cuda_bfloat16, test/test_nestedtensor.py::TestNestedTensorDeviceTypeCUDA::test_jagged_min_dtypes_cuda_float64, test/test_nestedtensor.py::TestNestedTensorDeviceTypeCUDA::test_jagged_min_dtypes_cuda_int64, test/test_nestedtensor.py::TestNestedTensorDeviceTypeCUDA::test_jagged_min_dtypes_cuda_uint8, test/test_nestedtensor.py::TestNestedTensorDeviceTypeCUDA::test_linear_cuda_float32, test/test_nestedtensor.py::TestNestedTensorDeviceTypeCUDA::test_linear_noncontiguous_cuda_float32, test/test_nestedtensor.py::TestNestedTensorDeviceTypeCUDA::test_matmul_with_bmm_path_cuda_float64, test/test_nestedtensor.py::TestNestedTensorDeviceTypeCUDA::test_narrow_cuda_float16, test/test_nestedtensor.py::TestNestedTensorDeviceTypeCUDA::test_narrow_cuda_float32, test/test_nestedtensor.py::TestNestedTensorDeviceTypeCUDA::test_nested_tensor_add_in_place_cuda_float16, test/test_nestedtensor.py::TestNestedTensorDeviceTypeCUDA::test_nested_tensor_dense_elementwise_embedding_dim_128_cuda_float16, test/test_nestedtensor.py::TestNestedTensorDeviceTypeCUDA::test_nested_tensor_dense_elementwise_embedding_dim_128_cuda_float32, test/test_nestedtensor.py::TestNestedTensorDeviceTypeCUDA::test_nested_tensor_dense_elementwise_embedding_dim_384_cuda_float16, test/test_nestedtensor.py::TestNestedTensorDeviceTypeCUDA::test_nested_tensor_dense_elementwise_embedding_dim_8_cuda_float32, test/test_nestedtensor.py::TestNestedTensorDeviceTypeCUDA::test_nested_tensor_indexing_cuda_float16, test/test_nestedtensor.py::TestNestedTensorDeviceTypeCUDA::test_nested_tensor_indexing_noncontiguous_cuda_float16, test/test_nestedtensor.py::TestNestedTensorDeviceTypeCUDA::test_nested_tensor_mul_cuda_float16, test/test_nestedtensor.py::TestNestedTensorDeviceTypeCUDA::test_nested_tensor_split_with_sizes_cuda_float16, test/test_nestedtensor.py::TestNestedTensorDeviceTypeCUDA::test_serialization_requires_grad_False_weights_only_False_cuda_float64, test/test_nestedtensor.py::TestNestedTensorDeviceTypeCUDA::test_serialization_requires_grad_True_weights_only_False_cuda_float16, test/test_nestedtensor.py::TestNestedTensorDeviceTypeCUDA::test_serialization_requires_grad_True_weights_only_False_cuda_float32, test/test_nestedtensor.py::TestNestedTensorDeviceTypeCUDA::test_softmax_cuda_float64, test/test_nestedtensor.py::TestNestedTensorDeviceTypeCUDA::test_softmax_noncontiguous_cuda_float64, test/test_nestedtensor.py::TestNestedTensorDeviceTypeCUDA::test_squeeze_unsqueeze_cuda_float16, test/test_nestedtensor.py::TestNestedTensorDeviceTypeCUDA::test_to_padded_tensor_dim3_cuda_float32, test/test_nestedtensor.py::TestNestedTensorDeviceTypeCUDA::test_to_padded_tensor_noncontiguous_cuda_float32, test/test_nestedtensor.py::TestNestedTensorDeviceTypeCUDA::test_to_padded_tensor_noncontiguous_cuda_float64, test/test_nestedtensor.py::TestNestedTensorDeviceTypeCUDA::test_to_padded_tensor_simple_cuda_float32, test/test_nestedtensor.py::TestNestedTensorDeviceTypeCUDA::test_to_padded_tensor_zero_numel_errors_cuda_float32, test/test_nestedtensor.py::TestNestedTensorDeviceTypeCUDA::test_transpose_cuda_float32, test/test_nestedtensor.py::TestNestedTensorDeviceTypeCUDA::test_transpose_inference_mode_interaction_cuda_float64, test/test_nestedtensor.py::TestNestedTensorDeviceTypeCUDA::test_unary_funcs_abs__cuda, test/test_nestedtensor.py::TestNestedTensorDeviceTypeCUDA::test_unary_funcs_gelu__cuda, test/test_nestedtensor.py::TestNestedTensorDeviceTypeCUDA::test_unary_funcs_isnan_cuda, test/test_nestedtensor.py::TestNestedTensorDeviceTypeCUDA::test_unary_funcs_isposinf_cuda, test/test_nestedtensor.py::TestNestedTensorDeviceTypeCUDA::test_unary_funcs_relu__cuda, test/test_nestedtensor.py::TestNestedTensorDeviceTypeCUDA::test_unary_funcs_sgn_cuda, test/test_nestedtensor.py::TestNestedTensorDeviceTypeCUDA::test_unary_funcs_silu__cuda, test/test_nestedtensor.py::TestNestedTensorDeviceTypeCUDA::test_unary_funcs_sin_cuda, test/test_nestedtensor.py::TestNestedTensorDeviceTypeCUDA::test_unbind_noncontiguous_cuda_float16, test/test_nestedtensor.py::TestNestedTensorDeviceTypeCUDA::test_unbind_noncontiguous_cuda_float32, test/test_nestedtensor.py::TestNestedTensorDeviceTypeCUDA::test_unbind_noncontiguous_cuda_float64, test/test_nestedtensor.py::TestNestedTensorDeviceTypeCUDA::test_view_inference_mode_interaction_cuda_float64, test/test_nestedtensor.py::TestNestedTensorAutogradCUDA::test_backward_for_sub_op_cuda, test/test_nestedtensor.py::TestNestedTensorAutogradCUDA::test_layer_norm_backward_5d_size_4_cuda, test/test_nestedtensor.py::TestNestedTensorAutogradCUDA::test_layer_norm_backward_edge_case_cuda, test/test_nestedtensor.py::TestNestedTensorAutogradCUDA::test_layer_norm_backward_size_32_cuda, test/test_nestedtensor.py::TestNestedTensorAutogradCUDA::test_layer_norm_backward_size_513_cuda, test/test_nestedtensor.py::TestNestedTensorAutogradCUDA::test_nested_tensor_bmm_backward_cuda, test/test_nestedtensor.py::TestNestedTensorAutogradCUDA::test_nested_tensor_bmm_gradcheck_cuda, test/test_nestedtensor.py::TestNestedTensorAutogradCUDA::test_nested_tensor_linear_backward_cuda, test/test_nestedtensor.py::TestNestedTensorAutogradCUDA::test_nested_tensor_reshape_backward_cuda, test/test_nestedtensor.py::TestNestedTensorAutogradCUDA::test_nested_tensor_reshape_gradcheck_cuda, test/test_nestedtensor.py::TestNestedTensorAutogradCUDA::test_nested_tensor_softmax_cuda, test/test_nestedtensor.py::TestNestedTensorAutogradCUDA::test_nested_tensor_squeeze_backward_cuda, test/test_nestedtensor.py::TestNestedTensorAutogradCUDA::test_nested_tensor_unsqueeze_backward_cuda, test/test_nestedtensor.py::TestNestedTensorAutogradCUDA::test_nested_tensor_unsqueeze_gradcheck_cuda, test/test_nestedtensor.py::TestNestedTensorAutogradCUDA::test_set_requires_grad_from_mask_cuda, test/test_nestedtensor.py::TestNestedTensorAutogradCUDA::test_values_grad_with_broadcast_cuda, test/test_nestedtensor.py::TestNestedTensorSubclassCUDA::test_apply__cuda_float32, test/test_nestedtensor.py::TestNestedTensorSubclassCUDA::test_as_nested_tensor_from_tensor_dim_0_layout_jagged_requires_grad_False_contiguous_False_cuda_float32, test/test_nestedtensor.py::TestNestedTensorSubclassCUDA::test_as_nested_tensor_from_tensor_dim_0_layout_jagged_requires_grad_False_contiguous_False_cuda_float64, test/test_nestedtensor.py::TestNestedTensorSubclassCUDA::test_as_nested_tensor_from_tensor_dim_0_layout_jagged_requires_grad_True_contiguous_False_cuda_float16, test/test_nestedtensor.py::TestNestedTensorSubclassCUDA::test_as_nested_tensor_from_tensor_dim_0_layout_strided_requires_grad_False_contiguous_True_cuda_float32, test/test_nestedtensor.py::TestNestedTensorSubclassCUDA::test_as_nested_tensor_from_tensor_dim_0_layout_strided_requires_grad_False_contiguous_True_cuda_float64, test/test_nestedtensor.py::TestNestedTensorSubclassCUDA::test_as_nested_tensor_from_tensor_dim_0_layout_strided_requires_grad_True_contiguous_True_cuda_float32, test/test_nestedtensor.py::TestNestedTensorSubclassCUDA::test_as_nested_tensor_from_tensor_dim_0_layout_strided_requires_grad_True_contiguous_True_cuda_float64, test/test_nestedtensor.py::TestNestedTensorSubclassCUDA::test_as_nested_tensor_from_tensor_dim_1_layout_jagged_requires_grad_True_contiguous_False_cuda_float16, test/test_nestedtensor.py::TestNestedTensorSubclassCUDA::test_as_nested_tensor_from_tensor_dim_1_layout_strided_requires_grad_False_contiguous_False_cuda_float32, test/test_nestedtensor.py::TestNestedTensorSubclassCUDA::test_as_nested_tensor_from_tensor_dim_1_layout_strided_requires_grad_False_contiguous_False_cuda_float64, test/test_nestedtensor.py::TestNestedTensorSubclassCUDA::test_as_nested_tensor_from_tensor_dim_1_layout_strided_requires_grad_False_contiguous_True_cuda_float16, test/test_nestedtensor.py::TestNestedTensorSubclassCUDA::test_as_nested_tensor_from_tensor_dim_1_layout_strided_requires_grad_True_contiguous_False_cuda_float16, test/test_nestedtensor.py::TestNestedTensorSubclassCUDA::test_as_nested_tensor_from_tensor_dim_2_layout_jagged_requires_grad_False_contiguous_False_cuda_float16, test/test_nestedtensor.py::TestNestedTensorSubclassCUDA::test_as_nested_tensor_from_tensor_dim_2_layout_jagged_requires_grad_True_contiguous_False_cuda_float32, test/test_nestedtensor.py::TestNestedTensorSubclassCUDA::test_as_nested_tensor_from_tensor_dim_2_layout_jagged_requires_grad_True_contiguous_True_cuda_float32, test/test_nestedtensor.py::TestNestedTensorSubclassCUDA::test_as_nested_tensor_from_tensor_dim_2_layout_jagged_requires_grad_True_contiguous_True_cuda_float64, test/test_nestedtensor.py::TestNestedTensorSubclassCUDA::test_as_nested_tensor_from_tensor_dim_2_layout_strided_requires_grad_False_contiguous_False_cuda_float32, test/test_nestedtensor.py::TestNestedTensorSubclassCUDA::test_as_nested_tensor_from_tensor_dim_2_layout_strided_requires_grad_False_contiguous_False_cuda_float64, test/test_nestedtensor.py::TestNestedTensorSubclassCUDA::test_as_nested_tensor_from_tensor_dim_2_layout_strided_requires_grad_True_contiguous_False_cuda_float32, test/test_nestedtensor.py::TestNestedTensorSubclassCUDA::test_as_nested_tensor_from_tensor_dim_2_layout_strided_requires_grad_True_contiguous_True_cuda_float16, test/test_nestedtensor.py::TestNestedTensorSubclassCUDA::test_as_nested_tensor_from_tensor_dim_3_layout_jagged_requires_grad_False_contiguous_False_cuda_float16, test/test_nestedtensor.py::TestNestedTensorSubclassCUDA::test_as_nested_tensor_from_tensor_dim_3_layout_jagged_requires_grad_False_contiguous_False_cuda_float32, test/test_nestedtensor.py::TestNestedTensorSubclassCUDA::test_as_nested_tensor_from_tensor_dim_3_layout_jagged_requires_grad_True_contiguous_False_cuda_float32, test/test_nestedtensor.py::TestNestedTensorSubclassCUDA::test_as_nested_tensor_from_tensor_dim_3_layout_strided_requires_grad_False_contiguous_True_cuda_float32, test/test_nestedtensor.py::TestNestedTensorSubclassCUDA::test_as_nested_tensor_from_tensor_dim_3_layout_strided_requires_grad_True_contiguous_True_cuda_float32, test/test_nestedtensor.py::TestNestedTensorSubclassCUDA::test_as_nested_tensor_from_tensor_dim_4_layout_jagged_requires_grad_False_contiguous_True_cuda_float32, test/test_nestedtensor.py::TestNestedTensorSubclassCUDA::test_as_nested_tensor_from_tensor_dim_4_layout_jagged_requires_grad_True_contiguous_False_cuda_float64, test/test_nestedtensor.py::TestNestedTensorSubclassCUDA::test_as_nested_tensor_from_tensor_dim_4_layout_strided_requires_grad_False_contiguous_False_cuda_float64, test/test_nestedtensor.py::TestNestedTensorSubclassCUDA::test_as_nested_tensor_from_tensor_dim_4_layout_strided_requires_grad_False_contiguous_True_cuda_float64, test/test_nestedtensor.py::TestNestedTensorSubclassCUDA::test_as_nested_tensor_from_tensor_dim_4_layout_strided_requires_grad_True_contiguous_True_cuda_float16, test/test_nestedtensor.py::TestNestedTensorSubclassCUDA::test_as_nested_tensor_from_tensor_dim_4_layout_strided_requires_grad_True_contiguous_True_cuda_float32, test/test_nestedtensor.py::TestNestedTensorSubclassCUDA::test_as_nested_tensor_from_tensor_dim_4_layout_strided_requires_grad_True_contiguous_True_cuda_float64, test/test_nestedtensor.py::TestNestedTensorSubclassCUDA::test_binary_pointwise_with_nested_int_second_arg_cuda, test/test_nestedtensor.py::TestNestedTensorSubclassCUDA::test_broadcast_shapes_on_in_graph_constructed_njt_cuda_float32, test/test_nestedtensor.py::TestNestedTensorSubclassCUDA::test_chunk_cuda, test/test_nestedtensor.py::TestNestedTensorSubclassCUDA::test_compile_with_dynamic_min_seq_len_cuda_float32, test/test_nestedtensor.py::TestNestedTensorSubclassCUDA::test_compile_with_propagated_dynamic_max_seq_len_cuda_float32, test/test_nestedtensor.py::TestNestedTensorSubclassCUDA::test_flatten_decomp_cuda, test/test_nestedtensor.py::TestNestedTensorSubclassCUDA::test_index_put_error_cuda, test/test_nestedtensor.py::TestNestedTensorSubclassCUDA::test_is_same_size_cuda, test/test_nestedtensor.py::TestNestedTensorSubclassCUDA::test_jagged_layout_construction_as_nested_tensor_components_require_grad_False_cuda_float32, test/test_nestedtensor.py::TestNestedTensorSubclassCUDA::test_jagged_layout_construction_as_nested_tensor_components_require_grad_False_cuda_float64, test/test_nestedtensor.py::TestNestedTensorSubclassCUDA::test_jagged_layout_construction_as_nested_tensor_components_require_grad_True_cuda_float16, test/test_nestedtensor.py::TestNestedTensorSubclassCUDA::test_jagged_layout_construction_as_nested_tensor_components_require_grad_True_cuda_float64, test/test_nestedtensor.py::TestNestedTensorSubclassCUDA::test_jagged_layout_construction_nested_tensor_requires_grad_False_components_require_grad_False_cuda_float32, test/test_nestedtensor.py::TestNestedTensorSubclassCUDA::test_jagged_layout_construction_nested_tensor_requires_grad_False_components_require_grad_False_cuda_float64, test/test_nestedtensor.py::TestNestedTensorSubclassCUDA::test_jagged_layout_construction_nested_tensor_requires_grad_True_components_require_grad_True_cuda_float64, test/test_nestedtensor.py::TestNestedTensorSubclassCUDA::test_jagged_op_different_output_shape_dim_mean_keepdim_False_requires_grad_False_components_require_grad_False_cuda_float32, test/test_nestedtensor.py::TestNestedTensorSubclassCUDA::test_jagged_op_different_output_shape_dim_mean_keepdim_False_requires_grad_False_components_require_grad_True_cuda_float32, test/test_nestedtensor.py::TestNestedTensorSubclassCUDA::test_jagged_op_different_output_shape_dim_mean_keepdim_False_requires_grad_True_components_require_grad_False_cuda_float32, test/test_nestedtensor.py::TestNestedTensorSubclassCUDA::test_jagged_op_different_output_shape_dim_sum_keepdim_False_requires_grad_True_components_require_grad_False_cuda_float32, test/test_nestedtensor.py::TestNestedTensorSubclassCUDA::test_jagged_op_different_output_shape_dim_sum_keepdim_False_requires_grad_True_components_require_grad_True_cuda_float32, test/test_nestedtensor.py::TestNestedTensorSubclassCUDA::test_jagged_op_different_output_shape_dim_sum_keepdim_True_requires_grad_True_components_require_grad_False_cuda_float32, test/test_nestedtensor.py::TestNestedTensorSubclassCUDA::test_jagged_padded_dense_conversion_kernels_cuda_float16, test/test_nestedtensor.py::TestNestedTensorSubclassCUDA::test_jagged_padded_dense_conversion_kernels_cuda_float32, test/test_nestedtensor.py::TestNestedTensorSubclassCUDA::test_jagged_view_from_values_offsets_requires_grad_False_values_is_view_False_cuda_float32, test/test_nestedtensor.py::TestNestedTensorSubclassCUDA::test_jagged_view_from_values_offsets_requires_grad_False_values_is_view_True_cuda_float16, test/test_nestedtensor.py::TestNestedTensorSubclassCUDA::test_jagged_view_from_values_offsets_requires_grad_False_values_is_view_True_cuda_float64, test/test_nestedtensor.py::TestNestedTensorSubclassCUDA::test_jagged_view_from_values_offsets_requires_grad_True_values_is_view_False_cuda_float16, test/test_nestedtensor.py::TestNestedTensorSubclassCUDA::test_jagged_view_from_values_offsets_requires_grad_True_values_is_view_False_cuda_float32, test/test_nestedtensor.py::TestNestedTensorSubclassCUDA::test_layer_norm_2d_input_requires_grad_True_components_require_grad_True_cuda_float32, test/test_nestedtensor.py::TestNestedTensorSubclassCUDA::test_layer_norm_operate_on_batch_dim_requires_grad_False_components_require_grad_False_cuda_float32, test/test_nestedtensor.py::TestNestedTensorSubclassCUDA::test_layer_norm_reduce_ragged_idx_1_requires_grad_True_components_require_grad_True_cuda_float32, test/test_nestedtensor.py::TestNestedTensorSubclassCUDA::test_layer_norm_with_lengths_requires_grad_False_components_require_grad_False_cuda_float32, test/test_nestedtensor.py::TestNestedTensorSubclassCUDA::test_layer_norm_with_lengths_requires_grad_True_components_require_grad_False_cuda_float32, test/test_nestedtensor.py::TestNestedTensorSubclassCUDA::test_layout_under_torch_dispatch_mode_cuda, test/test_nestedtensor.py::TestNestedTensorSubclassCUDA::test_like_value_ones_like_cuda, test/test_nestedtensor.py::TestNestedTensorSubclassCUDA::test_like_value_randn_like_cuda, test/test_nestedtensor.py::TestNestedTensorSubclassCUDA::test_like_value_zeros_like_cuda, test/test_nestedtensor.py::TestNestedTensorSubclassCUDA::test_linear_backward_memory_usage_cuda_float32, test/test_nestedtensor.py::TestNestedTensorSubclassCUDA::test_linear_nt_dim_4_cuda, test/test_nestedtensor.py::TestNestedTensorSubclassCUDA::test_linear_nt_dim_5_cuda, test/test_nestedtensor.py::TestNestedTensorSubclassCUDA::test_nested_tensor_from_jagged_pass_min_max_False_cuda_float32, test/test_nestedtensor.py::TestNestedTensorSubclassCUDA::test_noncontiguous_pointwise_cuda, test/test_nestedtensor.py::TestNestedTensorSubclassCUDA::test_noncontiguous_to_noncontig_with_holes_cuda_float16, test/test_nestedtensor.py::TestNestedTensorSubclassCUDA::test_noncontiguous_to_noncontig_with_holes_cuda_float32, test/test_nestedtensor.py::TestNestedTensorSubclassCUDA::test_noncontiguous_to_noncontig_with_holes_cuda_float64, test/test_nestedtensor.py::TestNestedTensorSubclassCUDA::test_op_dim_reduce_batch_only_different_output_shape_mean_keepdim_False_requires_grad_False_components_require_grad_False_cuda_float32, test/test_nestedtensor.py::TestNestedTensorSubclassCUDA::test_op_dim_reduce_batch_only_different_output_shape_mean_keepdim_True_requires_grad_True_components_require_grad_False_cuda_float32, test/test_nestedtensor.py::TestNestedTensorSubclassCUDA::test_op_dim_reduce_batch_only_different_output_shape_mean_keepdim_True_requires_grad_True_components_require_grad_True_cuda_float32, test/test_nestedtensor.py::TestNestedTensorSubclassCUDA::test_op_dim_reduce_batch_only_different_output_shape_sum_keepdim_False_requires_grad_False_components_require_grad_False_cuda_float32, test/test_nestedtensor.py::TestNestedTensorSubclassCUDA::test_op_dim_reduce_batch_only_different_output_shape_sum_keepdim_False_requires_grad_True_components_require_grad_False_cuda_float32, test/test_nestedtensor.py::TestNestedTensorSubclassCUDA::test_op_dim_reduce_batch_only_different_output_shape_sum_keepdim_True_requires_grad_False_components_require_grad_True_cuda_float32, test/test_nestedtensor.py::TestNestedTensorSubclassCUDA::test_op_dim_reduce_batch_only_different_output_shape_sum_keepdim_True_requires_grad_True_components_require_grad_False_cuda_float32, test/test_nestedtensor.py::TestNestedTensorSubclassCUDA::test_op_dim_reduce_ragged_idx_1_different_output_shape_mean_keepdim_True_requires_grad_True_components_require_grad_False_cuda_float32, test/test_nestedtensor.py::TestNestedTensorSubclassCUDA::test_op_dim_reduce_ragged_idx_1_different_output_shape_sum_keepdim_True_requires_grad_True_components_require_grad_False_cuda_float32, test/test_nestedtensor.py::TestNestedTensorSubclassCUDA::test_op_dim_reduce_ragged_idx_greater_than_1_different_output_shape_mean_transpose_offset_1_keepdim_False_requires_grad_False_components_require_grad_False_cuda_float32, test/test_nestedtensor.py::TestNestedTensorSubclassCUDA::test_op_dim_reduce_ragged_idx_greater_than_1_different_output_shape_mean_transpose_offset_1_keepdim_False_requires_grad_False_components_require_grad_True_cuda_float32, test/test_nestedtensor.py::TestNestedTensorSubclassCUDA::test_op_dim_reduce_ragged_idx_greater_than_1_different_output_shape_mean_transpose_offset_1_keepdim_True_requires_grad_True_components_require_grad_False_cuda_float32, test/test_nestedtensor.py::TestNestedTensorSubclassCUDA::test_op_dim_reduce_ragged_idx_greater_than_1_different_output_shape_sum_transpose_offset_1_keepdim_False_requires_grad_True_components_require_grad_False_cuda_float32, test/test_nestedtensor.py::TestNestedTensorSubclassCUDA::test_op_dim_reduce_ragged_idx_greater_than_1_different_output_shape_sum_transpose_offset_1_keepdim_True_requires_grad_False_components_require_grad_False_cuda_float32, test/test_nestedtensor.py::TestNestedTensorSubclassCUDA::test_op_dim_reduce_ragged_idx_greater_than_1_different_output_shape_sum_transpose_offset_1_keepdim_True_requires_grad_False_components_require_grad_True_cuda_float32, test/test_nestedtensor.py::TestNestedTensorSubclassCUDA::test_op_dim_reduce_ragged_idx_greater_than_1_different_output_shape_sum_transpose_offset_1_keepdim_True_requires_grad_True_components_require_grad_False_cuda_float32, test/test_nestedtensor.py::TestNestedTensorSubclassCUDA::test_op_dim_reduce_ragged_idx_greater_than_1_different_output_shape_sum_transpose_offset_1_keepdim_True_requires_grad_True_components_require_grad_True_cuda_float32, test/test_nestedtensor.py::TestNestedTensorSubclassCUDA::test_op_dim_reduce_ragged_idx_greater_than_1_different_output_shape_sum_transpose_offset_2_keepdim_False_requires_grad_False_components_require_grad_False_cuda_float32, test/test_nestedtensor.py::TestNestedTensorSubclassCUDA::test_op_dim_reduce_ragged_idx_greater_than_1_different_output_shape_sum_transpose_offset_2_keepdim_False_requires_grad_True_components_require_grad_False_cuda_float32, test/test_nestedtensor.py::TestNestedTensorSubclassCUDA::test_op_dim_reduce_ragged_idx_greater_than_1_different_output_shape_sum_transpose_offset_2_keepdim_False_requires_grad_True_components_require_grad_True_cuda_float32, test/test_nestedtensor.py::TestNestedTensorSubclassCUDA::test_op_dim_reduce_ragged_idx_greater_than_1_different_output_shape_sum_transpose_offset_2_keepdim_True_requires_grad_True_components_require_grad_False_cuda_float32, test/test_nestedtensor.py::TestNestedTensorSubclassCUDA::test_op_dim_transpose_non_ragged_dim_different_output_shape_mean_keepdim_False_requires_grad_True_components_require_grad_True_cuda_float32, test/test_nestedtensor.py::TestNestedTensorSubclassCUDA::test_op_dim_transpose_non_ragged_dim_different_output_shape_mean_keepdim_True_requires_grad_False_components_require_grad_False_cuda_float32, test/test_nestedtensor.py::TestNestedTensorSubclassCUDA::test_op_dim_transpose_non_ragged_dim_different_output_shape_mean_keepdim_True_requires_grad_False_components_require_grad_True_cuda_float32, test/test_nestedtensor.py::TestNestedTensorSubclassCUDA::test_op_dim_transpose_non_ragged_dim_different_output_shape_sum_keepdim_False_requires_grad_False_components_require_grad_False_cuda_float32, test/test_nestedtensor.py::TestNestedTensorSubclassCUDA::test_op_dim_transpose_non_ragged_dim_different_output_shape_sum_keepdim_False_requires_grad_False_components_require_grad_True_cuda_float32, test/test_nestedtensor.py::TestNestedTensorSubclassCUDA::test_op_dim_transpose_non_ragged_dim_different_output_shape_sum_keepdim_True_requires_grad_True_components_require_grad_False_cuda_float32, test/test_nestedtensor.py::TestNestedTensorSubclassCUDA::test_op_dim_with_lengths_different_output_shape_mean_keepdim_False_requires_grad_True_components_require_grad_False_cuda_float32, test/test_nestedtensor.py::TestNestedTensorSubclassCUDA::test_op_dim_with_lengths_different_output_shape_mean_keepdim_True_requires_grad_True_components_require_grad_True_cuda_float32, test/test_nestedtensor.py::TestNestedTensorSubclassCUDA::test_op_dim_with_lengths_different_output_shape_sum_keepdim_True_requires_grad_False_components_require_grad_True_cuda_float32, test/test_nestedtensor.py::TestNestedTensorSubclassCUDA::test_op_dim_with_lengths_different_output_shape_sum_keepdim_True_requires_grad_True_components_require_grad_False_cuda_float32, test/test_nestedtensor.py::TestNestedTensorSubclassCUDA::test_op_dim_with_lengths_different_output_shape_sum_keepdim_True_requires_grad_True_components_require_grad_True_cuda_float32, test/test_nestedtensor.py::TestNestedTensorSubclassCUDA::test_pin_memory_cuda, test/test_nestedtensor.py::TestNestedTensorSubclassCUDA::test_sdpa_cuda_bfloat16, test/test_nestedtensor.py::TestNestedTensorSubclassCUDA::test_sdpa_cuda_float16, test/test_nestedtensor.py::TestNestedTensorSubclassCUDA::test_serialization_contig_weights_only_False_cuda_float32, test/test_nestedtensor.py::TestNestedTensorSubclassCUDA::test_serialization_contig_weights_only_True_cuda_float32, test/test_nestedtensor.py::TestNestedTensorSubclassCUDA::test_serialization_noncontig_with_holes_weights_only_False_cuda_float32, test/test_nestedtensor.py::TestNestedTensorSubclassCUDA::test_softmax_dim_reduce_ragged_idx_1_requires_grad_True_components_require_grad_False_cuda_float32, test/test_nestedtensor.py::TestNestedTensorSubclassCUDA::test_softmax_dim_reduce_ragged_idx_greater_than_1_same_output_shape_transpose_offset_1_requires_grad_False_components_require_grad_True_cuda_float32, test/test_nestedtensor.py::TestNestedTensorSubclassCUDA::test_softmax_dim_reduce_ragged_idx_greater_than_1_same_output_shape_transpose_offset_2_requires_grad_True_components_require_grad_False_cuda_float32, test/test_nestedtensor.py::TestNestedTensorSubclassCUDA::test_softmax_dim_requires_grad_False_components_require_grad_False_log_softmax_cuda_float32, test/test_nestedtensor.py::TestNestedTensorSubclassCUDA::test_softmax_dim_requires_grad_False_components_require_grad_False_softmax_cuda_float32, test/test_nestedtensor.py::TestNestedTensorSubclassCUDA::test_softmax_dim_requires_grad_False_components_require_grad_True_log_softmax_cuda_float32, test/test_nestedtensor.py::TestNestedTensorSubclassCUDA::test_softmax_dim_requires_grad_False_components_require_grad_True_softmax_cuda_float32, test/test_nestedtensor.py::TestNestedTensorSubclassCUDA::test_softmax_dim_requires_grad_True_components_require_grad_True_softmax_cuda_float32, test/test_nestedtensor.py::TestNestedTensorSubclassCUDA::test_softmax_dim_transpose_non_ragged_dim_requires_grad_False_components_require_grad_False_cuda_float32, test/test_nestedtensor.py::TestNestedTensorSubclassCUDA::test_softmax_dim_with_lengths_requires_grad_False_components_require_grad_True_cuda_float32, test/test_nestedtensor.py::TestNestedTensorSubclassCUDA::test_softmax_dim_with_lengths_requires_grad_True_components_require_grad_False_cuda_float32, test/test_nestedtensor.py::TestNestedTensorSubclassCUDA::test_softmax_dim_with_lengths_requires_grad_True_components_require_grad_True_cuda_float32, test/test_nestedtensor.py::TestNestedTensorSubclassCUDA::test_softmax_reduce_batch_dim_requires_grad_True_components_require_grad_False_log_softmax_cuda_float32, test/test_nestedtensor.py::TestNestedTensorSubclassCUDA::test_softmax_reduce_batch_dim_requires_grad_True_components_require_grad_False_softmax_cuda_float32, test/test_nestedtensor.py::TestNestedTensorSubclassCUDA::test_softmax_reduce_batch_dim_requires_grad_True_components_require_grad_True_log_softmax_cuda_float32, test/test_nestedtensor.py::TestNestedTensorSubclassCUDA::test_softmax_reduce_batch_dim_requires_grad_True_components_require_grad_True_softmax_cuda_float32, test/test_nestedtensor.py::TestNestedTensorSubclassCUDA::test_specialize_dynamic_shape_cuda, test/test_nestedtensor.py::TestNestedTensorSubclassCUDA::test_specialize_dynamic_shape_recompile_cuda, test/test_nestedtensor.py::TestNestedTensorSubclassCUDA::test_sum_dim_reduce_batch_and_non_batch_keepdim_False_requires_grad_True_components_require_grad_False_cuda_float32, test/test_nestedtensor.py::TestNestedTensorSubclassCUDA::test_sum_dim_reduce_batch_and_non_batch_keepdim_True_requires_grad_False_components_require_grad_False_cuda_float32, test/test_nestedtensor.py::TestNestedTensorSubclassCUDA::test_sum_dim_reduce_batch_and_non_batch_keepdim_True_requires_grad_False_components_require_grad_True_cuda_float32, test/test_nestedtensor.py::TestNestedTensorSubclassCUDA::test_sum_dim_reduce_ragged_and_non_batch_keepdim_False_requires_grad_True_components_require_grad_True_cuda_float32, test/test_nestedtensor.py::TestNestedTensorSubclassCUDA::test_sum_dim_reduce_ragged_and_non_batch_keepdim_True_requires_grad_False_components_require_grad_False_cuda_float32, test/test_nestedtensor.py::TestNestedTensorSubclassCUDA::test_sum_dim_reduce_ragged_and_non_batch_keepdim_True_requires_grad_True_components_require_grad_False_cuda_float32, test/test_nestedtensor.py::TestNestedTensorSubclassCUDA::test_tensor_attributes_cuda, test/test_nestedtensor.py::TestNestedTensorSubclassCUDA::test_to_padded_tensor_compile_nt_dim_2_requires_grad_True_cuda_float16, test/test_nestedtensor.py::TestNestedTensorSubclassCUDA::test_to_padded_tensor_compile_nt_dim_3_requires_grad_True_cuda_float16, test/test_nestedtensor.py::TestNestedTensorSubclassCUDA::test_to_padded_tensor_compile_nt_dim_4_requires_grad_True_cuda_float16, test/test_nestedtensor.py::TestNestedTensorSubclassCUDA::test_to_padded_tensor_nt_dim_2_requires_grad_False_cuda_bool, test/test_nestedtensor.py::TestNestedTensorSubclassCUDA::test_to_padded_tensor_nt_dim_2_requires_grad_False_cuda_float16, test/test_nestedtensor.py::TestNestedTensorSubclassCUDA::test_to_padded_tensor_nt_dim_2_requires_grad_True_cuda_float16, test/test_nestedtensor.py::TestNestedTensorSubclassCUDA::test_to_padded_tensor_nt_dim_2_requires_grad_True_cuda_float64, test/test_nestedtensor.py::TestNestedTensorSubclassCUDA::test_to_padded_tensor_nt_dim_3_requires_grad_False_cuda_float16, test/test_nestedtensor.py::TestNestedTensorSubclassCUDA::test_to_padded_tensor_nt_dim_3_requires_grad_False_cuda_float64, test/test_nestedtensor.py::TestNestedTensorSubclassCUDA::test_to_padded_tensor_nt_dim_3_requires_grad_True_cuda_float16, test/test_nestedtensor.py::TestNestedTensorSubclassCUDA::test_to_padded_tensor_nt_dim_4_requires_grad_False_cuda_float16, test/test_nestedtensor.py::TestNestedTensorSubclassCUDA::test_to_padded_tensor_nt_dim_4_requires_grad_False_cuda_float64, test/test_nestedtensor.py::TestNestedTensorSubclassCUDA::test_to_padded_tensor_nt_dim_4_requires_grad_True_cuda_bool, test/test_nestedtensor.py::TestNestedTensorSubclassCUDA::test_to_padded_tensor_nt_dim_4_requires_grad_True_cuda_float16, test/test_nestedtensor.py::TestNestedTensorSubclassCUDA::test_unary_pointwise_cuda, test/test_nestedtensor.py::TestNestedTensorSubclassCUDA::test_unbind_backward_cuda_float16, test/test_nestedtensor.py::TestNestedTensorSubclassCUDA::test_unbind_lengths_cuda, test/test_nestedtensor.py::TestNestedTensorSubclassCUDA::test_unbind_lengths_ragged_idx_0_cuda, test/test_nestedtensor.py::TestNestedTensorSubclassCUDA::test_unbind_lengths_ragged_idx_1_cuda, test/test_nestedtensor.py::TestNestedTensorSubclassCUDA::test_unbind_transpose_ragged_idx_3_cuda, test/test_nestedtensor.py::TestNestedTensorSubclassCUDA::test_unbind_transpose_ragged_idx_last_dim_cuda, test/test_nestedtensor.py::TestNestedTensorSubclassCUDA::test_unsafe_view_cuda, test/test_nestedtensor.py::TestNestedTensorSubclassCUDA::test_views_inherit_ragged_dim_cuda, test/test_nestedtensor.py::TestNestedTensorOpInfoCUDA::test_backward___radd___cuda_float32, test/test_nestedtensor.py::TestNestedTensorOpInfoCUDA::test_backward___rpow___cuda_float32, test/test_nestedtensor.py::TestNestedTensorOpInfoCUDA::test_backward_add_cuda_float32, test/test_nestedtensor.py::TestNestedTensorOpInfoCUDA::test_backward_amax_cuda_float32, test/test_nestedtensor.py::TestNestedTensorOpInfoCUDA::test_backward_bfloat16_cuda_float32, test/test_nestedtensor.py::TestNestedTensorOpInfoCUDA::test_backward_cfloat_cuda_float32, test/test_nestedtensor.py::TestNestedTensorOpInfoCUDA::test_backward_chalf_cuda_float32, test/test_nestedtensor.py::TestNestedTensorOpInfoCUDA::test_backward_clamp_min_cuda_float32, test/test_nestedtensor.py::TestNestedTensorOpInfoCUDA::test_backward_clone_cuda_float32, test/test_nestedtensor.py::TestNestedTensorOpInfoCUDA::test_backward_complex_cuda_float32, test/test_nestedtensor.py::TestNestedTensorOpInfoCUDA::test_backward_conj_physical_cuda_float32, test/test_nestedtensor.py::TestNestedTensorOpInfoCUDA::test_backward_copysign_cuda_float32, test/test_nestedtensor.py::TestNestedTensorOpInfoCUDA::test_backward_cosh_cuda_float32, test/test_nestedtensor.py::TestNestedTensorOpInfoCUDA::test_backward_deg2rad_cuda_float32, test/test_nestedtensor.py::TestNestedTensorOpInfoCUDA::test_backward_digamma_cuda_float32, test/test_nestedtensor.py::TestNestedTensorOpInfoCUDA::test_backward_div_floor_rounding_cuda_float32, test/test_nestedtensor.py::TestNestedTensorOpInfoCUDA::test_backward_div_no_rounding_mode_cuda_float32, test/test_nestedtensor.py::TestNestedTensorOpInfoCUDA::test_backward_erf_cuda_float32, test/test_nestedtensor.py::TestNestedTensorOpInfoCUDA::test_backward_erfc_cuda_float32, test/test_nestedtensor.py::TestNestedTensorOpInfoCUDA::test_backward_exp_cuda_float32, test/test_nestedtensor.py::TestNestedTensorOpInfoCUDA::test_backward_expm1_cuda_float32, test/test_nestedtensor.py::TestNestedTensorOpInfoCUDA::test_backward_frac_cuda_float32, test/test_nestedtensor.py::TestNestedTensorOpInfoCUDA::test_backward_frexp_cuda_float32, test/test_nestedtensor.py::TestNestedTensorOpInfoCUDA::test_backward_hypot_cuda_float32, test/test_nestedtensor.py::TestNestedTensorOpInfoCUDA::test_backward_ldexp_cuda_float32, test/test_nestedtensor.py::TestNestedTensorOpInfoCUDA::test_backward_linalg_vector_norm_cuda_float32, test/test_nestedtensor.py::TestNestedTensorOpInfoCUDA::test_backward_log10_cuda_float32, test/test_nestedtensor.py::TestNestedTensorOpInfoCUDA::test_backward_log_cuda_float32, test/test_nestedtensor.py::TestNestedTensorOpInfoCUDA::test_backward_logit_cuda_float32, test/test_nestedtensor.py::TestNestedTensorOpInfoCUDA::test_backward_masked_logsumexp_cuda_float32, test/test_nestedtensor.py::TestNestedTensorOpInfoCUDA::test_backward_masked_select_cuda_float32, test/test_nestedtensor.py::TestNestedTensorOpInfoCUDA::test_backward_masked_sum_cuda_float32, test/test_nestedtensor.py::TestNestedTensorOpInfoCUDA::test_backward_matmul_cuda_float32, test/test_nestedtensor.py::TestNestedTensorOpInfoCUDA::test_backward_max_binary_cuda_float32, test/test_nestedtensor.py::TestNestedTensorOpInfoCUDA::test_backward_max_reduction_with_dim_cuda_float32, test/test_nestedtensor.py::TestNestedTensorOpInfoCUDA::test_backward_maximum_cuda_float32, test/test_nestedtensor.py::TestNestedTensorOpInfoCUDA::test_backward_mean_cuda_float32, test/test_nestedtensor.py::TestNestedTensorOpInfoCUDA::test_backward_minimum_cuda_float32, test/test_nestedtensor.py::TestNestedTensorOpInfoCUDA::test_backward_mul_cuda_float32, test/test_nestedtensor.py::TestNestedTensorOpInfoCUDA::test_backward_nansum_cuda_float32, test/test_nestedtensor.py::TestNestedTensorOpInfoCUDA::test_backward_nn_functional_celu_cuda_float32, test/test_nestedtensor.py::TestNestedTensorOpInfoCUDA::test_backward_nn_functional_elu_cuda_float32, test/test_nestedtensor.py::TestNestedTensorOpInfoCUDA::test_backward_nn_functional_hardshrink_cuda_float32, test/test_nestedtensor.py::TestNestedTensorOpInfoCUDA::test_backward_nn_functional_hardtanh_cuda_float32, test/test_nestedtensor.py::TestNestedTensorOpInfoCUDA::test_backward_nn_functional_linear_cuda_float32, test/test_nestedtensor.py::TestNestedTensorOpInfoCUDA::test_backward_nn_functional_logsigmoid_cuda_float32, test/test_nestedtensor.py::TestNestedTensorOpInfoCUDA::test_backward_nn_functional_relu6_cuda_float32, test/test_nestedtensor.py::TestNestedTensorOpInfoCUDA::test_backward_nn_functional_relu_cuda_float32, test/test_nestedtensor.py::TestNestedTensorOpInfoCUDA::test_backward_nn_functional_selu_cuda_float32, test/test_nestedtensor.py::TestNestedTensorOpInfoCUDA::test_backward_nn_functional_silu_cuda_float32, test/test_nestedtensor.py::TestNestedTensorOpInfoCUDA::test_backward_nn_functional_softplus_cuda_float32, test/test_nestedtensor.py::TestNestedTensorOpInfoCUDA::test_backward_nn_functional_softshrink_cuda_float32, test/test_nestedtensor.py::TestNestedTensorOpInfoCUDA::test_backward_polar_cuda_float32, test/test_nestedtensor.py::TestNestedTensorOpInfoCUDA::test_backward_polygamma_polygamma_n_2_cuda_float32, test/test_nestedtensor.py::TestNestedTensorOpInfoCUDA::test_backward_polygamma_polygamma_n_4_cuda_float32, test/test_nestedtensor.py::TestNestedTensorOpInfoCUDA::test_backward_real_cuda_float32, test/test_nestedtensor.py::TestNestedTensorOpInfoCUDA::test_backward_reciprocal_cuda_float32, test/test_nestedtensor.py::TestNestedTensorOpInfoCUDA::test_backward_rsqrt_cuda_float32, test/test_nestedtensor.py::TestNestedTensorOpInfoCUDA::test_backward_sign_cuda_float32, test/test_nestedtensor.py::TestNestedTensorOpInfoCUDA::test_backward_special_erfcx_cuda_float32, test/test_nestedtensor.py::TestNestedTensorOpInfoCUDA::test_backward_special_i1_cuda_float32, test/test_nestedtensor.py::TestNestedTensorOpInfoCUDA::test_backward_special_i1e_cuda_float32, test/test_nestedtensor.py::TestNestedTensorOpInfoCUDA::test_backward_special_ndtr_cuda_float32, test/test_nestedtensor.py::TestNestedTensorOpInfoCUDA::test_backward_special_ndtri_cuda_float32, test/test_nestedtensor.py::TestNestedTensorOpInfoCUDA::test_backward_split_cuda_float32, test/test_nestedtensor.py::TestNestedTensorOpInfoCUDA::test_backward_split_with_sizes_cuda_float32, test/test_nestedtensor.py::TestNestedTensorOpInfoCUDA::test_backward_square_cuda_float32, test/test_nestedtensor.py::TestNestedTensorOpInfoCUDA::test_backward_std_cuda_float32, test/test_nestedtensor.py::TestNestedTensorOpInfoCUDA::test_backward_sub_cuda_float32, test/test_nestedtensor.py::TestNestedTensorOpInfoCUDA::test_backward_true_divide_cuda_float32, test/test_nestedtensor.py::TestNestedTensorOpInfoCUDA::test_backward_trunc_cuda_float32, test/test_nestedtensor.py::TestNestedTensorOpInfoCUDA::test_backward_where_cuda_float32, test/test_nestedtensor.py::TestNestedTensorOpInfoCUDA::test_compile_backward___rdiv___cuda_float32, test/test_nestedtensor.py::TestNestedTensorOpInfoCUDA::test_compile_backward_add_cuda_float32, test/test_nestedtensor.py::TestNestedTensorOpInfoCUDA::test_compile_backward_angle_cuda_float32, test/test_nestedtensor.py::TestNestedTensorOpInfoCUDA::test_compile_backward_asinh_cuda_float32, test/test_nestedtensor.py::TestNestedTensorOpInfoCUDA::test_compile_backward_atan2_cuda_float32, test/test_nestedtensor.py::TestNestedTensorOpInfoCUDA::test_compile_backward_clamp_max_cuda_float32, test/test_nestedtensor.py::TestNestedTensorOpInfoCUDA::test_compile_backward_clone_cuda_float32, test/test_nestedtensor.py::TestNestedTensorOpInfoCUDA::test_compile_backward_conj_cuda_float32, test/test_nestedtensor.py::TestNestedTensorOpInfoCUDA::test_compile_backward_copysign_cuda_float32, test/test_nestedtensor.py::TestNestedTensorOpInfoCUDA::test_compile_backward_cosh_cuda_float32, test/test_nestedtensor.py::TestNestedTensorOpInfoCUDA::test_compile_backward_div_trunc_rounding_cuda_float32, test/test_nestedtensor.py::TestNestedTensorOpInfoCUDA::test_compile_backward_erfc_cuda_float32, test/test_nestedtensor.py::TestNestedTensorOpInfoCUDA::test_compile_backward_expm1_cuda_float32, test/test_nestedtensor.py::TestNestedTensorOpInfoCUDA::test_compile_backward_fill_cuda_float32, test/test_nestedtensor.py::TestNestedTensorOpInfoCUDA::test_compile_backward_float_cuda_float32, test/test_nestedtensor.py::TestNestedTensorOpInfoCUDA::test_compile_backward_float_power_cuda_float32, test/test_nestedtensor.py::TestNestedTensorOpInfoCUDA::test_compile_backward_floor_cuda_float32, test/test_nestedtensor.py::TestNestedTensorOpInfoCUDA::test_compile_backward_fmod_cuda_float32, test/test_nestedtensor.py::TestNestedTensorOpInfoCUDA::test_compile_backward_i0_cuda_float32, test/test_nestedtensor.py::TestNestedTensorOpInfoCUDA::test_compile_backward_ldexp_cuda_float32, test/test_nestedtensor.py::TestNestedTensorOpInfoCUDA::test_compile_backward_lgamma_cuda_float32, test/test_nestedtensor.py::TestNestedTensorOpInfoCUDA::test_compile_backward_log_cuda_float32, test/test_nestedtensor.py::TestNestedTensorOpInfoCUDA::test_compile_backward_logaddexp_cuda_float32, test/test_nestedtensor.py::TestNestedTensorOpInfoCUDA::test_compile_backward_masked_amax_cuda_float32, test/test_nestedtensor.py::TestNestedTensorOpInfoCUDA::test_compile_backward_masked_mean_cuda_float32, test/test_nestedtensor.py::TestNestedTensorOpInfoCUDA::test_compile_backward_masked_norm_cuda_float32, test/test_nestedtensor.py::TestNestedTensorOpInfoCUDA::test_compile_backward_masked_prod_cuda_float32, test/test_nestedtensor.py::TestNestedTensorOpInfoCUDA::test_compile_backward_masked_std_cuda_float32, test/test_nestedtensor.py::TestNestedTensorOpInfoCUDA::test_compile_backward_masked_sum_cuda_float32, test/test_nestedtensor.py::TestNestedTensorOpInfoCUDA::test_compile_backward_max_reduction_with_dim_cuda_float32, test/test_nestedtensor.py::TestNestedTensorOpInfoCUDA::test_compile_backward_mean_cuda_float32, test/test_nestedtensor.py::TestNestedTensorOpInfoCUDA::test_compile_backward_min_binary_cuda_float32, test/test_nestedtensor.py::TestNestedTensorOpInfoCUDA::test_compile_backward_mvlgamma_mvlgamma_p_1_cuda_float32, test/test_nestedtensor.py::TestNestedTensorOpInfoCUDA::test_compile_backward_mvlgamma_mvlgamma_p_3_cuda_float32, test/test_nestedtensor.py::TestNestedTensorOpInfoCUDA::test_compile_backward_nn_functional_hardsigmoid_cuda_float32, test/test_nestedtensor.py::TestNestedTensorOpInfoCUDA::test_compile_backward_nn_functional_hardtanh_cuda_float32, test/test_nestedtensor.py::TestNestedTensorOpInfoCUDA::test_compile_backward_nn_functional_logsigmoid_cuda_float32, test/test_nestedtensor.py::TestNestedTensorOpInfoCUDA::test_compile_backward_nn_functional_prelu_cuda_float32, test/test_nestedtensor.py::TestNestedTensorOpInfoCUDA::test_compile_backward_nn_functional_relu_cuda_float32, test/test_nestedtensor.py::TestNestedTensorOpInfoCUDA::test_compile_backward_nn_functional_softplus_cuda_float32, test/test_nestedtensor.py::TestNestedTensorOpInfoCUDA::test_compile_backward_nn_functional_softsign_cuda_float32, test/test_nestedtensor.py::TestNestedTensorOpInfoCUDA::test_compile_backward_nn_functional_threshold_cuda_float32, test/test_nestedtensor.py::TestNestedTensorOpInfoCUDA::test_compile_backward_polar_cuda_float32, test/test_nestedtensor.py::TestNestedTensorOpInfoCUDA::test_compile_backward_polygamma_polygamma_n_1_cuda_float32, test/test_nestedtensor.py::TestNestedTensorOpInfoCUDA::test_compile_backward_polygamma_polygamma_n_2_cuda_float32, test/test_nestedtensor.py::TestNestedTensorOpInfoCUDA::test_compile_backward_polygamma_polygamma_n_4_cuda_float32, test/test_nestedtensor.py::TestNestedTensorOpInfoCUDA::test_compile_backward_pow_cuda_float32, test/test_nestedtensor.py::TestNestedTensorOpInfoCUDA::test_compile_backward_prod_cuda_float32, test/test_nestedtensor.py::TestNestedTensorOpInfoCUDA::test_compile_backward_rad2deg_cuda_float32, test/test_nestedtensor.py::TestNestedTensorOpInfoCUDA::test_compile_backward_real_cuda_float32, test/test_nestedtensor.py::TestNestedTensorOpInfoCUDA::test_compile_backward_reciprocal_cuda_float32, test/test_nestedtensor.py::TestNestedTensorOpInfoCUDA::test_compile_backward_round_cuda_float32, test/test_nestedtensor.py::TestNestedTensorOpInfoCUDA::test_compile_backward_round_decimals_3_cuda_float32, test/test_nestedtensor.py::TestNestedTensorOpInfoCUDA::test_compile_backward_round_decimals_neg_3_cuda_float32, test/test_nestedtensor.py::TestNestedTensorOpInfoCUDA::test_compile_backward_select_cuda_float32, test/test_nestedtensor.py::TestNestedTensorOpInfoCUDA::test_compile_backward_sinc_cuda_float32, test/test_nestedtensor.py::TestNestedTensorOpInfoCUDA::test_compile_backward_special_ndtri_cuda_float32, test/test_nestedtensor.py::TestNestedTensorOpInfoCUDA::test_compile_backward_special_xlog1py_cuda_float32, test/test_nestedtensor.py::TestNestedTensorOpInfoCUDA::test_compile_backward_squeeze_cuda_float32, test/test_nestedtensor.py::TestNestedTensorOpInfoCUDA::test_compile_backward_std_cuda_float32, test/test_nestedtensor.py::TestNestedTensorOpInfoCUDA::test_compile_backward_std_unbiased_cuda_float32, test/test_nestedtensor.py::TestNestedTensorOpInfoCUDA::test_compile_backward_sub_cuda_float32, test/test_nestedtensor.py::TestNestedTensorOpInfoCUDA::test_compile_backward_var_unbiased_cuda_float32, test/test_nestedtensor.py::TestNestedTensorOpInfoCUDA::test_compile_forward___rmul___cuda_float32, test/test_nestedtensor.py::TestNestedTensorOpInfoCUDA::test_compile_forward_acosh_cuda_float32, test/test_nestedtensor.py::TestNestedTensorOpInfoCUDA::test_compile_forward_all_cuda_float32, test/test_nestedtensor.py::TestNestedTensorOpInfoCUDA::test_compile_forward_amax_cuda_float32, test/test_nestedtensor.py::TestNestedTensorOpInfoCUDA::test_compile_forward_angle_cuda_float32, test/test_nestedtensor.py::TestNestedTensorOpInfoCUDA::test_compile_forward_any_cuda_float32, test/test_nestedtensor.py::TestNestedTensorOpInfoCUDA::test_compile_forward_asinh_cuda_float32, test/test_nestedtensor.py::TestNestedTensorOpInfoCUDA::test_compile_forward_atan2_cuda_float32, test/test_nestedtensor.py::TestNestedTensorOpInfoCUDA::test_compile_forward_bmm_cuda_float32, test/test_nestedtensor.py::TestNestedTensorOpInfoCUDA::test_compile_forward_cfloat_cuda_float32, test/test_nestedtensor.py::TestNestedTensorOpInfoCUDA::test_compile_forward_clone_cuda_float32, test/test_nestedtensor.py::TestNestedTensorOpInfoCUDA::test_compile_forward_complex_cuda_float32, test/test_nestedtensor.py::TestNestedTensorOpInfoCUDA::test_compile_forward_count_nonzero_cuda_float32, test/test_nestedtensor.py::TestNestedTensorOpInfoCUDA::test_compile_forward_div_no_rounding_mode_cuda_float32, test/test_nestedtensor.py::TestNestedTensorOpInfoCUDA::test_compile_forward_exp2_cuda_float32, test/test_nestedtensor.py::TestNestedTensorOpInfoCUDA::test_compile_forward_exp_cuda_float32, test/test_nestedtensor.py::TestNestedTensorOpInfoCUDA::test_compile_forward_fill_cuda_float32, test/test_nestedtensor.py::TestNestedTensorOpInfoCUDA::test_compile_forward_float_power_cuda_float32, test/test_nestedtensor.py::TestNestedTensorOpInfoCUDA::test_compile_forward_half_cuda_float32, test/test_nestedtensor.py::TestNestedTensorOpInfoCUDA::test_compile_forward_heaviside_cuda_float32, test/test_nestedtensor.py::TestNestedTensorOpInfoCUDA::test_compile_forward_hypot_cuda_float32, test/test_nestedtensor.py::TestNestedTensorOpInfoCUDA::test_compile_forward_igammac_cuda_float32, test/test_nestedtensor.py::TestNestedTensorOpInfoCUDA::test_compile_forward_int_cuda_float32, test/test_nestedtensor.py::TestNestedTensorOpInfoCUDA::test_compile_forward_isneginf_cuda_float32, test/test_nestedtensor.py::TestNestedTensorOpInfoCUDA::test_compile_forward_isposinf_cuda_float32, test/test_nestedtensor.py::TestNestedTensorOpInfoCUDA::test_compile_forward_isreal_cuda_float32, test/test_nestedtensor.py::TestNestedTensorOpInfoCUDA::test_compile_forward_ldexp_cuda_float32, test/test_nestedtensor.py::TestNestedTensorOpInfoCUDA::test_compile_forward_le_cuda_float32, test/test_nestedtensor.py::TestNestedTensorOpInfoCUDA::test_compile_forward_log10_cuda_float32, test/test_nestedtensor.py::TestNestedTensorOpInfoCUDA::test_compile_forward_log1p_cuda_float32, test/test_nestedtensor.py::TestNestedTensorOpInfoCUDA::test_compile_forward_log2_cuda_float32, test/test_nestedtensor.py::TestNestedTensorOpInfoCUDA::test_compile_forward_logical_not_cuda_float32, test/test_nestedtensor.py::TestNestedTensorOpInfoCUDA::test_compile_forward_logical_or_cuda_float32, test/test_nestedtensor.py::TestNestedTensorOpInfoCUDA::test_compile_forward_logical_xor_cuda_float32, test/test_nestedtensor.py::TestNestedTensorOpInfoCUDA::test_compile_forward_logit_cuda_float32, test/test_nestedtensor.py::TestNestedTensorOpInfoCUDA::test_compile_forward_masked_amax_cuda_float32, test/test_nestedtensor.py::TestNestedTensorOpInfoCUDA::test_compile_forward_masked_amin_cuda_float32, test/test_nestedtensor.py::TestNestedTensorOpInfoCUDA::test_compile_forward_masked_mean_cuda_float32, test/test_nestedtensor.py::TestNestedTensorOpInfoCUDA::test_compile_forward_masked_norm_cuda_float32, test/test_nestedtensor.py::TestNestedTensorOpInfoCUDA::test_compile_forward_masked_sum_cuda_float32, test/test_nestedtensor.py::TestNestedTensorOpInfoCUDA::test_compile_forward_masked_var_cuda_float32, test/test_nestedtensor.py::TestNestedTensorOpInfoCUDA::test_compile_forward_matmul_cuda_float32, test/test_nestedtensor.py::TestNestedTensorOpInfoCUDA::test_compile_forward_max_reduction_with_dim_cuda_float32, test/test_nestedtensor.py::TestNestedTensorOpInfoCUDA::test_compile_forward_min_binary_cuda_float32, test/test_nestedtensor.py::TestNestedTensorOpInfoCUDA::test_compile_forward_mul_cuda_float32, test/test_nestedtensor.py::TestNestedTensorOpInfoCUDA::test_compile_forward_mvlgamma_mvlgamma_p_1_cuda_float32, test/test_nestedtensor.py::TestNestedTensorOpInfoCUDA::test_compile_forward_mvlgamma_mvlgamma_p_5_cuda_float32, test/test_nestedtensor.py::TestNestedTensorOpInfoCUDA::test_compile_forward_nan_to_num_cuda_float32, test/test_nestedtensor.py::TestNestedTensorOpInfoCUDA::test_compile_forward_neg_cuda_float32, test/test_nestedtensor.py::TestNestedTensorOpInfoCUDA::test_compile_forward_nn_functional_celu_cuda_float32, test/test_nestedtensor.py::TestNestedTensorOpInfoCUDA::test_compile_forward_nn_functional_embedding_bag_cuda_float32, test/test_nestedtensor.py::TestNestedTensorOpInfoCUDA::test_compile_forward_nn_functional_embedding_cuda_float32, test/test_nestedtensor.py::TestNestedTensorOpInfoCUDA::test_compile_forward_nn_functional_hardshrink_cuda_float32, test/test_nestedtensor.py::TestNestedTensorOpInfoCUDA::test_compile_forward_nn_functional_linear_cuda_float32, test/test_nestedtensor.py::TestNestedTensorOpInfoCUDA::test_compile_forward_nn_functional_logsigmoid_cuda_float32, test/test_nestedtensor.py::TestNestedTensorOpInfoCUDA::test_compile_forward_nn_functional_mish_cuda_float32, test/test_nestedtensor.py::TestNestedTensorOpInfoCUDA::test_compile_forward_nn_functional_silu_cuda_float32, test/test_nestedtensor.py::TestNestedTensorOpInfoCUDA::test_compile_forward_nn_functional_softshrink_cuda_float32, test/test_nestedtensor.py::TestNestedTensorOpInfoCUDA::test_compile_forward_polygamma_polygamma_n_3_cuda_float32, test/test_nestedtensor.py::TestNestedTensorOpInfoCUDA::test_compile_forward_positive_cuda_float32, test/test_nestedtensor.py::TestNestedTensorOpInfoCUDA::test_compile_forward_real_cuda_float32, test/test_nestedtensor.py::TestNestedTensorOpInfoCUDA::test_compile_forward_round_cuda_float32, test/test_nestedtensor.py::TestNestedTensorOpInfoCUDA::test_compile_forward_round_decimals_0_cuda_float32, test/test_nestedtensor.py::TestNestedTensorOpInfoCUDA::test_compile_forward_select_cuda_float32, test/test_nestedtensor.py::TestNestedTensorOpInfoCUDA::test_compile_forward_short_cuda_float32, test/test_nestedtensor.py::TestNestedTensorOpInfoCUDA::test_compile_forward_sigmoid_cuda_float32, test/test_nestedtensor.py::TestNestedTensorOpInfoCUDA::test_compile_forward_sin_cuda_float32, test/test_nestedtensor.py::TestNestedTensorOpInfoCUDA::test_compile_forward_sinc_cuda_float32, test/test_nestedtensor.py::TestNestedTensorOpInfoCUDA::test_compile_forward_sinh_cuda_float32, test/test_nestedtensor.py::TestNestedTensorOpInfoCUDA::test_compile_forward_special_airy_ai_cuda_float32, test/test_nestedtensor.py::TestNestedTensorOpInfoCUDA::test_compile_forward_special_chebyshev_polynomial_t_cuda_float32, test/test_nestedtensor.py::TestNestedTensorOpInfoCUDA::test_compile_forward_special_chebyshev_polynomial_u_cuda_float32, test/test_nestedtensor.py::TestNestedTensorOpInfoCUDA::test_compile_forward_special_chebyshev_polynomial_v_cuda_float32, test/test_nestedtensor.py::TestNestedTensorOpInfoCUDA::test_compile_forward_special_erfcx_cuda_float32, test/test_nestedtensor.py::TestNestedTensorOpInfoCUDA::test_compile_forward_special_i0e_cuda_float32, test/test_nestedtensor.py::TestNestedTensorOpInfoCUDA::test_compile_forward_special_laguerre_polynomial_l_cuda_float32, test/test_nestedtensor.py::TestNestedTensorOpInfoCUDA::test_compile_forward_special_legendre_polynomial_p_cuda_float32, test/test_nestedtensor.py::TestNestedTensorOpInfoCUDA::test_compile_forward_special_log_ndtr_cuda_float32, test/test_nestedtensor.py::TestNestedTensorOpInfoCUDA::test_compile_forward_special_modified_bessel_i1_cuda_float32, test/test_nestedtensor.py::TestNestedTensorOpInfoCUDA::test_compile_forward_special_scaled_modified_bessel_k0_cuda_float32, test/test_nestedtensor.py::TestNestedTensorOpInfoCUDA::test_compile_forward_special_shifted_chebyshev_polynomial_v_cuda_float32, test/test_nestedtensor.py::TestNestedTensorOpInfoCUDA::test_compile_forward_std_cuda_float32, test/test_nestedtensor.py::TestNestedTensorOpInfoCUDA::test_compile_forward_std_unbiased_cuda_float32, test/test_nestedtensor.py::TestNestedTensorOpInfoCUDA::test_compile_forward_sub_cuda_float32, test/test_nestedtensor.py::TestNestedTensorOpInfoCUDA::test_compile_forward_sum_cuda_float32, test/test_nestedtensor.py::TestNestedTensorOpInfoCUDA::test_compile_forward_tan_cuda_float32, test/test_nestedtensor.py::TestNestedTensorOpInfoCUDA::test_compile_forward_xlogy_cuda_float32, test/test_nestedtensor.py::TestNestedTensorOpInfoCUDA::test_forward___rmod___cuda_float32, test/test_nestedtensor.py::TestNestedTensorOpInfoCUDA::test_forward_atan2_cuda_float32, test/test_nestedtensor.py::TestNestedTensorOpInfoCUDA::test_forward_bmm_cuda_float32, test/test_nestedtensor.py::TestNestedTensorOpInfoCUDA::test_forward_byte_cuda_float32, test/test_nestedtensor.py::TestNestedTensorOpInfoCUDA::test_forward_ceil_cuda_float32, test/test_nestedtensor.py::TestNestedTensorOpInfoCUDA::test_forward_cfloat_cuda_float32, test/test_nestedtensor.py::TestNestedTensorOpInfoCUDA::test_forward_clone_cuda_float32, test/test_nestedtensor.py::TestNestedTensorOpInfoCUDA::test_forward_conj_physical_cuda_float32, test/test_nestedtensor.py::TestNestedTensorOpInfoCUDA::test_forward_copysign_cuda_float32, test/test_nestedtensor.py::TestNestedTensorOpInfoCUDA::test_forward_cos_cuda_float32, test/test_nestedtensor.py::TestNestedTensorOpInfoCUDA::test_forward_div_floor_rounding_cuda_float32, test/test_nestedtensor.py::TestNestedTensorOpInfoCUDA::test_forward_div_trunc_rounding_cuda_float32, test/test_nestedtensor.py::TestNestedTensorOpInfoCUDA::test_forward_double_cuda_float32, test/test_nestedtensor.py::TestNestedTensorOpInfoCUDA::test_forward_erfc_cuda_float32, test/test_nestedtensor.py::TestNestedTensorOpInfoCUDA::test_forward_expm1_cuda_float32, test/test_nestedtensor.py::TestNestedTensorOpInfoCUDA::test_forward_floor_cuda_float32, test/test_nestedtensor.py::TestNestedTensorOpInfoCUDA::test_forward_floor_divide_cuda_float32, test/test_nestedtensor.py::TestNestedTensorOpInfoCUDA::test_forward_fmax_cuda_float32, test/test_nestedtensor.py::TestNestedTensorOpInfoCUDA::test_forward_heaviside_cuda_float32, test/test_nestedtensor.py::TestNestedTensorOpInfoCUDA::test_forward_int_cuda_float32, test/test_nestedtensor.py::TestNestedTensorOpInfoCUDA::test_forward_isinf_cuda_float32, test/test_nestedtensor.py::TestNestedTensorOpInfoCUDA::test_forward_isnan_cuda_float32, test/test_nestedtensor.py::TestNestedTensorOpInfoCUDA::test_forward_isneginf_cuda_float32, test/test_nestedtensor.py::TestNestedTensorOpInfoCUDA::test_forward_isreal_cuda_float32, test/test_nestedtensor.py::TestNestedTensorOpInfoCUDA::test_forward_jiterator_binary_cuda_float32, test/test_nestedtensor.py::TestNestedTensorOpInfoCUDA::test_forward_le_cuda_float32, test/test_nestedtensor.py::TestNestedTensorOpInfoCUDA::test_forward_logical_and_cuda_float32, test/test_nestedtensor.py::TestNestedTensorOpInfoCUDA::test_forward_masked_mean_cuda_float32, test/test_nestedtensor.py::TestNestedTensorOpInfoCUDA::test_forward_masked_std_cuda_float32, test/test_nestedtensor.py::TestNestedTensorOpInfoCUDA::test_forward_matmul_cuda_float32, test/test_nestedtensor.py::TestNestedTensorOpInfoCUDA::test_forward_min_reduction_with_dim_cuda_float32, test/test_nestedtensor.py::TestNestedTensorOpInfoCUDA::test_forward_narrow_cuda_float32, test/test_nestedtensor.py::TestNestedTensorOpInfoCUDA::test_forward_neg_cuda_float32, test/test_nestedtensor.py::TestNestedTensorOpInfoCUDA::test_forward_nn_functional_hardshrink_cuda_float32, test/test_nestedtensor.py::TestNestedTensorOpInfoCUDA::test_forward_nn_functional_hardsigmoid_cuda_float32, test/test_nestedtensor.py::TestNestedTensorOpInfoCUDA::test_forward_nn_functional_hardtanh_cuda_float32, test/test_nestedtensor.py::TestNestedTensorOpInfoCUDA::test_forward_nn_functional_logsigmoid_cuda_float32, test/test_nestedtensor.py::TestNestedTensorOpInfoCUDA::test_forward_nn_functional_mish_cuda_float32, test/test_nestedtensor.py::TestNestedTensorOpInfoCUDA::test_forward_nn_functional_prelu_cuda_float32, test/test_nestedtensor.py::TestNestedTensorOpInfoCUDA::test_forward_nn_functional_relu_cuda_float32, test/test_nestedtensor.py::TestNestedTensorOpInfoCUDA::test_forward_nn_functional_rms_norm_cuda_float32, test/test_nestedtensor.py::TestNestedTensorOpInfoCUDA::test_forward_nn_functional_softshrink_cuda_float32, test/test_nestedtensor.py::TestNestedTensorOpInfoCUDA::test_forward_polar_cuda_float32, test/test_nestedtensor.py::TestNestedTensorOpInfoCUDA::test_forward_polygamma_polygamma_n_1_cuda_float32, test/test_nestedtensor.py::TestNestedTensorOpInfoCUDA::test_forward_polygamma_polygamma_n_3_cuda_float32, test/test_nestedtensor.py::TestNestedTensorOpInfoCUDA::test_forward_real_cuda_float32, test/test_nestedtensor.py::TestNestedTensorOpInfoCUDA::test_forward_reciprocal_cuda_float32, test/test_nestedtensor.py::TestNestedTensorOpInfoCUDA::test_forward_round_decimals_0_cuda_float32, test/test_nestedtensor.py::TestNestedTensorOpInfoCUDA::test_forward_round_decimals_neg_3_cuda_float32, test/test_nestedtensor.py::TestNestedTensorOpInfoCUDA::test_forward_rsqrt_cuda_float32, test/test_nestedtensor.py::TestNestedTensorOpInfoCUDA::test_forward_sgn_cuda_float32, test/test_nestedtensor.py::TestNestedTensorOpInfoCUDA::test_forward_sin_cuda_float32, test/test_nestedtensor.py::TestNestedTensorOpInfoCUDA::test_forward_special_bessel_y0_cuda_float32, test/test_nestedtensor.py::TestNestedTensorOpInfoCUDA::test_forward_special_i1_cuda_float32, test/test_nestedtensor.py::TestNestedTensorOpInfoCUDA::test_forward_special_laguerre_polynomial_l_cuda_float32, test/test_nestedtensor.py::TestNestedTensorOpInfoCUDA::test_forward_special_legendre_polynomial_p_cuda_float32, test/test_nestedtensor.py::TestNestedTensorOpInfoCUDA::test_forward_special_modified_bessel_k1_cuda_float32, test/test_nestedtensor.py::TestNestedTensorOpInfoCUDA::test_forward_special_polygamma_special_polygamma_n_0_cuda_float32, test/test_nestedtensor.py::TestNestedTensorOpInfoCUDA::test_forward_special_scaled_modified_bessel_k0_cuda_float32, test/test_nestedtensor.py::TestNestedTensorOpInfoCUDA::test_forward_special_scaled_modified_bessel_k1_cuda_float32, test/test_nestedtensor.py::TestNestedTensorOpInfoCUDA::test_forward_special_shifted_chebyshev_polynomial_v_cuda_float32, test/test_nestedtensor.py::TestNestedTensorOpInfoCUDA::test_forward_special_spherical_bessel_j0_cuda_float32, test/test_nestedtensor.py::TestNestedTensorOpInfoCUDA::test_forward_special_xlog1py_cuda_float32, test/test_nestedtensor.py::TestNestedTensorOpInfoCUDA::test_forward_special_zeta_cuda_float32, test/test_nestedtensor.py::TestNestedTensorOpInfoCUDA::test_forward_split_cuda_float32, test/test_nestedtensor.py::TestNestedTensorOpInfoCUDA::test_forward_split_with_sizes_cuda_float32, test/test_nestedtensor.py::TestNestedTensorOpInfoCUDA::test_forward_squeeze_cuda_float32, test/test_nestedtensor.py::TestNestedTensorOpInfoCUDA::test_forward_std_cuda_float32, test/test_nestedtensor.py::TestNestedTensorOpInfoCUDA::test_forward_tan_cuda_float32, test/test_nestedtensor.py::TestNestedTensorOpInfoCUDA::test_forward_tanh_cuda_float32, test/test_nestedtensor.py::TestNestedTensorOpInfoCUDA::test_forward_true_divide_cuda_float32, test/test_nestedtensor.py::TestNestedTensorOpInfoCUDA::test_forward_unflatten_cuda_float32, test/test_nestedtensor.py::TestNestedTensorOpInfoCUDA::test_nested_tensor_input_mutation_backward_cuda 2025-12-04T12:22:50.2874828Z 2025-12-04T12:22:50.2875052Z Finished test_nestedtensor 3/3 ... [2025-12-04 12:22:50.251327][9818.193543763], took 10.18min 2025-12-04T12:22:50.2875755Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/test_nestedtensor/test_nestedtensor-3292279b9435c183.xml 2025-12-04T12:22:50.8515283Z Uploading artifacts took 0.47 seconds 2025-12-04T12:22:50.8518417Z Running test_ops 7/9 ... [2025-12-04 12:22:50.851651][9818.793866089] 2025-12-04T12:22:50.8518823Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2025-12-04T12:22:50.8522648Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'test_ops.py', '--shard-id=7', '--num-shards=9', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-12-04 12:22:50.852037] 2025-12-04T12:47:10.4331560Z 2025-12-04T12:47:10.4332196Z test_ops 7/9 was successful, full logs can be found in artifacts with path test/test-reports/test_ops_7.9_4196a3b6f281d326_.log 2025-12-04T12:47:10.5231878Z Running 3837 items in this shard: test/test_ops.py::TestSelfKwarg::test_self_kwargs, test/test_ops.py::TestCommonCUDA::test_compare_cpu_H_cuda_float32, test/test_ops.py::TestCommonCUDA::test_compare_cpu___ror___cuda_int64, test/test_ops.py::TestCommonCUDA::test_compare_cpu___rxor___cuda_int64, test/test_ops.py::TestCommonCUDA::test_compare_cpu__batch_norm_with_update_cuda_float32, test/test_ops.py::TestCommonCUDA::test_compare_cpu__refs_T_cuda_float32, test/test_ops.py::TestCommonCUDA::test_compare_cpu__refs__conversions_double_cuda_float32, test/test_ops.py::TestCommonCUDA::test_compare_cpu__refs_atan2_cuda_float32, test/test_ops.py::TestCommonCUDA::test_compare_cpu__refs_column_stack_cuda_float32, test/test_ops.py::TestCommonCUDA::test_compare_cpu__refs_diagonal_scatter_cuda_float32, test/test_ops.py::TestCommonCUDA::test_compare_cpu__refs_div_trunc_rounding_cuda_float32, test/test_ops.py::TestCommonCUDA::test_compare_cpu__refs_hypot_cuda_float32, test/test_ops.py::TestCommonCUDA::test_compare_cpu__refs_log_normal_cuda_float32, test/test_ops.py::TestCommonCUDA::test_compare_cpu__refs_logspace_tensor_overload_cuda_float32, test/test_ops.py::TestCommonCUDA::test_compare_cpu__refs_new_ones_cuda_float32, test/test_ops.py::TestCommonCUDA::test_compare_cpu__refs_nn_functional_huber_loss_cuda_float32, test/test_ops.py::TestCommonCUDA::test_compare_cpu__refs_nn_functional_leaky_relu_cuda_float32, test/test_ops.py::TestCommonCUDA::test_compare_cpu__refs_nn_functional_margin_ranking_loss_cuda_float32, test/test_ops.py::TestCommonCUDA::test_compare_cpu__refs_nn_functional_softmax_with_dtype_cuda_float32, test/test_ops.py::TestCommonCUDA::test_compare_cpu__refs_renorm_cuda_float32, test/test_ops.py::TestCommonCUDA::test_compare_cpu__refs_std_mean_cuda_float32, test/test_ops.py::TestCommonCUDA::test_compare_cpu_addmm_cuda_float32, test/test_ops.py::TestCommonCUDA::test_compare_cpu_atleast_1d_cuda_float32, test/test_ops.py::TestCommonCUDA::test_compare_cpu_atleast_2d_cuda_float32, test/test_ops.py::TestCommonCUDA::test_compare_cpu_bucketize_cuda_float32, test/test_ops.py::TestCommonCUDA::test_compare_cpu_cfloat_cuda_float32, test/test_ops.py::TestCommonCUDA::test_compare_cpu_cov_cuda_float32, test/test_ops.py::TestCommonCUDA::test_compare_cpu_diagonal_copy_cuda_float32, test/test_ops.py::TestCommonCUDA::test_compare_cpu_expand_cuda_float32, test/test_ops.py::TestCommonCUDA::test_compare_cpu_float_cuda_float32, test/test_ops.py::TestCommonCUDA::test_compare_cpu_geqrf_cuda_float32, test/test_ops.py::TestCommonCUDA::test_compare_cpu_grid_sampler_2d_cuda_float32, test/test_ops.py::TestCommonCUDA::test_compare_cpu_igamma_cuda_float32, test/test_ops.py::TestCommonCUDA::test_compare_cpu_int_cuda_float32, test/test_ops.py::TestCommonCUDA::test_compare_cpu_ldexp_cuda_float32, test/test_ops.py::TestCommonCUDA::test_compare_cpu_linalg_ldl_solve_cuda_float32, test/test_ops.py::TestCommonCUDA::test_compare_cpu_linalg_matrix_power_cuda_float32, test/test_ops.py::TestCommonCUDA::test_compare_cpu_logsumexp_cuda_float32, test/test_ops.py::TestCommonCUDA::test_compare_cpu_mT_cuda_float32, test/test_ops.py::TestCommonCUDA::test_compare_cpu_masked_logaddexp_cuda_float32, test/test_ops.py::TestCommonCUDA::test_compare_cpu_median_cuda_float32, test/test_ops.py::TestCommonCUDA::test_compare_cpu_new_empty_strided_cuda_float32, test/test_ops.py::TestCommonCUDA::test_compare_cpu_new_ones_cuda_float32, test/test_ops.py::TestCommonCUDA::test_compare_cpu_nn_functional_avg_pool2d_cuda_float32, test/test_ops.py::TestCommonCUDA::test_compare_cpu_nn_functional_embedding_bag_cuda_float32, test/test_ops.py::TestCommonCUDA::test_compare_cpu_nn_functional_hinge_embedding_loss_cuda_float32, test/test_ops.py::TestCommonCUDA::test_compare_cpu_nn_functional_huber_loss_cuda_float32, test/test_ops.py::TestCommonCUDA::test_compare_cpu_nn_functional_leaky_relu_cuda_float32, test/test_ops.py::TestCommonCUDA::test_compare_cpu_nn_functional_max_pool2d_cuda_float32, test/test_ops.py::TestCommonCUDA::test_compare_cpu_nn_functional_nll_loss_cuda_float32, test/test_ops.py::TestCommonCUDA::test_compare_cpu_nn_functional_pixel_shuffle_cuda_float32, test/test_ops.py::TestCommonCUDA::test_compare_cpu_nn_functional_pixel_unshuffle_cuda_float32, test/test_ops.py::TestCommonCUDA::test_compare_cpu_nn_functional_relu6_cuda_float32, test/test_ops.py::TestCommonCUDA::test_compare_cpu_nn_functional_softmin_cuda_float32, test/test_ops.py::TestCommonCUDA::test_compare_cpu_reshape_as_cuda_float32, test/test_ops.py::TestCommonCUDA::test_compare_cpu_scalar_tensor_cuda_float32, test/test_ops.py::TestCommonCUDA::test_compare_cpu_scatter_cuda_float32, test/test_ops.py::TestCommonCUDA::test_compare_cpu_softmax_cuda_float32, test/test_ops.py::TestCommonCUDA::test_compare_cpu_split_cuda_float32, test/test_ops.py::TestCommonCUDA::test_compare_cpu_svd_cuda_float32, test/test_ops.py::TestCommonCUDA::test_compare_cpu_take_along_dim_cuda_float32, test/test_ops.py::TestCommonCUDA::test_compare_cpu_unsqueeze_cuda_float32, test/test_ops.py::TestCommonCUDA::test_compare_cpu_view_cuda_float32, test/test_ops.py::TestCommonCUDA::test_complex_half_reference_testing_T_cuda_complex32, test/test_ops.py::TestCommonCUDA::test_complex_half_reference_testing_conj_physical_cuda_complex32, test/test_ops.py::TestCommonCUDA::test_complex_half_reference_testing_empty_cuda_complex32, test/test_ops.py::TestCommonCUDA::test_complex_half_reference_testing_fft_ifft_cuda_complex32, test/test_ops.py::TestCommonCUDA::test_complex_half_reference_testing_new_zeros_cuda_complex32, test/test_ops.py::TestCommonCUDA::test_complex_half_reference_testing_nonzero_static_cuda_complex32, test/test_ops.py::TestCommonCUDA::test_complex_half_reference_testing_ones_cuda_complex32, test/test_ops.py::TestCommonCUDA::test_complex_half_reference_testing_positive_cuda_complex32, test/test_ops.py::TestCommonCUDA::test_complex_half_reference_testing_rand_like_cuda_complex32, test/test_ops.py::TestCommonCUDA::test_complex_half_reference_testing_randn_like_cuda_complex32, test/test_ops.py::TestCommonCUDA::test_complex_half_reference_testing_squeeze_multiple_cuda_complex32, test/test_ops.py::TestCommonCUDA::test_complex_half_reference_testing_tanh_cuda_complex32, test/test_ops.py::TestCommonCUDA::test_complex_half_reference_testing_unflatten_cuda_complex32, test/test_ops.py::TestCommonCUDA::test_complex_half_reference_testing_where_cuda_complex32, test/test_ops.py::TestCommonCUDA::test_dtypes_H_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes__batch_norm_with_update_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes__refs_T_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes__refs_asin_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes__refs_asinh_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes__refs_bitwise_and_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes__refs_cat_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes__refs_clamp_min_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes__refs_diagonal_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes__refs_expm1_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes__refs_fft_ihfft2_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes__refs_fft_ihfftn_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes__refs_igamma_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes__refs_index_select_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes__refs_item_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes__refs_lcm_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes__refs_linspace_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes__refs_log_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes__refs_masked_fill_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes__refs_minimum_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes__refs_ne_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes__refs_nn_functional_channel_shuffle_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes__refs_nn_functional_gelu_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes__refs_nn_functional_pdist_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes__refs_nn_functional_softmin_with_dtype_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes__refs_round_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes__refs_sin_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes__refs_special_i1_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes__refs_special_log_ndtr_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes__refs_special_spherical_bessel_j0_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes__refs_squeeze_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes__refs_vdot_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes__unsafe_masked_index_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes_acosh_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes_allclose_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes_aminmax_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes_as_strided_partial_views_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes_asin_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes_asinh_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes_block_diag_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes_chunk_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes_combinations_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes_complex_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes_cov_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes_diag_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes_diagonal_copy_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes_diagonal_scatter_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes_digamma_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes_empty_strided_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes_expand_as_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes_fft_fftn_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes_fmax_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes_fmod_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes_gather_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes_gt_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes_index_copy_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes_index_fill_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes_index_reduce_amax_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes_isnan_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes_jiterator_binary_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes_kthvalue_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes_ldexp_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes_lerp_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes_linalg_cholesky_ex_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes_logdet_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes_logit_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes_lu_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes_lu_unpack_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes_masked_amax_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes_masked_cumsum_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes_masked_fill_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes_masked_select_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes_max_binary_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes_min_reduction_no_dim_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes_mode_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes_nanmedian_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes_native_layer_norm_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes_new_empty_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes_new_zeros_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes_nn_functional_kl_div_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes_nn_functional_margin_ranking_loss_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes_nn_functional_max_unpool3d_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes_nn_functional_multilabel_margin_loss_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes_nn_functional_rms_norm_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes_nn_functional_softplus_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes_nn_functional_softsign_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes_ormqr_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes_rad2deg_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes_repeat_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes_resolve_neg_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes_round_decimals_neg_3_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes_scatter_reduce_prod_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes_softmax_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes_special_entr_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes_special_hermite_polynomial_he_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes_special_modified_bessel_k0_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes_special_polygamma_special_polygamma_n_0_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes_split_with_sizes_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes_to_sparse_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes_trapezoid_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes_triu_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes_true_divide_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes_unique_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes_unsqueeze_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes_var_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes_var_unbiased_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes_vstack_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes_zero__cuda, test/test_ops.py::TestCommonCUDA::test_errors___rand___cuda, test/test_ops.py::TestCommonCUDA::test_errors_arange_cuda, test/test_ops.py::TestCommonCUDA::test_errors_atan2_cuda, test/test_ops.py::TestCommonCUDA::test_errors_bitwise_and_cuda, test/test_ops.py::TestCommonCUDA::test_errors_cov_cuda, test/test_ops.py::TestCommonCUDA::test_errors_eye_cuda, test/test_ops.py::TestCommonCUDA::test_errors_fft_ifft2_cuda, test/test_ops.py::TestCommonCUDA::test_errors_fft_irfft2_cuda, test/test_ops.py::TestCommonCUDA::test_errors_gather_cuda, test/test_ops.py::TestCommonCUDA::test_errors_index_add_cuda, test/test_ops.py::TestCommonCUDA::test_errors_kthvalue_cuda, test/test_ops.py::TestCommonCUDA::test_errors_lcm_cuda, test/test_ops.py::TestCommonCUDA::test_errors_lt_cuda, test/test_ops.py::TestCommonCUDA::test_errors_movedim_cuda, test/test_ops.py::TestCommonCUDA::test_errors_multinomial_cuda, test/test_ops.py::TestCommonCUDA::test_errors_nn_functional_embedding_cuda, test/test_ops.py::TestCommonCUDA::test_errors_nn_functional_group_norm_cuda, test/test_ops.py::TestCommonCUDA::test_errors_nn_functional_max_pool3d_cuda, test/test_ops.py::TestCommonCUDA::test_errors_nn_functional_multi_margin_loss_cuda, test/test_ops.py::TestCommonCUDA::test_errors_nn_functional_triplet_margin_with_distance_loss_cuda, test/test_ops.py::TestCommonCUDA::test_errors_signal_windows_hann_cuda, test/test_ops.py::TestCommonCUDA::test_errors_sparse_randn_like_layout0_cuda, test/test_ops.py::TestCommonCUDA::test_errors_special_laguerre_polynomial_l_cuda, test/test_ops.py::TestCommonCUDA::test_errors_special_shifted_chebyshev_polynomial_v_cuda, test/test_ops.py::TestCommonCUDA::test_errors_sub_cuda, test/test_ops.py::TestCommonCUDA::test_errors_vdot_cuda, test/test_ops.py::TestCommonCUDA::test_meta_consistency_out_dtype_mismatch__native_batch_norm_legit_cuda_float32, test/test_ops.py::TestCommonCUDA::test_meta_consistency_out_dtype_mismatch_abs_cuda_float32, test/test_ops.py::TestCommonCUDA::test_meta_consistency_out_dtype_mismatch_acosh_cuda_float32, test/test_ops.py::TestCommonCUDA::test_meta_consistency_out_dtype_mismatch_addmm_decomposed_cuda_float32, test/test_ops.py::TestCommonCUDA::test_meta_consistency_out_dtype_mismatch_amin_cuda_float32, test/test_ops.py::TestCommonCUDA::test_meta_consistency_out_dtype_mismatch_baddbmm_cuda_float32, test/test_ops.py::TestCommonCUDA::test_meta_consistency_out_dtype_mismatch_cosh_cuda_float32, test/test_ops.py::TestCommonCUDA::test_meta_consistency_out_dtype_mismatch_div_trunc_rounding_cuda_float32, test/test_ops.py::TestCommonCUDA::test_meta_consistency_out_dtype_mismatch_dot_cuda_float32, test/test_ops.py::TestCommonCUDA::test_meta_consistency_out_dtype_mismatch_fft_fft2_cuda_float32, test/test_ops.py::TestCommonCUDA::test_meta_consistency_out_dtype_mismatch_fft_ifftn_cuda_float32, test/test_ops.py::TestCommonCUDA::test_meta_consistency_out_dtype_mismatch_float_power_cuda_float32, test/test_ops.py::TestCommonCUDA::test_meta_consistency_out_dtype_mismatch_igammac_cuda_float32, test/test_ops.py::TestCommonCUDA::test_meta_consistency_out_dtype_mismatch_ldexp_cuda_float32, test/test_ops.py::TestCommonCUDA::test_meta_consistency_out_dtype_mismatch_linalg_cholesky_cuda_float32, test/test_ops.py::TestCommonCUDA::test_meta_consistency_out_dtype_mismatch_linalg_cross_cuda_float32, test/test_ops.py::TestCommonCUDA::test_meta_consistency_out_dtype_mismatch_linalg_det_cuda_float32, test/test_ops.py::TestCommonCUDA::test_meta_consistency_out_dtype_mismatch_linalg_eigh_cuda_float32, test/test_ops.py::TestCommonCUDA::test_meta_consistency_out_dtype_mismatch_linalg_lstsq_cuda_float32, test/test_ops.py::TestCommonCUDA::test_meta_consistency_out_dtype_mismatch_linalg_pinv_cuda_float32, test/test_ops.py::TestCommonCUDA::test_meta_consistency_out_dtype_mismatch_linalg_solve_triangular_cuda_float32, test/test_ops.py::TestCommonCUDA::test_meta_consistency_out_dtype_mismatch_min_reduction_no_dim_cuda_float32, test/test_ops.py::TestCommonCUDA::test_meta_consistency_out_dtype_mismatch_min_reduction_with_dim_cuda_float32, test/test_ops.py::TestCommonCUDA::test_meta_consistency_out_dtype_mismatch_minimum_cuda_float32, test/test_ops.py::TestCommonCUDA::test_meta_consistency_out_dtype_mismatch_mm_cuda_float32, test/test_ops.py::TestCommonCUDA::test_meta_consistency_out_dtype_mismatch_nansum_cuda_float32, test/test_ops.py::TestCommonCUDA::test_meta_consistency_out_dtype_mismatch_nn_functional_avg_pool3d_cuda_float32, test/test_ops.py::TestCommonCUDA::test_meta_consistency_out_dtype_mismatch_permute_copy_cuda_float32, test/test_ops.py::TestCommonCUDA::test_meta_consistency_out_dtype_mismatch_polygamma_polygamma_n_1_cuda_float32, test/test_ops.py::TestCommonCUDA::test_meta_consistency_out_dtype_mismatch_polygamma_polygamma_n_4_cuda_float32, test/test_ops.py::TestCommonCUDA::test_meta_consistency_out_dtype_mismatch_scatter_reduce_amax_cuda_float32, test/test_ops.py::TestCommonCUDA::test_meta_consistency_out_dtype_mismatch_sinc_cuda_float32, test/test_ops.py::TestCommonCUDA::test_meta_consistency_out_dtype_mismatch_special_hermite_polynomial_h_cuda_float32, test/test_ops.py::TestCommonCUDA::test_meta_consistency_out_dtype_mismatch_special_hermite_polynomial_he_cuda_float32, test/test_ops.py::TestCommonCUDA::test_meta_consistency_out_dtype_mismatch_special_ndtr_cuda_float32, test/test_ops.py::TestCommonCUDA::test_meta_consistency_out_dtype_mismatch_svd_cuda_float32, test/test_ops.py::TestCommonCUDA::test_meta_consistency_out_dtype_mismatch_triangular_solve_cuda_float32, test/test_ops.py::TestCommonCUDA::test_meta_consistency_out_dtype_mismatch_triu_cuda_float32, test/test_ops.py::TestCommonCUDA::test_multiple_devices_T_cuda_int64, test/test_ops.py::TestCommonCUDA::test_multiple_devices___rdiv___cuda_int64, test/test_ops.py::TestCommonCUDA::test_multiple_devices___rpow___cuda_int64, test/test_ops.py::TestCommonCUDA::test_multiple_devices__segment_reduce_lengths_cuda_float32, test/test_ops.py::TestCommonCUDA::test_multiple_devices__unsafe_masked_index_put_accumulate_cuda_float32, test/test_ops.py::TestCommonCUDA::test_multiple_devices_add_cuda_float32, test/test_ops.py::TestCommonCUDA::test_multiple_devices_argsort_cuda_float32, test/test_ops.py::TestCommonCUDA::test_multiple_devices_bool_cuda_int64, test/test_ops.py::TestCommonCUDA::test_multiple_devices_cat_cuda_float32, test/test_ops.py::TestCommonCUDA::test_multiple_devices_cdouble_cuda_int64, test/test_ops.py::TestCommonCUDA::test_multiple_devices_chalf_cuda_int64, test/test_ops.py::TestCommonCUDA::test_multiple_devices_cosh_cuda_int64, test/test_ops.py::TestCommonCUDA::test_multiple_devices_count_nonzero_cuda_int64, test/test_ops.py::TestCommonCUDA::test_multiple_devices_cummax_cuda_float32, test/test_ops.py::TestCommonCUDA::test_multiple_devices_cummin_cuda_float32, test/test_ops.py::TestCommonCUDA::test_multiple_devices_cumsum_cuda_int64, test/test_ops.py::TestCommonCUDA::test_multiple_devices_cumulative_trapezoid_cuda_int64, test/test_ops.py::TestCommonCUDA::test_multiple_devices_diag_cuda_float32, test/test_ops.py::TestCommonCUDA::test_multiple_devices_diagflat_cuda_float32, test/test_ops.py::TestCommonCUDA::test_multiple_devices_dist_cuda_float32, test/test_ops.py::TestCommonCUDA::test_multiple_devices_dstack_cuda_float32, test/test_ops.py::TestCommonCUDA::test_multiple_devices_erfinv_cuda_int64, test/test_ops.py::TestCommonCUDA::test_multiple_devices_expand_as_cuda_int64, test/test_ops.py::TestCommonCUDA::test_multiple_devices_eye_cuda_float32, test/test_ops.py::TestCommonCUDA::test_multiple_devices_fft_hfft_cuda_float32, test/test_ops.py::TestCommonCUDA::test_multiple_devices_fft_ifft_cuda_int64, test/test_ops.py::TestCommonCUDA::test_multiple_devices_fft_ihfftn_cuda_float32, test/test_ops.py::TestCommonCUDA::test_multiple_devices_fft_irfft2_cuda_int64, test/test_ops.py::TestCommonCUDA::test_multiple_devices_fft_irfftn_cuda_int64, test/test_ops.py::TestCommonCUDA::test_multiple_devices_fft_rfftn_cuda_int64, test/test_ops.py::TestCommonCUDA::test_multiple_devices_flip_cuda_int64, test/test_ops.py::TestCommonCUDA::test_multiple_devices_fmax_cuda_int64, test/test_ops.py::TestCommonCUDA::test_multiple_devices_frac_cuda_float32, test/test_ops.py::TestCommonCUDA::test_multiple_devices_half_cuda_float32, test/test_ops.py::TestCommonCUDA::test_multiple_devices_hash_tensor_cuda_float32, test/test_ops.py::TestCommonCUDA::test_multiple_devices_hash_tensor_cuda_int64, test/test_ops.py::TestCommonCUDA::test_multiple_devices_igamma_cuda_float32, test/test_ops.py::TestCommonCUDA::test_multiple_devices_isin_cuda_float32, test/test_ops.py::TestCommonCUDA::test_multiple_devices_jiterator_binary_cuda_int64, test/test_ops.py::TestCommonCUDA::test_multiple_devices_jiterator_binary_return_by_ref_cuda_int64, test/test_ops.py::TestCommonCUDA::test_multiple_devices_kthvalue_cuda_float32, test/test_ops.py::TestCommonCUDA::test_multiple_devices_linalg_det_cuda_float32, test/test_ops.py::TestCommonCUDA::test_multiple_devices_linalg_lu_factor_ex_cuda_float32, test/test_ops.py::TestCommonCUDA::test_multiple_devices_log10_cuda_int64, test/test_ops.py::TestCommonCUDA::test_multiple_devices_logaddexp_cuda_float32, test/test_ops.py::TestCommonCUDA::test_multiple_devices_logical_not_cuda_int64, test/test_ops.py::TestCommonCUDA::test_multiple_devices_logical_or_cuda_int64, test/test_ops.py::TestCommonCUDA::test_multiple_devices_logspace_tensor_overload_cuda_int64, test/test_ops.py::TestCommonCUDA::test_multiple_devices_mT_cuda_float32, test/test_ops.py::TestCommonCUDA::test_multiple_devices_masked_cumprod_cuda_float32, test/test_ops.py::TestCommonCUDA::test_multiple_devices_masked_cumsum_cuda_float32, test/test_ops.py::TestCommonCUDA::test_multiple_devices_masked_mean_cuda_float32, test/test_ops.py::TestCommonCUDA::test_multiple_devices_masked_scatter_cuda_float32, test/test_ops.py::TestCommonCUDA::test_multiple_devices_masked_scatter_cuda_int64, test/test_ops.py::TestCommonCUDA::test_multiple_devices_masked_var_cuda_float32, test/test_ops.py::TestCommonCUDA::test_multiple_devices_masked_var_cuda_int64, test/test_ops.py::TestCommonCUDA::test_multiple_devices_matrix_exp_cuda_float32, test/test_ops.py::TestCommonCUDA::test_multiple_devices_min_reduction_no_dim_cuda_float32, test/test_ops.py::TestCommonCUDA::test_multiple_devices_minimum_cuda_float32, test/test_ops.py::TestCommonCUDA::test_multiple_devices_minimum_cuda_int64, test/test_ops.py::TestCommonCUDA::test_multiple_devices_movedim_cuda_int64, test/test_ops.py::TestCommonCUDA::test_multiple_devices_msort_cuda_int64, test/test_ops.py::TestCommonCUDA::test_multiple_devices_mv_cuda_float32, test/test_ops.py::TestCommonCUDA::test_multiple_devices_nanquantile_cuda_float32, test/test_ops.py::TestCommonCUDA::test_multiple_devices_nansum_cuda_float32, test/test_ops.py::TestCommonCUDA::test_multiple_devices_nn_functional_batch_norm_without_cudnn_cuda_float32, test/test_ops.py::TestCommonCUDA::test_multiple_devices_nn_functional_conv2d_cuda_float32, test/test_ops.py::TestCommonCUDA::test_multiple_devices_nn_functional_conv_transpose2d_cuda_float32, test/test_ops.py::TestCommonCUDA::test_multiple_devices_nn_functional_ctc_loss_cuda_float32, test/test_ops.py::TestCommonCUDA::test_multiple_devices_nn_functional_hardsigmoid_cuda_float32, test/test_ops.py::TestCommonCUDA::test_multiple_devices_nn_functional_hardtanh_cuda_int64, test/test_ops.py::TestCommonCUDA::test_multiple_devices_nn_functional_layer_norm_cuda_float32, test/test_ops.py::TestCommonCUDA::test_multiple_devices_nn_functional_margin_ranking_loss_cuda_float32, test/test_ops.py::TestCommonCUDA::test_multiple_devices_nn_functional_margin_ranking_loss_cuda_int64, test/test_ops.py::TestCommonCUDA::test_multiple_devices_nn_functional_max_unpool1d_grad_cuda_float32, test/test_ops.py::TestCommonCUDA::test_multiple_devices_nn_functional_max_unpool2d_cuda_float32, test/test_ops.py::TestCommonCUDA::test_multiple_devices_nn_functional_normalize_cuda_float32, test/test_ops.py::TestCommonCUDA::test_multiple_devices_nn_functional_pairwise_distance_cuda_int64, test/test_ops.py::TestCommonCUDA::test_multiple_devices_nn_functional_poisson_nll_loss_cuda_int64, test/test_ops.py::TestCommonCUDA::test_multiple_devices_nn_functional_silu_cuda_float32, test/test_ops.py::TestCommonCUDA::test_multiple_devices_nn_functional_softmin_with_dtype_cuda_int64, test/test_ops.py::TestCommonCUDA::test_multiple_devices_nn_functional_upsample_bilinear_cuda_float32, test/test_ops.py::TestCommonCUDA::test_multiple_devices_nonzero_cuda_float32, test/test_ops.py::TestCommonCUDA::test_multiple_devices_polar_cuda_float32, test/test_ops.py::TestCommonCUDA::test_multiple_devices_polygamma_polygamma_n_0_cuda_float32, test/test_ops.py::TestCommonCUDA::test_multiple_devices_polygamma_polygamma_n_1_cuda_int64, test/test_ops.py::TestCommonCUDA::test_multiple_devices_polygamma_polygamma_n_3_cuda_int64, test/test_ops.py::TestCommonCUDA::test_multiple_devices_positive_cuda_int64, test/test_ops.py::TestCommonCUDA::test_multiple_devices_put_cuda_int64, test/test_ops.py::TestCommonCUDA::test_multiple_devices_rad2deg_cuda_int64, test/test_ops.py::TestCommonCUDA::test_multiple_devices_ravel_cuda_int64, test/test_ops.py::TestCommonCUDA::test_multiple_devices_real_cuda_int64, test/test_ops.py::TestCommonCUDA::test_multiple_devices_reshape_as_cuda_int64, test/test_ops.py::TestCommonCUDA::test_multiple_devices_resize__cuda_int64, test/test_ops.py::TestCommonCUDA::test_multiple_devices_round_decimals_0_cuda_float32, test/test_ops.py::TestCommonCUDA::test_multiple_devices_scatter_cuda_int64, test/test_ops.py::TestCommonCUDA::test_multiple_devices_scatter_reduce_amax_cuda_int64, test/test_ops.py::TestCommonCUDA::test_multiple_devices_searchsorted_cuda_int64, test/test_ops.py::TestCommonCUDA::test_multiple_devices_select_cuda_float32, test/test_ops.py::TestCommonCUDA::test_multiple_devices_sigmoid_cuda_float32, test/test_ops.py::TestCommonCUDA::test_multiple_devices_sigmoid_cuda_int64, test/test_ops.py::TestCommonCUDA::test_multiple_devices_sign_cuda_float32, test/test_ops.py::TestCommonCUDA::test_multiple_devices_signal_windows_hamming_cuda_float32, test/test_ops.py::TestCommonCUDA::test_multiple_devices_special_chebyshev_polynomial_t_cuda_float32, test/test_ops.py::TestCommonCUDA::test_multiple_devices_special_chebyshev_polynomial_w_cuda_int64, test/test_ops.py::TestCommonCUDA::test_multiple_devices_special_hermite_polynomial_h_cuda_int64, test/test_ops.py::TestCommonCUDA::test_multiple_devices_special_hermite_polynomial_he_cuda_float32, test/test_ops.py::TestCommonCUDA::test_multiple_devices_special_modified_bessel_k0_cuda_float32, test/test_ops.py::TestCommonCUDA::test_multiple_devices_special_modified_bessel_k0_cuda_int64, test/test_ops.py::TestCommonCUDA::test_multiple_devices_special_scaled_modified_bessel_k0_cuda_float32, test/test_ops.py::TestCommonCUDA::test_multiple_devices_special_spherical_bessel_j0_cuda_int64, test/test_ops.py::TestCommonCUDA::test_multiple_devices_split_with_sizes_cuda_int64, test/test_ops.py::TestCommonCUDA::test_multiple_devices_sqrt_cuda_int64, test/test_ops.py::TestCommonCUDA::test_multiple_devices_squeeze_cuda_int64, test/test_ops.py::TestCommonCUDA::test_multiple_devices_std_cuda_float32, test/test_ops.py::TestCommonCUDA::test_multiple_devices_t_copy_cuda_float32, test/test_ops.py::TestCommonCUDA::test_multiple_devices_take_cuda_float32, test/test_ops.py::TestCommonCUDA::test_multiple_devices_tan_cuda_float32, test/test_ops.py::TestCommonCUDA::test_multiple_devices_to_cuda_float32, test/test_ops.py::TestCommonCUDA::test_multiple_devices_triangular_solve_cuda_float32, test/test_ops.py::TestCommonCUDA::test_multiple_devices_triu_cuda_int64, test/test_ops.py::TestCommonCUDA::test_multiple_devices_trunc_cuda_int64, test/test_ops.py::TestCommonCUDA::test_multiple_devices_unbind_copy_cuda_float32, test/test_ops.py::TestCommonCUDA::test_multiple_devices_unfold_copy_cuda_int64, test/test_ops.py::TestCommonCUDA::test_multiple_devices_unsafe_split_cuda_int64, test/test_ops.py::TestCommonCUDA::test_multiple_devices_var_mean_unbiased_cuda_float32, test/test_ops.py::TestCommonCUDA::test_multiple_devices_view_copy_cuda_float32, test/test_ops.py::TestCommonCUDA::test_multiple_devices_vsplit_cuda_int64, test/test_ops.py::TestCommonCUDA::test_multiple_devices_zero__cuda_int64, test/test_ops.py::TestCommonCUDA::test_non_standard_bool_values___getitem___cuda_bool, test/test_ops.py::TestCommonCUDA::test_non_standard_bool_values__unsafe_masked_index_cuda_bool, test/test_ops.py::TestCommonCUDA::test_non_standard_bool_values_abs_cuda_bool, test/test_ops.py::TestCommonCUDA::test_non_standard_bool_values_all_cuda_bool, test/test_ops.py::TestCommonCUDA::test_non_standard_bool_values_any_cuda_bool, test/test_ops.py::TestCommonCUDA::test_non_standard_bool_values_as_strided_scatter_cuda_bool, test/test_ops.py::TestCommonCUDA::test_non_standard_bool_values_atan_cuda_bool, test/test_ops.py::TestCommonCUDA::test_non_standard_bool_values_clamp_min_cuda_bool, test/test_ops.py::TestCommonCUDA::test_non_standard_bool_values_contiguous_cuda_bool, test/test_ops.py::TestCommonCUDA::test_non_standard_bool_values_diagonal_copy_cuda_bool, test/test_ops.py::TestCommonCUDA::test_non_standard_bool_values_diff_cuda_bool, test/test_ops.py::TestCommonCUDA::test_non_standard_bool_values_div_no_rounding_mode_cuda_bool, test/test_ops.py::TestCommonCUDA::test_non_standard_bool_values_empty_strided_cuda_bool, test/test_ops.py::TestCommonCUDA::test_non_standard_bool_values_fft_fft2_cuda_bool, test/test_ops.py::TestCommonCUDA::test_non_standard_bool_values_fft_ifft2_cuda_bool, test/test_ops.py::TestCommonCUDA::test_non_standard_bool_values_fft_rfft2_cuda_bool, test/test_ops.py::TestCommonCUDA::test_non_standard_bool_values_fliplr_cuda_bool, test/test_ops.py::TestCommonCUDA::test_non_standard_bool_values_half_cuda_bool, test/test_ops.py::TestCommonCUDA::test_non_standard_bool_values_heaviside_cuda_bool, test/test_ops.py::TestCommonCUDA::test_non_standard_bool_values_i0_cuda_bool, test/test_ops.py::TestCommonCUDA::test_non_standard_bool_values_isposinf_cuda_bool, test/test_ops.py::TestCommonCUDA::test_non_standard_bool_values_jiterator_4inputs_with_extra_args_cuda_bool, test/test_ops.py::TestCommonCUDA::test_non_standard_bool_values_jiterator_binary_cuda_bool, test/test_ops.py::TestCommonCUDA::test_non_standard_bool_values_jiterator_unary_cuda_bool, test/test_ops.py::TestCommonCUDA::test_non_standard_bool_values_linalg_diagonal_cuda_bool, test/test_ops.py::TestCommonCUDA::test_non_standard_bool_values_log_cuda_bool, test/test_ops.py::TestCommonCUDA::test_non_standard_bool_values_maximum_cuda_bool, test/test_ops.py::TestCommonCUDA::test_non_standard_bool_values_mode_cuda_bool, test/test_ops.py::TestCommonCUDA::test_non_standard_bool_values_movedim_cuda_bool, test/test_ops.py::TestCommonCUDA::test_non_standard_bool_values_new_empty_strided_cuda_bool, test/test_ops.py::TestCommonCUDA::test_non_standard_bool_values_ones_cuda_bool, test/test_ops.py::TestCommonCUDA::test_non_standard_bool_values_outer_cuda_bool, test/test_ops.py::TestCommonCUDA::test_non_standard_bool_values_polygamma_polygamma_n_0_cuda_bool, test/test_ops.py::TestCommonCUDA::test_non_standard_bool_values_polygamma_polygamma_n_3_cuda_bool, test/test_ops.py::TestCommonCUDA::test_non_standard_bool_values_polygamma_polygamma_n_4_cuda_bool, test/test_ops.py::TestCommonCUDA::test_non_standard_bool_values_rsqrt_cuda_bool, test/test_ops.py::TestCommonCUDA::test_non_standard_bool_values_scatter_add_cuda_bool, test/test_ops.py::TestCommonCUDA::test_non_standard_bool_values_special_airy_ai_cuda_bool, test/test_ops.py::TestCommonCUDA::test_non_standard_bool_values_special_chebyshev_polynomial_t_cuda_bool, test/test_ops.py::TestCommonCUDA::test_non_standard_bool_values_special_ndtr_cuda_bool, test/test_ops.py::TestCommonCUDA::test_non_standard_bool_values_sqrt_cuda_bool, test/test_ops.py::TestCommonCUDA::test_non_standard_bool_values_transpose_cuda_bool, test/test_ops.py::TestCommonCUDA::test_non_standard_bool_values_triu_cuda_bool, test/test_ops.py::TestCommonCUDA::test_non_standard_bool_values_unflatten_cuda_bool, test/test_ops.py::TestCommonCUDA::test_non_standard_bool_values_unfold_copy_cuda_bool, test/test_ops.py::TestCommonCUDA::test_non_standard_bool_values_unsafe_split_cuda_bool, test/test_ops.py::TestCommonCUDA::test_non_standard_bool_values_unsqueeze_copy_cuda_bool, test/test_ops.py::TestCommonCUDA::test_non_standard_bool_values_view_cuda_bool, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples___rdiv___cuda_float32, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples___rmatmul___cuda_float32, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples___rmod___cuda_int64, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples___rmul___cuda_float32, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples__chunk_cat_cuda_float32, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples__unsafe_masked_index_cuda_float32, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_abs_cuda_int64, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_amin_cuda_int64, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_as_strided_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_as_strided_cuda_float32, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_as_strided_cuda_int64, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_as_strided_scatter_cuda_int64, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_asinh_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_asinh_cuda_int64, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_atanh_cuda_float32, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_atanh_cuda_int64, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_broadcast_shapes_cuda_float32, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_clamp_min_cuda_int64, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_column_stack_cuda_float32, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_combinations_cuda_float32, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_complex_cuda_float32, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_conj_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_copysign_cuda_float32, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_cumulative_trapezoid_cuda_int64, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_diag_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_diff_cuda_int64, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_dot_cuda_float32, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_erf_cuda_float32, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_erf_cuda_int64, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_expm1_cuda_float32, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_fft_fft_cuda_float32, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_fft_hfft2_cuda_int64, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_fft_hfft_cuda_int64, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_fft_ifft_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_fft_ifft_cuda_int64, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_fft_ihfftn_cuda_int64, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_fft_rfft2_cuda_int64, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_floor_divide_cuda_float32, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_fmin_cuda_int64, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_gather_cuda_float32, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_ge_cuda_int64, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_gradient_cuda_float32, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_heaviside_cuda_float32, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_histc_cuda_float32, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_hstack_cuda_float32, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_index_copy_cuda_float32, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_index_fill_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_index_reduce_amax_cuda_float32, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_index_reduce_amax_cuda_int64, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_int_cuda_int64, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_item_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_jiterator_4inputs_with_extra_args_cuda_float32, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_jiterator_4inputs_with_extra_args_cuda_int64, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_jiterator_unary_cuda_int64, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_lerp_cuda_float32, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_linalg_diagonal_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_linalg_eigh_cuda_float32, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_linalg_lstsq_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_linalg_lu_factor_cuda_float32, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_linalg_lu_solve_cuda_float32, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_linalg_norm_subgradients_at_zero_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_linalg_pinv_singular_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_linalg_pinv_singular_cuda_float32, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_linalg_svd_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_linalg_svd_cuda_float32, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_log2_cuda_int64, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_logcumsumexp_cuda_float32, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_logdet_cuda_float32, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_masked_argmax_cuda_float32, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_masked_cumprod_cuda_int64, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_masked_log_softmax_cuda_float32, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_masked_median_cuda_float32, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_masked_scatter_cuda_float32, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_masked_select_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_masked_sum_cuda_float32, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_masked_sum_cuda_int64, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_masked_var_cuda_int64, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_max_reduction_no_dim_cuda_int64, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_max_reduction_with_dim_cuda_int64, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_median_cuda_float32, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_meshgrid_list_of_tensors_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_nanmedian_cuda_int64, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_narrow_cuda_float32, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_ne_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_new_empty_strided_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_new_empty_strided_cuda_float32, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_new_ones_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_nn_functional_binary_cross_entropy_cuda_float32, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_nn_functional_ctc_loss_cuda_float32, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_nn_functional_dropout_cuda_float32, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_nn_functional_feature_alpha_dropout_without_train_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_nn_functional_feature_alpha_dropout_without_train_cuda_int64, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_nn_functional_fractional_max_pool2d_cuda_float32, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_nn_functional_glu_cuda_float32, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_nn_functional_group_norm_cuda_float32, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_nn_functional_interpolate_trilinear_cuda_float32, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_nn_functional_kl_div_cuda_float32, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_nn_functional_local_response_norm_cuda_float32, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_nn_functional_max_unpool1d_cuda_float32, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_nn_functional_max_unpool1d_grad_cuda_float32, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_nn_functional_max_unpool3d_grad_cuda_float32, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_nn_functional_multi_head_attention_forward_cuda_float32, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_nn_functional_pad_reflect_cuda_int64, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_nn_functional_pairwise_distance_cuda_float32, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_nn_functional_pairwise_distance_cuda_int64, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_nn_functional_pixel_shuffle_cuda_float32, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_nn_functional_softmin_with_dtype_cuda_int64, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_nn_functional_softplus_cuda_float32, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_nn_functional_softsign_cuda_float32, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_nonzero_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_nonzero_static_cuda_int64, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_norm_cuda_float32, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_normal_number_mean_cuda_float32, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_pca_lowrank_cuda_float32, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_permute_copy_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_pinverse_cuda_float32, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_polygamma_polygamma_n_1_cuda_float32, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_positive_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_put_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_qr_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_remainder_cuda_float32, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_resize__cuda_int64, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_resolve_conj_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_resolve_conj_cuda_float32, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_scatter_add_cuda_float32, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_scatter_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_select_cuda_float32, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_sgn_cuda_int64, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_signal_windows_bartlett_cuda_float32, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_signbit_cuda_int64, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_sinh_cuda_float32, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_slice_cuda_float32, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_sort_cuda_int64, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_special_bessel_j0_cuda_int64, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_special_bessel_y1_cuda_int64, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_special_shifted_chebyshev_polynomial_t_cuda_int64, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_special_spherical_bessel_j0_cuda_int64, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_split_with_sizes_copy_cuda_float32, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_split_with_sizes_copy_cuda_int64, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_sqrt_cuda_int64, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_squeeze_cuda_int64, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_squeeze_multiple_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_std_mean_cuda_float32, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_std_unbiased_cuda_float32, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_stft_cuda_float32, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_sum_to_size_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_svd_cuda_float32, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_t_cuda_int64, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_take_cuda_float32, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_tan_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_tanh_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_tensor_split_cuda_float32, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_tile_cuda_int64, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_to_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_transpose_copy_cuda_float32, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_tril_cuda_int64, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_triu_cuda_int64, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_trunc_cuda_float32, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_unique_consecutive_cuda_int64, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_unique_cuda_float32, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_unravel_index_cuda_int64, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_var_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_view_as_cuda_float32, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_zeros_cuda_float32, test/test_ops.py::TestCommonCUDA::test_numpy_ref_allclose_cuda_float64, test/test_ops.py::TestCommonCUDA::test_numpy_ref_argwhere_cuda_float64, test/test_ops.py::TestCommonCUDA::test_numpy_ref_broadcast_tensors_cuda_int64, test/test_ops.py::TestCommonCUDA::test_numpy_ref_equal_cuda_int64, test/test_ops.py::TestCommonCUDA::test_numpy_ref_item_cuda_int64, test/test_ops.py::TestCommonCUDA::test_numpy_ref_linalg_cross_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_numpy_ref_linalg_vander_cuda_int64, test/test_ops.py::TestCommonCUDA::test_numpy_ref_meshgrid_variadic_tensors_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_numpy_ref_nn_functional_conv_transpose2d_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_numpy_ref_nn_functional_gelu_cuda_float64, test/test_ops.py::TestCommonCUDA::test_numpy_ref_nn_functional_smooth_l1_loss_cuda_float64, test/test_ops.py::TestCommonCUDA::test_numpy_ref_signal_windows_cosine_cuda_float64, test/test_ops.py::TestCommonCUDA::test_numpy_ref_squeeze_multiple_cuda_int64, test/test_ops.py::TestCommonCUDA::test_numpy_ref_view_copy_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_out_T_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out___getitem___cuda_float32, test/test_ops.py::TestCommonCUDA::test_out___rand___cuda_int64, test/test_ops.py::TestCommonCUDA::test_out__native_batch_norm_legit_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out__refs__conversions_bfloat16_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out__refs_addr_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out__refs_all_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out__refs_as_strided_copy_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out__refs_atan2_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out__refs_atleast_1d_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out__refs_constant_pad_nd_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out__refs_copysign_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out__refs_count_nonzero_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out__refs_cumprod_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out__refs_digamma_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out__refs_dot_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out__refs_dstack_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out__refs_erfinv_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out__refs_exp_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out__refs_eye_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out__refs_hsplit_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out__refs_isnan_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out__refs_le_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out__refs_linalg_matrix_norm_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out__refs_linspace_tensor_overload_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out__refs_logical_not_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out__refs_meshgrid_list_of_tensors_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out__refs_narrow_copy_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out__refs_nn_functional_group_norm_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out__refs_prod_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out__refs_sinc_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out__refs_special_log_softmax_with_dtype_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out__refs_special_multigammaln_mvlgamma_p_1_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out__refs_special_spherical_bessel_j0_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out__refs_special_xlog1py_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out__refs_special_zeta_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out__refs_unsqueeze_copy_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out__refs_vdot_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out__refs_vsplit_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_acosh_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_addmm_decomposed_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_amax_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_aminmax_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_as_strided_copy_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_asinh_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_atleast_3d_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_bernoulli_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_bincount_cuda_int64, test/test_ops.py::TestCommonCUDA::test_out_bitwise_left_shift_cuda_int64, test/test_ops.py::TestCommonCUDA::test_out_block_diag_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_cholesky_inverse_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_complex_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_deg2rad_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_empty_like_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_fft_ihfft2_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_fft_irfft2_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_fft_rfftn_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_isinf_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_jiterator_unary_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_kron_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_linalg_householder_product_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_linalg_lu_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_linalg_qr_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_linalg_vecdot_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_lu_unpack_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_mT_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_masked_median_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_masked_normalize_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_masked_prod_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_matmul_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_mul_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_mv_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_mvlgamma_mvlgamma_p_5_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_nan_to_num_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_nanmedian_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_new_full_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_nn_functional_alpha_dropout_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_nn_functional_dropout_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_nn_functional_fractional_max_pool3d_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_nn_functional_glu_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_nn_functional_group_norm_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_nn_functional_layer_norm_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_nn_functional_max_pool1d_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_nn_functional_multi_margin_loss_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_nn_functional_multilabel_margin_loss_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_nn_functional_pad_replicate_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_nn_functional_softmin_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_nn_functional_softmin_with_dtype_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_nn_functional_triplet_margin_loss_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_nn_functional_unfold_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_positive_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_put_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_reciprocal_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_repeat_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_requires_grad_error_acos_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_requires_grad_error_as_strided_copy_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_out_requires_grad_error_atanh_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_out_requires_grad_error_baddbmm_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_out_requires_grad_error_cat_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_requires_grad_error_cholesky_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_requires_grad_error_cholesky_solve_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_out_requires_grad_error_cumsum_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_requires_grad_error_diagonal_copy_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_out_requires_grad_error_fft_fft_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_out_requires_grad_error_fft_fft_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_requires_grad_error_fft_hfft2_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_out_requires_grad_error_fft_hfft2_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_requires_grad_error_fft_irfft2_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_requires_grad_error_full_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_requires_grad_error_gather_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_out_requires_grad_error_i0_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_requires_grad_error_index_add_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_out_requires_grad_error_ldexp_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_requires_grad_error_linalg_cholesky_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_out_requires_grad_error_linalg_cond_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_out_requires_grad_error_linalg_eigh_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_out_requires_grad_error_linalg_eigvalsh_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_requires_grad_error_linalg_lu_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_requires_grad_error_linalg_lu_factor_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_out_requires_grad_error_linalg_matrix_power_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_out_requires_grad_error_linalg_matrix_power_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_requires_grad_error_linalg_solve_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_out_requires_grad_error_linalg_vecdot_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_out_requires_grad_error_log2_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_out_requires_grad_error_logcumsumexp_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_out_requires_grad_error_min_reduction_with_dim_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_requires_grad_error_mm_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_out_requires_grad_error_mul_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_requires_grad_error_nanmean_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_out_requires_grad_error_neg_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_requires_grad_error_nn_functional_normalize_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_out_requires_grad_error_normal_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_requires_grad_error_polygamma_polygamma_n_2_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_requires_grad_error_pow_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_requires_grad_error_sgn_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_out_requires_grad_error_sinc_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_requires_grad_error_slice_scatter_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_requires_grad_error_take_along_dim_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_requires_grad_error_tensordot_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_out_requires_grad_error_transpose_copy_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_requires_grad_error_unbind_copy_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_out_round_decimals_neg_3_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_rsqrt_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_scalar_tensor_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_sgn_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_signal_windows_blackman_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_sinc_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_sort_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_special_chebyshev_polynomial_t_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_special_i1e_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_special_laguerre_polynomial_l_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_special_shifted_chebyshev_polynomial_t_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_special_shifted_chebyshev_polynomial_w_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_special_spherical_bessel_j0_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_squeeze_copy_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_stack_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_std_mean_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_tile_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_torch__scaled_mm_v2_cuda_float8_e4m3fn, test/test_ops.py::TestCommonCUDA::test_out_torch_ops_aten__safe_softmax_default_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_transpose_copy_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_triu_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_unsafe_chunk_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_view_as_complex_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_warning___rand___cuda, test/test_ops.py::TestCommonCUDA::test_out_warning___rdiv___cuda, test/test_ops.py::TestCommonCUDA::test_out_warning___rmul___cuda, test/test_ops.py::TestCommonCUDA::test_out_warning___ror___cuda, test/test_ops.py::TestCommonCUDA::test_out_warning__refs__conversions_bfloat16_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning__refs__conversions_chalf_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning__refs__conversions_char_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning__refs__conversions_complex_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning__refs_acosh_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning__refs_arange_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning__refs_broadcast_to_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning__refs_clamp_min_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning__refs_conj_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning__refs_cumprod_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning__refs_div_floor_rounding_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning__refs_dsplit_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning__refs_erfinv_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning__refs_fft_hfft_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning__refs_fft_hfftn_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning__refs_fft_ifft_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning__refs_fft_rfftn_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning__refs_fmod_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning__refs_linalg_matrix_norm_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning__refs_linspace_tensor_overload_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning__refs_meshgrid_list_of_tensors_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning__refs_nn_functional_relu_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning__refs_nn_functional_smooth_l1_loss_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning__refs_signbit_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning__refs_sinh_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning__refs_special_i0e_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning__refs_special_ndtri_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning__refs_stack_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning__refs_take_along_dim_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning__refs_tanh_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning__refs_trace_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning__refs_unbind_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning__refs_unfold_copy_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning__refs_var_mean_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning_addcdiv_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning_all_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning_aminmax_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning_as_strided_partial_views_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning_as_strided_scatter_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning_atanh_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning_bincount_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning_bitwise_left_shift_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning_broadcast_to_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning_cdouble_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning_cholesky_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning_cholesky_inverse_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning_conj_physical_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning_cummin_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning_empty_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning_empty_like_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning_fft_fftshift_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning_fft_hfft2_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning_fft_hfftn_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning_fft_ihfft2_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning_fft_irfft_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning_fmod_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning_grid_sampler_3d_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning_gt_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning_histogram_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning_hypot_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning_igamma_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning_index_fill_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning_isneginf_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning_lcm_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning_linalg_cond_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning_linalg_matrix_rank_hermitian_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning_linalg_vander_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning_logaddexp2_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning_logical_not_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning_logical_or_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning_logsumexp_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning_masked_argmax_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning_masked_argmin_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning_masked_logaddexp_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning_masked_logsumexp_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning_masked_normalize_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning_max_reduction_no_dim_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning_narrow_copy_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning_nn_functional_adaptive_max_pool2d_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning_nn_functional_conv1d_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning_nn_functional_conv_transpose2d_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning_nn_functional_cosine_similarity_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning_nn_functional_dropout_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning_nn_functional_elu_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning_nn_functional_feature_alpha_dropout_with_train_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning_nn_functional_fractional_max_pool3d_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning_nn_functional_group_norm_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning_nn_functional_interpolate_linear_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning_nn_functional_relu_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning_nn_functional_soft_margin_loss_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning_nn_functional_triplet_margin_with_distance_loss_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning_ones_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning_prod_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning_randn_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning_ravel_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning_rot90_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning_searchsorted_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning_signal_windows_bartlett_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning_softmax_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning_special_chebyshev_polynomial_t_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning_special_legendre_polynomial_p_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning_special_scaled_modified_bessel_k0_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning_split_with_sizes_copy_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning_split_with_sizes_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning_stack_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning_std_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning_tensordot_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning_unbind_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning_unfold_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning_uniform_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning_var_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning_var_mean_unbiased_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning_view_as_complex_cuda, test/test_ops.py::TestCommonCUDA::test_out_zeros_cuda_float32, test/test_ops.py::TestCommonCUDA::test_promotes_int_to_float_acos_cuda_int32, test/test_ops.py::TestCommonCUDA::test_promotes_int_to_float_atanh_cuda_bool, test/test_ops.py::TestCommonCUDA::test_promotes_int_to_float_deg2rad_cuda_int8, test/test_ops.py::TestCommonCUDA::test_promotes_int_to_float_erfinv_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_promotes_int_to_float_exp2_cuda_int32, test/test_ops.py::TestCommonCUDA::test_promotes_int_to_float_expm1_cuda_int64, test/test_ops.py::TestCommonCUDA::test_promotes_int_to_float_expm1_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_promotes_int_to_float_i0_cuda_int64, test/test_ops.py::TestCommonCUDA::test_promotes_int_to_float_ldexp_cuda_int64, test/test_ops.py::TestCommonCUDA::test_promotes_int_to_float_ldexp_cuda_int8, test/test_ops.py::TestCommonCUDA::test_promotes_int_to_float_log10_cuda_int32, test/test_ops.py::TestCommonCUDA::test_promotes_int_to_float_log1p_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_promotes_int_to_float_log_cuda_int32, test/test_ops.py::TestCommonCUDA::test_promotes_int_to_float_logit_cuda_int16, test/test_ops.py::TestCommonCUDA::test_promotes_int_to_float_masked_std_cuda_int32, test/test_ops.py::TestCommonCUDA::test_promotes_int_to_float_polygamma_polygamma_n_0_cuda_int16, test/test_ops.py::TestCommonCUDA::test_promotes_int_to_float_polygamma_polygamma_n_0_cuda_int32, test/test_ops.py::TestCommonCUDA::test_promotes_int_to_float_polygamma_polygamma_n_1_cuda_int16, test/test_ops.py::TestCommonCUDA::test_promotes_int_to_float_polygamma_polygamma_n_3_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_promotes_int_to_float_sinc_cuda_int16, test/test_ops.py::TestCommonCUDA::test_promotes_int_to_float_special_chebyshev_polynomial_t_cuda_int64, test/test_ops.py::TestCommonCUDA::test_promotes_int_to_float_special_chebyshev_polynomial_w_cuda_int64, test/test_ops.py::TestCommonCUDA::test_promotes_int_to_float_special_legendre_polynomial_p_cuda_int16, test/test_ops.py::TestCommonCUDA::test_promotes_int_to_float_special_shifted_chebyshev_polynomial_u_cuda_bool, test/test_ops.py::TestCommonCUDA::test_promotes_int_to_float_special_shifted_chebyshev_polynomial_u_cuda_int64, test/test_ops.py::TestCommonCUDA::test_promotes_int_to_float_special_shifted_chebyshev_polynomial_u_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_promotes_int_to_float_special_shifted_chebyshev_polynomial_v_cuda_int64, test/test_ops.py::TestCommonCUDA::test_promotes_int_to_float_special_shifted_chebyshev_polynomial_v_cuda_int8, test/test_ops.py::TestCommonCUDA::test_promotes_int_to_float_special_shifted_chebyshev_polynomial_w_cuda_int32, test/test_ops.py::TestCommonCUDA::test_promotes_int_to_float_special_shifted_chebyshev_polynomial_w_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_promotes_int_to_float_true_divide_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_promotes_int_to_float_xlogy_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_T_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref__refs__conversions_bfloat16_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref__refs__conversions_bool_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref__refs__conversions_bool_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref__refs__conversions_byte_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref__refs__conversions_byte_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref__refs__conversions_byte_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref__refs__conversions_cdouble_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref__refs__conversions_cfloat_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref__refs__conversions_chalf_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref__refs__conversions_chalf_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref__refs__conversions_char_cuda_complex32, test/test_ops.py::TestCommonCUDA::test_python_ref__refs__conversions_char_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref__refs__conversions_char_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref__refs__conversions_double_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref__refs__conversions_double_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref__refs__conversions_float_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref__refs__conversions_float_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref__refs__conversions_int_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref__refs__conversions_long_cuda_complex32, test/test_ops.py::TestCommonCUDA::test_python_ref__refs__conversions_long_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref__refs__conversions_polar_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref__refs__conversions_polar_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref__refs__conversions_short_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_abs_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_acos_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_acosh_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_add_cuda_complex32, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_add_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_add_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_addcmul_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_addr_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_addr_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_addr_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_all_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_allclose_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_amin_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_amin_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_amin_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_any_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_any_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_any_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_arange_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_as_strided_copy_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_as_strided_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_as_strided_scatter_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_asin_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_asin_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_asin_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_asinh_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_atan2_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_atan2_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_atanh_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_atleast_1d_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_atleast_1d_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_atleast_2d_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_atleast_3d_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_bitwise_and_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_bitwise_and_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_bitwise_left_shift_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_bitwise_not_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_bitwise_xor_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_broadcast_tensors_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_broadcast_tensors_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_broadcast_to_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_broadcast_to_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_broadcast_to_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_bucketize_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_bucketize_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_cat_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_cat_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_cat_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_chunk_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_clamp_max_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_clamp_max_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_clamp_max_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_column_stack_cuda_complex32, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_conj_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_conj_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_conj_physical_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_constant_pad_nd_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_copysign_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_copysign_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_cos_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_cosh_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_cumprod_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_cumprod_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_cumprod_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_cumprod_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_cumsum_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_diag_embed_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_diagonal_copy_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_diagonal_copy_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_digamma_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_div_floor_rounding_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_div_no_rounding_mode_cuda_complex32, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_div_no_rounding_mode_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_dsplit_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_dstack_cuda_complex32, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_dstack_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_empty_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_empty_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_empty_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_empty_like_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_empty_like_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_empty_strided_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_empty_strided_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_eq_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_equal_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_equal_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_erf_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_erf_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_erf_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_erfc_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_erfc_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_erfc_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_erfc_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_exp_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_expand_as_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_expand_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_expand_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_expm1_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_expm1_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_expm1_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_exponential_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_eye_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_fft_fft2_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_fft_fft2_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_fft_fft_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_fft_fftn_cuda_complex32, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_fft_fftn_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_fft_fftn_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_fft_fftshift_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_fft_fftshift_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_fft_hfft2_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_fft_hfft2_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_fft_hfft_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_fft_hfftn_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_fft_ifftn_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_fft_ifftn_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_fft_ifftn_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_fft_ifftn_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_fft_ifftshift_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_fft_ihfftn_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_fft_irfft_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_fft_irfft_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_fft_irfftn_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_fft_irfftn_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_fft_rfft2_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_fft_rfft2_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_fill_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_flip_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_fliplr_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_float_power_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_float_power_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_floor_divide_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_fmax_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_fmin_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_fmod_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_fmod_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_gcd_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_ge_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_ge_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_geometric_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_geometric_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_geometric_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_gt_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_heaviside_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_heaviside_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_hsplit_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_hstack_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_hstack_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_hypot_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_i0_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_i0_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_index_add_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_index_copy_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_index_copy_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_index_copy_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_index_fill_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_index_fill_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_index_select_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_index_select_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_isclose_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_isclose_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_isfinite_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_isinf_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_isnan_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_isnan_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_isneginf_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_isposinf_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_isreal_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_isreal_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_item_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_item_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_le_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_le_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_lgamma_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_lgamma_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_linalg_cross_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_linalg_diagonal_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_linalg_matrix_norm_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_linalg_norm_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_linalg_svdvals_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_linalg_svdvals_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_linalg_vector_norm_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_linspace_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_linspace_tensor_overload_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_linspace_tensor_overload_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_log1p_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_log_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_log_softmax_with_dtype_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_log_softmax_with_dtype_cuda_complex32, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_log_softmax_with_dtype_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_logaddexp2_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_logaddexp_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_logical_and_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_logical_and_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_logical_and_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_logical_and_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_logical_not_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_logical_not_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_logical_or_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_logical_xor_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_logspace_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_logspace_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_logspace_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_logspace_tensor_overload_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_logspace_tensor_overload_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_logspace_tensor_overload_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_logsumexp_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_lt_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_mean_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_meshgrid_variadic_tensors_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_minimum_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_minimum_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_mul_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_mul_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_mul_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_narrow_copy_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_narrow_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_narrow_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_native_layer_norm_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_ne_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_neg_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_neg_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_neg_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_new_full_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_new_full_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_new_full_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_new_full_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_new_ones_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_new_ones_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_new_ones_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_new_ones_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_new_ones_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_new_zeros_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_new_zeros_cuda_complex32, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_new_zeros_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_nn_functional_celu_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_nn_functional_channel_shuffle_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_nn_functional_channel_shuffle_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_nn_functional_dropout_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_nn_functional_gelu_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_nn_functional_group_norm_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_nn_functional_huber_loss_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_nn_functional_layer_norm_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_nn_functional_log_softmax_with_dtype_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_nn_functional_log_softmax_with_dtype_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_nn_functional_log_softmax_with_dtype_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_nn_functional_margin_ranking_loss_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_nn_functional_margin_ranking_loss_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_nn_functional_mish_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_nn_functional_mish_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_nn_functional_pairwise_distance_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_nn_functional_pixel_shuffle_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_nn_functional_pixel_unshuffle_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_nn_functional_pixel_unshuffle_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_nn_functional_pixel_unshuffle_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_nn_functional_poisson_nll_loss_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_nn_functional_poisson_nll_loss_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_nn_functional_softmin_with_dtype_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_nn_functional_softmin_with_dtype_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_nn_functional_softplus_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_nn_functional_threshold_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_nn_functional_triplet_margin_loss_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_norm_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_permute_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_permute_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_positive_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_positive_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_pow_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_prod_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_prod_cuda_complex32, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_prod_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_prod_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_rad2deg_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_rad2deg_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_randn_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_ravel_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_real_cuda_complex32, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_reshape_as_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_reshape_as_cuda_complex32, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_roll_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_rot90_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_rot90_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_rsqrt_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_rsqrt_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_rsub_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_rsub_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_select_scatter_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_select_scatter_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_sgn_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_sgn_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_sgn_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_sgn_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_sign_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_signbit_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_sin_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_sin_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_sin_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_sin_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_sin_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_sinc_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_sinh_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_sinh_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_softmax_with_dtype_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_special_bessel_j1_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_special_bessel_j1_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_special_entr_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_special_entr_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_special_erfcx_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_special_i0e_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_special_i0e_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_special_i1_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_special_log_softmax_with_dtype_cuda_complex32, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_special_log_softmax_with_dtype_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_special_log_softmax_with_dtype_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_special_logit_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_special_multigammaln_mvlgamma_p_1_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_special_multigammaln_mvlgamma_p_3_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_special_multigammaln_mvlgamma_p_3_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_special_ndtr_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_special_ndtr_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_special_ndtri_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_special_softmax_with_dtype_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_special_softmax_with_dtype_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_special_spherical_bessel_j0_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_special_xlog1py_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_special_xlog1py_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_special_xlog1py_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_split_with_sizes_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_split_with_sizes_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_split_with_sizes_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_sqrt_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_sqrt_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_square_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_square_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_squeeze_copy_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_squeeze_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_squeeze_cuda_complex32, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_squeeze_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_squeeze_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_squeeze_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_squeeze_multiple_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_squeeze_multiple_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_stack_cuda_complex32, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_stack_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_stack_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_stack_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_std_mean_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_std_mean_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_stft_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_sub_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_sum_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_sum_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_sum_to_size_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_t_copy_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_t_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_take_along_dim_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_take_along_dim_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_tanh_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_tanh_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_to_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_trace_cuda_complex32, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_transpose_copy_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_transpose_copy_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_transpose_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_transpose_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_transpose_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_transpose_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_transpose_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_tril_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_tril_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_triu_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_triu_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_triu_indices_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_trunc_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_trunc_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_trunc_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_unbind_copy_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_unbind_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_unflatten_cuda_complex32, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_unflatten_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_unflatten_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_unflatten_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_unflatten_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_unfold_copy_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_unfold_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_unsqueeze_copy_cuda_complex32, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_unsqueeze_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_var_mean_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_var_mean_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_view_as_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_view_as_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_view_as_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_view_copy_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_view_copy_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_view_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_view_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_vsplit_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_vsplit_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_vstack_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_xlogy_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_xlogy_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_zeros_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_errors__refs__conversions_polar_cuda, test/test_ops.py::TestCommonCUDA::test_python_ref_errors__refs_amin_cuda, test/test_ops.py::TestCommonCUDA::test_python_ref_errors__refs_bitwise_and_cuda, test/test_ops.py::TestCommonCUDA::test_python_ref_errors__refs_bucketize_cuda, test/test_ops.py::TestCommonCUDA::test_python_ref_errors__refs_ge_cuda, test/test_ops.py::TestCommonCUDA::test_python_ref_errors__refs_hstack_cuda, test/test_ops.py::TestCommonCUDA::test_python_ref_errors__refs_index_select_cuda, test/test_ops.py::TestCommonCUDA::test_python_ref_errors__refs_isclose_cuda, test/test_ops.py::TestCommonCUDA::test_python_ref_errors__refs_linalg_diagonal_cuda, test/test_ops.py::TestCommonCUDA::test_python_ref_errors__refs_lt_cuda, test/test_ops.py::TestCommonCUDA::test_python_ref_errors__refs_ne_cuda, test/test_ops.py::TestCommonCUDA::test_python_ref_errors__refs_nn_functional_gelu_cuda, test/test_ops.py::TestCommonCUDA::test_python_ref_errors__refs_nn_functional_group_norm_cuda, test/test_ops.py::TestCommonCUDA::test_python_ref_errors__refs_nn_functional_hardtanh_cuda, test/test_ops.py::TestCommonCUDA::test_python_ref_errors__refs_normal__in_place_cuda, test/test_ops.py::TestCommonCUDA::test_python_ref_errors__refs_special_xlog1py_cuda, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_T_executor_aten_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_T_executor_aten_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs__conversions_bfloat16_executor_aten_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs__conversions_bfloat16_executor_aten_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs__conversions_bfloat16_executor_aten_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs__conversions_bfloat16_executor_aten_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs__conversions_bfloat16_executor_aten_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs__conversions_bool_executor_aten_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs__conversions_byte_executor_aten_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs__conversions_byte_executor_aten_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs__conversions_cdouble_executor_aten_cuda_complex32, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs__conversions_cdouble_executor_aten_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs__conversions_cdouble_executor_aten_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs__conversions_cfloat_executor_aten_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs__conversions_chalf_executor_aten_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs__conversions_char_executor_aten_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs__conversions_double_executor_aten_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs__conversions_double_executor_aten_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs__conversions_double_executor_aten_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs__conversions_float_executor_aten_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs__conversions_float_executor_aten_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs__conversions_float_executor_aten_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs__conversions_half_executor_aten_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs__conversions_long_executor_aten_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs__conversions_long_executor_aten_cuda_complex32, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs__conversions_long_executor_aten_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs__conversions_long_executor_aten_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs__conversions_long_executor_aten_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs__conversions_short_executor_aten_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_abs_executor_aten_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_acos_executor_aten_cuda_complex32, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_acos_executor_aten_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_acos_executor_aten_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_acosh_executor_aten_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_acosh_executor_aten_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_acosh_executor_aten_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_add_executor_aten_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_add_executor_aten_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_addcmul_executor_aten_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_addcmul_executor_aten_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_addr_executor_aten_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_addr_executor_aten_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_allclose_executor_aten_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_allclose_executor_aten_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_any_executor_aten_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_any_executor_aten_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_as_strided_copy_executor_aten_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_as_strided_executor_aten_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_as_strided_executor_aten_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_as_strided_executor_aten_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_as_strided_executor_aten_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_as_strided_partial_views_executor_aten_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_as_strided_scatter_executor_aten_cuda_complex32, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_asin_executor_aten_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_asin_executor_aten_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_asin_executor_aten_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_asinh_executor_aten_cuda_complex32, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_atan2_executor_aten_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_atan2_executor_aten_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_atanh_executor_aten_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_atleast_1d_executor_aten_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_atleast_1d_executor_aten_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_atleast_2d_executor_aten_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_atleast_2d_executor_aten_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_bitwise_and_executor_aten_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_bitwise_left_shift_executor_aten_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_bitwise_or_executor_aten_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_bitwise_right_shift_executor_aten_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_block_diag_executor_aten_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_broadcast_shapes_executor_aten_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_broadcast_to_executor_aten_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_broadcast_to_executor_aten_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_bucketize_executor_aten_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_cat_executor_aten_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_chunk_executor_aten_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_clamp_min_executor_aten_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_clone_executor_aten_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_clone_executor_aten_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_clone_executor_aten_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_column_stack_executor_aten_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_column_stack_executor_aten_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_conj_executor_aten_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_conj_physical_executor_aten_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_conj_physical_executor_aten_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_conj_physical_executor_aten_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_constant_pad_nd_executor_aten_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_contiguous_executor_aten_cuda_complex32, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_contiguous_executor_aten_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_copysign_executor_aten_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_copysign_executor_aten_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_cos_executor_aten_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_count_nonzero_executor_aten_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_count_nonzero_executor_aten_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_count_nonzero_executor_aten_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_cumprod_executor_aten_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_deg2rad_executor_aten_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_diag_embed_executor_aten_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_diag_embed_executor_aten_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_diag_embed_executor_aten_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_diag_embed_executor_aten_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_diag_executor_aten_cuda_complex32, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_diag_executor_aten_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_diagonal_copy_executor_aten_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_diagonal_copy_executor_aten_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_diagonal_copy_executor_aten_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_diagonal_executor_aten_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_diagonal_scatter_executor_aten_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_digamma_executor_aten_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_digamma_executor_aten_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_div_floor_rounding_executor_aten_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_div_no_rounding_mode_executor_aten_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_div_no_rounding_mode_executor_aten_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_div_no_rounding_mode_executor_aten_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_dot_executor_aten_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_dstack_executor_aten_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_empty_executor_aten_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_empty_executor_aten_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_empty_executor_aten_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_empty_like_executor_aten_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_empty_strided_executor_aten_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_empty_strided_executor_aten_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_empty_strided_executor_aten_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_erf_executor_aten_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_erf_executor_aten_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_erfc_executor_aten_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_erfc_executor_aten_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_erfinv_executor_aten_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_erfinv_executor_aten_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_erfinv_executor_aten_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_exp2_executor_aten_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_exp_executor_aten_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_exp_executor_aten_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_expand_copy_executor_aten_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_expm1_executor_aten_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_exponential_executor_aten_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_eye_executor_aten_cuda_float8_e5m2, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_eye_executor_aten_cuda_float8_e5m2fnuz, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_eye_executor_aten_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_fft_fft_executor_aten_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_fft_fft_executor_aten_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_fft_fft_executor_aten_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_fft_fft_executor_aten_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_fft_fftn_executor_aten_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_fft_fftshift_executor_aten_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_fft_fftshift_executor_aten_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_fft_fftshift_executor_aten_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_fft_fftshift_executor_aten_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_fft_hfft2_executor_aten_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_fft_hfft_executor_aten_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_fft_hfftn_executor_aten_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_fft_ifft2_executor_aten_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_fft_ifft2_executor_aten_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_fft_ifft_executor_aten_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_fft_ifft_executor_aten_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_fft_ifftn_executor_aten_cuda_complex32, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_fft_ifftn_executor_aten_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_fft_ifftn_executor_aten_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_fft_ifftshift_executor_aten_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_fft_ifftshift_executor_aten_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_fft_ifftshift_executor_aten_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_fft_ihfft2_executor_aten_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_fft_ihfft_executor_aten_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_fft_ihfftn_executor_aten_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_fft_irfft2_executor_aten_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_fft_irfftn_executor_aten_cuda_complex32, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_fft_rfftn_executor_aten_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_fill_executor_aten_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_flatten_executor_aten_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_flatten_executor_aten_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_flip_executor_aten_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_fliplr_executor_aten_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_fliplr_executor_aten_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_fliplr_executor_aten_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_fliplr_executor_aten_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_flipud_executor_aten_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_float_power_executor_aten_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_fmax_executor_aten_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_fmax_executor_aten_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_frexp_executor_aten_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_gcd_executor_aten_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_ge_executor_aten_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_ge_executor_aten_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_geometric_executor_aten_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_heaviside_executor_aten_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_heaviside_executor_aten_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_hsplit_executor_aten_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_hsplit_executor_aten_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_i0_executor_aten_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_index_copy_executor_aten_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_index_copy_executor_aten_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_index_copy_executor_aten_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_index_copy_executor_aten_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_index_fill_executor_aten_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_index_select_executor_aten_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_isclose_executor_aten_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_isfinite_executor_aten_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_isfinite_executor_aten_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_isinf_executor_aten_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_isnan_executor_aten_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_isposinf_executor_aten_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_isposinf_executor_aten_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_isposinf_executor_aten_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_isreal_executor_aten_cuda_complex32, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_isreal_executor_aten_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_item_executor_aten_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_item_executor_aten_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_linalg_diagonal_executor_aten_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_linalg_diagonal_executor_aten_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_linalg_svdvals_executor_aten_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_linalg_vecdot_executor_aten_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_linalg_vecdot_executor_aten_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_log10_executor_aten_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_log1p_executor_aten_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_log1p_executor_aten_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_log1p_executor_aten_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_log_executor_aten_cuda_complex32, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_log_softmax_with_dtype_executor_aten_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_log_softmax_with_dtype_executor_aten_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_logical_and_executor_aten_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_logical_and_executor_aten_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_logical_or_executor_aten_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_logspace_executor_aten_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_logspace_executor_aten_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_logsumexp_executor_aten_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_lt_executor_aten_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_masked_fill_executor_aten_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_masked_fill_executor_aten_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_masked_fill_executor_aten_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_maximum_executor_aten_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_maximum_executor_aten_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_meshgrid_list_of_tensors_executor_aten_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_meshgrid_list_of_tensors_executor_aten_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_minimum_executor_aten_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_minimum_executor_aten_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_movedim_executor_aten_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_mul_executor_aten_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_mul_executor_aten_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_nan_to_num_executor_aten_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_narrow_copy_executor_aten_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_narrow_copy_executor_aten_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_narrow_copy_executor_aten_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_native_layer_norm_executor_aten_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_ne_executor_aten_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_neg_executor_aten_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_new_empty_executor_aten_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_new_empty_executor_aten_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_new_empty_strided_executor_aten_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_new_empty_strided_executor_aten_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_new_empty_strided_executor_aten_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_new_empty_strided_executor_aten_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_new_full_executor_aten_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_new_zeros_executor_aten_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_nn_functional_channel_shuffle_executor_aten_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_nn_functional_channel_shuffle_executor_aten_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_nn_functional_hinge_embedding_loss_executor_aten_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_nn_functional_layer_norm_executor_aten_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_nn_functional_leaky_relu_executor_aten_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_nn_functional_log_softmax_with_dtype_executor_aten_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_nn_functional_mse_loss_executor_aten_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_nn_functional_nll_loss_executor_aten_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_nn_functional_pairwise_distance_executor_aten_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_nn_functional_pixel_shuffle_executor_aten_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_nn_functional_pixel_shuffle_executor_aten_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_nn_functional_pixel_unshuffle_executor_aten_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_nn_functional_poisson_nll_loss_executor_aten_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_nn_functional_prelu_executor_aten_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_nn_functional_relu_executor_aten_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_nn_functional_relu_executor_aten_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_nn_functional_selu_executor_aten_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_nn_functional_softmax_with_dtype_executor_aten_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_nn_functional_softmax_with_dtype_executor_aten_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_nn_functional_softmax_with_dtype_executor_aten_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_nn_functional_tanhshrink_executor_aten_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_nn_functional_tanhshrink_executor_aten_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_norm_executor_aten_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_normal__in_place_executor_aten_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_normal__in_place_executor_aten_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_ones_executor_aten_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_permute_copy_executor_aten_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_permute_executor_aten_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_positive_executor_aten_cuda_complex32, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_positive_executor_aten_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_positive_executor_aten_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_positive_executor_aten_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_pow_executor_aten_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_pow_executor_aten_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_pow_executor_aten_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_prod_executor_aten_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_prod_executor_aten_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_prod_executor_aten_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_prod_executor_aten_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_prod_executor_aten_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_prod_executor_aten_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_prod_executor_aten_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_rad2deg_executor_aten_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_randn_executor_aten_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_real_executor_aten_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_real_executor_aten_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_real_executor_aten_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_reciprocal_executor_aten_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_renorm_executor_aten_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_repeat_executor_aten_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_repeat_executor_aten_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_reshape_as_executor_aten_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_reshape_executor_aten_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_reshape_executor_aten_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_roll_executor_aten_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_rot90_executor_aten_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_rot90_executor_aten_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_round_executor_aten_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_rsqrt_executor_aten_cuda_complex32, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_select_scatter_executor_aten_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_sgn_executor_aten_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_sgn_executor_aten_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_sgn_executor_aten_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_sigmoid_executor_aten_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_sigmoid_executor_aten_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_signbit_executor_aten_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_signbit_executor_aten_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_sin_executor_aten_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_sin_executor_aten_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_sin_executor_aten_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_sinc_executor_aten_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_softmax_with_dtype_executor_aten_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_special_i1_executor_aten_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_special_i1e_executor_aten_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_special_i1e_executor_aten_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_special_i1e_executor_aten_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_special_logit_executor_aten_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_special_multigammaln_mvlgamma_p_1_executor_aten_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_special_multigammaln_mvlgamma_p_3_executor_aten_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_special_multigammaln_mvlgamma_p_3_executor_aten_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_special_multigammaln_mvlgamma_p_5_executor_aten_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_special_ndtr_executor_aten_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_special_ndtr_executor_aten_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_special_ndtri_executor_aten_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_special_softmax_with_dtype_executor_aten_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_special_softmax_with_dtype_executor_aten_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_special_spherical_bessel_j0_executor_aten_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_special_spherical_bessel_j0_executor_aten_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_special_spherical_bessel_j0_executor_aten_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_special_zeta_executor_aten_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_split_with_sizes_executor_aten_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_split_with_sizes_executor_aten_cuda_complex32, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_sqrt_executor_aten_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_square_executor_aten_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_square_executor_aten_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_squeeze_copy_executor_aten_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_squeeze_executor_aten_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_squeeze_multiple_executor_aten_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_squeeze_multiple_executor_aten_cuda_complex32, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_squeeze_multiple_executor_aten_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_stack_executor_aten_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_std_mean_executor_aten_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_sub_executor_aten_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_sum_executor_aten_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_sum_executor_aten_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_sum_to_size_executor_aten_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_sum_to_size_executor_aten_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_sum_to_size_executor_aten_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_t_copy_executor_aten_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_t_executor_aten_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_t_executor_aten_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_take_along_dim_executor_aten_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_tan_executor_aten_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_tan_executor_aten_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_tanh_executor_aten_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_tensor_split_executor_aten_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_to_executor_aten_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_trace_executor_aten_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_trace_executor_aten_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_transpose_copy_executor_aten_cuda_complex32, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_transpose_copy_executor_aten_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_transpose_executor_aten_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_transpose_executor_aten_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_tril_executor_aten_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_triu_executor_aten_cuda_complex32, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_triu_executor_aten_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_triu_indices_executor_aten_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_triu_indices_executor_aten_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_true_divide_executor_aten_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_unbind_copy_executor_aten_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_unbind_copy_executor_aten_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_unbind_copy_executor_aten_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_unbind_executor_aten_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_unflatten_executor_aten_cuda_complex32, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_unflatten_executor_aten_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_unfold_copy_executor_aten_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_unfold_copy_executor_aten_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_unfold_executor_aten_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_unfold_executor_aten_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_unsqueeze_copy_executor_aten_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_unsqueeze_copy_executor_aten_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_unsqueeze_executor_aten_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_var_executor_aten_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_var_mean_executor_aten_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_view_as_executor_aten_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_view_copy_executor_aten_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_view_copy_executor_aten_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_view_copy_executor_aten_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_view_executor_aten_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_view_executor_aten_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_vsplit_executor_aten_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_vsplit_executor_aten_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_vsplit_executor_aten_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_vstack_executor_aten_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_vstack_executor_aten_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_where_executor_aten_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_where_executor_aten_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_xlogy_executor_aten_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_zeros_executor_aten_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_zeros_executor_aten_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_zeros_executor_aten_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_T_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_T_cuda_complex32, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_T_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs__conversions_bool_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs__conversions_byte_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs__conversions_byte_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs__conversions_cdouble_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs__conversions_cfloat_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs__conversions_cfloat_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs__conversions_cfloat_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs__conversions_cfloat_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs__conversions_chalf_cuda_complex32, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs__conversions_char_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs__conversions_char_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs__conversions_char_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs__conversions_float_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs__conversions_float_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs__conversions_float_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs__conversions_half_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs__conversions_int_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs__conversions_long_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs__conversions_short_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_abs_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_abs_cuda_complex32, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_add_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_add_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_addcmul_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_addcmul_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_addcmul_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_alias_copy_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_alias_copy_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_alias_copy_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_allclose_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_allclose_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_amax_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_arange_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_as_strided_copy_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_as_strided_copy_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_as_strided_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_as_strided_partial_views_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_as_strided_partial_views_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_as_strided_scatter_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_asin_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_asin_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_asin_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_asinh_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_atan_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_atanh_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_atanh_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_atanh_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_atleast_1d_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_atleast_1d_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_atleast_3d_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_atleast_3d_cuda_complex32, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_atleast_3d_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_bitwise_and_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_bitwise_not_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_bitwise_not_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_bitwise_or_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_bitwise_right_shift_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_broadcast_tensors_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_broadcast_tensors_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_broadcast_to_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_bucketize_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_cat_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_cauchy_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_chunk_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_chunk_cuda_complex32, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_chunk_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_clamp_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_clamp_max_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_clamp_max_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_clamp_min_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_clone_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_clone_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_clone_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_column_stack_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_column_stack_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_conj_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_constant_pad_nd_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_contiguous_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_contiguous_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_copysign_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_cosh_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_count_nonzero_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_count_nonzero_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_cumprod_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_cumprod_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_cumsum_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_cumsum_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_cumsum_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_deg2rad_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_diag_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_diag_embed_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_diagonal_copy_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_diagonal_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_digamma_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_digamma_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_digamma_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_div_floor_rounding_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_div_no_rounding_mode_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_div_trunc_rounding_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_dstack_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_dstack_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_dstack_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_empty_cuda_complex32, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_empty_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_empty_like_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_empty_strided_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_eq_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_erf_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_erfc_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_erfinv_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_expand_as_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_expand_copy_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_expand_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_expand_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_expm1_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_eye_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_eye_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_eye_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_fft_fft_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_fft_fftshift_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_fft_fftshift_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_fft_hfft2_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_fft_hfft_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_fft_hfftn_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_fft_ifft2_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_fft_ifft_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_fft_ifftn_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_fft_ifftshift_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_fft_ihfft2_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_fft_ihfft_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_fft_ihfft_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_fft_ihfft_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_fft_ihfft_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_fft_ihfftn_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_fft_ihfftn_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_fft_ihfftn_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_fft_irfft2_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_fft_irfft_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_fft_irfft_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_fft_irfftn_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_fft_irfftn_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_fft_rfft_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_fill_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_flatten_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_flip_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_fliplr_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_fliplr_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_fliplr_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_flipud_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_flipud_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_float_power_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_float_power_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_floor_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_fmax_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_fmax_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_fmin_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_frac_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_gcd_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_ge_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_hsplit_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_hsplit_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_hsplit_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_hstack_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_hstack_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_igammac_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_index_fill_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_index_fill_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_index_select_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_index_select_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_isclose_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_isclose_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_isclose_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_isfinite_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_isneginf_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_isposinf_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_isreal_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_item_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_lerp_cuda_complex32, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_lgamma_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_lgamma_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_linalg_norm_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_linalg_svd_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_linalg_svdvals_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_linalg_vecdot_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_linalg_vecdot_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_linspace_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_linspace_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_linspace_tensor_overload_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_log10_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_log10_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_log10_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_log10_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_log1p_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_log2_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_log2_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_log2_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_log_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_log_normal_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_log_softmax_with_dtype_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_logical_and_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_logical_not_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_logical_not_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_logical_xor_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_logical_xor_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_logical_xor_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_logical_xor_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_logspace_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_logsumexp_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_logsumexp_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_logsumexp_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_masked_fill_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_masked_fill_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_maximum_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_mean_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_mean_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_mean_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_meshgrid_list_of_tensors_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_meshgrid_list_of_tensors_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_meshgrid_list_of_tensors_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_minimum_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_minimum_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_mul_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_mul_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_mul_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_narrow_copy_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_narrow_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_narrow_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_narrow_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_native_layer_norm_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_ne_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_ne_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_new_empty_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_new_empty_strided_cuda_complex32, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_new_empty_strided_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_new_full_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_new_zeros_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_new_zeros_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_new_zeros_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_nn_functional_channel_shuffle_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_nn_functional_elu_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_nn_functional_gelu_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_nn_functional_gelu_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_nn_functional_gelu_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_nn_functional_hardtanh_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_nn_functional_hardtanh_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_nn_functional_l1_loss_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_nn_functional_layer_norm_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_nn_functional_leaky_relu_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_nn_functional_log_softmax_with_dtype_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_nn_functional_margin_ranking_loss_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_nn_functional_pairwise_distance_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_nn_functional_pairwise_distance_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_nn_functional_pdist_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_nn_functional_pixel_shuffle_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_nn_functional_pixel_shuffle_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_nn_functional_pixel_unshuffle_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_nn_functional_poisson_nll_loss_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_nn_functional_relu6_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_nn_functional_selu_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_nn_functional_softmax_with_dtype_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_nn_functional_softmax_with_dtype_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_nn_functional_threshold_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_nn_functional_triplet_margin_loss_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_normal_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_ones_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_permute_copy_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_permute_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_permute_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_positive_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_prod_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_rad2deg_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_ravel_cuda_complex32, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_ravel_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_ravel_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_real_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_reciprocal_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_reciprocal_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_remainder_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_remainder_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_renorm_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_repeat_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_rot90_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_rsqrt_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_rsqrt_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_select_scatter_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_select_scatter_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_select_scatter_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_sgn_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_sigmoid_cuda_complex32, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_sigmoid_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_signbit_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_sin_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_sin_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_sinc_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_softmax_with_dtype_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_special_bessel_j0_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_special_i0e_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_special_i0e_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_special_i1_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_special_i1e_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_special_log_ndtr_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_special_log_ndtr_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_special_log_softmax_with_dtype_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_special_logit_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_special_multigammaln_mvlgamma_p_1_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_special_multigammaln_mvlgamma_p_3_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_special_multigammaln_mvlgamma_p_3_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_special_multigammaln_mvlgamma_p_5_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_special_multigammaln_mvlgamma_p_5_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_special_ndtr_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_special_ndtri_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_special_ndtri_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_special_ndtri_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_special_softmax_with_dtype_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_special_xlog1py_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_special_zeta_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_split_with_sizes_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_split_with_sizes_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_square_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_square_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_squeeze_copy_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_squeeze_copy_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_squeeze_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_squeeze_multiple_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_squeeze_multiple_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_squeeze_multiple_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_stack_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_stack_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_stack_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_stack_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_std_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_std_mean_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_stft_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_sum_to_size_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_sum_to_size_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_t_copy_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_t_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_t_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_take_along_dim_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_tan_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_tanh_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_to_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_trace_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_transpose_copy_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_transpose_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_transpose_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_transpose_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_tril_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_tril_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_tril_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_tril_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_triu_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_triu_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_triu_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_true_divide_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_true_divide_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_trunc_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_trunc_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_unbind_copy_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_unbind_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_unflatten_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_unflatten_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_unsqueeze_copy_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_unsqueeze_copy_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_unsqueeze_copy_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_unsqueeze_copy_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_unsqueeze_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_var_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_var_mean_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_view_as_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_view_as_cuda_complex32, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_view_copy_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_view_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_view_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_where_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_where_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_where_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_xlogy_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_zeros_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_zeros_cuda_complex32, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_T_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_T_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_T_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_T_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs__conversions_bfloat16_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs__conversions_bfloat16_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs__conversions_bool_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs__conversions_cdouble_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs__conversions_cfloat_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs__conversions_cfloat_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs__conversions_char_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs__conversions_char_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs__conversions_char_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs__conversions_complex_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs__conversions_double_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs__conversions_double_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs__conversions_half_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs__conversions_half_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs__conversions_half_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs__conversions_long_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs__conversions_short_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_acos_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_acos_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_acos_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_acosh_cuda_complex32, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_add_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_addcmul_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_addcmul_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_addcmul_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_addr_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_all_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_allclose_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_amin_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_as_strided_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_as_strided_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_as_strided_partial_views_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_asinh_cuda_complex32, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_asinh_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_atanh_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_atanh_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_atleast_1d_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_atleast_1d_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_atleast_3d_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_atleast_3d_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_atleast_3d_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_bitwise_or_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_bitwise_or_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_bitwise_xor_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_broadcast_tensors_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_broadcast_tensors_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_broadcast_tensors_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_broadcast_to_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_bucketize_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_cat_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_ceil_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_ceil_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_chunk_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_chunk_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_clamp_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_clamp_max_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_clamp_max_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_clamp_min_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_clone_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_conj_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_conj_physical_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_constant_pad_nd_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_constant_pad_nd_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_contiguous_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_contiguous_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_contiguous_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_contiguous_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_copysign_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_cos_cuda_complex32, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_count_nonzero_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_count_nonzero_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_count_nonzero_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_count_nonzero_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_cumprod_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_deg2rad_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_diag_embed_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_diagonal_copy_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_diagonal_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_diagonal_scatter_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_diagonal_scatter_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_digamma_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_div_floor_rounding_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_div_floor_rounding_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_div_no_rounding_mode_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_dsplit_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_dstack_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_dstack_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_dstack_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_empty_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_empty_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_empty_like_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_empty_like_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_empty_strided_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_empty_strided_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_empty_strided_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_eq_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_equal_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_equal_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_erf_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_erf_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_erf_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_erfinv_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_erfinv_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_erfinv_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_exp2_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_exp2_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_exp_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_exp_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_exp_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_expand_as_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_expand_as_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_expand_copy_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_expand_copy_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_expand_copy_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_expand_copy_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_expm1_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_eye_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_eye_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_eye_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_fft_fft2_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_fft_fft_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_fft_fftn_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_fft_fftn_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_fft_fftshift_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_fft_fftshift_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_fft_hfft2_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_fft_hfft2_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_fft_hfft2_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_fft_hfft_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_fft_hfftn_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_fft_hfftn_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_fft_ifft_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_fft_ifftshift_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_fft_ifftshift_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_fft_ihfftn_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_fft_irfft2_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_fft_irfft2_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_fft_irfft_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_fft_irfft_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_fft_irfftn_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_fft_irfftn_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_fft_rfft2_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_fft_rfft2_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_fft_rfft_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_fft_rfft_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_fill_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_fill_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_flip_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_flip_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_flip_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_fliplr_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_fliplr_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_fliplr_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_flipud_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_float_power_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_floor_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_floor_divide_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_floor_divide_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_floor_divide_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_ge_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_ge_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_geometric_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_geometric_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_geometric_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_hsplit_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_hstack_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_hstack_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_i0_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_index_add_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_index_add_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_index_add_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_index_fill_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_index_fill_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_index_fill_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_isclose_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_isclose_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_isfinite_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_isfinite_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_isinf_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_isnan_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_isnan_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_isnan_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_isneginf_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_isneginf_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_isneginf_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_isposinf_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_isposinf_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_isreal_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_item_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_le_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_le_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_linalg_diagonal_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_linalg_diagonal_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_linalg_matrix_norm_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_linalg_vecdot_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_linalg_vector_norm_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_linspace_tensor_overload_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_linspace_tensor_overload_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_linspace_tensor_overload_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_linspace_tensor_overload_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_log10_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_log10_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_log10_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_log1p_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_log2_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_log_softmax_with_dtype_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_log_softmax_with_dtype_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_log_softmax_with_dtype_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_logaddexp_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_logical_and_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_logical_not_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_logical_not_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_logical_or_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_logical_or_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_logsumexp_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_lt_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_masked_fill_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_masked_fill_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_maximum_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_maximum_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_meshgrid_list_of_tensors_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_meshgrid_list_of_tensors_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_meshgrid_variadic_tensors_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_meshgrid_variadic_tensors_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_minimum_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_minimum_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_minimum_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_minimum_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_minimum_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_mul_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_mul_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_mul_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_nan_to_num_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_narrow_copy_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_narrow_copy_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_narrow_copy_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_narrow_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_narrow_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_native_layer_norm_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_ne_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_ne_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_neg_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_new_empty_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_new_empty_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_new_empty_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_new_empty_strided_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_new_empty_strided_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_new_full_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_new_ones_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_new_ones_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_new_ones_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_new_zeros_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_new_zeros_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_new_zeros_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_new_zeros_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_nn_functional_celu_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_nn_functional_hardtanh_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_nn_functional_hardtanh_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_nn_functional_hardtanh_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_nn_functional_hinge_embedding_loss_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_nn_functional_huber_loss_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_nn_functional_leaky_relu_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_nn_functional_log_softmax_with_dtype_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_nn_functional_log_softmax_with_dtype_cuda_complex32, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_nn_functional_margin_ranking_loss_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_nn_functional_margin_ranking_loss_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_nn_functional_mse_loss_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_nn_functional_nll_loss_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_nn_functional_pairwise_distance_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_nn_functional_pairwise_distance_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_nn_functional_pixel_shuffle_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_nn_functional_pixel_shuffle_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_nn_functional_pixel_unshuffle_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_nn_functional_pixel_unshuffle_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_nn_functional_prelu_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_nn_functional_relu_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_nn_functional_relu_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_nn_functional_softmax_with_dtype_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_nn_functional_softmax_with_dtype_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_nn_functional_softmin_with_dtype_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_nn_functional_softmin_with_dtype_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_nn_functional_tanhshrink_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_nn_functional_tanhshrink_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_nn_functional_threshold_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_nn_functional_threshold_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_nn_functional_triplet_margin_loss_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_nn_functional_triplet_margin_loss_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_nn_functional_triplet_margin_loss_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_nn_functional_triplet_margin_loss_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_norm_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_norm_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_normal__in_place_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_normal_number_mean_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_ones_cuda_complex32, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_permute_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_permute_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_permute_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_pow_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_prod_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_prod_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_ravel_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_ravel_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_real_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_reciprocal_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_remainder_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_renorm_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_repeat_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_repeat_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_reshape_as_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_roll_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_roll_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_round_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_rsqrt_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_rsqrt_cuda_complex32, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_rsqrt_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_rsqrt_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_rsub_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_rsub_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_select_scatter_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_select_scatter_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_sigmoid_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_sigmoid_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_sign_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_sign_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_sign_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_sin_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_sinc_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_sinc_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_sinc_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_sinh_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_special_bessel_j0_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_special_entr_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_special_entr_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_special_erfcx_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_special_erfcx_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_special_i0e_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_special_i1e_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_special_i1e_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_special_log_softmax_with_dtype_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_special_logit_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_special_logit_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_special_multigammaln_mvlgamma_p_3_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_special_multigammaln_mvlgamma_p_5_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_special_ndtr_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_special_ndtri_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_special_ndtri_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_special_softmax_with_dtype_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_special_xlog1py_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_special_zeta_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_split_with_sizes_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_split_with_sizes_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_split_with_sizes_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_split_with_sizes_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_sqrt_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_square_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_squeeze_copy_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_squeeze_copy_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_std_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_std_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_std_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_std_mean_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_sub_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_sum_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_sum_to_size_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_t_copy_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_t_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_take_along_dim_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_take_along_dim_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_tan_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_tanh_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_tensor_split_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_trace_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_transpose_copy_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_transpose_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_transpose_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_tril_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_tril_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_tril_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_triu_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_trunc_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_unbind_copy_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_unbind_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_unflatten_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_unfold_copy_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_unfold_cuda_complex32, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_unfold_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_unsqueeze_copy_cuda_complex32, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_unsqueeze_copy_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_var_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_var_mean_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_vdot_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_vdot_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_view_as_complex_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_view_as_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_view_as_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_view_as_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_view_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_vsplit_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_vstack_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_vstack_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_where_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_where_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_where_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_xlogy_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_xlogy_cuda_int8, test/test_ops.py::TestCommonCUDA::test_reduction_ops_reduce_amax_cuda, test/test_ops.py::TestCommonCUDA::test_reduction_ops_reduce_argmin_cuda, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager___rmatmul___cuda_float32, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager___rmul___cuda_complex64, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager___rpow___cuda_float32, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager__batch_norm_with_update_cuda_float32, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager__upsample_bilinear2d_aa_cuda_float32, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_add_cuda_float32, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_addr_cuda_float32, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_argwhere_cuda_float32, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_atleast_1d_cuda_float32, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_atleast_2d_cuda_float32, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_baddbmm_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_bfloat16_cuda_float32, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_broadcast_shapes_cuda_float32, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_broadcast_to_cuda_float32, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_cdist_cuda_float32, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_clamp_cuda_float32, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_clamp_max_cuda_float32, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_column_stack_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_corrcoef_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_count_nonzero_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_cumsum_cuda_float32, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_diag_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_dstack_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_empty_cuda_float32, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_empty_permuted_cuda_float32, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_empty_strided_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_eq_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_exp2_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_exp2_cuda_float32, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_expand_cuda_float32, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_fft_ifft_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_float_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_floor_cuda_float32, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_frexp_cuda_float32, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_full_like_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_ge_cuda_float32, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_gradient_cuda_float32, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_hash_tensor_cuda_float32, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_histc_cuda_float32, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_hstack_cuda_float32, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_index_fill_cuda_float32, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_inner_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_isinf_cuda_float32, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_jiterator_2inputs_2outputs_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_linalg_det_cuda_float32, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_linalg_eigh_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_linalg_inv_ex_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_linalg_ldl_factor_ex_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_linalg_matrix_power_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_linalg_matrix_rank_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_linalg_tensorinv_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_linalg_vander_cuda_float32, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_log10_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_log1p_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_logaddexp_cuda_float32, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_logspace_cuda_float32, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_mT_cuda_float32, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_masked_amax_cuda_float32, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_masked_cumsum_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_masked_norm_cuda_float32, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_masked_scatter_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_masked_softmax_cuda_float32, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_matmul_cuda_float32, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_min_reduction_no_dim_cuda_float32, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_nan_to_num_cuda_float32, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_narrow_copy_cuda_float32, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_native_layer_norm_cuda_float32, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_nn_functional_adaptive_avg_pool2d_cuda_float32, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_nn_functional_adaptive_avg_pool3d_cuda_float32, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_nn_functional_adaptive_max_pool3d_cuda_float32, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_nn_functional_batch_norm_without_cudnn_cuda_float32, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_nn_functional_celu_cuda_float32, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_nn_functional_conv1d_cuda_float32, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_nn_functional_huber_loss_cuda_float32, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_nn_functional_max_unpool2d_cuda_float32, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_nn_functional_max_unpool3d_cuda_float32, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_nn_functional_mish_cuda_float32, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_nn_functional_mse_loss_cuda_float32, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_nn_functional_multi_margin_loss_cuda_float32, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_nn_functional_pixel_shuffle_cuda_float32, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_nn_functional_softplus_cuda_float32, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_nn_functional_triplet_margin_loss_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_norm_fro_cuda_float32, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_norm_inf_cuda_float32, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_ones_like_cuda_float32, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_permute_copy_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_pow_cuda_float32, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_randn_like_cuda_float32, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_remainder_cuda_float32, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_renorm_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_reshape_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_rot90_cuda_float32, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_round_cuda_float32, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_sgn_cuda_float32, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_sigmoid_cuda_float32, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_signal_windows_gaussian_cuda_float32, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_sinc_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_sinh_cuda_float32, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_sparse_sampled_addmm_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_special_airy_ai_cuda_float32, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_special_chebyshev_polynomial_w_cuda_float32, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_special_hermite_polynomial_h_cuda_float32, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_split_cuda_float32, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_split_list_args_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_split_list_args_cuda_float32, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_square_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_stack_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_sub_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_sub_cuda_float32, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_sum_to_size_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_svd_lowrank_cuda_float32, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_t_copy_cuda_float32, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_tan_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_tanh_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_tensordot_cuda_float32, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_to_sparse_cuda_float32, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_trapz_cuda_float32, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_true_divide_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_unbind_copy_cuda_float32, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_unfold_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_uniform_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_unique_cuda_float32, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_unsafe_chunk_cuda_float32, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_vdot_cuda_complex64, test/test_ops.py::TestCompositeComplianceCUDA::test_backward___getitem___cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_backward___rdiv___cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_backward___rpow___cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_backward__segment_reduce_lengths_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_backward_acosh_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_backward_addbmm_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_backward_addcdiv_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_backward_alias_copy_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_backward_angle_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_backward_asin_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_backward_atan_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_backward_atleast_3d_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_backward_broadcast_to_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_backward_clamp_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_backward_cosh_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_backward_cummax_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_backward_deg2rad_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_backward_fft_irfft2_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_backward_fft_rfft2_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_backward_float_power_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_backward_grid_sampler_2d_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_backward_index_reduce_mean_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_backward_lerp_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_backward_linalg_cholesky_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_backward_linalg_cross_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_backward_linalg_inv_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_backward_linalg_lu_factor_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_backward_linalg_vector_norm_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_backward_lu_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_backward_lu_solve_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_backward_masked_cumsum_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_backward_masked_log_softmax_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_backward_masked_normalize_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_backward_masked_std_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_backward_max_pool2d_with_indices_backward_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_backward_msort_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_backward_narrow_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_backward_nn_functional_adaptive_avg_pool3d_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_backward_nn_functional_avg_pool1d_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_backward_nn_functional_bilinear_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_backward_nn_functional_conv1d_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_backward_nn_functional_conv_transpose1d_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_backward_nn_functional_conv_transpose3d_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_backward_nn_functional_gaussian_nll_loss_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_backward_nn_functional_gelu_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_backward_nn_functional_instance_norm_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_backward_nn_functional_interpolate_bilinear_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_backward_nn_functional_max_unpool1d_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_backward_nn_functional_max_unpool3d_grad_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_backward_nn_functional_pad_reflect_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_backward_nn_functional_pixel_shuffle_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_backward_nn_functional_rrelu_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_backward_nn_functional_tanhshrink_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_backward_nn_functional_threshold_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_backward_nn_functional_triplet_margin_with_distance_loss_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_backward_norm_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_backward_normal_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_backward_polygamma_polygamma_n_1_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_backward_rad2deg_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_backward_reciprocal_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_backward_std_mean_unbiased_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_backward_t_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_backward_take_along_dim_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_backward_tensordot_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_backward_to_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_backward_torch_ops_aten__efficient_attention_forward_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_backward_true_divide_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_backward_unsqueeze_copy_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_cow_input___rdiv___cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_cow_input___rmul___cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_cow_input__upsample_bilinear2d_aa_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_cow_input_add_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_cow_input_as_strided_scatter_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_cow_input_asin_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_cow_input_atleast_3d_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_cow_input_broadcast_shapes_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_cow_input_byte_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_cow_input_cartesian_prod_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_cow_input_chalf_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_cow_input_column_stack_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_cow_input_copysign_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_cow_input_cosh_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_cow_input_cummin_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_cow_input_dist_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_cow_input_div_floor_rounding_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_cow_input_div_no_rounding_mode_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_cow_input_einsum_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_cow_input_empty_strided_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_cow_input_eq_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_cow_input_erfinv_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_cow_input_expand_as_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_cow_input_fft_fftn_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_cow_input_fft_fftshift_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_cow_input_fft_irfft2_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_cow_input_fft_rfft2_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_cow_input_fmin_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_cow_input_frac_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_cow_input_geometric_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_cow_input_linalg_lstsq_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_cow_input_linalg_lstsq_grad_oriented_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_cow_input_linalg_pinv_singular_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_cow_input_linalg_solve_triangular_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_cow_input_linalg_svd_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_cow_input_linspace_tensor_overload_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_cow_input_log2_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_cow_input_log_normal_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_cow_input_masked_mean_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_cow_input_max_binary_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_cow_input_max_reduction_no_dim_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_cow_input_mvlgamma_mvlgamma_p_1_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_cow_input_neg_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_cow_input_nn_functional_avg_pool1d_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_cow_input_nn_functional_conv2d_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_cow_input_nn_functional_conv_transpose2d_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_cow_input_nn_functional_dropout3d_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_cow_input_nn_functional_feature_alpha_dropout_without_train_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_cow_input_nn_functional_glu_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_cow_input_nn_functional_linear_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_cow_input_nn_functional_pixel_unshuffle_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_cow_input_nn_functional_prelu_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_cow_input_nn_functional_scaled_dot_product_attention_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_cow_input_nn_functional_softmin_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_cow_input_nn_functional_threshold_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_cow_input_nn_functional_triplet_margin_loss_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_cow_input_polygamma_polygamma_n_0_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_cow_input_put_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_cow_input_rad2deg_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_cow_input_round_decimals_0_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_cow_input_select_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_cow_input_signal_windows_kaiser_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_cow_input_signal_windows_nuttall_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_cow_input_sinc_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_cow_input_special_chebyshev_polynomial_u_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_cow_input_special_entr_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_cow_input_special_i1_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_cow_input_special_modified_bessel_k0_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_cow_input_special_polygamma_special_polygamma_n_0_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_cow_input_special_shifted_chebyshev_polynomial_u_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_cow_input_special_shifted_chebyshev_polynomial_w_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_cow_input_squeeze_copy_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_cow_input_stack_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_cow_input_std_unbiased_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_cow_input_tile_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_cow_input_unbind_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_cow_input_unflatten_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_cow_input_unsafe_split_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_cow_input_unsqueeze_copy_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_cow_input_view_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_forward_ad_T_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_forward_ad___radd___cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_forward_ad_acos_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_forward_ad_addmv_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_forward_ad_alias_copy_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_forward_ad_atan_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_forward_ad_atanh_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_forward_ad_cholesky_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_forward_ad_clamp_max_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_forward_ad_conj_physical_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_forward_ad_cumsum_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_forward_ad_dist_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_forward_ad_dsplit_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_forward_ad_exponential_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_forward_ad_fft_irfft2_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_forward_ad_fft_rfftn_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_forward_ad_fill_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_forward_ad_flipud_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_forward_ad_gather_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_forward_ad_ge_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_forward_ad_grid_sampler_2d_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_forward_ad_gt_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_forward_ad_igamma_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_forward_ad_index_select_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_forward_ad_isposinf_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_forward_ad_jiterator_binary_return_by_ref_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_forward_ad_le_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_forward_ad_linalg_cholesky_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_forward_ad_linalg_cross_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_forward_ad_linalg_diagonal_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_forward_ad_linalg_inv_ex_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_forward_ad_linalg_ldl_factor_ex_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_forward_ad_linalg_lstsq_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_forward_ad_linalg_lu_factor_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_forward_ad_linalg_multi_dot_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_forward_ad_linalg_solve_ex_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_forward_ad_linalg_vander_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_forward_ad_mH_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_forward_ad_masked_argmin_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_forward_ad_masked_sum_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_forward_ad_matmul_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_forward_ad_mode_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_forward_ad_nan_to_num_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_forward_ad_narrow_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_forward_ad_new_full_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_forward_ad_nn_functional_alpha_dropout_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_forward_ad_nn_functional_avg_pool2d_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_forward_ad_nn_functional_batch_norm_without_cudnn_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_forward_ad_nn_functional_feature_alpha_dropout_without_train_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_forward_ad_nn_functional_kl_div_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_forward_ad_nn_functional_leaky_relu_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_forward_ad_nn_functional_max_pool3d_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_forward_ad_nn_functional_multilabel_soft_margin_loss_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_forward_ad_nn_functional_scaled_dot_product_attention_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_forward_ad_nn_functional_selu_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_forward_ad_nn_functional_softplus_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_forward_ad_nn_functional_softshrink_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_forward_ad_pca_lowrank_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_forward_ad_rand_like_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_forward_ad_randn_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_forward_ad_resolve_neg_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_forward_ad_scatter_add_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_forward_ad_signal_windows_gaussian_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_forward_ad_slice_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_forward_ad_sparse_sampled_addmm_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_forward_ad_special_shifted_chebyshev_polynomial_u_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_forward_ad_special_shifted_chebyshev_polynomial_v_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_forward_ad_special_xlog1py_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_forward_ad_std_mean_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_forward_ad_to_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_forward_ad_unbind_copy_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_forward_ad_vsplit_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_operator___rmod___cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_operator___rsub___cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_operator__upsample_bilinear2d_aa_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_operator_add_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_operator_allclose_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_operator_amin_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_operator_any_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_operator_arange_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_operator_argsort_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_operator_argwhere_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_operator_as_strided_scatter_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_operator_bernoulli_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_operator_bfloat16_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_operator_bool_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_operator_byte_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_operator_clamp_max_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_operator_combinations_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_operator_count_nonzero_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_operator_cumulative_trapezoid_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_operator_diff_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_operator_dist_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_operator_einsum_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_operator_empty_like_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_operator_empty_strided_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_operator_equal_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_operator_expand_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_operator_fft_fft2_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_operator_fft_ifftn_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_operator_fft_ifftshift_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_operator_fft_ihfftn_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_operator_frac_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_operator_gather_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_operator_half_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_operator_i0_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_operator_index_add_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_operator_inner_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_operator_linalg_cholesky_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_operator_linalg_eig_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_operator_linalg_eigvalsh_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_operator_linalg_norm_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_operator_linalg_tensorinv_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_operator_linalg_vecdot_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_operator_mH_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_operator_mT_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_operator_masked_argmin_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_operator_masked_normalize_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_operator_meshgrid_variadic_tensors_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_operator_min_reduction_no_dim_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_operator_minimum_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_operator_nanmedian_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_operator_nanquantile_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_operator_narrow_copy_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_operator_narrow_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_operator_native_layer_norm_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_operator_nn_functional_adaptive_avg_pool1d_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_operator_nn_functional_bilinear_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_operator_nn_functional_interpolate_linear_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_operator_nn_functional_interpolate_nearest_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_operator_nn_functional_pad_replicate_negative_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_operator_nn_functional_relu_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_operator_nn_functional_tanhshrink_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_operator_nn_functional_triplet_margin_loss_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_operator_norm_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_operator_normal_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_operator_ones_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_operator_ormqr_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_operator_put_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_operator_reciprocal_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_operator_reshape_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_operator_round_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_operator_short_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_operator_signal_windows_general_cosine_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_operator_signal_windows_kaiser_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_operator_signbit_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_operator_special_bessel_j0_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_operator_special_chebyshev_polynomial_t_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_operator_special_chebyshev_polynomial_w_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_operator_special_entr_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_operator_special_scaled_modified_bessel_k0_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_operator_split_list_args_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_operator_split_with_sizes_copy_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_operator_squeeze_multiple_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_operator_std_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_operator_sub_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_operator_trapz_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_operator_true_divide_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_operator_unbind_copy_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_operator_unbind_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_operator_var_mean_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_operator_var_unbiased_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_operator_view_copy_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_view_replay___rpow___cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_view_replay__unsafe_masked_index_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_view_replay_addmm_decomposed_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_view_replay_argmin_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_view_replay_argwhere_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_view_replay_asin_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_view_replay_bfloat16_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_view_replay_block_diag_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_view_replay_bmm_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_view_replay_cdist_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_view_replay_cdouble_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_view_replay_chalf_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_view_replay_clamp_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_view_replay_clamp_min_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_view_replay_cos_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_view_replay_diagonal_copy_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_view_replay_div_floor_rounding_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_view_replay_empty_permuted_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_view_replay_exp2_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_view_replay_expand_as_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_view_replay_expand_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_view_replay_expm1_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_view_replay_fft_hfftn_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_view_replay_fft_ifftn_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_view_replay_fft_ihfftn_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_view_replay_fft_irfft2_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_view_replay_flip_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_view_replay_index_add_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_view_replay_index_put_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_view_replay_isposinf_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_view_replay_linalg_matrix_rank_hermitian_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_view_replay_linalg_solve_triangular_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_view_replay_linalg_vecdot_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_view_replay_linspace_tensor_overload_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_view_replay_log_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_view_replay_logsumexp_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_view_replay_long_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_view_replay_masked_logsumexp_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_view_replay_masked_sum_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_view_replay_max_pool2d_with_indices_backward_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_view_replay_maximum_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_view_replay_meshgrid_variadic_tensors_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_view_replay_minimum_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_view_replay_neg_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_view_replay_nn_functional_elu_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_view_replay_nn_functional_embedding_bag_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_view_replay_nn_functional_embedding_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_view_replay_nn_functional_fractional_max_pool3d_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_view_replay_nn_functional_margin_ranking_loss_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_view_replay_nn_functional_max_unpool1d_grad_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_view_replay_nn_functional_multi_head_attention_forward_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_view_replay_nn_functional_selu_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_view_replay_nn_functional_upsample_bilinear_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_view_replay_norm_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_view_replay_polar_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_view_replay_randint_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_view_replay_randn_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_view_replay_renorm_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_view_replay_searchsorted_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_view_replay_signal_windows_blackman_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_view_replay_signal_windows_hann_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_view_replay_signal_windows_kaiser_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_view_replay_sparse_sampled_addmm_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_view_replay_special_bessel_y0_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_view_replay_special_chebyshev_polynomial_w_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_view_replay_special_i1e_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_view_replay_special_scaled_modified_bessel_k1_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_view_replay_split_with_sizes_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_view_replay_sum_to_size_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_view_replay_true_divide_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_view_replay_unique_consecutive_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_view_replay_var_cuda_float32, test/test_ops.py::TestMathBitsCUDA::test_conj_view___rmatmul___cuda_complex64, test/test_ops.py::TestMathBitsCUDA::test_conj_view__refs__conversions_half_cuda_complex64, test/test_ops.py::TestMathBitsCUDA::test_conj_view__refs_acosh_cuda_complex64, test/test_ops.py::TestMathBitsCUDA::test_conj_view__refs_as_strided_partial_views_cuda_complex64, test/test_ops.py::TestMathBitsCUDA::test_conj_view__refs_asinh_cuda_complex64, test/test_ops.py::TestMathBitsCUDA::test_conj_view__refs_cosh_cuda_complex64, test/test_ops.py::TestMathBitsCUDA::test_conj_view__refs_diag_cuda_complex64, test/test_ops.py::TestMathBitsCUDA::test_conj_view__refs_diag_embed_cuda_complex64, test/test_ops.py::TestMathBitsCUDA::test_conj_view__refs_empty_cuda_complex64, test/test_ops.py::TestMathBitsCUDA::test_conj_view__refs_equal_cuda_complex64, test/test_ops.py::TestMathBitsCUDA::test_conj_view__refs_expand_as_cuda_complex64, test/test_ops.py::TestMathBitsCUDA::test_conj_view__refs_expm1_cuda_complex64, test/test_ops.py::TestMathBitsCUDA::test_conj_view__refs_fft_hfft2_cuda_complex64, test/test_ops.py::TestMathBitsCUDA::test_conj_view__refs_fft_ifft2_cuda_complex64, test/test_ops.py::TestMathBitsCUDA::test_conj_view__refs_linalg_cross_cuda_complex64, test/test_ops.py::TestMathBitsCUDA::test_conj_view__refs_linalg_svdvals_cuda_complex64, test/test_ops.py::TestMathBitsCUDA::test_conj_view__refs_log_cuda_complex64, test/test_ops.py::TestMathBitsCUDA::test_conj_view__refs_narrow_cuda_complex64, test/test_ops.py::TestMathBitsCUDA::test_conj_view__refs_nn_functional_softmax_with_dtype_cuda_complex64, test/test_ops.py::TestMathBitsCUDA::test_conj_view__refs_ones_cuda_complex64, test/test_ops.py::TestMathBitsCUDA::test_conj_view__refs_real_cuda_complex64, test/test_ops.py::TestMathBitsCUDA::test_conj_view__refs_sinc_cuda_complex64, test/test_ops.py::TestMathBitsCUDA::test_conj_view__refs_sum_cuda_complex64, test/test_ops.py::TestMathBitsCUDA::test_conj_view_as_strided_copy_cuda_complex64, test/test_ops.py::TestMathBitsCUDA::test_conj_view_cdouble_cuda_complex64, test/test_ops.py::TestMathBitsCUDA::test_conj_view_conj_cuda_complex64, test/test_ops.py::TestMathBitsCUDA::test_conj_view_constant_pad_nd_cuda_complex64, test/test_ops.py::TestMathBitsCUDA::test_conj_view_cos_cuda_complex64, test/test_ops.py::TestMathBitsCUDA::test_conj_view_cumprod_cuda_complex64, test/test_ops.py::TestMathBitsCUDA::test_conj_view_cumsum_cuda_complex64, test/test_ops.py::TestMathBitsCUDA::test_conj_view_diagflat_cuda_complex64, test/test_ops.py::TestMathBitsCUDA::test_conj_view_diagonal_copy_cuda_complex64, test/test_ops.py::TestMathBitsCUDA::test_conj_view_eq_cuda_complex64, test/test_ops.py::TestMathBitsCUDA::test_conj_view_geqrf_cuda_complex64, test/test_ops.py::TestMathBitsCUDA::test_conj_view_half_cuda_complex64, test/test_ops.py::TestMathBitsCUDA::test_conj_view_imag_cuda_complex64, test/test_ops.py::TestMathBitsCUDA::test_conj_view_inner_cuda_complex64, test/test_ops.py::TestMathBitsCUDA::test_conj_view_isfinite_cuda_complex64, test/test_ops.py::TestMathBitsCUDA::test_conj_view_istft_cuda_complex64, test/test_ops.py::TestMathBitsCUDA::test_conj_view_linalg_det_cuda_complex64, test/test_ops.py::TestMathBitsCUDA::test_conj_view_linalg_matrix_power_cuda_complex64, test/test_ops.py::TestMathBitsCUDA::test_conj_view_linalg_qr_cuda_complex64, test/test_ops.py::TestMathBitsCUDA::test_conj_view_linalg_solve_ex_cuda_complex64, test/test_ops.py::TestMathBitsCUDA::test_conj_view_linalg_tensorinv_cuda_complex64, test/test_ops.py::TestMathBitsCUDA::test_conj_view_log1p_cuda_complex64, test/test_ops.py::TestMathBitsCUDA::test_conj_view_masked_select_cuda_complex64, test/test_ops.py::TestMathBitsCUDA::test_conj_view_masked_std_cuda_complex64, test/test_ops.py::TestMathBitsCUDA::test_conj_view_nn_functional_rms_norm_cuda_complex64, test/test_ops.py::TestMathBitsCUDA::test_conj_view_permute_copy_cuda_complex64, test/test_ops.py::TestMathBitsCUDA::test_conj_view_randn_cuda_complex64, test/test_ops.py::TestMathBitsCUDA::test_conj_view_resize__cuda_complex64, test/test_ops.py::TestMathBitsCUDA::test_conj_view_sin_cuda_complex64, test/test_ops.py::TestMathBitsCUDA::test_conj_view_sinc_cuda_complex64, test/test_ops.py::TestMathBitsCUDA::test_conj_view_split_with_sizes_copy_cuda_complex64, test/test_ops.py::TestMathBitsCUDA::test_conj_view_t_cuda_complex64, test/test_ops.py::TestMathBitsCUDA::test_conj_view_to_sparse_cuda_complex64, test/test_ops.py::TestMathBitsCUDA::test_conj_view_trace_cuda_complex64, test/test_ops.py::TestMathBitsCUDA::test_conj_view_transpose_copy_cuda_complex64, test/test_ops.py::TestMathBitsCUDA::test_conj_view_trapz_cuda_complex64, test/test_ops.py::TestMathBitsCUDA::test_conj_view_unfold_copy_cuda_complex64, test/test_ops.py::TestMathBitsCUDA::test_conj_view_var_cuda_complex64, test/test_ops.py::TestMathBitsCUDA::test_conj_view_vdot_cuda_complex64, test/test_ops.py::TestMathBitsCUDA::test_conj_view_where_cuda_complex64, test/test_ops.py::TestMathBitsCUDA::test_neg_conj_view__refs__conversions_chalf_cuda_complex128, test/test_ops.py::TestMathBitsCUDA::test_neg_conj_view__refs__conversions_double_cuda_complex128, test/test_ops.py::TestMathBitsCUDA::test_neg_conj_view__refs_addcdiv_cuda_complex128, test/test_ops.py::TestMathBitsCUDA::test_neg_conj_view__refs_expand_as_cuda_complex128, test/test_ops.py::TestMathBitsCUDA::test_neg_conj_view__refs_fft_ifft2_cuda_complex128, test/test_ops.py::TestMathBitsCUDA::test_neg_conj_view__refs_fft_ifftn_cuda_complex128, test/test_ops.py::TestMathBitsCUDA::test_neg_conj_view__refs_fft_ifftshift_cuda_complex128, test/test_ops.py::TestMathBitsCUDA::test_neg_conj_view__refs_fill_cuda_complex128, test/test_ops.py::TestMathBitsCUDA::test_neg_conj_view__refs_flipud_cuda_complex128, test/test_ops.py::TestMathBitsCUDA::test_neg_conj_view__refs_index_add_cuda_complex128, test/test_ops.py::TestMathBitsCUDA::test_neg_conj_view__refs_item_cuda_complex128, test/test_ops.py::TestMathBitsCUDA::test_neg_conj_view__refs_linalg_cross_cuda_complex128, test/test_ops.py::TestMathBitsCUDA::test_neg_conj_view__refs_linalg_vecdot_cuda_complex128, test/test_ops.py::TestMathBitsCUDA::test_neg_conj_view__refs_linspace_cuda_complex128, test/test_ops.py::TestMathBitsCUDA::test_neg_conj_view__refs_narrow_copy_cuda_complex128, test/test_ops.py::TestMathBitsCUDA::test_neg_conj_view__refs_new_empty_strided_cuda_complex128, test/test_ops.py::TestMathBitsCUDA::test_neg_conj_view__refs_new_zeros_cuda_complex128, test/test_ops.py::TestMathBitsCUDA::test_neg_conj_view__refs_nn_functional_l1_loss_cuda_complex128, test/test_ops.py::TestMathBitsCUDA::test_neg_conj_view__refs_normal__in_place_cuda_complex128, test/test_ops.py::TestMathBitsCUDA::test_neg_conj_view__refs_ones_cuda_complex128, test/test_ops.py::TestMathBitsCUDA::test_neg_conj_view__refs_repeat_cuda_complex128, test/test_ops.py::TestMathBitsCUDA::test_neg_conj_view__refs_roll_cuda_complex128, test/test_ops.py::TestMathBitsCUDA::test_neg_conj_view__refs_softmax_with_dtype_cuda_complex128, test/test_ops.py::TestMathBitsCUDA::test_neg_conj_view__refs_std_cuda_complex128, test/test_ops.py::TestMathBitsCUDA::test_neg_conj_view__refs_unsqueeze_copy_cuda_complex128, test/test_ops.py::TestMathBitsCUDA::test_neg_conj_view_abs_cuda_complex128, test/test_ops.py::TestMathBitsCUDA::test_neg_conj_view_add_cuda_complex128, test/test_ops.py::TestMathBitsCUDA::test_neg_conj_view_addr_cuda_complex128, test/test_ops.py::TestMathBitsCUDA::test_neg_conj_view_any_cuda_complex128, test/test_ops.py::TestMathBitsCUDA::test_neg_conj_view_as_strided_cuda_complex128, test/test_ops.py::TestMathBitsCUDA::test_neg_conj_view_cartesian_prod_cuda_complex128, test/test_ops.py::TestMathBitsCUDA::test_neg_conj_view_chunk_cuda_complex128, test/test_ops.py::TestMathBitsCUDA::test_neg_conj_view_conj_cuda_complex128, test/test_ops.py::TestMathBitsCUDA::test_neg_conj_view_cumprod_cuda_complex128, test/test_ops.py::TestMathBitsCUDA::test_neg_conj_view_cumsum_cuda_complex128, test/test_ops.py::TestMathBitsCUDA::test_neg_conj_view_diagonal_cuda_complex128, test/test_ops.py::TestMathBitsCUDA::test_neg_conj_view_diff_cuda_complex128, test/test_ops.py::TestMathBitsCUDA::test_neg_conj_view_dist_cuda_complex128, test/test_ops.py::TestMathBitsCUDA::test_neg_conj_view_empty_strided_cuda_complex128, test/test_ops.py::TestMathBitsCUDA::test_neg_conj_view_fft_irfft_cuda_complex128, test/test_ops.py::TestMathBitsCUDA::test_neg_conj_view_fliplr_cuda_complex128, test/test_ops.py::TestMathBitsCUDA::test_neg_conj_view_float_power_cuda_complex128, test/test_ops.py::TestMathBitsCUDA::test_neg_conj_view_gather_cuda_complex128, test/test_ops.py::TestMathBitsCUDA::test_neg_conj_view_half_cuda_complex128, test/test_ops.py::TestMathBitsCUDA::test_neg_conj_view_hsplit_cuda_complex128, test/test_ops.py::TestMathBitsCUDA::test_neg_conj_view_imag_cuda_complex128, test/test_ops.py::TestMathBitsCUDA::test_neg_conj_view_isfinite_cuda_complex128, test/test_ops.py::TestMathBitsCUDA::test_neg_conj_view_isinf_cuda_complex128, test/test_ops.py::TestMathBitsCUDA::test_neg_conj_view_kron_cuda_complex128, test/test_ops.py::TestMathBitsCUDA::test_neg_conj_view_linalg_eigvals_cuda_complex128, test/test_ops.py::TestMathBitsCUDA::test_neg_conj_view_linalg_lstsq_cuda_complex128, test/test_ops.py::TestMathBitsCUDA::test_neg_conj_view_linalg_lu_solve_cuda_complex128, test/test_ops.py::TestMathBitsCUDA::test_neg_conj_view_linalg_matrix_power_cuda_complex128, test/test_ops.py::TestMathBitsCUDA::test_neg_conj_view_linalg_multi_dot_cuda_complex128, test/test_ops.py::TestMathBitsCUDA::test_neg_conj_view_linalg_pinv_cuda_complex128, test/test_ops.py::TestMathBitsCUDA::test_neg_conj_view_log1p_cuda_complex128, test/test_ops.py::TestMathBitsCUDA::test_neg_conj_view_log2_cuda_complex128, test/test_ops.py::TestMathBitsCUDA::test_neg_conj_view_masked_std_cuda_complex128, test/test_ops.py::TestMathBitsCUDA::test_neg_conj_view_masked_var_cuda_complex128, test/test_ops.py::TestMathBitsCUDA::test_neg_conj_view_narrow_copy_cuda_complex128, test/test_ops.py::TestMathBitsCUDA::test_neg_conj_view_ne_cuda_complex128, test/test_ops.py::TestMathBitsCUDA::test_neg_conj_view_neg_cuda_complex128, test/test_ops.py::TestMathBitsCUDA::test_neg_conj_view_nn_functional_conv1d_cuda_complex128, test/test_ops.py::TestMathBitsCUDA::test_neg_conj_view_nn_functional_conv2d_cuda_complex128, test/test_ops.py::TestMathBitsCUDA::test_neg_conj_view_nn_functional_conv_transpose2d_cuda_complex128, test/test_ops.py::TestMathBitsCUDA::test_neg_conj_view_nn_functional_rms_norm_cuda_complex128, test/test_ops.py::TestMathBitsCUDA::test_neg_conj_view_nonzero_cuda_complex128, test/test_ops.py::TestMathBitsCUDA::test_neg_conj_view_ormqr_cuda_complex128, test/test_ops.py::TestMathBitsCUDA::test_neg_conj_view_put_cuda_complex128, test/test_ops.py::TestMathBitsCUDA::test_neg_conj_view_reshape_as_cuda_complex128, test/test_ops.py::TestMathBitsCUDA::test_neg_conj_view_scalar_tensor_cuda_complex128, test/test_ops.py::TestMathBitsCUDA::test_neg_conj_view_scatter_add_cuda_complex128, test/test_ops.py::TestMathBitsCUDA::test_neg_conj_view_sin_cuda_complex128, test/test_ops.py::TestMathBitsCUDA::test_neg_conj_view_squeeze_copy_cuda_complex128, test/test_ops.py::TestMathBitsCUDA::test_neg_conj_view_stack_cuda_complex128, test/test_ops.py::TestMathBitsCUDA::test_neg_conj_view_tensor_split_cuda_complex128, test/test_ops.py::TestMathBitsCUDA::test_neg_conj_view_trapz_cuda_complex128, test/test_ops.py::TestMathBitsCUDA::test_neg_conj_view_true_divide_cuda_complex128, test/test_ops.py::TestMathBitsCUDA::test_neg_conj_view_unbind_cuda_complex128, test/test_ops.py::TestMathBitsCUDA::test_neg_conj_view_unsafe_chunk_cuda_complex128, test/test_ops.py::TestMathBitsCUDA::test_neg_conj_view_unsafe_split_cuda_complex128, test/test_ops.py::TestMathBitsCUDA::test_neg_conj_view_zeros_like_cuda_complex128, test/test_ops.py::TestMathBitsCUDA::test_neg_view_H_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view__refs_T_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view__refs_acos_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view__refs_chunk_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view__refs_clamp_max_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view__refs_erfinv_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view__refs_expand_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view__refs_fft_ifft_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view__refs_hypot_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view__refs_isnan_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view__refs_isreal_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view__refs_linalg_diagonal_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view__refs_nn_functional_nll_loss_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view__refs_nn_functional_pairwise_distance_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view__refs_prod_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view__refs_rad2deg_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view__refs_reciprocal_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view__refs_remainder_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view__refs_sgn_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view__refs_sinc_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view__refs_special_multigammaln_mvlgamma_p_3_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view__refs_squeeze_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view__refs_stft_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view__refs_trunc_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view__refs_unflatten_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view__refs_unfold_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view__refs_var_mean_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view__softmax_backward_data_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view_addbmm_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view_addmm_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view_addmm_decomposed_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view_addr_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view_argwhere_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view_bfloat16_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view_block_diag_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view_cartesian_prod_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view_chunk_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view_clamp_min_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view_cos_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view_count_nonzero_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view_diagonal_scatter_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view_empty_strided_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view_fliplr_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view_heaviside_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view_i0_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view_igamma_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view_igammac_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view_index_add_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view_linalg_det_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view_linalg_norm_subgradients_at_zero_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view_linalg_svdvals_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view_linalg_vander_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view_linalg_vecdot_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view_linalg_vector_norm_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view_log_softmax_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view_logical_xor_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view_long_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view_masked_fill_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view_masked_logaddexp_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view_masked_scatter_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view_masked_sum_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view_max_pool2d_with_indices_backward_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view_meshgrid_variadic_tensors_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view_minimum_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view_mv_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view_nanmedian_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view_nansum_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view_native_batch_norm_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view_nn_functional_avg_pool1d_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view_nn_functional_avg_pool2d_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view_nn_functional_binary_cross_entropy_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view_nn_functional_conv3d_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view_nn_functional_feature_alpha_dropout_with_train_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view_nn_functional_grid_sample_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view_nn_functional_hardsigmoid_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view_nn_functional_instance_norm_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view_nn_functional_max_unpool3d_grad_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view_nn_functional_mse_loss_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view_nn_functional_pad_replicate_negative_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view_nn_functional_pixel_unshuffle_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view_nn_functional_poisson_nll_loss_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view_nn_functional_smooth_l1_loss_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view_nn_functional_soft_margin_loss_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view_nn_functional_softmin_with_dtype_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view_nn_functional_softshrink_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view_nn_functional_softsign_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view_nn_functional_threshold_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view_norm_nuc_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view_normal_in_place_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view_prod_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view_randn_like_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view_repeat_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view_round_decimals_3_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view_signal_windows_cosine_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view_signal_windows_gaussian_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view_signal_windows_general_hamming_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view_signbit_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view_sinh_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view_special_airy_ai_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view_special_bessel_j0_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view_special_chebyshev_polynomial_w_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view_special_erfcx_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view_special_shifted_chebyshev_polynomial_t_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view_special_shifted_chebyshev_polynomial_u_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view_tan_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view_topk_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view_trace_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view_tril_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view_unsqueeze_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view_var_mean_unbiased_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view_vdot_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view_view_as_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view_vsplit_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view_vstack_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view_where_cuda_float64, test/test_ops.py::TestFakeTensorCUDA::test_fake___rmul___cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake__segment_reduce_lengths_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_addbmm_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_amax_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_angle_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_arange_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_atan_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_autocast___rsub___cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_autocast___rxor___cuda_int64, test/test_ops.py::TestFakeTensorCUDA::test_fake_autocast__unsafe_masked_index_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_autocast_add_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_autocast_addcdiv_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_autocast_as_strided_scatter_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_autocast_cat_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_autocast_cfloat_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_autocast_cov_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_autocast_cross_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_autocast_div_floor_rounding_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_autocast_fft_irfft2_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_autocast_fft_irfft_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_autocast_flipud_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_autocast_gather_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_autocast_geqrf_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_autocast_grid_sampler_2d_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_autocast_histc_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_autocast_hstack_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_autocast_hypot_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_autocast_index_put_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_autocast_index_reduce_mean_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_autocast_isclose_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_autocast_jiterator_unary_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_autocast_lcm_cuda_int64, test/test_ops.py::TestFakeTensorCUDA::test_fake_autocast_linalg_eigvalsh_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_autocast_linalg_matrix_rank_hermitian_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_autocast_linalg_tensorinv_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_autocast_linspace_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_autocast_log2_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_autocast_logaddexp2_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_autocast_logdet_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_autocast_masked_log_softmax_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_autocast_masked_softmax_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_autocast_masked_var_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_autocast_max_reduction_with_dim_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_autocast_meshgrid_list_of_tensors_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_autocast_mode_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_autocast_narrow_copy_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_autocast_neg_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_autocast_new_empty_strided_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_autocast_nn_functional_adaptive_avg_pool3d_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_autocast_nn_functional_batch_norm_without_cudnn_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_autocast_nn_functional_dropout2d_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_autocast_nn_functional_glu_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_autocast_nn_functional_interpolate_area_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_autocast_nn_functional_interpolate_linear_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_autocast_nn_functional_interpolate_nearest_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_autocast_nn_functional_multilabel_margin_loss_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_autocast_nn_functional_pad_replicate_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_autocast_nn_functional_relu_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_autocast_nn_functional_softmin_with_dtype_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_autocast_nn_functional_softshrink_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_autocast_nn_functional_upsample_nearest_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_autocast_norm_inf_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_autocast_norm_nuc_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_autocast_pinverse_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_autocast_qr_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_autocast_remainder_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_autocast_reshape_as_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_autocast_reshape_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_autocast_scalar_tensor_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_autocast_select_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_autocast_signal_windows_bartlett_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_autocast_signal_windows_hann_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_autocast_special_chebyshev_polynomial_w_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_autocast_special_entr_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_autocast_special_i1e_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_autocast_special_laguerre_polynomial_l_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_autocast_special_scaled_modified_bessel_k1_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_autocast_var_mean_unbiased_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_bitwise_and_cuda_int64, test/test_ops.py::TestFakeTensorCUDA::test_fake_clone_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_amp___rdiv___cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_amp___rmod___cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_amp__segment_reduce_lengths_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_amp_abs_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_amp_as_strided_scatter_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_amp_asin_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_amp_cat_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_amp_complex_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_amp_conj_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_amp_conj_physical_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_amp_div_floor_rounding_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_amp_div_no_rounding_mode_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_amp_expand_copy_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_amp_fft_fftn_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_amp_flatten_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_amp_flipud_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_amp_hsplit_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_amp_index_put_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_amp_linalg_lstsq_grad_oriented_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_amp_linalg_qr_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_amp_linalg_vander_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_amp_logaddexp2_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_amp_lu_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_amp_masked_amax_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_amp_masked_fill_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_amp_masked_prod_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_amp_max_reduction_with_dim_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_amp_mm_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_amp_mode_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_amp_msort_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_amp_nn_functional_adaptive_avg_pool1d_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_amp_nn_functional_adaptive_avg_pool3d_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_amp_nn_functional_conv3d_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_amp_nn_functional_dropout_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_amp_nn_functional_embedding_bag_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_amp_nn_functional_feature_alpha_dropout_with_train_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_amp_nn_functional_hardshrink_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_amp_nn_functional_interpolate_area_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_amp_nn_functional_max_unpool3d_grad_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_amp_nn_functional_mse_loss_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_amp_nn_functional_multi_head_attention_forward_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_amp_nn_functional_nll_loss_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_amp_nn_functional_pad_constant_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_amp_nn_functional_pad_replicate_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_amp_nn_functional_pad_replicate_negative_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_amp_nn_functional_prelu_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_amp_nn_functional_rms_norm_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_amp_nn_functional_scaled_dot_product_attention_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_amp_nn_functional_silu_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_amp_nn_functional_softmin_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_amp_nn_functional_softsign_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_amp_norm_fro_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_amp_outer_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_amp_polar_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_amp_put_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_amp_ravel_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_amp_reciprocal_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_amp_select_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_amp_slice_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_amp_split_list_args_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_amp_svd_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_amp_t_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_amp_topk_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_amp_transpose_copy_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_amp_unsafe_chunk_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_amp_view_as_complex_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_no_amp_T_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_no_amp___getitem___cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_no_amp___rmod___cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_no_amp_as_strided_partial_views_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_no_amp_as_strided_scatter_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_no_amp_broadcast_tensors_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_no_amp_cdouble_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_no_amp_ceil_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_no_amp_cfloat_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_no_amp_combinations_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_no_amp_complex_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_no_amp_conj_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_no_amp_dist_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_no_amp_div_floor_rounding_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_no_amp_fft_fftshift_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_no_amp_fft_hfft_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_no_amp_flip_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_no_amp_fliplr_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_no_amp_float_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_no_amp_hstack_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_no_amp_index_put_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_no_amp_logaddexp2_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_no_amp_logsumexp_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_no_amp_lu_solve_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_no_amp_masked_amin_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_no_amp_masked_normalize_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_no_amp_masked_select_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_no_amp_matrix_exp_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_no_amp_min_binary_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_no_amp_nn_functional_alpha_dropout_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_no_amp_nn_functional_batch_norm_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_no_amp_nn_functional_batch_norm_without_cudnn_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_no_amp_nn_functional_bilinear_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_no_amp_nn_functional_conv_transpose2d_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_no_amp_nn_functional_cosine_embedding_loss_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_no_amp_nn_functional_cosine_similarity_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_no_amp_nn_functional_dropout2d_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_no_amp_nn_functional_feature_alpha_dropout_with_train_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_no_amp_nn_functional_fractional_max_pool2d_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_no_amp_nn_functional_hardsigmoid_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_no_amp_nn_functional_hinge_embedding_loss_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_no_amp_nn_functional_interpolate_bilinear_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_no_amp_nn_functional_interpolate_trilinear_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_no_amp_nn_functional_kl_div_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_no_amp_nn_functional_mse_loss_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_no_amp_nn_functional_softshrink_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_no_amp_polygamma_polygamma_n_4_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_no_amp_pow_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_no_amp_quantile_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_no_amp_sqrt_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_no_amp_std_mean_unbiased_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_no_amp_t_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_no_amp_tan_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_no_amp_tanh_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_no_amp_trace_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_no_amp_unsqueeze_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_no_amp_vsplit_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_diagonal_scatter_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_dist_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_div_no_rounding_mode_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_expm1_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_fliplr_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_frac_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_frexp_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_geqrf_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_gradient_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_gt_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_igammac_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_index_put_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_jiterator_unary_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_kthvalue_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_linalg_det_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_linalg_eigvals_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_linalg_ldl_solve_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_linalg_multi_dot_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_linalg_norm_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_linalg_pinv_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_linalg_slogdet_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_linalg_solve_ex_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_linalg_tensorinv_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_linspace_tensor_overload_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_log_normal_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_logaddexp_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_logical_not_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_logspace_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_logspace_tensor_overload_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_logsumexp_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_long_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_masked_fill_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_masked_norm_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_masked_prod_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_maximum_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_median_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_min_reduction_with_dim_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_minimum_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_neg_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_new_full_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_nn_functional_avg_pool3d_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_nn_functional_fractional_max_pool3d_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_nn_functional_hardtanh_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_nn_functional_interpolate_area_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_nn_functional_interpolate_nearest_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_nn_functional_max_unpool1d_grad_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_nn_functional_multi_head_attention_forward_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_nn_functional_pad_circular_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_nn_functional_pad_replicate_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_nn_functional_relu_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_nn_functional_scaled_dot_product_attention_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_nn_functional_silu_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_ones_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_outer_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_randint_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_randint_like_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_real_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_scatter_reduce_amax_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_short_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_signal_windows_blackman_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_signal_windows_gaussian_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_sinc_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_softmax_with_dtype_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_special_airy_ai_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_special_chebyshev_polynomial_t_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_special_i1_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_special_legendre_polynomial_p_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_special_scaled_modified_bessel_k0_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_special_spherical_bessel_j0_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_split_with_sizes_copy_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_torch__scaled_mm_cuda_float8_e4m3fn, test/test_ops.py::TestFakeTensorCUDA::test_fake_tril_indices_cuda_int64, test/test_ops.py::TestFakeTensorCUDA::test_fake_triu_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_triu_indices_cuda_int64, test/test_ops.py::TestFakeTensorCUDA::test_fake_unique_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_unsqueeze_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_pointwise_ops___rmul___cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_pointwise_ops__unsafe_masked_index_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_pointwise_ops__unsafe_masked_index_put_accumulate_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_pointwise_ops_addmm_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_pointwise_ops_amax_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_pointwise_ops_any_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_pointwise_ops_argwhere_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_pointwise_ops_as_strided_partial_views_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_pointwise_ops_as_strided_scatter_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_pointwise_ops_asinh_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_pointwise_ops_atleast_2d_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_pointwise_ops_baddbmm_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_pointwise_ops_bitwise_or_cuda_int64, test/test_ops.py::TestFakeTensorCUDA::test_pointwise_ops_broadcast_tensors_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_pointwise_ops_ceil_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_pointwise_ops_chunk_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_pointwise_ops_clamp_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_pointwise_ops_conj_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_pointwise_ops_cosh_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_pointwise_ops_count_nonzero_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_pointwise_ops_cummin_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_pointwise_ops_dsplit_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_pointwise_ops_empty_strided_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_pointwise_ops_exponential_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_pointwise_ops_fft_fft2_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_pointwise_ops_fft_hfft2_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_pointwise_ops_fft_irfft_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_pointwise_ops_fft_rfftn_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_pointwise_ops_flatten_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_pointwise_ops_fmax_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_pointwise_ops_geqrf_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_pointwise_ops_grid_sampler_3d_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_pointwise_ops_isinf_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_pointwise_ops_isnan_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_pointwise_ops_ldexp_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_pointwise_ops_linalg_lu_factor_ex_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_pointwise_ops_linalg_matrix_rank_hermitian_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_pointwise_ops_linalg_norm_subgradients_at_zero_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_pointwise_ops_linalg_solve_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_pointwise_ops_linspace_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_pointwise_ops_log_normal_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_pointwise_ops_logical_xor_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_pointwise_ops_logsumexp_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_pointwise_ops_mH_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_pointwise_ops_masked_mean_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_pointwise_ops_matmul_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_pointwise_ops_max_pool2d_with_indices_backward_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_pointwise_ops_maximum_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_pointwise_ops_meshgrid_variadic_tensors_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_pointwise_ops_mvlgamma_mvlgamma_p_3_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_pointwise_ops_narrow_copy_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_pointwise_ops_neg_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_pointwise_ops_new_empty_strided_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_pointwise_ops_new_full_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_pointwise_ops_nn_functional_adaptive_max_pool1d_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_pointwise_ops_nn_functional_conv2d_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_pointwise_ops_nn_functional_conv_transpose3d_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_pointwise_ops_nn_functional_cross_entropy_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_pointwise_ops_nn_functional_fractional_max_pool2d_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_pointwise_ops_nn_functional_fractional_max_pool3d_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_pointwise_ops_nn_functional_gelu_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_pointwise_ops_nn_functional_group_norm_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_pointwise_ops_nn_functional_hardtanh_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_pointwise_ops_nn_functional_huber_loss_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_pointwise_ops_nn_functional_margin_ranking_loss_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_pointwise_ops_nn_functional_normalize_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_pointwise_ops_nn_functional_poisson_nll_loss_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_pointwise_ops_nn_functional_rrelu_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_pointwise_ops_nn_functional_softsign_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_pointwise_ops_nn_functional_unfold_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_pointwise_ops_pow_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_pointwise_ops_qr_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_pointwise_ops_repeat_interleave_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_pointwise_ops_rsub_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_pointwise_ops_signal_windows_blackman_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_pointwise_ops_signal_windows_nuttall_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_pointwise_ops_signbit_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_pointwise_ops_special_i0e_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_pointwise_ops_special_i1e_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_pointwise_ops_special_modified_bessel_k1_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_pointwise_ops_special_polygamma_special_polygamma_n_0_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_pointwise_ops_tan_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_pointwise_ops_topk_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_pointwise_ops_torch__scaled_mm_cuda_float8_e4m3fn, test/test_ops.py::TestFakeTensorCUDA::test_pointwise_ops_torch_ops_aten__safe_softmax_default_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_pointwise_ops_transpose_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_pointwise_ops_triu_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_pointwise_ops_unflatten_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_pointwise_ops_unsqueeze_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_pointwise_ops_view_as_complex_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_strided_layout__refs_linspace_tensor_overload_cuda_complex128, test/test_ops.py::TestFakeTensorCUDA::test_strided_layout__refs_linspace_tensor_overload_cuda_int32, test/test_ops.py::TestFakeTensorCUDA::test_strided_layout__refs_linspace_tensor_overload_cuda_int64, test/test_ops.py::TestFakeTensorCUDA::test_strided_layout__refs_logspace_cuda_float16, test/test_ops.py::TestFakeTensorCUDA::test_strided_layout__refs_logspace_tensor_overload_cuda_int8, test/test_ops.py::TestFakeTensorCUDA::test_strided_layout__refs_logspace_tensor_overload_cuda_uint8, test/test_ops.py::TestFakeTensorCUDA::test_strided_layout__refs_ones_cuda_float64, test/test_ops.py::TestFakeTensorCUDA::test_strided_layout__refs_ones_cuda_int16, test/test_ops.py::TestFakeTensorCUDA::test_strided_layout_full_cuda_int32, test/test_ops.py::TestFakeTensorCUDA::test_strided_layout_full_cuda_int8, test/test_ops.py::TestFakeTensorCUDA::test_strided_layout_linspace_tensor_overload_cuda_float64, test/test_ops.py::TestFakeTensorCUDA::test_strided_layout_linspace_tensor_overload_cuda_int64, test/test_ops.py::TestFakeTensorCUDA::test_strided_layout_logspace_cuda_complex64, test/test_ops.py::TestFakeTensorCUDA::test_strided_layout_logspace_cuda_float64, test/test_ops.py::TestFakeTensorCUDA::test_strided_layout_logspace_tensor_overload_cuda_float16, test/test_ops.py::TestFakeTensorCUDA::test_strided_layout_zeros_cuda_bfloat16, test/test_ops.py::TestTagsCUDA::test_tags__native_batch_norm_legit_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags__refs__conversions_cfloat_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags__refs__conversions_half_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags__refs_addcdiv_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags__refs_bitwise_left_shift_cuda_int64, test/test_ops.py::TestTagsCUDA::test_tags__refs_bitwise_not_cuda_int64, test/test_ops.py::TestTagsCUDA::test_tags__refs_cumsum_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags__refs_div_floor_rounding_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags__refs_dstack_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags__refs_erfc_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags__refs_expm1_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags__refs_fliplr_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags__refs_gcd_cuda_int64, test/test_ops.py::TestTagsCUDA::test_tags__refs_igammac_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags__refs_lerp_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags__refs_linalg_cross_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags__refs_linspace_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags__refs_linspace_tensor_overload_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags__refs_logical_not_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags__refs_maximum_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags__refs_mul_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags__refs_nextafter_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags__refs_nn_functional_huber_loss_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags__refs_nn_functional_layer_norm_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags__refs_nn_functional_pdist_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags__refs_nn_functional_tanhshrink_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags__refs_permute_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags__refs_rad2deg_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags__refs_reshape_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags__refs_round_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags__refs_rsub_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags__refs_special_i1e_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags__refs_special_multigammaln_mvlgamma_p_1_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags__refs_special_ndtr_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags__refs_stack_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags__refs_tril_indices_cuda_int64, test/test_ops.py::TestTagsCUDA::test_tags_acosh_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags_addcmul_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags_bfloat16_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags_bitwise_not_cuda_int64, test/test_ops.py::TestTagsCUDA::test_tags_bool_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags_cdouble_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags_cholesky_inverse_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags_cholesky_solve_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags_clone_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags_contiguous_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags_diagonal_copy_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags_empty_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags_empty_like_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags_empty_permuted_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags_empty_strided_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags_erf_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags_expm1_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags_fft_hfft2_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags_fft_ihfft2_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags_fft_ihfft_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags_fft_irfft_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags_fmax_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags_full_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags_ge_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags_geometric_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags_hash_tensor_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags_histc_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags_hsplit_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags_inner_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags_isinf_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags_isreal_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags_item_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags_jiterator_binary_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags_linalg_lu_factor_ex_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags_linalg_pinv_hermitian_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags_linalg_svdvals_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags_linalg_vecdot_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags_long_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags_mT_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags_masked_amin_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags_max_pool2d_with_indices_backward_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags_mean_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags_meshgrid_variadic_tensors_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags_min_binary_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags_mode_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags_narrow_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags_nn_functional_bilinear_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags_nn_functional_ctc_loss_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags_nn_functional_dropout3d_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags_nn_functional_elu_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags_nn_functional_embedding_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags_nn_functional_fractional_max_pool3d_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags_nn_functional_grid_sample_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags_nn_functional_huber_loss_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags_nn_functional_interpolate_trilinear_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags_nn_functional_l1_loss_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags_nn_functional_local_response_norm_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags_nn_functional_max_unpool3d_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags_nn_functional_prelu_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags_nn_functional_relu6_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags_nn_functional_rrelu_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags_polygamma_polygamma_n_3_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags_prod_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags_put_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags_randn_like_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags_select_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags_signal_windows_nuttall_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags_softmax_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags_sort_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags_special_bessel_j0_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags_special_hermite_polynomial_he_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags_special_ndtr_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags_special_scaled_modified_bessel_k0_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags_std_mean_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags_std_mean_unbiased_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags_std_unbiased_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags_svd_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags_trapz_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags_unsafe_split_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags_var_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags_zero__cuda_float32, test/test_ops.py::TestForwardADWithScalarsCUDA::test_0d_tensor_with_python_scalar_add_cuda_float32 2025-12-04T12:47:10.6117060Z 2025-12-04T12:47:10.6117273Z Finished test_ops 7/9 ... [2025-12-04 12:47:10.436788][11278.379000062], took 24.33min 2025-12-04T12:47:10.6118046Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/test_ops/test_ops-383153bca27cf48b.xml 2025-12-04T12:47:11.1943421Z Uploading artifacts took 0.54 seconds 2025-12-04T12:47:11.1946342Z Running functorch/test_ops 2/8 ... [2025-12-04 12:47:11.194390][11279.136604999] 2025-12-04T12:47:11.1946795Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2025-12-04T12:47:11.1949644Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'functorch/test_ops.py', '--shard-id=2', '--num-shards=8', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-12-04 12:47:11.194718] 2025-12-04T12:57:41.9217587Z 2025-12-04T12:57:41.9218484Z functorch/test_ops 2/8 was successful, full logs can be found in artifacts with path test/test-reports/functorch.test_ops_2.8_bb188d7ff7219e71_.log 2025-12-04T12:57:41.9550406Z Running 1245 items in this shard: test/functorch/test_ops.py::TestOperatorsCUDA::test_grad_ForwardHasDefaultArgsAutogradFunction_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_grad__segment_reduce_offsets_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_grad__unsafe_masked_index_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_grad__upsample_bilinear2d_aa_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_grad_abs_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_grad_amax_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_grad_argmin_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_grad_argsort_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_grad_asin_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_grad_atan_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_grad_atanh_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_grad_bool_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_grad_broadcast_shapes_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_grad_byte_functorch_no_channels_last_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_grad_cat_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_grad_chalf_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_grad_combinations_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_grad_diagflat_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_grad_diagonal_copy_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_grad_dstack_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_grad_empty_permuted_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_grad_expand_as_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_grad_expand_copy_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_grad_expand_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_grad_fft_fftshift_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_grad_fft_irfft2_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_grad_frexp_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_grad_gather_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_grad_hsplit_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_grad_hypot_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_grad_igamma_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_grad_inner_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_grad_int_functorch_no_channels_last_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_grad_isfinite_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_grad_isneginf_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_grad_item_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_grad_kthvalue_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_grad_linalg_lstsq_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_grad_linalg_pinv_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_grad_linalg_solve_ex_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_grad_linalg_vector_norm_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_grad_logdet_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_grad_logical_not_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_grad_logical_xor_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_grad_long_functorch_no_channels_last_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_grad_masked_amin_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_grad_masked_mean_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_grad_masked_normalize_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_grad_masked_scatter_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_grad_max_binary_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_grad_minimum_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_grad_movedim_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_grad_mul_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_grad_multinomial_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_grad_mvlgamma_mvlgamma_p_3_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_grad_nanmedian_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_grad_new_zeros_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_grad_nn_functional_adaptive_max_pool3d_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_grad_nn_functional_channel_shuffle_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_grad_nn_functional_conv2d_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_grad_nn_functional_conv2d_stride_with_bias_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_grad_nn_functional_conv3d_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_grad_nn_functional_conv_transpose1d_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_grad_nn_functional_cosine_embedding_loss_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_grad_nn_functional_cross_entropy_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_grad_nn_functional_fractional_max_pool3d_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_grad_nn_functional_grid_sample_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_grad_nn_functional_hardsigmoid_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_grad_nn_functional_instance_norm_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_grad_nn_functional_interpolate_bilinear_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_grad_nn_functional_linear_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_grad_nn_functional_max_unpool2d_grad_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_grad_nn_functional_nll_loss_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_grad_nn_functional_normalize_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_grad_nn_functional_poisson_nll_loss_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_grad_nn_functional_prelu_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_grad_nn_functional_upsample_bilinear_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_grad_real_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_grad_remainder_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_grad_repeat_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_grad_reshape_as_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_grad_round_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_grad_round_decimals_0_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_grad_rsub_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_grad_scalar_tensor_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_grad_sgn_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_grad_signal_windows_general_cosine_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_grad_special_chebyshev_polynomial_v_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_grad_split_list_args_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_grad_std_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_grad_sub_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_grad_torch_ops_aten__efficient_attention_forward_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_grad_unflatten_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_grad_unique_consecutive_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_grad_unsqueeze_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_grad_view_as_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_grad_where_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_grad_xlogy_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_jvp_ForwardHasDefaultArgsAutogradFunction_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_jvp_SortGenVmapAutogradFunction_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_jvp_ZeroGradientsGenVmapAutogradFunction_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_jvp_argmax_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_jvp_as_strided_partial_views_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_jvp_atan_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_jvp_bfloat16_functorch_no_channels_last_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_jvp_byte_functorch_no_channels_last_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_jvp_cdist_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_jvp_diag_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_jvp_diff_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_jvp_empty_strided_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_jvp_expand_as_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_jvp_expand_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_jvp_fft_hfft_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_jvp_fft_ifft_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_jvp_fft_ihfft_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_jvp_fft_ihfftn_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_jvp_fft_irfft_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_jvp_flatten_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_jvp_flip_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_jvp_fmax_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_jvp_frexp_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_jvp_full_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_jvp_grid_sampler_2d_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_jvp_index_fill_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_jvp_index_put_functorch_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_jvp_isfinite_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_jvp_isposinf_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_jvp_jiterator_2inputs_2outputs_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_jvp_linalg_diagonal_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_jvp_linalg_lu_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_jvp_linalg_lu_solve_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_jvp_linalg_norm_subgradients_at_zero_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_jvp_linalg_svdvals_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_jvp_log_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_jvp_logaddexp2_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_jvp_lu_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_jvp_mH_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_jvp_masked_softmax_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_jvp_masked_std_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_jvp_matmul_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_jvp_matrix_exp_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_jvp_meshgrid_list_of_tensors_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_jvp_min_reduction_with_dim_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_jvp_mvlgamma_mvlgamma_p_5_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_jvp_nanmean_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_jvp_new_zeros_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_jvp_nn_functional_adaptive_avg_pool3d_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_jvp_nn_functional_alpha_dropout_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_jvp_nn_functional_avg_pool1d_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_jvp_nn_functional_avg_pool2d_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_jvp_nn_functional_avg_pool3d_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_jvp_nn_functional_conv2d_stride_groups_with_bias_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_jvp_nn_functional_conv2d_strided_padding_dilation_with_bias_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_jvp_nn_functional_conv_transpose2d_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_jvp_nn_functional_hinge_embedding_loss_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_jvp_nn_functional_interpolate_nearest-exact_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_jvp_nn_functional_l1_loss_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_jvp_nn_functional_max_pool1d_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_jvp_nn_functional_pad_replicate_negative_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_jvp_nn_functional_pairwise_distance_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_jvp_nn_functional_prelu_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_jvp_nn_functional_silu_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_jvp_nn_functional_softshrink_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_jvp_nonzero_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_jvp_norm_inf_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_jvp_ops_aten_index_put_functorch_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_jvp_permute_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_jvp_quantile_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_jvp_rad2deg_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_jvp_real_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_jvp_reshape_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_jvp_rot90_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_jvp_scatter_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_jvp_scatter_reduce_amax_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_jvp_short_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_jvp_signal_windows_gaussian_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_jvp_signal_windows_general_cosine_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_jvp_signal_windows_hann_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_jvp_special_bessel_j0_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_jvp_special_chebyshev_polynomial_w_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_jvp_special_ndtri_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_jvp_std_mean_unbiased_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_jvp_sum_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_jvp_t_copy_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_jvp_unflatten_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_jvp_var_unbiased_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_jvp_view_as_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_jvp_view_copy_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_jvpjvpvmap_NumpyCubeNotComposableAutogradFunction_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_jvpjvpvmap_SortGenVmapAutogradFunction_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_jvpvjp_CubeGenVmapAutogradFunction_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_jvpvjp_NumpyTakeAutogradFunction_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_jvpvjp___getitem___functorch_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_jvpvjp_addr_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_jvpvjp_argsort_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_jvpvjp_atan2_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_jvpvjp_atleast_3d_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_jvpvjp_bucketize_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_jvpvjp_byte_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_jvpvjp_cat_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_jvpvjp_cdist_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_jvpvjp_cholesky_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_jvpvjp_column_stack_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_jvpvjp_conj_physical_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_jvpvjp_cross_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_jvpvjp_cumsum_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_jvpvjp_diagonal_scatter_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_jvpvjp_dsplit_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_jvpvjp_einsum_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_jvpvjp_empty_permuted_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_jvpvjp_erf_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_jvpvjp_expand_as_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_jvpvjp_fft_hfft_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_jvpvjp_fft_rfft_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_jvpvjp_float_power_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_jvpvjp_fmod_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_jvpvjp_grid_sampler_2d_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_jvpvjp_index_reduce_mean_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_jvpvjp_jiterator_binary_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_jvpvjp_lerp_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_jvpvjp_linalg_diagonal_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_jvpvjp_linalg_lu_factor_ex_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_jvpvjp_linalg_matrix_rank_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_jvpvjp_log10_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_jvpvjp_long_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_jvpvjp_masked_normalize_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_jvpvjp_masked_softmax_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_jvpvjp_min_reduction_with_dim_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_jvpvjp_msort_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_jvpvjp_mul_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_jvpvjp_narrow_copy_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_jvpvjp_new_empty_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_jvpvjp_nextafter_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_jvpvjp_nn_functional_adaptive_avg_pool1d_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_jvpvjp_nn_functional_adaptive_max_pool1d_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_jvpvjp_nn_functional_conv2d_stride_groups_with_bias_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_jvpvjp_nn_functional_conv2d_stride_padding_no_bias_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_jvpvjp_nn_functional_conv2d_stride_padding_with_bias_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_jvpvjp_nn_functional_cosine_similarity_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_jvpvjp_nn_functional_ctc_loss_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_jvpvjp_nn_functional_dropout2d_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_jvpvjp_nn_functional_elu_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_jvpvjp_nn_functional_embedding_functorch_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_jvpvjp_nn_functional_hardsigmoid_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_jvpvjp_nn_functional_interpolate_linear_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_jvpvjp_nn_functional_interpolate_nearest_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_jvpvjp_nn_functional_logsigmoid_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_jvpvjp_nn_functional_max_unpool1d_grad_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_jvpvjp_nn_functional_max_unpool3d_grad_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_jvpvjp_nn_functional_prelu_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_jvpvjp_nn_functional_scaled_dot_product_attention_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_jvpvjp_nn_functional_selu_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_jvpvjp_nn_functional_softplus_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_jvpvjp_norm_fro_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_jvpvjp_real_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_jvpvjp_repeat_interleave_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_jvpvjp_scatter_reduce_mean_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_jvpvjp_sgn_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_jvpvjp_sign_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_jvpvjp_signal_windows_hamming_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_jvpvjp_signbit_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_jvpvjp_sinc_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_jvpvjp_sort_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_jvpvjp_special_hermite_polynomial_h_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_jvpvjp_special_i1e_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_jvpvjp_special_modified_bessel_k0_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_jvpvjp_special_modified_bessel_k1_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_jvpvjp_special_polygamma_special_polygamma_n_0_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_jvpvjp_special_scaled_modified_bessel_k1_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_jvpvjp_special_shifted_chebyshev_polynomial_t_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_jvpvjp_special_zeta_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_jvpvjp_split_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_jvpvjp_split_list_args_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_jvpvjp_std_unbiased_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_jvpvjp_t_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_jvpvjp_tile_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_jvpvjp_view_as_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_jvpvjp_zero__cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_jvpvjpvmap_CubeGenVmapAutogradFunction_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_jvpvjpvmap_NumpyExpMarkDirtyAutogradFunction_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_jvpvjpvmap_ScaleGradGenVmapAutogradFunction_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_jvpvjpvmap_SelectGenVmapAutogradFunction_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_jvpvmap_ScaleGradGenVmapAutogradFunction_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_jvpvmap_SelectGenVmapAutogradFunction_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_ordered_bool_raises_argmin_cuda_bool, test/functorch/test_ops.py::TestOperatorsCUDA::test_ordered_complex_raises_amin_cuda_complex64, test/functorch/test_ops.py::TestOperatorsCUDA::test_ordered_complex_raises_clamp_cuda_complex32, test/functorch/test_ops.py::TestOperatorsCUDA::test_ordered_complex_raises_maximum_cuda_complex32, test/functorch/test_ops.py::TestOperatorsCUDA::test_ordered_complex_raises_sort_cuda_complex128, test/functorch/test_ops.py::TestOperatorsCUDA::test_view_then_inplace_T_grad_op_vjp_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_view_then_inplace_list_return_split_list_args_grad_op_vjp_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_view_then_inplace_permute_grad_op_jvp_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_view_then_inplace_resolve_neg_grad_op_jvp_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_view_then_inplace_select_grad_op_vjp_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_view_then_inplace_view_as_complex_grad_op_jvp_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_view_then_inplace_view_as_grad_op_jvp_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_view_then_inplace_view_as_grad_op_vjp_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vjp_ForwardHasDefaultArgsAutogradFunction_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vjp_NumpyMulAutogradFunction_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vjp_T_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vjp__unsafe_masked_index_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vjp_aminmax_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vjp_as_strided_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vjp_as_strided_scatter_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vjp_baddbmm_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vjp_cholesky_solve_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vjp_conj_physical_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vjp_constant_pad_nd_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vjp_copysign_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vjp_diagflat_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vjp_diagonal_copy_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vjp_expand_as_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vjp_fft_fft_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vjp_fmax_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vjp_frac_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vjp_half_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vjp_isneginf_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vjp_lgamma_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vjp_linalg_eig_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vjp_linalg_matrix_rank_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vjp_linalg_matrix_rank_hermitian_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vjp_linalg_qr_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vjp_log10_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vjp_log2_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vjp_logdet_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vjp_masked_logaddexp_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vjp_masked_scatter_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vjp_masked_select_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vjp_max_binary_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vjp_maximum_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vjp_mul_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vjp_mvlgamma_mvlgamma_p_5_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vjp_nextafter_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vjp_nn_functional_adaptive_avg_pool1d_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vjp_nn_functional_adaptive_avg_pool2d_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vjp_nn_functional_adaptive_max_pool2d_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vjp_nn_functional_alpha_dropout_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vjp_nn_functional_celu_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vjp_nn_functional_cosine_similarity_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vjp_nn_functional_dropout_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vjp_nn_functional_embedding_functorch_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vjp_nn_functional_gelu_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vjp_nn_functional_hardtanh_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vjp_nn_functional_interpolate_linear_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vjp_nn_functional_interpolate_trilinear_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vjp_nn_functional_max_unpool1d_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vjp_nn_functional_mish_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vjp_nn_functional_multilabel_margin_loss_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vjp_nn_functional_pad_replicate_negative_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vjp_nn_functional_poisson_nll_loss_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vjp_nn_functional_soft_margin_loss_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vjp_nn_functional_upsample_bilinear_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vjp_norm_fro_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vjp_norm_nuc_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vjp_pca_lowrank_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vjp_pow_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vjp_put_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vjp_qr_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vjp_repeat_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vjp_round_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vjp_scatter_reduce_amin_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vjp_select_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vjp_short_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vjp_signal_windows_cosine_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vjp_special_hermite_polynomial_he_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vjp_special_shifted_chebyshev_polynomial_u_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vjp_special_shifted_chebyshev_polynomial_w_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vjp_sum_to_size_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vjp_to_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vjp_torch_ops_aten__safe_softmax_default_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vjp_transpose_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vjp_unbind_copy_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vjp_unique_consecutive_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vjp_view_as_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vjp_view_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vjpvjp_ScaleGradGenVmapAutogradFunction_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vjpvjp_SelectAutogradFunction_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vjpvjp___rmatmul___cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vjpvjp_addcmul_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vjpvjp_alias_copy_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vjpvjp_argwhere_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vjpvjp_atan_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vjpvjp_block_diag_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vjpvjp_bucketize_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vjpvjp_char_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vjpvjp_deg2rad_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vjpvjp_empty_like_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vjpvjp_erfc_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vjpvjp_exp2_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vjpvjp_fft_ifftn_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vjpvjp_fft_ihfft2_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vjpvjp_fft_irfftn_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vjpvjp_fill_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vjpvjp_flatten_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vjpvjp_geqrf_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vjpvjp_gradient_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vjpvjp_heaviside_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vjpvjp_index_put_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vjpvjp_index_reduce_amax_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vjpvjp_index_reduce_mean_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vjpvjp_inner_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vjpvjp_int_functorch_no_channels_last_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vjpvjp_isfinite_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vjpvjp_le_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vjpvjp_linalg_inv_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vjpvjp_linalg_qr_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vjpvjp_linalg_tensorinv_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vjpvjp_linalg_vector_norm_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vjpvjp_log1p_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vjpvjp_log_softmax_with_dtype_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vjpvjp_logaddexp_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vjpvjp_logit_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vjpvjp_logsumexp_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vjpvjp_long_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vjpvjp_masked_scatter_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vjpvjp_meshgrid_list_of_tensors_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vjpvjp_mm_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vjpvjp_mode_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vjpvjp_multinomial_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vjpvjp_mv_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vjpvjp_mvlgamma_mvlgamma_p_1_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vjpvjp_new_empty_strided_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vjpvjp_nn_functional_adaptive_avg_pool2d_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vjpvjp_nn_functional_adaptive_max_pool2d_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vjpvjp_nn_functional_avg_pool1d_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vjpvjp_nn_functional_batch_norm_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vjpvjp_nn_functional_conv2d_stride_no_bias_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vjpvjp_nn_functional_conv_transpose3d_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vjpvjp_nn_functional_dropout2d_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vjpvjp_nn_functional_dropout_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vjpvjp_nn_functional_grid_sample_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vjpvjp_nn_functional_hardswish_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vjpvjp_nn_functional_hardtanh_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vjpvjp_nn_functional_interpolate_nearest-exact_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vjpvjp_nn_functional_max_unpool2d_grad_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vjpvjp_nn_functional_max_unpool3d_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vjpvjp_nn_functional_multi_margin_loss_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vjpvjp_nn_functional_pixel_unshuffle_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vjpvjp_nn_functional_softplus_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vjpvjp_nn_functional_threshold_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vjpvjp_norm_fro_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vjpvjp_polygamma_polygamma_n_3_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vjpvjp_rad2deg_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vjpvjp_rsub_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vjpvjp_signbit_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vjpvjp_sinc_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vjpvjp_sinh_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vjpvjp_special_bessel_y0_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vjpvjp_special_hermite_polynomial_h_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vjpvjp_special_hermite_polynomial_he_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vjpvjp_special_i1e_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vjpvjp_std_mean_unbiased_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vjpvjp_sum_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vjpvjp_t_copy_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vjpvjp_trapz_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vjpvjp_triangular_solve_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vjpvjp_where_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vjpvjp_xlogy_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vjpvjp_zeros_like_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vjpvjpvmap_CubeGenVmapAutogradFunction_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vjpvjpvmap_NumpyTakeAutogradFunction_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vjpvjpvmap_SelectAutogradFunction_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vjpvmap_H_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vjpvmap_SelectGenVmapAutogradFunction_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vjpvmap__upsample_bilinear2d_aa_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vjpvmap_amin_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vjpvmap_byte_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vjpvmap_cdist_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vjpvmap_conj_physical_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vjpvmap_cummax_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vjpvmap_cummin_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vjpvmap_cumsum_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vjpvmap_diagonal_scatter_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vjpvmap_div_floor_rounding_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vjpvmap_dstack_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vjpvmap_erfinv_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vjpvmap_fft_hfftn_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vjpvmap_fmod_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vjpvmap_frexp_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vjpvmap_geometric_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vjpvmap_geqrf_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vjpvmap_igamma_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vjpvmap_isreal_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vjpvmap_jiterator_2inputs_2outputs_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vjpvmap_linalg_diagonal_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vjpvmap_linalg_ldl_factor_ex_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vjpvmap_linalg_lu_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vjpvmap_linalg_multi_dot_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vjpvmap_logical_not_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vjpvmap_logit_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vjpvmap_logspace_tensor_overload_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vjpvmap_masked_amax_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vjpvmap_masked_median_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vjpvmap_masked_std_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vjpvmap_minimum_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vjpvmap_nanquantile_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vjpvmap_narrow_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vjpvmap_new_full_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vjpvmap_nextafter_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vjpvmap_nn_functional_channel_shuffle_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vjpvmap_nn_functional_conv2d_stride_padding_with_bias_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vjpvmap_nn_functional_interpolate_linear_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vjpvmap_nn_functional_l1_loss_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vjpvmap_nn_functional_local_response_norm_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vjpvmap_nn_functional_margin_ranking_loss_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vjpvmap_nn_functional_max_pool3d_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vjpvmap_nn_functional_max_unpool2d_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vjpvmap_nn_functional_multilabel_soft_margin_loss_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vjpvmap_nn_functional_pad_constant_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vjpvmap_nn_functional_pairwise_distance_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vjpvmap_nn_functional_softmin_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vjpvmap_nn_functional_softshrink_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vjpvmap_nn_functional_triplet_margin_loss_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vjpvmap_norm_inf_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vjpvmap_permute_copy_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vjpvmap_polygamma_polygamma_n_2_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vjpvmap_polygamma_polygamma_n_4_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vjpvmap_reshape_as_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vjpvmap_reshape_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vjpvmap_resolve_neg_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vjpvmap_round_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vjpvmap_scatter_reduce_amin_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vjpvmap_short_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vjpvmap_short_functorch_no_channels_last_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vjpvmap_sin_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vjpvmap_sinc_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vjpvmap_sinh_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vjpvmap_slice_scatter_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vjpvmap_sparse_mm_reduce_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vjpvmap_special_bessel_y0_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vjpvmap_special_i0e_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vjpvmap_special_modified_bessel_i1_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vjpvmap_sum_to_size_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vjpvmap_t_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vjpvmap_to_sparse_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vjpvmap_unbind_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vjpvmap_unsqueeze_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vjpvmap_var_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vjpvmap_var_mean_unbiased_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vjpvmap_where_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmap_autograd_grad_NumpyCubeAutogradFunction_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmap_autograd_grad_NumpyTakeAutogradFunction_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmap_autograd_grad_SelectGenVmapAutogradFunction_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmap_autograd_grad_ZeroGradientsGenVmapAutogradFunction_cuda_float64, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmap_autograd_grad___radd___cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmap_autograd_grad__native_batch_norm_legit_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmap_autograd_grad__native_batch_norm_legit_cuda_float64, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmap_autograd_grad__segment_reduce_lengths_cuda_float64, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmap_autograd_grad__unsafe_masked_index_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmap_autograd_grad_addbmm_cuda_float64, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmap_autograd_grad_addcdiv_cuda_float64, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmap_autograd_grad_alias_copy_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmap_autograd_grad_any_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmap_autograd_grad_atleast_1d_cuda_float64, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmap_autograd_grad_block_diag_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmap_autograd_grad_bool_functorch_no_channels_last_cuda_float64, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmap_autograd_grad_ceil_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmap_autograd_grad_cfloat_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmap_autograd_grad_char_cuda_float64, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmap_autograd_grad_clamp_max_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmap_autograd_grad_column_stack_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmap_autograd_grad_combinations_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmap_autograd_grad_complex_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmap_autograd_grad_cos_cuda_float64, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmap_autograd_grad_cross_cuda_float64, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmap_autograd_grad_cummax_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmap_autograd_grad_diag_embed_cuda_float64, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmap_autograd_grad_diagonal_cuda_float64, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmap_autograd_grad_diff_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmap_autograd_grad_div_no_rounding_mode_cuda_float64, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmap_autograd_grad_dstack_cuda_float64, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmap_autograd_grad_empty_permuted_cuda_float64, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmap_autograd_grad_erfinv_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmap_autograd_grad_exp2_cuda_float64, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmap_autograd_grad_expand_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmap_autograd_grad_fft_fft2_cuda_float64, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmap_autograd_grad_fft_fft_cuda_float64, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmap_autograd_grad_fft_fftn_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmap_autograd_grad_fft_fftn_cuda_float64, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmap_autograd_grad_fft_hfftn_cuda_float64, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmap_autograd_grad_fft_ifft_cuda_float64, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmap_autograd_grad_fft_ihfftn_cuda_float64, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmap_autograd_grad_fft_rfft2_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmap_autograd_grad_flatten_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmap_autograd_grad_fliplr_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmap_autograd_grad_fmax_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmap_autograd_grad_geometric_cuda_float64, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmap_autograd_grad_gradient_cuda_float64, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmap_autograd_grad_heaviside_cuda_float64, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmap_autograd_grad_index_copy_cuda_float64, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmap_autograd_grad_index_reduce_amin_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmap_autograd_grad_index_select_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmap_autograd_grad_isinf_cuda_float64, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmap_autograd_grad_isnan_cuda_float64, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmap_autograd_grad_isreal_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmap_autograd_grad_jiterator_2inputs_2outputs_cuda_float64, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmap_autograd_grad_kron_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmap_autograd_grad_le_cuda_float64, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmap_autograd_grad_linalg_cholesky_cuda_float64, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmap_autograd_grad_linalg_cholesky_ex_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmap_autograd_grad_linalg_norm_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmap_autograd_grad_linalg_pinv_hermitian_cuda_float64, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmap_autograd_grad_linalg_solve_cuda_float64, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmap_autograd_grad_linalg_tensorsolve_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmap_autograd_grad_log1p_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmap_autograd_grad_log2_cuda_float64, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmap_autograd_grad_log_cuda_float64, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmap_autograd_grad_log_softmax_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmap_autograd_grad_logaddexp2_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmap_autograd_grad_logical_not_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmap_autograd_grad_logical_xor_cuda_float64, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmap_autograd_grad_lu_solve_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmap_autograd_grad_masked_amin_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmap_autograd_grad_masked_cumprod_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmap_autograd_grad_masked_cumsum_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmap_autograd_grad_masked_logaddexp_cuda_float64, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmap_autograd_grad_masked_mean_cuda_float64, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmap_autograd_grad_masked_prod_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmap_autograd_grad_masked_sum_cuda_float64, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmap_autograd_grad_matmul_cuda_float64, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmap_autograd_grad_median_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmap_autograd_grad_meshgrid_variadic_tensors_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmap_autograd_grad_min_reduction_no_dim_cuda_float64, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmap_autograd_grad_minimum_cuda_float64, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmap_autograd_grad_msort_cuda_float64, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmap_autograd_grad_multinomial_cuda_float64, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmap_autograd_grad_nan_to_num_cuda_float64, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmap_autograd_grad_nanmean_cuda_float64, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmap_autograd_grad_nansum_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmap_autograd_grad_new_zeros_cuda_float64, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmap_autograd_grad_nn_functional_adaptive_avg_pool1d_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmap_autograd_grad_nn_functional_adaptive_avg_pool1d_cuda_float64, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmap_autograd_grad_nn_functional_adaptive_avg_pool2d_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmap_autograd_grad_nn_functional_alpha_dropout_cuda_float64, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmap_autograd_grad_nn_functional_avg_pool1d_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmap_autograd_grad_nn_functional_avg_pool2d_cuda_float64, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmap_autograd_grad_nn_functional_batch_norm_cuda_float64, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmap_autograd_grad_nn_functional_conv2d_stride_depthwise_with_bias_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmap_autograd_grad_nn_functional_conv2d_stride_no_bias_cuda_float64, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmap_autograd_grad_nn_functional_conv2d_stride_padding_no_bias_cuda_float64, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmap_autograd_grad_nn_functional_conv2d_stride_padding_with_bias_cuda_float64, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmap_autograd_grad_nn_functional_conv2d_strided_padding_dilation_with_bias_cuda_float64, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmap_autograd_grad_nn_functional_ctc_loss_cuda_float64, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmap_autograd_grad_nn_functional_dropout3d_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmap_autograd_grad_nn_functional_embedding_bag_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmap_autograd_grad_nn_functional_feature_alpha_dropout_without_train_cuda_float64, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmap_autograd_grad_nn_functional_fractional_max_pool2d_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmap_autograd_grad_nn_functional_fractional_max_pool2d_cuda_float64, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmap_autograd_grad_nn_functional_fractional_max_pool3d_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmap_autograd_grad_nn_functional_grid_sample_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmap_autograd_grad_nn_functional_grid_sample_cuda_float64, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmap_autograd_grad_nn_functional_group_norm_cuda_float64, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmap_autograd_grad_nn_functional_interpolate_bicubic_cuda_float64, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmap_autograd_grad_nn_functional_interpolate_trilinear_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmap_autograd_grad_nn_functional_max_unpool1d_grad_cuda_float64, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmap_autograd_grad_nn_functional_multilabel_soft_margin_loss_cuda_float64, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmap_autograd_grad_nn_functional_pixel_unshuffle_cuda_float64, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmap_autograd_grad_nn_functional_triplet_margin_with_distance_loss_cuda_float64, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmap_autograd_grad_nn_functional_upsample_nearest_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmap_autograd_grad_nonzero_cuda_float64, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmap_autograd_grad_normal_in_place_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmap_autograd_grad_normal_in_place_cuda_float64, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmap_autograd_grad_ops_aten__new_zeros_with_same_feature_meta_functorchonly_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmap_autograd_grad_ormqr_cuda_float64, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmap_autograd_grad_pca_lowrank_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmap_autograd_grad_pca_lowrank_cuda_float64, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmap_autograd_grad_pinverse_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmap_autograd_grad_pinverse_cuda_float64, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmap_autograd_grad_polar_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmap_autograd_grad_polygamma_polygamma_n_0_cuda_float64, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmap_autograd_grad_positive_cuda_float64, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmap_autograd_grad_put_cuda_float64, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmap_autograd_grad_qr_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmap_autograd_grad_rad2deg_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmap_autograd_grad_randint_like_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmap_autograd_grad_randn_like_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmap_autograd_grad_repeat_interleave_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmap_autograd_grad_resolve_neg_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmap_autograd_grad_roll_cuda_float64, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmap_autograd_grad_scatter_reduce_sum_cuda_float64, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmap_autograd_grad_select_scatter_cuda_float64, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmap_autograd_grad_short_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmap_autograd_grad_short_functorch_no_channels_last_cuda_float64, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmap_autograd_grad_sigmoid_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmap_autograd_grad_sign_cuda_float64, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmap_autograd_grad_signal_windows_general_hamming_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmap_autograd_grad_signal_windows_hamming_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmap_autograd_grad_signal_windows_nuttall_cuda_float64, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmap_autograd_grad_signbit_cuda_float64, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmap_autograd_grad_sinc_cuda_float64, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmap_autograd_grad_sinh_cuda_float64, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmap_autograd_grad_slice_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmap_autograd_grad_softmax_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmap_autograd_grad_softmax_with_dtype_cuda_float64, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmap_autograd_grad_sparse_mm_reduce_cuda_float64, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmap_autograd_grad_sparse_sampled_addmm_cuda_float64, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmap_autograd_grad_special_chebyshev_polynomial_u_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmap_autograd_grad_special_chebyshev_polynomial_v_cuda_float64, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmap_autograd_grad_special_i0e_cuda_float64, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmap_autograd_grad_special_laguerre_polynomial_l_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmap_autograd_grad_special_modified_bessel_i1_cuda_float64, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmap_autograd_grad_special_modified_bessel_k0_cuda_float64, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmap_autograd_grad_special_modified_bessel_k1_cuda_float64, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmap_autograd_grad_special_ndtri_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmap_autograd_grad_special_ndtri_cuda_float64, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmap_autograd_grad_special_shifted_chebyshev_polynomial_w_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmap_autograd_grad_square_cuda_float64, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmap_autograd_grad_std_cuda_float64, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmap_autograd_grad_std_mean_cuda_float64, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmap_autograd_grad_std_mean_unbiased_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmap_autograd_grad_tensordot_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmap_autograd_grad_tile_cuda_float64, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmap_autograd_grad_torch_ops_aten__safe_softmax_default_cuda_float64, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmap_autograd_grad_transpose_cuda_float64, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmap_autograd_grad_triangular_solve_cuda_float64, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmap_autograd_grad_tril_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmap_autograd_grad_true_divide_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmap_autograd_grad_unbind_copy_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmap_autograd_grad_unbind_cuda_float64, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmap_autograd_grad_var_cuda_float64, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmap_autograd_grad_view_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmap_autograd_grad_xlogy_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmapjvpall_NumpyCubeAutogradFunction_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmapjvpall___getitem___functorch_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmapjvpall___rdiv___cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmapjvpall___rmatmul___cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmapjvpall___rsub___cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmapjvpall_addcdiv_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmapjvpall_addcmul_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmapjvpall_addmv_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmapjvpall_addr_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmapjvpall_angle_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmapjvpall_argmin_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmapjvpall_asinh_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmapjvpall_atleast_1d_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmapjvpall_atleast_3d_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmapjvpall_bfloat16_functorch_no_channels_last_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmapjvpall_cartesian_prod_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmapjvpall_clone_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmapjvpall_complex_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmapjvpall_constant_pad_nd_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmapjvpall_diagflat_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmapjvpall_diagonal_copy_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmapjvpall_div_trunc_rounding_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmapjvpall_double_functorch_no_channels_last_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmapjvpall_dstack_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmapjvpall_empty_permuted_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmapjvpall_fft_hfft2_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmapjvpall_fft_hfft_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmapjvpall_fft_ihfft_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmapjvpall_fft_rfftn_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmapjvpall_flip_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmapjvpall_float_power_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmapjvpall_floor_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmapjvpall_frac_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmapjvpall_geometric_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmapjvpall_geqrf_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmapjvpall_half_functorch_no_channels_last_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmapjvpall_has_batch_rule_SelectGenVmapAutogradFunction_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmapjvpall_has_batch_rule__softmax_backward_data_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmapjvpall_has_batch_rule__unsafe_masked_index_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmapjvpall_has_batch_rule_argmax_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmapjvpall_has_batch_rule_argsort_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmapjvpall_has_batch_rule_as_strided_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmapjvpall_has_batch_rule_atan_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmapjvpall_has_batch_rule_bool_functorch_no_channels_last_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmapjvpall_has_batch_rule_bucketize_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmapjvpall_has_batch_rule_cartesian_prod_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmapjvpall_has_batch_rule_cdouble_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmapjvpall_has_batch_rule_cholesky_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmapjvpall_has_batch_rule_clamp_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmapjvpall_has_batch_rule_column_stack_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmapjvpall_has_batch_rule_conj_physical_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmapjvpall_has_batch_rule_contiguous_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmapjvpall_has_batch_rule_cov_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmapjvpall_has_batch_rule_cummax_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmapjvpall_has_batch_rule_cummin_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmapjvpall_has_batch_rule_cumsum_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmapjvpall_has_batch_rule_div_trunc_rounding_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmapjvpall_has_batch_rule_double_functorch_no_channels_last_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmapjvpall_has_batch_rule_dsplit_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmapjvpall_has_batch_rule_fft_ifftshift_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmapjvpall_has_batch_rule_flip_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmapjvpall_has_batch_rule_flipud_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmapjvpall_has_batch_rule_float_functorch_no_channels_last_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmapjvpall_has_batch_rule_fmod_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmapjvpall_has_batch_rule_hash_tensor_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmapjvpall_has_batch_rule_index_copy_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmapjvpall_has_batch_rule_index_put_functorch_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmapjvpall_has_batch_rule_index_reduce_prod_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmapjvpall_has_batch_rule_jiterator_binary_return_by_ref_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmapjvpall_has_batch_rule_linalg_diagonal_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmapjvpall_has_batch_rule_linalg_pinv_singular_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmapjvpall_has_batch_rule_linalg_slogdet_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmapjvpall_has_batch_rule_linalg_svdvals_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmapjvpall_has_batch_rule_linalg_tensorinv_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmapjvpall_has_batch_rule_log2_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmapjvpall_has_batch_rule_lu_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmapjvpall_has_batch_rule_lu_solve_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmapjvpall_has_batch_rule_masked_fill_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmapjvpall_has_batch_rule_masked_fill_functorch_Scalar_only_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmapjvpall_has_batch_rule_new_empty_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmapjvpall_has_batch_rule_nn_functional_adaptive_avg_pool2d_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmapjvpall_has_batch_rule_nn_functional_alpha_dropout_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmapjvpall_has_batch_rule_nn_functional_ctc_loss_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmapjvpall_has_batch_rule_nn_functional_embedding_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmapjvpall_has_batch_rule_nn_functional_glu_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmapjvpall_has_batch_rule_nn_functional_interpolate_nearest-exact_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmapjvpall_has_batch_rule_nn_functional_interpolate_trilinear_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmapjvpall_has_batch_rule_nn_functional_margin_ranking_loss_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmapjvpall_has_batch_rule_nn_functional_max_pool1d_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmapjvpall_has_batch_rule_nn_functional_max_unpool3d_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmapjvpall_has_batch_rule_nn_functional_multi_head_attention_forward_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmapjvpall_has_batch_rule_nn_functional_nll_loss_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmapjvpall_has_batch_rule_nn_functional_pad_replicate_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmapjvpall_has_batch_rule_nn_functional_pdist_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmapjvpall_has_batch_rule_norm_inf_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmapjvpall_has_batch_rule_rad2deg_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmapjvpall_has_batch_rule_resolve_conj_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmapjvpall_has_batch_rule_scatter_reduce_sum_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmapjvpall_has_batch_rule_signal_windows_blackman_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmapjvpall_has_batch_rule_signal_windows_cosine_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmapjvpall_has_batch_rule_signal_windows_gaussian_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmapjvpall_has_batch_rule_signal_windows_nuttall_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmapjvpall_has_batch_rule_slice_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmapjvpall_has_batch_rule_special_chebyshev_polynomial_u_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmapjvpall_has_batch_rule_special_scaled_modified_bessel_k0_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmapjvpall_has_batch_rule_squeeze_multiple_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmapjvpall_has_batch_rule_std_mean_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmapjvpall_has_batch_rule_trace_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmapjvpall_has_batch_rule_triangular_solve_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmapjvpall_has_batch_rule_unbind_copy_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmapjvpall_has_batch_rule_unique_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmapjvpall_has_batch_rule_var_unbiased_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmapjvpall_has_batch_rule_view_as_complex_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmapjvpall_has_batch_rule_view_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmapjvpall_has_batch_rule_where_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmapjvpall_hstack_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmapjvpall_index_copy_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmapjvpall_index_reduce_amax_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmapjvpall_index_reduce_amin_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmapjvpall_isposinf_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmapjvpall_jiterator_binary_return_by_ref_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmapjvpall_kthvalue_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmapjvpall_lerp_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmapjvpall_linalg_lstsq_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmapjvpall_linalg_matrix_norm_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmapjvpall_linalg_qr_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmapjvpall_linalg_solve_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmapjvpall_linalg_solve_triangular_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmapjvpall_linspace_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmapjvpall_linspace_tensor_overload_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmapjvpall_log_softmax_with_dtype_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmapjvpall_logaddexp2_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmapjvpall_logical_or_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmapjvpall_logical_xor_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmapjvpall_logspace_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmapjvpall_masked_logsumexp_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmapjvpall_masked_softmax_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmapjvpall_meshgrid_list_of_tensors_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmapjvpall_mvlgamma_mvlgamma_p_1_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmapjvpall_nan_to_num_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmapjvpall_nansum_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmapjvpall_nn_functional_avg_pool1d_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmapjvpall_nn_functional_batch_norm_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmapjvpall_nn_functional_binary_cross_entropy_with_logits_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmapjvpall_nn_functional_conv2d_stride_padding_no_bias_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmapjvpall_nn_functional_conv_transpose3d_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmapjvpall_nn_functional_cosine_similarity_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmapjvpall_nn_functional_gaussian_nll_loss_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmapjvpall_nn_functional_glu_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmapjvpall_nn_functional_instance_norm_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmapjvpall_nn_functional_l1_loss_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmapjvpall_nn_functional_linear_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmapjvpall_nn_functional_pixel_unshuffle_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmapjvpall_ops_aten__new_zeros_with_same_feature_meta_functorchonly_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmapjvpall_polygamma_polygamma_n_4_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmapjvpall_rand_like_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmapjvpall_randn_like_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmapjvpall_reshape_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmapjvpall_rsqrt_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmapjvpall_signal_windows_gaussian_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmapjvpall_signal_windows_nuttall_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmapjvpall_special_hermite_polynomial_he_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmapjvpall_special_xlog1py_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmapjvpall_split_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmapjvpall_square_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmapjvpall_squeeze_copy_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmapjvpall_stft_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmapjvpall_triangular_solve_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmapjvpall_unfold_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmapjvpall_var_mean_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmapjvpall_view_copy_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmapjvpvjp_NumpyCubeAutogradFunction_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmapjvpvjp_NumpyCubeNotComposableAutogradFunction_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmapjvpvjp_NumpySortAutogradFunction_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmapjvpvjp_SelectGenVmapAutogradFunction_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmapjvpvjp___rmul___cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmapjvpvjp_acosh_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmapjvpvjp_all_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmapjvpvjp_atan2_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmapjvpvjp_atleast_2d_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmapjvpvjp_bernoulli_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmapjvpvjp_cartesian_prod_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmapjvpvjp_cdouble_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmapjvpvjp_char_functorch_no_channels_last_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmapjvpvjp_chunk_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmapjvpvjp_clamp_max_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmapjvpvjp_contiguous_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmapjvpvjp_cummin_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmapjvpvjp_cumsum_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmapjvpvjp_digamma_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmapjvpvjp_div_no_rounding_mode_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmapjvpvjp_div_trunc_rounding_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmapjvpvjp_einsum_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmapjvpvjp_empty_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmapjvpvjp_erfc_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmapjvpvjp_erfinv_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmapjvpvjp_expand_as_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmapjvpvjp_expand_copy_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmapjvpvjp_expm1_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmapjvpvjp_fft_rfft2_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmapjvpvjp_frac_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmapjvpvjp_grid_sampler_2d_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmapjvpvjp_isnan_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmapjvpvjp_linalg_cond_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmapjvpvjp_linalg_diagonal_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmapjvpvjp_linalg_householder_product_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmapjvpvjp_linalg_ldl_factor_ex_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmapjvpvjp_linalg_lstsq_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmapjvpvjp_log_softmax_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmapjvpvjp_logit_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmapjvpvjp_lu_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmapjvpvjp_masked_fill_functorch_Scalar_only_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmapjvpvjp_masked_median_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmapjvpvjp_max_binary_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmapjvpvjp_minimum_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmapjvpvjp_mv_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmapjvpvjp_native_batch_norm_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmapjvpvjp_native_dropout_backward_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmapjvpvjp_nn_functional_adaptive_avg_pool1d_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmapjvpvjp_nn_functional_binary_cross_entropy_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmapjvpvjp_nn_functional_cosine_similarity_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmapjvpvjp_nn_functional_embedding_functorch_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmapjvpvjp_nn_functional_fractional_max_pool2d_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmapjvpvjp_nn_functional_gelu_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmapjvpvjp_nn_functional_hardswish_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmapjvpvjp_nn_functional_huber_loss_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmapjvpvjp_nn_functional_instance_norm_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmapjvpvjp_nn_functional_interpolate_trilinear_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmapjvpvjp_nn_functional_layer_norm_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmapjvpvjp_nn_functional_max_unpool2d_grad_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmapjvpvjp_nn_functional_multi_margin_loss_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmapjvpvjp_nn_functional_rms_norm_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmapjvpvjp_nn_functional_silu_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmapjvpvjp_norm_nuc_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmapjvpvjp_normal_number_mean_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmapjvpvjp_ones_like_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmapjvpvjp_put_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmapjvpvjp_rad2deg_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmapjvpvjp_resolve_conj_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmapjvpvjp_rsqrt_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmapjvpvjp_scatter_add_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmapjvpvjp_scatter_reduce_mean_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmapjvpvjp_select_scatter_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmapjvpvjp_short_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmapjvpvjp_sign_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmapjvpvjp_signal_windows_exponential_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmapjvpvjp_signal_windows_gaussian_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmapjvpvjp_sinc_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmapjvpvjp_sinh_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmapjvpvjp_special_bessel_y0_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmapjvpvjp_special_entr_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmapjvpvjp_special_i1e_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmapjvpvjp_split_with_sizes_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmapjvpvjp_squeeze_copy_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmapjvpvjp_std_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmapjvpvjp_svd_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmapjvpvjp_trapz_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmapjvpvjp_triangular_solve_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmapjvpvjp_unfold_copy_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmapjvpvjp_uniform_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmapjvpvjp_vdot_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmapjvpvjp_view_as_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmapjvpvmap_NumpyMulAutogradFunction_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmapjvpvmap_NumpySortAutogradFunction_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmapjvpvmap_NumpyTakeAutogradFunction_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmapjvpvmap_SortGenVmapAutogradFunction_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmapvjp_NumpyMulAutogradFunction_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmapvjp___rmul___cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmapvjp___rsub___cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmapvjp__batch_norm_with_update_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmapvjp_acosh_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmapvjp_add_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmapvjp_amax_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmapvjp_aminmax_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmapvjp_argsort_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmapvjp_as_strided_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmapvjp_atan_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmapvjp_atleast_3d_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmapvjp_bmm_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmapvjp_cauchy_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmapvjp_ceil_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmapvjp_combinations_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmapvjp_cummax_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmapvjp_deg2rad_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmapvjp_diff_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmapvjp_div_trunc_rounding_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmapvjp_empty_permuted_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmapvjp_empty_strided_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmapvjp_equal_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmapvjp_erfc_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmapvjp_expand_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmapvjp_expm1_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmapvjp_fft_fft2_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmapvjp_fft_fftshift_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmapvjp_fft_hfftn_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmapvjp_fft_ihfftn_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmapvjp_flatten_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmapvjp_float_functorch_no_channels_last_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmapvjp_frac_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmapvjp_grid_sampler_2d_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmapvjp_has_batch_rule_H_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmapvjp_has_batch_rule_NumpySortAutogradFunction_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmapvjp_has_batch_rule_addcmul_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmapvjp_has_batch_rule_angle_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmapvjp_has_batch_rule_baddbmm_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmapvjp_has_batch_rule_byte_functorch_no_channels_last_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmapvjp_has_batch_rule_cfloat_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmapvjp_has_batch_rule_char_functorch_no_channels_last_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmapvjp_has_batch_rule_cummax_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmapvjp_has_batch_rule_cumprod_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmapvjp_has_batch_rule_cumsum_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmapvjp_has_batch_rule_diagflat_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmapvjp_has_batch_rule_dist_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmapvjp_has_batch_rule_double_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmapvjp_has_batch_rule_dstack_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmapvjp_has_batch_rule_equal_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmapvjp_has_batch_rule_fft_fft2_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmapvjp_has_batch_rule_fliplr_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmapvjp_has_batch_rule_floor_divide_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmapvjp_has_batch_rule_frexp_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmapvjp_has_batch_rule_geometric_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmapvjp_has_batch_rule_gradient_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmapvjp_has_batch_rule_half_functorch_no_channels_last_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmapvjp_has_batch_rule_histc_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmapvjp_has_batch_rule_index_add_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmapvjp_has_batch_rule_index_select_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmapvjp_has_batch_rule_inner_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmapvjp_has_batch_rule_lgamma_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmapvjp_has_batch_rule_linalg_cholesky_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmapvjp_has_batch_rule_linalg_cross_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmapvjp_has_batch_rule_linalg_eigh_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmapvjp_has_batch_rule_linalg_lu_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmapvjp_has_batch_rule_linalg_pinv_singular_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmapvjp_has_batch_rule_log_normal_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmapvjp_has_batch_rule_logaddexp_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmapvjp_has_batch_rule_logcumsumexp_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmapvjp_has_batch_rule_logical_not_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmapvjp_has_batch_rule_masked_argmin_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmapvjp_has_batch_rule_masked_norm_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmapvjp_has_batch_rule_masked_select_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmapvjp_has_batch_rule_min_reduction_no_dim_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmapvjp_has_batch_rule_nanmean_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmapvjp_has_batch_rule_nanmedian_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmapvjp_has_batch_rule_new_empty_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmapvjp_has_batch_rule_new_empty_strided_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmapvjp_has_batch_rule_new_full_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmapvjp_has_batch_rule_new_ones_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmapvjp_has_batch_rule_nn_functional_adaptive_max_pool2d_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmapvjp_has_batch_rule_nn_functional_avg_pool3d_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmapvjp_has_batch_rule_nn_functional_conv2d_stride_depthwise_with_bias_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmapvjp_has_batch_rule_nn_functional_conv_transpose3d_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmapvjp_has_batch_rule_nn_functional_elu_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmapvjp_has_batch_rule_nn_functional_interpolate_nearest-exact_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmapvjp_has_batch_rule_nn_functional_local_response_norm_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmapvjp_has_batch_rule_nn_functional_max_pool1d_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmapvjp_has_batch_rule_nn_functional_multilabel_margin_loss_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmapvjp_has_batch_rule_nn_functional_relu6_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmapvjp_has_batch_rule_nn_functional_upsample_nearest_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmapvjp_has_batch_rule_permute_copy_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmapvjp_has_batch_rule_qr_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmapvjp_has_batch_rule_randn_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmapvjp_has_batch_rule_repeat_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmapvjp_has_batch_rule_roll_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmapvjp_has_batch_rule_round_decimals_neg_3_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmapvjp_has_batch_rule_searchsorted_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmapvjp_has_batch_rule_select_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmapvjp_has_batch_rule_sigmoid_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmapvjp_has_batch_rule_signal_windows_bartlett_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmapvjp_has_batch_rule_signal_windows_cosine_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmapvjp_has_batch_rule_signal_windows_hann_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmapvjp_has_batch_rule_sinc_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmapvjp_has_batch_rule_slice_scatter_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmapvjp_has_batch_rule_sparse_mm_reduce_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmapvjp_has_batch_rule_special_airy_ai_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmapvjp_has_batch_rule_special_bessel_j0_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmapvjp_has_batch_rule_special_chebyshev_polynomial_t_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmapvjp_has_batch_rule_special_erfcx_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmapvjp_has_batch_rule_special_xlog1py_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmapvjp_has_batch_rule_split_with_sizes_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmapvjp_has_batch_rule_squeeze_multiple_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmapvjp_has_batch_rule_sum_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmapvjp_has_batch_rule_svd_lowrank_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmapvjp_has_batch_rule_tensordot_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmapvjp_has_batch_rule_to_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmapvjp_has_batch_rule_trace_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmapvjp_has_batch_rule_var_mean_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmapvjp_has_batch_rule_var_unbiased_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmapvjp_has_batch_rule_vdot_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmapvjp_has_batch_rule_view_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmapvjp_has_batch_rule_xlogy_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmapvjp_i0_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmapvjp_isnan_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmapvjp_linalg_cond_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmapvjp_linalg_ldl_factor_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmapvjp_linalg_matrix_norm_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmapvjp_linalg_matrix_power_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmapvjp_linalg_tensorinv_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmapvjp_log_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmapvjp_logspace_tensor_overload_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmapvjp_long_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmapvjp_masked_cumsum_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmapvjp_masked_fill_functorch_Scalar_only_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmapvjp_masked_logsumexp_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmapvjp_nanquantile_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmapvjp_nextafter_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmapvjp_nn_functional_batch_norm_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmapvjp_nn_functional_conv2d_strided_padding_dilation_no_bias_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmapvjp_nn_functional_glu_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmapvjp_nn_functional_logsigmoid_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmapvjp_nn_functional_max_pool2d_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmapvjp_nn_functional_multilabel_margin_loss_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmapvjp_nn_functional_pairwise_distance_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmapvjp_nn_functional_selu_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmapvjp_nn_functional_softmin_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmapvjp_nn_functional_tanhshrink_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmapvjp_nn_functional_triplet_margin_loss_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmapvjp_norm_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmapvjp_norm_inf_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmapvjp_norm_nuc_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmapvjp_normal_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmapvjp_normal_in_place_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmapvjp_ops_aten_index_put_functorch_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmapvjp_ormqr_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmapvjp_round_decimals_3_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmapvjp_scatter_reduce_mean_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmapvjp_searchsorted_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmapvjp_sgn_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmapvjp_slice_scatter_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmapvjp_sparse_mm_reduce_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmapvjp_special_bessel_y1_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmapvjp_sum_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmapvjp_trace_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmapvjp_transpose_copy_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmapvjp_unsafe_split_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmapvjp_view_as_complex_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmapvjp_view_copy_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmapvjpvjp_H_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmapvjpvjp_NumpyExpMarkDirtyAutogradFunction_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmapvjpvjp_NumpySortAutogradFunction_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmapvjpvjp_ScaleGradGenVmapAutogradFunction_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmapvjpvjp_T_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmapvjpvjp___rpow___cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmapvjpvjp__native_batch_norm_legit_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmapvjpvjp_alias_copy_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmapvjpvjp_as_strided_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmapvjpvjp_as_strided_scatter_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmapvjpvjp_atan_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmapvjpvjp_broadcast_tensors_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmapvjpvjp_byte_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmapvjpvjp_clamp_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmapvjpvjp_clamp_max_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmapvjpvjp_copysign_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmapvjpvjp_dist_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmapvjpvjp_double_functorch_no_channels_last_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmapvjpvjp_dsplit_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmapvjpvjp_exponential_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmapvjpvjp_fft_fft_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmapvjpvjp_fft_fftn_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmapvjpvjp_fill_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmapvjpvjp_frexp_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmapvjpvjp_full_like_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmapvjpvjp_index_add_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmapvjpvjp_index_copy_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmapvjpvjp_index_put_functorch_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmapvjpvjp_isposinf_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmapvjpvjp_isreal_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmapvjpvjp_jiterator_2inputs_2outputs_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmapvjpvjp_linalg_eigvals_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmapvjpvjp_linalg_ldl_solve_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmapvjpvjp_linalg_lstsq_grad_oriented_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmapvjpvjp_linalg_multi_dot_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmapvjpvjp_linalg_pinv_singular_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmapvjpvjp_linalg_solve_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmapvjpvjp_linalg_svd_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmapvjpvjp_linalg_vector_norm_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmapvjpvjp_logical_and_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmapvjpvjp_masked_amax_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmapvjpvjp_masked_cumprod_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmapvjpvjp_masked_fill_functorch_Scalar_only_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmapvjpvjp_masked_logaddexp_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmapvjpvjp_meshgrid_variadic_tensors_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmapvjpvjp_mvlgamma_mvlgamma_p_3_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmapvjpvjp_nansum_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmapvjpvjp_narrow_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmapvjpvjp_native_batch_norm_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmapvjpvjp_new_empty_strided_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmapvjpvjp_nn_functional_adaptive_avg_pool2d_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmapvjpvjp_nn_functional_alpha_dropout_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmapvjpvjp_nn_functional_avg_pool1d_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmapvjpvjp_nn_functional_avg_pool2d_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmapvjpvjp_nn_functional_batch_norm_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmapvjpvjp_nn_functional_conv_transpose1d_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmapvjpvjp_nn_functional_conv_transpose2d_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmapvjpvjp_nn_functional_ctc_loss_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmapvjpvjp_nn_functional_feature_alpha_dropout_without_train_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmapvjpvjp_nn_functional_max_unpool2d_grad_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmapvjpvjp_nn_functional_max_unpool3d_grad_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmapvjpvjp_nn_functional_mse_loss_functorch_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmapvjpvjp_nn_functional_pad_constant_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmapvjpvjp_nn_functional_pad_replicate_negative_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmapvjpvjp_nn_functional_softshrink_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmapvjpvjp_ops_aten__new_zeros_with_same_feature_meta_functorchonly_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmapvjpvjp_outer_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmapvjpvjp_pca_lowrank_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmapvjpvjp_scatter_reduce_amin_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmapvjpvjp_signal_windows_cosine_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmapvjpvjp_sinh_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmapvjpvjp_special_hermite_polynomial_he_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmapvjpvjp_special_modified_bessel_i0_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmapvjpvjp_special_modified_bessel_k0_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmapvjpvjp_split_with_sizes_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmapvjpvjp_std_mean_unbiased_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmapvjpvjp_svd_lowrank_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmapvjpvjp_t_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmapvjpvjp_take_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmapvjpvjp_tanh_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmapvjpvjp_topk_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmapvjpvjp_unbind_copy_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmapvjpvjp_unique_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmapvjpvjp_view_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmapvjpvjp_zeros_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmapvjpvmap_NumpyMulAutogradFunction_cuda_float32, test/functorch/test_ops.py::TestOperatorsCUDA::test_vmapvjpvmap_SelectGenVmapAutogradFunction_cuda_float32 2025-12-04T12:57:41.9872150Z 2025-12-04T12:57:41.9872377Z Finished functorch/test_ops 2/8 ... [2025-12-04 12:57:41.923189][11909.865405854], took 10.51min 2025-12-04T12:57:41.9873112Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/functorch.test_ops/functorch.test_ops-78f655d60a12ab9a.xml 2025-12-04T12:57:42.0435921Z Running functorch/test_vmap 1/1 ... [2025-12-04 12:57:42.043342][11909.985559952] 2025-12-04T12:57:42.0436362Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2025-12-04T12:57:42.0439128Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'functorch/test_vmap.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-12-04 12:57:42.043653] 2025-12-04T13:04:29.4477000Z 2025-12-04T13:04:29.4478022Z functorch/test_vmap 1/1 was successful, full logs can be found in artifacts with path test/test-reports/functorch.test_vmap_1.1_e45df0fc68d2af64_.log 2025-12-04T13:04:29.5154671Z Running 2142 items in this shard: test/functorch/test_vmap.py::TestVmapAPI::test_accepts_nested_inputs, test/functorch/test_vmap.py::TestVmapAPI::test_backward_unsupported_interaction, test/functorch/test_vmap.py::TestVmapAPI::test_batch_rule_does_not_need_to_handle_no_batched_input, test/functorch/test_vmap.py::TestVmapAPI::test_batched_gradient_basic, test/functorch/test_vmap.py::TestVmapAPI::test_checkpoint, test/functorch/test_vmap.py::TestVmapAPI::test_constant_function, test/functorch/test_vmap.py::TestVmapAPI::test_data_attribute, test/functorch/test_vmap.py::TestVmapAPI::test_data_dependent_control_flow_throws, test/functorch/test_vmap.py::TestVmapAPI::test_decomposition_under_python_dispatcher, test/functorch/test_vmap.py::TestVmapAPI::test_different_map_dim_size_raises, test/functorch/test_vmap.py::TestVmapAPI::test_fallback_does_not_warn_by_default, test/functorch/test_vmap.py::TestVmapAPI::test_fallback_masked_fill, test/functorch/test_vmap.py::TestVmapAPI::test_fallback_multiple_returns, test/functorch/test_vmap.py::TestVmapAPI::test_fallback_warning, test/functorch/test_vmap.py::TestVmapAPI::test_fallback_warns_when_warnings_are_enabled, test/functorch/test_vmap.py::TestVmapAPI::test_fallback_with_undefined_grad, test/functorch/test_vmap.py::TestVmapAPI::test_fallback_zero_dim, test/functorch/test_vmap.py::TestVmapAPI::test_func_with_no_inputs, test/functorch/test_vmap.py::TestVmapAPI::test_func_with_no_tensors, test/functorch/test_vmap.py::TestVmapAPI::test_functools_partial, test/functorch/test_vmap.py::TestVmapAPI::test_grad_unsupported_interaction, test/functorch/test_vmap.py::TestVmapAPI::test_in_dim_not_in_tensor_err_msg, test/functorch/test_vmap.py::TestVmapAPI::test_in_dims_wrong_type_err_msg, test/functorch/test_vmap.py::TestVmapAPI::test_inplace_fallback_nary_different_levels, test/functorch/test_vmap.py::TestVmapAPI::test_inplace_fallback_nary_same_levels, test/functorch/test_vmap.py::TestVmapAPI::test_inplace_fallback_unary, test/functorch/test_vmap.py::TestVmapAPI::test_integer_in_dim_but_not_tensor_input_err_msg, test/functorch/test_vmap.py::TestVmapAPI::test_item_throws, test/functorch/test_vmap.py::TestVmapAPI::test_multiple_inputs, test/functorch/test_vmap.py::TestVmapAPI::test_multiple_out_dims, test/functorch/test_vmap.py::TestVmapAPI::test_multiple_outputs, test/functorch/test_vmap.py::TestVmapAPI::test_multiple_outputs2, test/functorch/test_vmap.py::TestVmapAPI::test_nested_negative_in_dims, test/functorch/test_vmap.py::TestVmapAPI::test_nested_non_default_in_dims, test/functorch/test_vmap.py::TestVmapAPI::test_nested_out_dims, test/functorch/test_vmap.py::TestVmapAPI::test_nested_with_diag_embed, test/functorch/test_vmap.py::TestVmapAPI::test_nested_with_different_map_dim, test/functorch/test_vmap.py::TestVmapAPI::test_nested_with_same_map_dim, test/functorch/test_vmap.py::TestVmapAPI::test_nn_module, test/functorch/test_vmap.py::TestVmapAPI::test_non_default_in_dims_out_dims, test/functorch/test_vmap.py::TestVmapAPI::test_non_tensor_output_raises, test/functorch/test_vmap.py::TestVmapAPI::test_non_zero_in_dims, test/functorch/test_vmap.py::TestVmapAPI::test_none_in_dims, test/functorch/test_vmap.py::TestVmapAPI::test_nonzero_out_dims, test/functorch/test_vmap.py::TestVmapAPI::test_noop_in_inner_vmap, test/functorch/test_vmap.py::TestVmapAPI::test_not_enough_in_dims_err_msg, test/functorch/test_vmap.py::TestVmapAPI::test_out_dim_out_of_bounds_err_msg, test/functorch/test_vmap.py::TestVmapAPI::test_out_dims_and_num_outputs_mismatch_err_msg, test/functorch/test_vmap.py::TestVmapAPI::test_out_dims_edge_case, test/functorch/test_vmap.py::TestVmapAPI::test_out_dims_must_be_int_or_collection_of_int_err_msg, test/functorch/test_vmap.py::TestVmapAPI::test_out_dims_none, test/functorch/test_vmap.py::TestVmapAPI::test_out_dims_none_tuple, test/functorch/test_vmap.py::TestVmapAPI::test_out_dims_normal_tensor, test/functorch/test_vmap.py::TestVmapAPI::test_pytree_odict_returns, test/functorch/test_vmap.py::TestVmapAPI::test_pytree_returns, test/functorch/test_vmap.py::TestVmapAPI::test_pytree_returns_broadcast_nested, test/functorch/test_vmap.py::TestVmapAPI::test_pytree_returns_broadcast_simple, test/functorch/test_vmap.py::TestVmapAPI::test_pytree_returns_outdims, test/functorch/test_vmap.py::TestVmapAPI::test_reshape_dim_into, test/functorch/test_vmap.py::TestVmapAPI::test_reshape_dim_outof, test/functorch/test_vmap.py::TestVmapAPI::test_restore_vmap_no_vmapped_inputs, test/functorch/test_vmap.py::TestVmapAPI::test_restore_vmap_pytree_input_output, test/functorch/test_vmap.py::TestVmapAPI::test_restore_vmap_unexpanded_outputs, test/functorch/test_vmap.py::TestVmapAPI::test_single_input, test/functorch/test_vmap.py::TestVmapAPI::test_unsupported_op_err_msg, test/functorch/test_vmap.py::TestVmapAPI::test_vmap_autocast_cpu, test/functorch/test_vmap.py::TestVmapAPI::test_vmap_autocast_cuda, test/functorch/test_vmap.py::TestVmapOperators::test_T_numpy, test/functorch/test_vmap.py::TestVmapOperators::test_adaptive_avg_pool2d, test/functorch/test_vmap.py::TestVmapOperators::test_argmax_dim, test/functorch/test_vmap.py::TestVmapOperators::test_arithmetic_add, test/functorch/test_vmap.py::TestVmapOperators::test_arithmetic_add_dunder, test/functorch/test_vmap.py::TestVmapOperators::test_arithmetic_div, test/functorch/test_vmap.py::TestVmapOperators::test_arithmetic_div_dunder, test/functorch/test_vmap.py::TestVmapOperators::test_arithmetic_mul, test/functorch/test_vmap.py::TestVmapOperators::test_arithmetic_mul_dunder, test/functorch/test_vmap.py::TestVmapOperators::test_arithmetic_pow, test/functorch/test_vmap.py::TestVmapOperators::test_arithmetic_pow_dunder, test/functorch/test_vmap.py::TestVmapOperators::test_arithmetic_sub, test/functorch/test_vmap.py::TestVmapOperators::test_arithmetic_sub_dunder, test/functorch/test_vmap.py::TestVmapOperators::test_as_strided, test/functorch/test_vmap.py::TestVmapOperators::test_bmm, test/functorch/test_vmap.py::TestVmapOperators::test_cat, test/functorch/test_vmap.py::TestVmapOperators::test_chunk, test/functorch/test_vmap.py::TestVmapOperators::test_chunk_vmap_in_dim_0_out_dim_0_randomness_error, test/functorch/test_vmap.py::TestVmapOperators::test_chunk_vmap_in_dim_0_out_dim_0_randomness_same, test/functorch/test_vmap.py::TestVmapOperators::test_chunk_vmap_in_dim_0_out_dim_1_randomness_error, test/functorch/test_vmap.py::TestVmapOperators::test_chunk_vmap_in_dim_0_out_dim_1_randomness_same, test/functorch/test_vmap.py::TestVmapOperators::test_chunk_vmap_in_dim_0_out_dim_2_randomness_error, test/functorch/test_vmap.py::TestVmapOperators::test_chunk_vmap_in_dim_0_out_dim_2_randomness_same, test/functorch/test_vmap.py::TestVmapOperators::test_chunk_vmap_in_dim_1_out_dim_0_randomness_error, test/functorch/test_vmap.py::TestVmapOperators::test_chunk_vmap_in_dim_1_out_dim_0_randomness_same, test/functorch/test_vmap.py::TestVmapOperators::test_chunk_vmap_in_dim_1_out_dim_1_randomness_error, test/functorch/test_vmap.py::TestVmapOperators::test_chunk_vmap_in_dim_1_out_dim_1_randomness_same, test/functorch/test_vmap.py::TestVmapOperators::test_chunk_vmap_in_dim_1_out_dim_2_randomness_error, test/functorch/test_vmap.py::TestVmapOperators::test_chunk_vmap_in_dim_1_out_dim_2_randomness_same, test/functorch/test_vmap.py::TestVmapOperators::test_chunk_vmap_in_dim_2_out_dim_0_randomness_error, test/functorch/test_vmap.py::TestVmapOperators::test_chunk_vmap_in_dim_2_out_dim_0_randomness_same, test/functorch/test_vmap.py::TestVmapOperators::test_chunk_vmap_in_dim_2_out_dim_1_randomness_error, test/functorch/test_vmap.py::TestVmapOperators::test_chunk_vmap_in_dim_2_out_dim_1_randomness_same, test/functorch/test_vmap.py::TestVmapOperators::test_chunk_vmap_in_dim_2_out_dim_2_randomness_error, test/functorch/test_vmap.py::TestVmapOperators::test_chunk_vmap_in_dim_2_out_dim_2_randomness_same, test/functorch/test_vmap.py::TestVmapOperators::test_clamp, test/functorch/test_vmap.py::TestVmapOperators::test_clamp_inplace_variant_clamp_max_, test/functorch/test_vmap.py::TestVmapOperators::test_clamp_inplace_variant_clamp_min_, test/functorch/test_vmap.py::TestVmapOperators::test_clamp_variant_clamp_max, test/functorch/test_vmap.py::TestVmapOperators::test_clamp_variant_clamp_min, test/functorch/test_vmap.py::TestVmapOperators::test_clone, test/functorch/test_vmap.py::TestVmapOperators::test_comparison_ops, test/functorch/test_vmap.py::TestVmapOperators::test_conj, test/functorch/test_vmap.py::TestVmapOperators::test_conj_bit, test/functorch/test_vmap.py::TestVmapOperators::test_contiguous, test/functorch/test_vmap.py::TestVmapOperators::test_conv2d, test/functorch/test_vmap.py::TestVmapOperators::test_copy_, test/functorch/test_vmap.py::TestVmapOperators::test_cross_batch_size_three, test/functorch/test_vmap.py::TestVmapOperators::test_diagonal, test/functorch/test_vmap.py::TestVmapOperators::test_dot, test/functorch/test_vmap.py::TestVmapOperators::test_expand_as, test/functorch/test_vmap.py::TestVmapOperators::test_fill_and_zero_inplace, test/functorch/test_vmap.py::TestVmapOperators::test_imag, test/functorch/test_vmap.py::TestVmapOperators::test_is_complex, test/functorch/test_vmap.py::TestVmapOperators::test_is_contiguous, test/functorch/test_vmap.py::TestVmapOperators::test_is_floating_point, test/functorch/test_vmap.py::TestVmapOperators::test_mean, test/functorch/test_vmap.py::TestVmapOperators::test_mean_dim, test/functorch/test_vmap.py::TestVmapOperators::test_mm, test/functorch/test_vmap.py::TestVmapOperators::test_mode_key, test/functorch/test_vmap.py::TestVmapOperators::test_movedim, test/functorch/test_vmap.py::TestVmapOperators::test_mv, test/functorch/test_vmap.py::TestVmapOperators::test_narrow, test/functorch/test_vmap.py::TestVmapOperators::test_new_empty, test/functorch/test_vmap.py::TestVmapOperators::test_new_empty_strided, test/functorch/test_vmap.py::TestVmapOperators::test_new_zeros, test/functorch/test_vmap.py::TestVmapOperators::test_nll_loss, test/functorch/test_vmap.py::TestVmapOperators::test_one_hot, test/functorch/test_vmap.py::TestVmapOperators::test_real, test/functorch/test_vmap.py::TestVmapOperators::test_repeat, test/functorch/test_vmap.py::TestVmapOperators::test_reshape, test/functorch/test_vmap.py::TestVmapOperators::test_reshape_as, test/functorch/test_vmap.py::TestVmapOperators::test_result_type, test/functorch/test_vmap.py::TestVmapOperators::test_roll_no_dims, test/functorch/test_vmap.py::TestVmapOperators::test_select, test/functorch/test_vmap.py::TestVmapOperators::test_silu_backward, test/functorch/test_vmap.py::TestVmapOperators::test_slice, test/functorch/test_vmap.py::TestVmapOperators::test_slogdet, test/functorch/test_vmap.py::TestVmapOperators::test_split, test/functorch/test_vmap.py::TestVmapOperators::test_squeeze, test/functorch/test_vmap.py::TestVmapOperators::test_stack, test/functorch/test_vmap.py::TestVmapOperators::test_stride, test/functorch/test_vmap.py::TestVmapOperators::test_sum, test/functorch/test_vmap.py::TestVmapOperators::test_sum_dim, test/functorch/test_vmap.py::TestVmapOperators::test_t, test/functorch/test_vmap.py::TestVmapOperators::test_tensor_split, test/functorch/test_vmap.py::TestVmapOperators::test_to, test/functorch/test_vmap.py::TestVmapOperators::test_trace, test/functorch/test_vmap.py::TestVmapOperators::test_transpose, test/functorch/test_vmap.py::TestVmapOperators::test_unary_pointwise_abs, test/functorch/test_vmap.py::TestVmapOperators::test_unary_pointwise_acos, test/functorch/test_vmap.py::TestVmapOperators::test_unary_pointwise_asin, test/functorch/test_vmap.py::TestVmapOperators::test_unary_pointwise_atan, test/functorch/test_vmap.py::TestVmapOperators::test_unary_pointwise_ceil, test/functorch/test_vmap.py::TestVmapOperators::test_unary_pointwise_cos, test/functorch/test_vmap.py::TestVmapOperators::test_unary_pointwise_cosh, test/functorch/test_vmap.py::TestVmapOperators::test_unary_pointwise_digamma, test/functorch/test_vmap.py::TestVmapOperators::test_unary_pointwise_exp, test/functorch/test_vmap.py::TestVmapOperators::test_unary_pointwise_expm1, test/functorch/test_vmap.py::TestVmapOperators::test_unary_pointwise_floor, test/functorch/test_vmap.py::TestVmapOperators::test_unary_pointwise_frac, test/functorch/test_vmap.py::TestVmapOperators::test_unary_pointwise_lgamma, test/functorch/test_vmap.py::TestVmapOperators::test_unary_pointwise_log, test/functorch/test_vmap.py::TestVmapOperators::test_unary_pointwise_log10, test/functorch/test_vmap.py::TestVmapOperators::test_unary_pointwise_log1p, test/functorch/test_vmap.py::TestVmapOperators::test_unary_pointwise_log2, test/functorch/test_vmap.py::TestVmapOperators::test_unary_pointwise_neg, test/functorch/test_vmap.py::TestVmapOperators::test_unary_pointwise_reciprocal, test/functorch/test_vmap.py::TestVmapOperators::test_unary_pointwise_relu, test/functorch/test_vmap.py::TestVmapOperators::test_unary_pointwise_round, test/functorch/test_vmap.py::TestVmapOperators::test_unary_pointwise_rsqrt, test/functorch/test_vmap.py::TestVmapOperators::test_unary_pointwise_sigmoid, test/functorch/test_vmap.py::TestVmapOperators::test_unary_pointwise_sign, test/functorch/test_vmap.py::TestVmapOperators::test_unary_pointwise_sin, test/functorch/test_vmap.py::TestVmapOperators::test_unary_pointwise_sinh, test/functorch/test_vmap.py::TestVmapOperators::test_unary_pointwise_sqrt, test/functorch/test_vmap.py::TestVmapOperators::test_unary_pointwise_tan, test/functorch/test_vmap.py::TestVmapOperators::test_unary_pointwise_tanh, test/functorch/test_vmap.py::TestVmapOperators::test_unary_pointwise_trunc, test/functorch/test_vmap.py::TestVmapOperators::test_unbind, test/functorch/test_vmap.py::TestVmapOperators::test_unfold, test/functorch/test_vmap.py::TestVmapOperators::test_unsafe_view, test/functorch/test_vmap.py::TestVmapOperators::test_unsqueeze, test/functorch/test_vmap.py::TestVmapOperators::test_view, test/functorch/test_vmap.py::TestVmapOperators::test_view_as, test/functorch/test_vmap.py::TestVmapOperators::test_view_as_complex, test/functorch/test_vmap.py::TestVmapOperators::test_view_as_real, test/functorch/test_vmap.py::TestVmapOperators::test_vmap_chunksize_composition_in_dim_0_out_dim_0_randomness_error, test/functorch/test_vmap.py::TestVmapOperators::test_vmap_chunksize_composition_in_dim_0_out_dim_0_randomness_same, test/functorch/test_vmap.py::TestVmapOperators::test_vmap_chunksize_composition_in_dim_0_out_dim_1_randomness_error, test/functorch/test_vmap.py::TestVmapOperators::test_vmap_chunksize_composition_in_dim_0_out_dim_1_randomness_same, test/functorch/test_vmap.py::TestVmapOperators::test_vmap_chunksize_composition_in_dim_1_out_dim_0_randomness_error, test/functorch/test_vmap.py::TestVmapOperators::test_vmap_chunksize_composition_in_dim_1_out_dim_0_randomness_same, test/functorch/test_vmap.py::TestVmapOperators::test_vmap_chunksize_composition_in_dim_1_out_dim_1_randomness_error, test/functorch/test_vmap.py::TestVmapOperators::test_vmap_chunksize_composition_in_dim_1_out_dim_1_randomness_same, test/functorch/test_vmap.py::TestVmapOperators::test_vmap_chunksize_error_in_dim_0_out_dim_0_randomness_error, test/functorch/test_vmap.py::TestVmapOperators::test_vmap_chunksize_error_in_dim_0_out_dim_0_randomness_same, test/functorch/test_vmap.py::TestVmapOperators::test_vmap_chunksize_error_in_dim_0_out_dim_1_randomness_error, test/functorch/test_vmap.py::TestVmapOperators::test_vmap_chunksize_error_in_dim_0_out_dim_1_randomness_same, test/functorch/test_vmap.py::TestVmapOperators::test_vmap_chunksize_error_in_dim_1_out_dim_0_randomness_error, test/functorch/test_vmap.py::TestVmapOperators::test_vmap_chunksize_error_in_dim_1_out_dim_0_randomness_same, test/functorch/test_vmap.py::TestVmapOperators::test_vmap_chunksize_error_in_dim_1_out_dim_1_randomness_error, test/functorch/test_vmap.py::TestVmapOperators::test_vmap_chunksize_error_in_dim_1_out_dim_1_randomness_same, test/functorch/test_vmap.py::TestVmapOperators::test_vmap_chunksize_in_dim_0_out_dim_0_randomness_error, test/functorch/test_vmap.py::TestVmapOperators::test_vmap_chunksize_in_dim_0_out_dim_0_randomness_same, test/functorch/test_vmap.py::TestVmapOperators::test_vmap_chunksize_in_dim_0_out_dim_1_randomness_error, test/functorch/test_vmap.py::TestVmapOperators::test_vmap_chunksize_in_dim_0_out_dim_1_randomness_same, test/functorch/test_vmap.py::TestVmapOperators::test_vmap_chunksize_in_dim_0_out_dim_2_randomness_error, test/functorch/test_vmap.py::TestVmapOperators::test_vmap_chunksize_in_dim_0_out_dim_2_randomness_same, test/functorch/test_vmap.py::TestVmapOperators::test_vmap_chunksize_in_dim_1_out_dim_0_randomness_error, test/functorch/test_vmap.py::TestVmapOperators::test_vmap_chunksize_in_dim_1_out_dim_0_randomness_same, test/functorch/test_vmap.py::TestVmapOperators::test_vmap_chunksize_in_dim_1_out_dim_1_randomness_error, test/functorch/test_vmap.py::TestVmapOperators::test_vmap_chunksize_in_dim_1_out_dim_1_randomness_same, test/functorch/test_vmap.py::TestVmapOperators::test_vmap_chunksize_in_dim_1_out_dim_2_randomness_error, test/functorch/test_vmap.py::TestVmapOperators::test_vmap_chunksize_in_dim_1_out_dim_2_randomness_same, test/functorch/test_vmap.py::TestVmapOperators::test_vmap_chunksize_in_dim_2_out_dim_0_randomness_error, test/functorch/test_vmap.py::TestVmapOperators::test_vmap_chunksize_in_dim_2_out_dim_0_randomness_same, test/functorch/test_vmap.py::TestVmapOperators::test_vmap_chunksize_in_dim_2_out_dim_1_randomness_error, test/functorch/test_vmap.py::TestVmapOperators::test_vmap_chunksize_in_dim_2_out_dim_1_randomness_same, test/functorch/test_vmap.py::TestVmapOperators::test_vmap_chunksize_in_dim_2_out_dim_2_randomness_error, test/functorch/test_vmap.py::TestVmapOperators::test_vmap_chunksize_in_dim_2_out_dim_2_randomness_same, test/functorch/test_vmap.py::TestVmapOperators::test_vmap_fallback_check, test/functorch/test_vmap.py::TestVmapOperators::test_vmap_fallback_check_ok, test/functorch/test_vmap.py::TestVmapOperators::test_weird_matmul_case, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_0d_tensor_index_put_inplace_False_cuda, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_0d_tensor_index_put_inplace_True_cuda, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_advanced_indexing_cuda, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_batch_norm_training_False_track_running_stats_False_affine_False_cuda, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_batch_norm_training_False_track_running_stats_False_affine_True_cuda, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_batch_norm_training_False_track_running_stats_True_affine_False_cuda, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_batch_norm_training_False_track_running_stats_True_affine_True_cuda, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_batch_norm_training_True_track_running_stats_False_affine_False_cuda, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_batch_norm_training_True_track_running_stats_False_affine_True_cuda, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_batch_norm_training_True_track_running_stats_True_affine_False_cuda, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_batch_norm_training_True_track_running_stats_True_affine_True_cuda, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_conv_double_backward_cuda, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_fill__Tensor_cuda, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_flatten_cuda, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_foo_like_cuda, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_group_norm_cuda, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_index_fill_cuda, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_index_put_cuda, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_inplace_on_view_cuda, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_isinf_cuda, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_isnan_cuda, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_linalg_eigh_cuda, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_linalg_svd_cuda, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_namedtuple_returns_cuda, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_nested_advanced_indexing_cuda, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_CubeGenVmapAutogradFunction_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_ForwardHasDefaultArgsAutogradFunction_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_H_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_MulGenVmapAutogradFunction_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_NumpyCatCustomOp_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_NumpyCubeAutogradFunction_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_NumpyCubeCustomOp_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_NumpyCubeNotComposableAutogradFunction_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_NumpyExpMarkDirtyAutogradFunction_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_NumpyMulAutogradFunction_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_NumpyMulCustomOp_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_NumpyMulScalarCustomOp_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_NumpyNMSCustomOp_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_NumpyNonzeroCustomOp_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_NumpySortAutogradFunction_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_NumpySortCustomOp_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_NumpySplitCopyCustomOp_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_NumpySplitCopyWithIntCustomOp_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_NumpyTakeAutogradFunction_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_NumpyTakeCustomOp_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_NumpyViewCopyCustomOp_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_ScaleGradGenVmapAutogradFunction_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_SelectAutogradFunction_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_SelectGenVmapAutogradFunction_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_SortGenVmapAutogradFunction_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_T_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_ZeroGradientsGenVmapAutogradFunction_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule___getitem___cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule___getitem___functorch_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule___radd___cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule___rand___cuda_int64, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule___rdiv___cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule___rmatmul___cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule___rmod___cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule___rmul___cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule___ror___cuda_int64, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule___rpow___cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule___rsub___cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule___rxor___cuda_int64, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule__batch_norm_with_update_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule__chunk_cat_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule__native_batch_norm_legit_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule__segment_reduce_lengths_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule__segment_reduce_offsets_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule__softmax_backward_data_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule__unsafe_masked_index_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule__unsafe_masked_index_put_accumulate_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule__upsample_bilinear2d_aa_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_abs_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_acos_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_acosh_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_add_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_addbmm_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_addcdiv_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_addcmul_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_addmm_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_addmm_decomposed_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_addmv_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_addr_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_alias_copy_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_all_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_allclose_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_amax_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_amin_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_aminmax_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_angle_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_any_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_arange_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_argmax_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_argmin_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_argsort_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_argwhere_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_as_strided_copy_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_as_strided_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_as_strided_partial_views_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_as_strided_scatter_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_asin_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_asinh_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_atan2_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_atan_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_atanh_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_atleast_1d_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_atleast_2d_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_atleast_3d_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_baddbmm_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_bernoulli_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_bfloat16_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_bfloat16_functorch_no_channels_last_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_bincount_cuda_int64, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_bitwise_and_cuda_int64, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_bitwise_left_shift_cuda_int64, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_bitwise_not_cuda_int64, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_bitwise_or_cuda_int64, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_bitwise_right_shift_cuda_int64, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_bitwise_xor_cuda_int64, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_block_diag_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_bmm_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_bool_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_bool_functorch_no_channels_last_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_broadcast_shapes_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_broadcast_tensors_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_broadcast_to_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_bucketize_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_byte_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_byte_functorch_no_channels_last_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_cartesian_prod_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_cat_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_cauchy_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_cdist_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_cdouble_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_ceil_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_cfloat_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_chalf_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_char_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_char_functorch_no_channels_last_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_cholesky_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_cholesky_inverse_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_cholesky_solve_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_chunk_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_clamp_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_clamp_max_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_clamp_min_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_clone_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_column_stack_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_combinations_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_complex_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_conj_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_conj_physical_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_constant_pad_nd_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_contiguous_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_copysign_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_corrcoef_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_cos_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_cosh_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_count_nonzero_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_cov_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_cross_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_cummax_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_cummin_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_cumprod_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_cumsum_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_cumulative_trapezoid_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_deg2rad_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_diag_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_diag_embed_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_diagflat_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_diagonal_copy_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_diagonal_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_diagonal_scatter_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_diff_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_digamma_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_dist_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_div_floor_rounding_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_div_no_rounding_mode_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_div_trunc_rounding_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_dot_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_double_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_double_functorch_no_channels_last_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_dsplit_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_dstack_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_einsum_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_empty_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_empty_like_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_empty_permuted_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_empty_strided_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_eq_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_equal_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_erf_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_erfc_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_erfinv_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_exp2_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_exp_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_expand_as_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_expand_copy_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_expand_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_expm1_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_exponential_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_eye_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_fft_fft2_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_fft_fft_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_fft_fftn_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_fft_fftshift_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_fft_hfft2_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_fft_hfft_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_fft_hfftn_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_fft_ifft2_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_fft_ifft_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_fft_ifftn_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_fft_ifftshift_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_fft_ihfft2_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_fft_ihfft_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_fft_ihfftn_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_fft_irfft2_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_fft_irfft_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_fft_irfftn_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_fft_rfft2_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_fft_rfft_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_fft_rfftn_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_fill_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_flatten_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_flip_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_fliplr_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_flipud_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_float_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_float_functorch_no_channels_last_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_float_power_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_floor_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_floor_divide_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_fmax_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_fmin_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_fmod_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_frac_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_frexp_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_full_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_full_like_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_gather_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_gcd_cuda_int64, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_ge_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_geometric_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_geqrf_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_gradient_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_grid_sampler_2d_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_grid_sampler_3d_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_gt_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_half_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_half_functorch_no_channels_last_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_hash_tensor_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_heaviside_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_histc_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_hsplit_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_hstack_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_hypot_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_i0_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_igamma_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_igammac_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_imag_cuda_complex64, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_index_add_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_index_copy_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_index_fill_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_index_put_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_index_put_functorch_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_index_reduce_amax_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_index_reduce_amin_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_index_reduce_mean_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_index_reduce_prod_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_index_select_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_inner_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_int_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_int_functorch_no_channels_last_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_isclose_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_isfinite_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_isin_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_isinf_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_isnan_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_isneginf_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_isposinf_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_isreal_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_istft_cuda_complex64, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_item_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_jiterator_2inputs_2outputs_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_jiterator_4inputs_with_extra_args_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_jiterator_binary_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_jiterator_binary_return_by_ref_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_jiterator_unary_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_kron_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_kthvalue_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_lcm_cuda_int64, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_ldexp_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_le_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_lerp_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_lgamma_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_linalg_cholesky_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_linalg_cholesky_ex_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_linalg_cond_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_linalg_cross_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_linalg_det_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_linalg_diagonal_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_linalg_eig_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_linalg_eigh_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_linalg_eigvals_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_linalg_eigvalsh_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_linalg_householder_product_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_linalg_inv_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_linalg_inv_ex_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_linalg_ldl_factor_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_linalg_ldl_factor_ex_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_linalg_ldl_solve_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_linalg_lstsq_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_linalg_lstsq_grad_oriented_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_linalg_lu_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_linalg_lu_factor_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_linalg_lu_factor_ex_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_linalg_lu_solve_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_linalg_matrix_norm_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_linalg_matrix_power_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_linalg_matrix_rank_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_linalg_matrix_rank_hermitian_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_linalg_multi_dot_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_linalg_norm_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_linalg_norm_subgradients_at_zero_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_linalg_pinv_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_linalg_pinv_hermitian_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_linalg_pinv_singular_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_linalg_qr_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_linalg_slogdet_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_linalg_solve_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_linalg_solve_ex_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_linalg_solve_triangular_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_linalg_svd_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_linalg_svdvals_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_linalg_tensorinv_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_linalg_tensorsolve_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_linalg_vander_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_linalg_vecdot_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_linalg_vector_norm_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_linspace_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_linspace_tensor_overload_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_log10_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_log1p_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_log2_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_log_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_log_normal_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_log_softmax_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_log_softmax_with_dtype_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_logaddexp2_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_logaddexp_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_logcumsumexp_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_logdet_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_logical_and_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_logical_not_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_logical_or_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_logical_xor_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_logit_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_logspace_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_logspace_tensor_overload_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_logsumexp_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_long_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_long_functorch_no_channels_last_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_lt_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_lu_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_lu_solve_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_lu_unpack_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_mH_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_mT_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_masked_amax_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_masked_amin_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_masked_argmax_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_masked_argmin_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_masked_cumprod_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_masked_cumsum_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_masked_fill_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_masked_fill_functorch_Scalar_only_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_masked_log_softmax_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_masked_logaddexp_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_masked_logsumexp_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_masked_mean_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_masked_median_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_masked_norm_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_masked_normalize_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_masked_prod_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_masked_scatter_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_masked_select_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_masked_softmax_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_masked_softmin_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_masked_std_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_masked_sum_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_masked_var_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_matmul_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_matrix_exp_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_max_binary_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_max_pool2d_with_indices_backward_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_max_reduction_no_dim_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_max_reduction_with_dim_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_maximum_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_mean_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_median_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_meshgrid_list_of_tensors_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_meshgrid_variadic_tensors_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_min_binary_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_min_reduction_no_dim_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_min_reduction_with_dim_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_minimum_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_mm_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_mode_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_movedim_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_msort_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_mul_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_multinomial_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_mv_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_mvlgamma_mvlgamma_p_1_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_mvlgamma_mvlgamma_p_3_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_mvlgamma_mvlgamma_p_5_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_nan_to_num_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_nanmean_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_nanmedian_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_nanquantile_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_nansum_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_narrow_copy_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_narrow_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_native_batch_norm_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_native_dropout_backward_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_native_layer_norm_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_ne_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_neg_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_new_empty_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_new_empty_strided_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_new_full_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_new_ones_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_new_zeros_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_nextafter_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_nn_functional_adaptive_avg_pool1d_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_nn_functional_adaptive_avg_pool2d_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_nn_functional_adaptive_avg_pool3d_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_nn_functional_adaptive_max_pool1d_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_nn_functional_adaptive_max_pool2d_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_nn_functional_adaptive_max_pool3d_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_nn_functional_alpha_dropout_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_nn_functional_avg_pool1d_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_nn_functional_avg_pool2d_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_nn_functional_avg_pool3d_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_nn_functional_batch_norm_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_nn_functional_batch_norm_without_cudnn_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_nn_functional_bilinear_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_nn_functional_binary_cross_entropy_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_nn_functional_binary_cross_entropy_with_logits_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_nn_functional_celu_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_nn_functional_channel_shuffle_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_nn_functional_conv1d_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_nn_functional_conv2d_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_nn_functional_conv2d_no_bias_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_nn_functional_conv2d_stride_depthwise_with_bias_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_nn_functional_conv2d_stride_groups_with_bias_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_nn_functional_conv2d_stride_no_bias_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_nn_functional_conv2d_stride_padding_no_bias_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_nn_functional_conv2d_stride_padding_with_bias_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_nn_functional_conv2d_stride_with_bias_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_nn_functional_conv2d_strided_padding_dilation_no_bias_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_nn_functional_conv2d_strided_padding_dilation_with_bias_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_nn_functional_conv2d_with_bias_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_nn_functional_conv3d_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_nn_functional_conv_transpose1d_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_nn_functional_conv_transpose2d_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_nn_functional_conv_transpose3d_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_nn_functional_cosine_embedding_loss_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_nn_functional_cosine_similarity_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_nn_functional_cross_entropy_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_nn_functional_ctc_loss_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_nn_functional_dropout2d_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_nn_functional_dropout3d_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_nn_functional_dropout_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_nn_functional_elu_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_nn_functional_embedding_bag_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_nn_functional_embedding_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_nn_functional_embedding_functorch_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_nn_functional_feature_alpha_dropout_with_train_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_nn_functional_feature_alpha_dropout_without_train_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_nn_functional_fractional_max_pool2d_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_nn_functional_fractional_max_pool3d_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_nn_functional_gaussian_nll_loss_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_nn_functional_gelu_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_nn_functional_glu_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_nn_functional_grid_sample_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_nn_functional_group_norm_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_nn_functional_hardshrink_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_nn_functional_hardsigmoid_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_nn_functional_hardswish_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_nn_functional_hardtanh_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_nn_functional_hinge_embedding_loss_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_nn_functional_huber_loss_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_nn_functional_instance_norm_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_nn_functional_interpolate_area_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_nn_functional_interpolate_bicubic_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_nn_functional_interpolate_bilinear_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_nn_functional_interpolate_linear_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_nn_functional_interpolate_nearest-exact_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_nn_functional_interpolate_nearest_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_nn_functional_interpolate_trilinear_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_nn_functional_kl_div_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_nn_functional_l1_loss_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_nn_functional_layer_norm_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_nn_functional_leaky_relu_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_nn_functional_linear_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_nn_functional_local_response_norm_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_nn_functional_logsigmoid_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_nn_functional_margin_ranking_loss_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_nn_functional_max_pool1d_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_nn_functional_max_pool2d_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_nn_functional_max_pool3d_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_nn_functional_max_unpool1d_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_nn_functional_max_unpool1d_grad_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_nn_functional_max_unpool2d_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_nn_functional_max_unpool2d_grad_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_nn_functional_max_unpool3d_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_nn_functional_max_unpool3d_grad_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_nn_functional_mish_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_nn_functional_mse_loss_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_nn_functional_mse_loss_functorch_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_nn_functional_multi_head_attention_forward_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_nn_functional_multi_margin_loss_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_nn_functional_multilabel_margin_loss_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_nn_functional_multilabel_soft_margin_loss_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_nn_functional_nll_loss_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_nn_functional_normalize_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_nn_functional_one_hot_cuda_int64, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_nn_functional_pad_circular_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_nn_functional_pad_constant_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_nn_functional_pad_reflect_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_nn_functional_pad_replicate_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_nn_functional_pad_replicate_negative_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_nn_functional_pairwise_distance_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_nn_functional_pdist_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_nn_functional_pixel_shuffle_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_nn_functional_pixel_unshuffle_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_nn_functional_poisson_nll_loss_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_nn_functional_prelu_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_nn_functional_relu6_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_nn_functional_relu_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_nn_functional_rms_norm_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_nn_functional_rrelu_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_nn_functional_scaled_dot_product_attention_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_nn_functional_selu_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_nn_functional_silu_complex_cuda_complex64, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_nn_functional_silu_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_nn_functional_smooth_l1_loss_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_nn_functional_soft_margin_loss_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_nn_functional_softmin_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_nn_functional_softmin_with_dtype_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_nn_functional_softplus_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_nn_functional_softshrink_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_nn_functional_softsign_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_nn_functional_tanhshrink_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_nn_functional_threshold_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_nn_functional_triplet_margin_loss_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_nn_functional_triplet_margin_with_distance_loss_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_nn_functional_unfold_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_nn_functional_upsample_bilinear_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_nn_functional_upsample_nearest_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_nonzero_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_nonzero_static_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_norm_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_norm_fro_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_norm_inf_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_norm_nuc_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_normal_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_normal_in_place_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_normal_number_mean_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_ones_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_ones_like_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_ops_aten__new_zeros_with_same_feature_meta_functorchonly_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_ops_aten_index_put_functorch_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_ormqr_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_outer_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_pca_lowrank_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_permute_copy_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_permute_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_pinverse_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_polar_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_polygamma_polygamma_n_0_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_polygamma_polygamma_n_1_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_polygamma_polygamma_n_2_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_polygamma_polygamma_n_3_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_polygamma_polygamma_n_4_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_positive_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_pow_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_prod_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_put_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_qr_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_quantile_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_rad2deg_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_rand_like_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_randint_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_randint_like_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_randn_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_randn_like_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_ravel_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_real_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_reciprocal_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_remainder_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_renorm_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_repeat_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_repeat_interleave_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_reshape_as_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_reshape_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_resize__cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_resize_as__cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_resolve_conj_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_resolve_neg_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_roll_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_rot90_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_round_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_round_decimals_0_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_round_decimals_3_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_round_decimals_neg_3_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_rsqrt_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_rsub_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_scalar_tensor_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_scatter_add_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_scatter_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_scatter_reduce_amax_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_scatter_reduce_amin_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_scatter_reduce_mean_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_scatter_reduce_prod_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_scatter_reduce_sum_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_searchsorted_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_select_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_select_scatter_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_sgn_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_short_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_short_functorch_no_channels_last_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_sigmoid_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_sign_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_signal_windows_bartlett_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_signal_windows_blackman_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_signal_windows_cosine_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_signal_windows_exponential_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_signal_windows_gaussian_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_signal_windows_general_cosine_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_signal_windows_general_hamming_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_signal_windows_hamming_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_signal_windows_hann_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_signal_windows_kaiser_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_signal_windows_nuttall_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_signbit_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_sin_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_sinc_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_sinh_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_slice_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_slice_scatter_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_softmax_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_softmax_with_dtype_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_sort_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_sparse_mm_reduce_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_sparse_sampled_addmm_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_special_airy_ai_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_special_bessel_j0_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_special_bessel_j1_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_special_bessel_y0_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_special_bessel_y1_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_special_chebyshev_polynomial_t_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_special_chebyshev_polynomial_u_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_special_chebyshev_polynomial_v_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_special_chebyshev_polynomial_w_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_special_entr_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_special_erfcx_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_special_hermite_polynomial_h_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_special_hermite_polynomial_he_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_special_i0e_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_special_i1_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_special_i1e_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_special_laguerre_polynomial_l_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_special_legendre_polynomial_p_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_special_log_ndtr_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_special_modified_bessel_i0_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_special_modified_bessel_i1_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_special_modified_bessel_k0_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_special_modified_bessel_k1_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_special_ndtr_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_special_ndtri_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_special_polygamma_special_polygamma_n_0_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_special_scaled_modified_bessel_k0_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_special_scaled_modified_bessel_k1_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_special_shifted_chebyshev_polynomial_t_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_special_shifted_chebyshev_polynomial_u_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_special_shifted_chebyshev_polynomial_v_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_special_shifted_chebyshev_polynomial_w_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_special_spherical_bessel_j0_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_special_xlog1py_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_special_zeta_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_split_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_split_list_args_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_split_with_sizes_copy_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_split_with_sizes_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_sqrt_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_square_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_squeeze_copy_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_squeeze_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_squeeze_multiple_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_stack_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_std_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_std_mean_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_std_mean_unbiased_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_std_unbiased_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_stft_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_sub_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_sum_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_sum_to_size_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_svd_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_svd_lowrank_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_t_copy_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_t_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_take_along_dim_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_take_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_tan_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_tanh_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_tensor_split_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_tensordot_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_tile_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_to_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_to_sparse_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_topk_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_torch__scaled_mm_cuda_float8_e4m3fn, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_torch__scaled_mm_v2_cuda_float8_e4m3fn, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_torch_ops_aten__efficient_attention_forward_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_torch_ops_aten__flash_attention_forward_cuda_float16, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_torch_ops_aten__safe_softmax_default_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_trace_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_transpose_copy_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_transpose_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_trapezoid_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_trapz_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_triangular_solve_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_tril_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_tril_indices_cuda_int64, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_triu_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_triu_indices_cuda_int64, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_true_divide_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_trunc_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_unbind_copy_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_unbind_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_unflatten_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_unfold_copy_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_unfold_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_uniform_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_unique_consecutive_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_unique_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_unravel_index_cuda_int64, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_unsafe_chunk_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_unsafe_split_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_unsqueeze_copy_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_unsqueeze_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_var_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_var_mean_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_var_mean_unbiased_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_var_unbiased_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_vdot_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_view_as_complex_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_view_as_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_view_as_real_cuda_complex64, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_view_copy_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_view_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_vsplit_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_vstack_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_where_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_xlogy_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_zero__cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_zeros_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_op_has_batch_rule_zeros_like_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_searchsorted_bucketize_cuda, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_slogdet_cuda, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_sum_scalar_cuda, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_torch_return_types_returns_cuda, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_escaped_error_cuda, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_CubeGenVmapAutogradFunction_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_ForwardHasDefaultArgsAutogradFunction_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_H_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_MulGenVmapAutogradFunction_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_NumpyCatCustomOp_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_NumpyCubeAutogradFunction_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_NumpyCubeCustomOp_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_NumpyCubeNotComposableAutogradFunction_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_NumpyExpMarkDirtyAutogradFunction_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_NumpyMulAutogradFunction_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_NumpyMulCustomOp_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_NumpyMulScalarCustomOp_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_NumpyNMSCustomOp_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_NumpyNonzeroCustomOp_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_NumpySortAutogradFunction_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_NumpySortCustomOp_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_NumpySplitCopyCustomOp_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_NumpySplitCopyWithIntCustomOp_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_NumpyTakeAutogradFunction_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_NumpyTakeCustomOp_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_NumpyViewCopyCustomOp_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_ScaleGradGenVmapAutogradFunction_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_SelectAutogradFunction_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_SelectGenVmapAutogradFunction_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_SortGenVmapAutogradFunction_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_T_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_ZeroGradientsGenVmapAutogradFunction_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive___getitem___cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive___getitem___functorch_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive___radd___cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive___rand___cuda_int64, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive___rdiv___cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive___rmatmul___cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive___rmod___cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive___rmul___cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive___ror___cuda_int64, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive___rpow___cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive___rsub___cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive___rxor___cuda_int64, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive__batch_norm_with_update_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive__chunk_cat_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive__native_batch_norm_legit_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive__segment_reduce_lengths_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive__segment_reduce_offsets_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive__softmax_backward_data_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive__unsafe_masked_index_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive__unsafe_masked_index_put_accumulate_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive__upsample_bilinear2d_aa_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_abs_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_acos_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_acosh_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_add_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_addbmm_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_addcdiv_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_addcmul_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_addmm_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_addmm_decomposed_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_addmv_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_addr_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_alias_copy_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_all_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_allclose_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_amax_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_amin_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_aminmax_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_angle_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_any_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_arange_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_argmax_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_argmin_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_argsort_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_argwhere_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_as_strided_copy_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_as_strided_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_as_strided_partial_views_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_as_strided_scatter_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_asin_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_asinh_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_atan2_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_atan_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_atanh_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_atleast_1d_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_atleast_2d_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_atleast_3d_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_baddbmm_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_bernoulli_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_bfloat16_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_bfloat16_functorch_no_channels_last_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_bincount_cuda_int64, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_bitwise_and_cuda_int64, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_bitwise_left_shift_cuda_int64, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_bitwise_not_cuda_int64, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_bitwise_or_cuda_int64, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_bitwise_right_shift_cuda_int64, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_bitwise_xor_cuda_int64, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_block_diag_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_bmm_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_bool_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_bool_functorch_no_channels_last_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_broadcast_shapes_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_broadcast_tensors_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_broadcast_to_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_bucketize_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_byte_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_byte_functorch_no_channels_last_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_cartesian_prod_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_cat_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_cauchy_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_cdist_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_cdouble_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_ceil_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_cfloat_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_chalf_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_char_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_char_functorch_no_channels_last_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_cholesky_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_cholesky_inverse_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_cholesky_solve_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_chunk_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_clamp_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_clamp_max_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_clamp_min_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_clone_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_column_stack_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_combinations_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_complex_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_conj_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_conj_physical_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_constant_pad_nd_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_contiguous_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_copysign_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_corrcoef_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_cos_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_cosh_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_count_nonzero_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_cov_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_cross_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_cummax_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_cummin_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_cumprod_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_cumsum_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_cumulative_trapezoid_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_deg2rad_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_diag_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_diag_embed_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_diagflat_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_diagonal_copy_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_diagonal_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_diagonal_scatter_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_diff_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_digamma_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_dist_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_div_floor_rounding_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_div_no_rounding_mode_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_div_trunc_rounding_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_dot_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_double_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_double_functorch_no_channels_last_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_dsplit_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_dstack_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_einsum_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_empty_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_empty_like_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_empty_permuted_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_empty_strided_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_eq_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_equal_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_erf_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_erfc_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_erfinv_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_exp2_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_exp_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_expand_as_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_expand_copy_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_expand_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_expm1_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_exponential_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_eye_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_fft_fft2_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_fft_fft_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_fft_fftn_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_fft_fftshift_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_fft_hfft2_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_fft_hfft_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_fft_hfftn_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_fft_ifft2_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_fft_ifft_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_fft_ifftn_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_fft_ifftshift_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_fft_ihfft2_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_fft_ihfft_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_fft_ihfftn_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_fft_irfft2_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_fft_irfft_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_fft_irfftn_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_fft_rfft2_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_fft_rfft_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_fft_rfftn_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_fill_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_flatten_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_flip_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_fliplr_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_flipud_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_float_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_float_functorch_no_channels_last_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_float_power_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_floor_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_floor_divide_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_fmax_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_fmin_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_fmod_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_frac_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_frexp_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_full_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_full_like_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_gather_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_gcd_cuda_int64, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_ge_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_geometric_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_geqrf_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_gradient_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_grid_sampler_2d_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_grid_sampler_3d_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_gt_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_half_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_half_functorch_no_channels_last_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_hash_tensor_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_heaviside_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_histc_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_hsplit_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_hstack_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_hypot_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_i0_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_igamma_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_igammac_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_imag_cuda_complex64, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_index_add_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_index_copy_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_index_fill_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_index_put_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_index_put_functorch_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_index_reduce_amax_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_index_reduce_amin_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_index_reduce_mean_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_index_reduce_prod_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_index_select_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_inner_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_int_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_int_functorch_no_channels_last_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_isclose_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_isfinite_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_isin_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_isinf_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_isnan_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_isneginf_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_isposinf_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_isreal_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_istft_cuda_complex64, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_item_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_jiterator_2inputs_2outputs_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_jiterator_4inputs_with_extra_args_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_jiterator_binary_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_jiterator_binary_return_by_ref_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_jiterator_unary_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_kron_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_kthvalue_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_lcm_cuda_int64, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_ldexp_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_le_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_lerp_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_lgamma_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_linalg_cholesky_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_linalg_cholesky_ex_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_linalg_cond_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_linalg_cross_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_linalg_det_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_linalg_diagonal_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_linalg_eig_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_linalg_eigh_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_linalg_eigvals_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_linalg_eigvalsh_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_linalg_householder_product_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_linalg_inv_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_linalg_inv_ex_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_linalg_ldl_factor_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_linalg_ldl_factor_ex_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_linalg_ldl_solve_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_linalg_lstsq_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_linalg_lstsq_grad_oriented_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_linalg_lu_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_linalg_lu_factor_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_linalg_lu_factor_ex_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_linalg_lu_solve_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_linalg_matrix_norm_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_linalg_matrix_power_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_linalg_matrix_rank_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_linalg_matrix_rank_hermitian_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_linalg_multi_dot_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_linalg_norm_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_linalg_norm_subgradients_at_zero_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_linalg_pinv_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_linalg_pinv_hermitian_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_linalg_pinv_singular_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_linalg_qr_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_linalg_slogdet_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_linalg_solve_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_linalg_solve_ex_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_linalg_solve_triangular_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_linalg_svd_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_linalg_svdvals_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_linalg_tensorinv_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_linalg_tensorsolve_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_linalg_vander_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_linalg_vecdot_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_linalg_vector_norm_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_linspace_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_linspace_tensor_overload_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_log10_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_log1p_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_log2_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_log_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_log_normal_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_log_softmax_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_log_softmax_with_dtype_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_logaddexp2_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_logaddexp_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_logcumsumexp_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_logdet_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_logical_and_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_logical_not_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_logical_or_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_logical_xor_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_logit_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_logspace_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_logspace_tensor_overload_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_logsumexp_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_long_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_long_functorch_no_channels_last_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_lt_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_lu_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_lu_solve_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_lu_unpack_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_mH_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_mT_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_masked_amax_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_masked_amin_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_masked_argmax_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_masked_argmin_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_masked_cumprod_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_masked_cumsum_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_masked_fill_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_masked_fill_functorch_Scalar_only_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_masked_log_softmax_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_masked_logaddexp_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_masked_logsumexp_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_masked_mean_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_masked_median_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_masked_norm_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_masked_normalize_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_masked_prod_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_masked_scatter_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_masked_select_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_masked_softmax_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_masked_softmin_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_masked_std_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_masked_sum_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_masked_var_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_matmul_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_matrix_exp_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_max_binary_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_max_pool2d_with_indices_backward_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_max_reduction_no_dim_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_max_reduction_with_dim_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_maximum_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_mean_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_median_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_meshgrid_list_of_tensors_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_meshgrid_variadic_tensors_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_min_binary_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_min_reduction_no_dim_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_min_reduction_with_dim_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_minimum_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_mm_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_mode_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_movedim_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_msort_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_mul_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_multinomial_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_mv_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_mvlgamma_mvlgamma_p_1_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_mvlgamma_mvlgamma_p_3_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_mvlgamma_mvlgamma_p_5_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_nan_to_num_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_nanmean_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_nanmedian_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_nanquantile_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_nansum_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_narrow_copy_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_narrow_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_native_batch_norm_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_native_dropout_backward_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_native_layer_norm_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_ne_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_neg_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_new_empty_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_new_empty_strided_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_new_full_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_new_ones_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_new_zeros_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_nextafter_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_nn_functional_adaptive_avg_pool1d_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_nn_functional_adaptive_avg_pool2d_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_nn_functional_adaptive_avg_pool3d_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_nn_functional_adaptive_max_pool1d_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_nn_functional_adaptive_max_pool2d_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_nn_functional_adaptive_max_pool3d_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_nn_functional_alpha_dropout_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_nn_functional_avg_pool1d_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_nn_functional_avg_pool2d_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_nn_functional_avg_pool3d_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_nn_functional_batch_norm_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_nn_functional_batch_norm_without_cudnn_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_nn_functional_bilinear_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_nn_functional_binary_cross_entropy_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_nn_functional_binary_cross_entropy_with_logits_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_nn_functional_celu_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_nn_functional_channel_shuffle_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_nn_functional_conv1d_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_nn_functional_conv2d_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_nn_functional_conv2d_no_bias_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_nn_functional_conv2d_stride_depthwise_with_bias_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_nn_functional_conv2d_stride_groups_with_bias_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_nn_functional_conv2d_stride_no_bias_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_nn_functional_conv2d_stride_padding_no_bias_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_nn_functional_conv2d_stride_padding_with_bias_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_nn_functional_conv2d_stride_with_bias_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_nn_functional_conv2d_strided_padding_dilation_no_bias_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_nn_functional_conv2d_strided_padding_dilation_with_bias_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_nn_functional_conv2d_with_bias_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_nn_functional_conv3d_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_nn_functional_conv_transpose1d_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_nn_functional_conv_transpose2d_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_nn_functional_conv_transpose3d_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_nn_functional_cosine_embedding_loss_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_nn_functional_cosine_similarity_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_nn_functional_cross_entropy_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_nn_functional_ctc_loss_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_nn_functional_dropout2d_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_nn_functional_dropout3d_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_nn_functional_dropout_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_nn_functional_elu_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_nn_functional_embedding_bag_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_nn_functional_embedding_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_nn_functional_embedding_functorch_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_nn_functional_feature_alpha_dropout_with_train_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_nn_functional_feature_alpha_dropout_without_train_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_nn_functional_fractional_max_pool2d_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_nn_functional_fractional_max_pool3d_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_nn_functional_gaussian_nll_loss_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_nn_functional_gelu_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_nn_functional_glu_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_nn_functional_grid_sample_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_nn_functional_group_norm_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_nn_functional_hardshrink_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_nn_functional_hardsigmoid_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_nn_functional_hardswish_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_nn_functional_hardtanh_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_nn_functional_hinge_embedding_loss_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_nn_functional_huber_loss_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_nn_functional_instance_norm_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_nn_functional_interpolate_area_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_nn_functional_interpolate_bicubic_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_nn_functional_interpolate_bilinear_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_nn_functional_interpolate_linear_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_nn_functional_interpolate_nearest-exact_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_nn_functional_interpolate_nearest_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_nn_functional_interpolate_trilinear_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_nn_functional_kl_div_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_nn_functional_l1_loss_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_nn_functional_layer_norm_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_nn_functional_leaky_relu_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_nn_functional_linear_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_nn_functional_local_response_norm_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_nn_functional_logsigmoid_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_nn_functional_margin_ranking_loss_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_nn_functional_max_pool1d_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_nn_functional_max_pool2d_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_nn_functional_max_pool3d_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_nn_functional_max_unpool1d_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_nn_functional_max_unpool1d_grad_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_nn_functional_max_unpool2d_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_nn_functional_max_unpool2d_grad_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_nn_functional_max_unpool3d_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_nn_functional_max_unpool3d_grad_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_nn_functional_mish_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_nn_functional_mse_loss_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_nn_functional_mse_loss_functorch_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_nn_functional_multi_head_attention_forward_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_nn_functional_multi_margin_loss_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_nn_functional_multilabel_margin_loss_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_nn_functional_multilabel_soft_margin_loss_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_nn_functional_nll_loss_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_nn_functional_normalize_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_nn_functional_one_hot_cuda_int64, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_nn_functional_pad_circular_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_nn_functional_pad_constant_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_nn_functional_pad_reflect_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_nn_functional_pad_replicate_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_nn_functional_pad_replicate_negative_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_nn_functional_pairwise_distance_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_nn_functional_pdist_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_nn_functional_pixel_shuffle_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_nn_functional_pixel_unshuffle_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_nn_functional_poisson_nll_loss_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_nn_functional_prelu_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_nn_functional_relu6_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_nn_functional_relu_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_nn_functional_rms_norm_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_nn_functional_rrelu_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_nn_functional_scaled_dot_product_attention_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_nn_functional_selu_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_nn_functional_silu_complex_cuda_complex64, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_nn_functional_silu_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_nn_functional_smooth_l1_loss_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_nn_functional_soft_margin_loss_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_nn_functional_softmin_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_nn_functional_softmin_with_dtype_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_nn_functional_softplus_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_nn_functional_softshrink_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_nn_functional_softsign_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_nn_functional_tanhshrink_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_nn_functional_threshold_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_nn_functional_triplet_margin_loss_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_nn_functional_triplet_margin_with_distance_loss_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_nn_functional_unfold_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_nn_functional_upsample_bilinear_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_nn_functional_upsample_nearest_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_nonzero_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_nonzero_static_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_norm_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_norm_fro_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_norm_inf_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_norm_nuc_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_normal_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_normal_in_place_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_normal_number_mean_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_ones_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_ones_like_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_ops_aten__new_zeros_with_same_feature_meta_functorchonly_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_ops_aten_index_put_functorch_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_ormqr_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_outer_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_pca_lowrank_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_permute_copy_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_permute_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_pinverse_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_polar_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_polygamma_polygamma_n_0_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_polygamma_polygamma_n_1_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_polygamma_polygamma_n_2_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_polygamma_polygamma_n_3_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_polygamma_polygamma_n_4_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_positive_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_pow_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_prod_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_put_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_qr_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_quantile_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_rad2deg_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_rand_like_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_randint_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_randint_like_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_randn_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_randn_like_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_ravel_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_real_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_reciprocal_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_remainder_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_renorm_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_repeat_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_repeat_interleave_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_reshape_as_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_reshape_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_resize__cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_resize_as__cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_resolve_conj_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_resolve_neg_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_roll_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_rot90_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_round_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_round_decimals_0_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_round_decimals_3_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_round_decimals_neg_3_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_rsqrt_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_rsub_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_scalar_tensor_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_scatter_add_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_scatter_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_scatter_reduce_amax_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_scatter_reduce_amin_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_scatter_reduce_mean_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_scatter_reduce_prod_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_scatter_reduce_sum_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_searchsorted_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_select_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_select_scatter_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_sgn_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_short_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_short_functorch_no_channels_last_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_sigmoid_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_sign_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_signal_windows_bartlett_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_signal_windows_blackman_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_signal_windows_cosine_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_signal_windows_exponential_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_signal_windows_gaussian_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_signal_windows_general_cosine_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_signal_windows_general_hamming_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_signal_windows_hamming_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_signal_windows_hann_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_signal_windows_kaiser_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_signal_windows_nuttall_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_signbit_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_sin_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_sinc_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_sinh_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_slice_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_slice_scatter_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_softmax_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_softmax_with_dtype_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_sort_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_sparse_mm_reduce_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_sparse_sampled_addmm_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_special_airy_ai_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_special_bessel_j0_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_special_bessel_j1_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_special_bessel_y0_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_special_bessel_y1_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_special_chebyshev_polynomial_t_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_special_chebyshev_polynomial_u_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_special_chebyshev_polynomial_v_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_special_chebyshev_polynomial_w_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_special_entr_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_special_erfcx_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_special_hermite_polynomial_h_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_special_hermite_polynomial_he_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_special_i0e_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_special_i1_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_special_i1e_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_special_laguerre_polynomial_l_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_special_legendre_polynomial_p_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_special_log_ndtr_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_special_modified_bessel_i0_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_special_modified_bessel_i1_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_special_modified_bessel_k0_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_special_modified_bessel_k1_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_special_ndtr_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_special_ndtri_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_special_polygamma_special_polygamma_n_0_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_special_scaled_modified_bessel_k0_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_special_scaled_modified_bessel_k1_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_special_shifted_chebyshev_polynomial_t_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_special_shifted_chebyshev_polynomial_u_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_special_shifted_chebyshev_polynomial_v_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_special_shifted_chebyshev_polynomial_w_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_special_spherical_bessel_j0_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_special_xlog1py_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_special_zeta_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_split_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_split_list_args_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_split_with_sizes_copy_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_split_with_sizes_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_sqrt_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_square_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_squeeze_copy_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_squeeze_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_squeeze_multiple_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_stack_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_std_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_std_mean_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_std_mean_unbiased_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_std_unbiased_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_stft_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_sub_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_sum_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_sum_to_size_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_svd_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_svd_lowrank_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_t_copy_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_t_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_take_along_dim_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_take_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_tan_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_tanh_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_tensor_split_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_tensordot_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_tile_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_to_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_to_sparse_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_topk_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_torch__scaled_mm_cuda_float8_e4m3fn, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_torch__scaled_mm_v2_cuda_float8_e4m3fn, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_torch_ops_aten__efficient_attention_forward_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_torch_ops_aten__flash_attention_forward_cuda_float16, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_torch_ops_aten__safe_softmax_default_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_trace_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_transpose_copy_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_transpose_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_trapezoid_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_trapz_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_triangular_solve_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_tril_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_tril_indices_cuda_int64, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_triu_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_triu_indices_cuda_int64, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_true_divide_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_trunc_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_unbind_copy_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_unbind_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_unflatten_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_unfold_copy_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_unfold_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_uniform_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_unique_consecutive_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_unique_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_unravel_index_cuda_int64, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_unsafe_chunk_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_unsafe_split_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_unsqueeze_copy_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_unsqueeze_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_var_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_var_mean_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_var_mean_unbiased_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_var_unbiased_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_vdot_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_view_as_complex_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_view_as_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_view_as_real_cuda_complex64, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_view_copy_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_view_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_vsplit_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_vstack_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_where_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_xlogy_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_zero__cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_zeros_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_zeros_like_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_linalg_failure_1D_input_linalg_cholesky_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_linalg_failure_1D_input_linalg_cholesky_ex_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_linalg_failure_1D_input_linalg_cond_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_linalg_failure_1D_input_linalg_cross_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_linalg_failure_1D_input_linalg_det_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_linalg_failure_1D_input_linalg_diagonal_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_linalg_failure_1D_input_linalg_eig_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_linalg_failure_1D_input_linalg_eigh_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_linalg_failure_1D_input_linalg_eigvals_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_linalg_failure_1D_input_linalg_eigvalsh_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_linalg_failure_1D_input_linalg_householder_product_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_linalg_failure_1D_input_linalg_inv_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_linalg_failure_1D_input_linalg_inv_ex_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_linalg_failure_1D_input_linalg_ldl_factor_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_linalg_failure_1D_input_linalg_ldl_factor_ex_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_linalg_failure_1D_input_linalg_ldl_solve_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_linalg_failure_1D_input_linalg_lstsq_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_linalg_failure_1D_input_linalg_lstsq_grad_oriented_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_linalg_failure_1D_input_linalg_lu_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_linalg_failure_1D_input_linalg_lu_factor_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_linalg_failure_1D_input_linalg_lu_factor_ex_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_linalg_failure_1D_input_linalg_lu_solve_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_linalg_failure_1D_input_linalg_matrix_norm_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_linalg_failure_1D_input_linalg_matrix_power_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_linalg_failure_1D_input_linalg_matrix_rank_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_linalg_failure_1D_input_linalg_matrix_rank_hermitian_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_linalg_failure_1D_input_linalg_multi_dot_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_linalg_failure_1D_input_linalg_norm_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_linalg_failure_1D_input_linalg_norm_subgradients_at_zero_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_linalg_failure_1D_input_linalg_pinv_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_linalg_failure_1D_input_linalg_pinv_hermitian_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_linalg_failure_1D_input_linalg_pinv_singular_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_linalg_failure_1D_input_linalg_qr_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_linalg_failure_1D_input_linalg_slogdet_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_linalg_failure_1D_input_linalg_solve_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_linalg_failure_1D_input_linalg_solve_ex_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_linalg_failure_1D_input_linalg_solve_triangular_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_linalg_failure_1D_input_linalg_svd_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_linalg_failure_1D_input_linalg_svdvals_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_linalg_failure_1D_input_linalg_tensorinv_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_linalg_failure_1D_input_linalg_tensorsolve_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_linalg_failure_1D_input_linalg_vander_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_linalg_failure_1D_input_linalg_vecdot_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_linalg_failure_1D_input_linalg_vector_norm_cuda_float32, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_multi_dot_failure_1D_input_cuda, test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_with_anomaly_detection_cuda, test/functorch/test_vmap.py::TestVmapBatchedGradientCUDA::test_add_cuda, test/functorch/test_vmap.py::TestVmapBatchedGradientCUDA::test_binary_cross_entropy_cuda, test/functorch/test_vmap.py::TestVmapBatchedGradientCUDA::test_diagonal_cuda, test/functorch/test_vmap.py::TestVmapBatchedGradientCUDA::test_div_cuda, test/functorch/test_vmap.py::TestVmapBatchedGradientCUDA::test_expand_cuda, test/functorch/test_vmap.py::TestVmapBatchedGradientCUDA::test_index_cuda, test/functorch/test_vmap.py::TestVmapBatchedGradientCUDA::test_inplace_manyview_cuda, test/functorch/test_vmap.py::TestVmapBatchedGradientCUDA::test_inplace_view_cuda, test/functorch/test_vmap.py::TestVmapBatchedGradientCUDA::test_lgamma_cuda, test/functorch/test_vmap.py::TestVmapBatchedGradientCUDA::test_log1p_cuda, test/functorch/test_vmap.py::TestVmapBatchedGradientCUDA::test_log_cuda, test/functorch/test_vmap.py::TestVmapBatchedGradientCUDA::test_log_softmax_cuda, test/functorch/test_vmap.py::TestVmapBatchedGradientCUDA::test_logsumexp_cuda, test/functorch/test_vmap.py::TestVmapBatchedGradientCUDA::test_max_cuda, test/functorch/test_vmap.py::TestVmapBatchedGradientCUDA::test_median_cuda, test/functorch/test_vmap.py::TestVmapBatchedGradientCUDA::test_min_cuda, test/functorch/test_vmap.py::TestVmapBatchedGradientCUDA::test_mul_cuda, test/functorch/test_vmap.py::TestVmapBatchedGradientCUDA::test_permute_cuda, test/functorch/test_vmap.py::TestVmapBatchedGradientCUDA::test_randomness_backend0_randomness_different_cuda, test/functorch/test_vmap.py::TestVmapBatchedGradientCUDA::test_randomness_backend0_randomness_error_cuda, test/functorch/test_vmap.py::TestVmapBatchedGradientCUDA::test_randomness_backend0_randomness_same_cuda, test/functorch/test_vmap.py::TestVmapBatchedGradientCUDA::test_randomness_backend1_randomness_different_cuda, test/functorch/test_vmap.py::TestVmapBatchedGradientCUDA::test_randomness_backend1_randomness_error_cuda, test/functorch/test_vmap.py::TestVmapBatchedGradientCUDA::test_randomness_backend1_randomness_same_cuda, test/functorch/test_vmap.py::TestVmapBatchedGradientCUDA::test_randomness_backend2_randomness_different_cuda, test/functorch/test_vmap.py::TestVmapBatchedGradientCUDA::test_randomness_backend2_randomness_error_cuda, test/functorch/test_vmap.py::TestVmapBatchedGradientCUDA::test_randomness_backend2_randomness_same_cuda, test/functorch/test_vmap.py::TestVmapBatchedGradientCUDA::test_randomness_backend3_randomness_different_cuda, test/functorch/test_vmap.py::TestVmapBatchedGradientCUDA::test_randomness_backend3_randomness_error_cuda, test/functorch/test_vmap.py::TestVmapBatchedGradientCUDA::test_randomness_backend3_randomness_same_cuda, test/functorch/test_vmap.py::TestVmapBatchedGradientCUDA::test_reshape_cuda, test/functorch/test_vmap.py::TestVmapBatchedGradientCUDA::test_sdpa_backend0_cuda, test/functorch/test_vmap.py::TestVmapBatchedGradientCUDA::test_sdpa_backend1_cuda, test/functorch/test_vmap.py::TestVmapBatchedGradientCUDA::test_sdpa_backend2_cuda, test/functorch/test_vmap.py::TestVmapBatchedGradientCUDA::test_sdpa_backend3_cuda, test/functorch/test_vmap.py::TestVmapBatchedGradientCUDA::test_select_cuda, test/functorch/test_vmap.py::TestVmapBatchedGradientCUDA::test_sigmoid_cuda, test/functorch/test_vmap.py::TestVmapBatchedGradientCUDA::test_slice_cuda, test/functorch/test_vmap.py::TestVmapBatchedGradientCUDA::test_stack_cuda, test/functorch/test_vmap.py::TestVmapBatchedGradientCUDA::test_sub_cuda, test/functorch/test_vmap.py::TestVmapBatchedGradientCUDA::test_threshold_cuda, test/functorch/test_vmap.py::TestVmapBatchedGradientCUDA::test_trace_cuda, test/functorch/test_vmap.py::TestVmapBatchedGradientCUDA::test_unrelated_output_cuda, test/functorch/test_vmap.py::TestVmapBatchedGradientCUDA::test_unrelated_output_multiple_grad_cuda, test/functorch/test_vmap.py::TestVmapBatchedGradientCUDA::test_vmap_fallback_check, test/functorch/test_vmap.py::TestVmapBatchedGradientCUDA::test_vmap_fallback_check_ok, test/functorch/test_vmap.py::TestVmapBatchedGradientCUDA::test_where_cuda, test/functorch/test_vmap.py::TestTransformFailureCUDA::test_fails_with_autograd_function_transform_grad_and_value_cuda, test/functorch/test_vmap.py::TestTransformFailureCUDA::test_fails_with_autograd_function_transform_grad_cuda, test/functorch/test_vmap.py::TestTransformFailureCUDA::test_fails_with_autograd_function_transform_jacfwd_cuda, test/functorch/test_vmap.py::TestTransformFailureCUDA::test_fails_with_autograd_function_transform_jacrev_cuda, test/functorch/test_vmap.py::TestTransformFailureCUDA::test_fails_with_autograd_function_transform_jvp_cuda, test/functorch/test_vmap.py::TestTransformFailureCUDA::test_fails_with_autograd_function_transform_vjp_cuda, test/functorch/test_vmap.py::TestTransformFailureCUDA::test_fails_with_autograd_function_transform_vmap_cuda, test/functorch/test_vmap.py::TestRandomnessCUDA::test_alpha_dropout_randomness_different_batched_input_first_cuda, test/functorch/test_vmap.py::TestRandomnessCUDA::test_alpha_dropout_randomness_different_batched_input_last_cuda, test/functorch/test_vmap.py::TestRandomnessCUDA::test_alpha_dropout_randomness_different_batched_input_none_cuda, test/functorch/test_vmap.py::TestRandomnessCUDA::test_alpha_dropout_randomness_error_batched_input_first_cuda, test/functorch/test_vmap.py::TestRandomnessCUDA::test_alpha_dropout_randomness_error_batched_input_last_cuda, test/functorch/test_vmap.py::TestRandomnessCUDA::test_alpha_dropout_randomness_error_batched_input_none_cuda, test/functorch/test_vmap.py::TestRandomnessCUDA::test_alpha_dropout_randomness_same_batched_input_first_cuda, test/functorch/test_vmap.py::TestRandomnessCUDA::test_alpha_dropout_randomness_same_batched_input_last_cuda, test/functorch/test_vmap.py::TestRandomnessCUDA::test_alpha_dropout_randomness_same_batched_input_none_cuda, test/functorch/test_vmap.py::TestRandomnessCUDA::test_bernoulli_in_place_use_generator_False_randomness_different_batched_input_first_batched_probability_first_cuda, test/functorch/test_vmap.py::TestRandomnessCUDA::test_bernoulli_in_place_use_generator_False_randomness_different_batched_input_first_batched_probability_last_cuda, test/functorch/test_vmap.py::TestRandomnessCUDA::test_bernoulli_in_place_use_generator_False_randomness_different_batched_input_first_batched_probability_none_cuda, test/functorch/test_vmap.py::TestRandomnessCUDA::test_bernoulli_in_place_use_generator_False_randomness_different_batched_input_last_batched_probability_first_cuda, test/functorch/test_vmap.py::TestRandomnessCUDA::test_bernoulli_in_place_use_generator_False_randomness_different_batched_input_last_batched_probability_last_cuda, test/functorch/test_vmap.py::TestRandomnessCUDA::test_bernoulli_in_place_use_generator_False_randomness_different_batched_input_last_batched_probability_none_cuda, test/functorch/test_vmap.py::TestRandomnessCUDA::test_bernoulli_in_place_use_generator_False_randomness_different_batched_input_none_batched_probability_first_cuda, test/functorch/test_vmap.py::TestRandomnessCUDA::test_bernoulli_in_place_use_generator_False_randomness_different_batched_input_none_batched_probability_last_cuda, test/functorch/test_vmap.py::TestRandomnessCUDA::test_bernoulli_in_place_use_generator_False_randomness_different_batched_input_none_batched_probability_none_cuda, test/functorch/test_vmap.py::TestRandomnessCUDA::test_bernoulli_in_place_use_generator_False_randomness_error_batched_input_first_batched_probability_first_cuda, test/functorch/test_vmap.py::TestRandomnessCUDA::test_bernoulli_in_place_use_generator_False_randomness_error_batched_input_first_batched_probability_last_cuda, test/functorch/test_vmap.py::TestRandomnessCUDA::test_bernoulli_in_place_use_generator_False_randomness_error_batched_input_first_batched_probability_none_cuda, test/functorch/test_vmap.py::TestRandomnessCUDA::test_bernoulli_in_place_use_generator_False_randomness_error_batched_input_last_batched_probability_first_cuda, test/functorch/test_vmap.py::TestRandomnessCUDA::test_bernoulli_in_place_use_generator_False_randomness_error_batched_input_last_batched_probability_last_cuda, test/functorch/test_vmap.py::TestRandomnessCUDA::test_bernoulli_in_place_use_generator_False_randomness_error_batched_input_last_batched_probability_none_cuda, test/functorch/test_vmap.py::TestRandomnessCUDA::test_bernoulli_in_place_use_generator_False_randomness_error_batched_input_none_batched_probability_first_cuda, test/functorch/test_vmap.py::TestRandomnessCUDA::test_bernoulli_in_place_use_generator_False_randomness_error_batched_input_none_batched_probability_last_cuda, test/functorch/test_vmap.py::TestRandomnessCUDA::test_bernoulli_in_place_use_generator_False_randomness_error_batched_input_none_batched_probability_none_cuda, test/functorch/test_vmap.py::TestRandomnessCUDA::test_bernoulli_in_place_use_generator_False_randomness_same_batched_input_first_batched_probability_first_cuda, test/functorch/test_vmap.py::TestRandomnessCUDA::test_bernoulli_in_place_use_generator_False_randomness_same_batched_input_first_batched_probability_last_cuda, test/functorch/test_vmap.py::TestRandomnessCUDA::test_bernoulli_in_place_use_generator_False_randomness_same_batched_input_first_batched_probability_none_cuda, test/functorch/test_vmap.py::TestRandomnessCUDA::test_bernoulli_in_place_use_generator_False_randomness_same_batched_input_last_batched_probability_first_cuda, test/functorch/test_vmap.py::TestRandomnessCUDA::test_bernoulli_in_place_use_generator_False_randomness_same_batched_input_last_batched_probability_last_cuda, test/functorch/test_vmap.py::TestRandomnessCUDA::test_bernoulli_in_place_use_generator_False_randomness_same_batched_input_last_batched_probability_none_cuda, test/functorch/test_vmap.py::TestRandomnessCUDA::test_bernoulli_in_place_use_generator_False_randomness_same_batched_input_none_batched_probability_first_cuda, test/functorch/test_vmap.py::TestRandomnessCUDA::test_bernoulli_in_place_use_generator_False_randomness_same_batched_input_none_batched_probability_last_cuda, test/functorch/test_vmap.py::TestRandomnessCUDA::test_bernoulli_in_place_use_generator_False_randomness_same_batched_input_none_batched_probability_none_cuda, test/functorch/test_vmap.py::TestRandomnessCUDA::test_bernoulli_in_place_use_generator_True_randomness_different_batched_input_first_batched_probability_first_cuda, test/functorch/test_vmap.py::TestRandomnessCUDA::test_bernoulli_in_place_use_generator_True_randomness_different_batched_input_first_batched_probability_last_cuda, test/functorch/test_vmap.py::TestRandomnessCUDA::test_bernoulli_in_place_use_generator_True_randomness_different_batched_input_first_batched_probability_none_cuda, test/functorch/test_vmap.py::TestRandomnessCUDA::test_bernoulli_in_place_use_generator_True_randomness_different_batched_input_last_batched_probability_first_cuda, test/functorch/test_vmap.py::TestRandomnessCUDA::test_bernoulli_in_place_use_generator_True_randomness_different_batched_input_last_batched_probability_last_cuda, test/functorch/test_vmap.py::TestRandomnessCUDA::test_bernoulli_in_place_use_generator_True_randomness_different_batched_input_last_batched_probability_none_cuda, test/functorch/test_vmap.py::TestRandomnessCUDA::test_bernoulli_in_place_use_generator_True_randomness_different_batched_input_none_batched_probability_first_cuda, test/functorch/test_vmap.py::TestRandomnessCUDA::test_bernoulli_in_place_use_generator_True_randomness_different_batched_input_none_batched_probability_last_cuda, test/functorch/test_vmap.py::TestRandomnessCUDA::test_bernoulli_in_place_use_generator_True_randomness_different_batched_input_none_batched_probability_none_cuda, test/functorch/test_vmap.py::TestRandomnessCUDA::test_bernoulli_in_place_use_generator_True_randomness_error_batched_input_first_batched_probability_first_cuda, test/functorch/test_vmap.py::TestRandomnessCUDA::test_bernoulli_in_place_use_generator_True_randomness_error_batched_input_first_batched_probability_last_cuda, test/functorch/test_vmap.py::TestRandomnessCUDA::test_bernoulli_in_place_use_generator_True_randomness_error_batched_input_first_batched_probability_none_cuda, test/functorch/test_vmap.py::TestRandomnessCUDA::test_bernoulli_in_place_use_generator_True_randomness_error_batched_input_last_batched_probability_first_cuda, test/functorch/test_vmap.py::TestRandomnessCUDA::test_bernoulli_in_place_use_generator_True_randomness_error_batched_input_last_batched_probability_last_cuda, test/functorch/test_vmap.py::TestRandomnessCUDA::test_bernoulli_in_place_use_generator_True_randomness_error_batched_input_last_batched_probability_none_cuda, test/functorch/test_vmap.py::TestRandomnessCUDA::test_bernoulli_in_place_use_generator_True_randomness_error_batched_input_none_batched_probability_first_cuda, test/functorch/test_vmap.py::TestRandomnessCUDA::test_bernoulli_in_place_use_generator_True_randomness_error_batched_input_none_batched_probability_last_cuda, test/functorch/test_vmap.py::TestRandomnessCUDA::test_bernoulli_in_place_use_generator_True_randomness_error_batched_input_none_batched_probability_none_cuda, test/functorch/test_vmap.py::TestRandomnessCUDA::test_bernoulli_in_place_use_generator_True_randomness_same_batched_input_first_batched_probability_first_cuda, test/functorch/test_vmap.py::TestRandomnessCUDA::test_bernoulli_in_place_use_generator_True_randomness_same_batched_input_first_batched_probability_last_cuda, test/functorch/test_vmap.py::TestRandomnessCUDA::test_bernoulli_in_place_use_generator_True_randomness_same_batched_input_first_batched_probability_none_cuda, test/functorch/test_vmap.py::TestRandomnessCUDA::test_bernoulli_in_place_use_generator_True_randomness_same_batched_input_last_batched_probability_first_cuda, test/functorch/test_vmap.py::TestRandomnessCUDA::test_bernoulli_in_place_use_generator_True_randomness_same_batched_input_last_batched_probability_last_cuda, test/functorch/test_vmap.py::TestRandomnessCUDA::test_bernoulli_in_place_use_generator_True_randomness_same_batched_input_last_batched_probability_none_cuda, test/functorch/test_vmap.py::TestRandomnessCUDA::test_bernoulli_in_place_use_generator_True_randomness_same_batched_input_none_batched_probability_first_cuda, test/functorch/test_vmap.py::TestRandomnessCUDA::test_bernoulli_in_place_use_generator_True_randomness_same_batched_input_none_batched_probability_last_cuda, test/functorch/test_vmap.py::TestRandomnessCUDA::test_bernoulli_in_place_use_generator_True_randomness_same_batched_input_none_batched_probability_none_cuda, test/functorch/test_vmap.py::TestRandomnessCUDA::test_chunk_vmap_in_dim_0_out_dim_0_cuda, test/functorch/test_vmap.py::TestRandomnessCUDA::test_chunk_vmap_in_dim_0_out_dim_1_cuda, test/functorch/test_vmap.py::TestRandomnessCUDA::test_chunk_vmap_in_dim_0_out_dim_2_cuda, test/functorch/test_vmap.py::TestRandomnessCUDA::test_chunk_vmap_in_dim_1_out_dim_0_cuda, test/functorch/test_vmap.py::TestRandomnessCUDA::test_chunk_vmap_in_dim_1_out_dim_1_cuda, test/functorch/test_vmap.py::TestRandomnessCUDA::test_chunk_vmap_in_dim_1_out_dim_2_cuda, test/functorch/test_vmap.py::TestRandomnessCUDA::test_chunk_vmap_in_dim_2_out_dim_0_cuda, test/functorch/test_vmap.py::TestRandomnessCUDA::test_chunk_vmap_in_dim_2_out_dim_1_cuda, test/functorch/test_vmap.py::TestRandomnessCUDA::test_chunk_vmap_in_dim_2_out_dim_2_cuda, test/functorch/test_vmap.py::TestRandomnessCUDA::test_dropout_randomness_different_batched_input_first_cuda, test/functorch/test_vmap.py::TestRandomnessCUDA::test_dropout_randomness_different_batched_input_last_cuda, test/functorch/test_vmap.py::TestRandomnessCUDA::test_dropout_randomness_different_batched_input_none_cuda, test/functorch/test_vmap.py::TestRandomnessCUDA::test_dropout_randomness_error_batched_input_first_cuda, test/functorch/test_vmap.py::TestRandomnessCUDA::test_dropout_randomness_error_batched_input_last_cuda, test/functorch/test_vmap.py::TestRandomnessCUDA::test_dropout_randomness_error_batched_input_none_cuda, test/functorch/test_vmap.py::TestRandomnessCUDA::test_dropout_randomness_same_batched_input_first_cuda, test/functorch/test_vmap.py::TestRandomnessCUDA::test_dropout_randomness_same_batched_input_last_cuda, test/functorch/test_vmap.py::TestRandomnessCUDA::test_dropout_randomness_same_batched_input_none_cuda, test/functorch/test_vmap.py::TestRandomnessCUDA::test_dropout_unbatched_randomness_different_cuda, test/functorch/test_vmap.py::TestRandomnessCUDA::test_dropout_unbatched_randomness_error_cuda, test/functorch/test_vmap.py::TestRandomnessCUDA::test_dropout_unbatched_randomness_same_cuda, test/functorch/test_vmap.py::TestRandomnessCUDA::test_factory_ops_randomness_different_use_generator_False_cuda, test/functorch/test_vmap.py::TestRandomnessCUDA::test_factory_ops_randomness_different_use_generator_True_cuda, test/functorch/test_vmap.py::TestRandomnessCUDA::test_factory_ops_randomness_error_use_generator_False_cuda, test/functorch/test_vmap.py::TestRandomnessCUDA::test_factory_ops_randomness_error_use_generator_True_cuda, test/functorch/test_vmap.py::TestRandomnessCUDA::test_factory_ops_randomness_same_use_generator_False_cuda, test/functorch/test_vmap.py::TestRandomnessCUDA::test_factory_ops_randomness_same_use_generator_True_cuda, test/functorch/test_vmap.py::TestRandomnessCUDA::test_feature_alpha_dropout_randomness_different_batched_input_first_cuda, test/functorch/test_vmap.py::TestRandomnessCUDA::test_feature_alpha_dropout_randomness_different_batched_input_last_cuda, test/functorch/test_vmap.py::TestRandomnessCUDA::test_feature_alpha_dropout_randomness_different_batched_input_none_cuda, test/functorch/test_vmap.py::TestRandomnessCUDA::test_feature_alpha_dropout_randomness_error_batched_input_first_cuda, test/functorch/test_vmap.py::TestRandomnessCUDA::test_feature_alpha_dropout_randomness_error_batched_input_last_cuda, test/functorch/test_vmap.py::TestRandomnessCUDA::test_feature_alpha_dropout_randomness_error_batched_input_none_cuda, test/functorch/test_vmap.py::TestRandomnessCUDA::test_feature_alpha_dropout_randomness_same_batched_input_first_cuda, test/functorch/test_vmap.py::TestRandomnessCUDA::test_feature_alpha_dropout_randomness_same_batched_input_last_cuda, test/functorch/test_vmap.py::TestRandomnessCUDA::test_feature_alpha_dropout_randomness_same_batched_input_none_cuda, test/functorch/test_vmap.py::TestRandomnessCUDA::test_feature_dropout_randomness_different_batched_input_first_dim_2_cuda, test/functorch/test_vmap.py::TestRandomnessCUDA::test_feature_dropout_randomness_different_batched_input_first_dim_3_cuda, test/functorch/test_vmap.py::TestRandomnessCUDA::test_feature_dropout_randomness_different_batched_input_last_dim_2_cuda, test/functorch/test_vmap.py::TestRandomnessCUDA::test_feature_dropout_randomness_different_batched_input_last_dim_3_cuda, test/functorch/test_vmap.py::TestRandomnessCUDA::test_feature_dropout_randomness_different_batched_input_none_dim_2_cuda, test/functorch/test_vmap.py::TestRandomnessCUDA::test_feature_dropout_randomness_different_batched_input_none_dim_3_cuda, test/functorch/test_vmap.py::TestRandomnessCUDA::test_feature_dropout_randomness_error_batched_input_first_dim_2_cuda, test/functorch/test_vmap.py::TestRandomnessCUDA::test_feature_dropout_randomness_error_batched_input_first_dim_3_cuda, test/functorch/test_vmap.py::TestRandomnessCUDA::test_feature_dropout_randomness_error_batched_input_last_dim_2_cuda, test/functorch/test_vmap.py::TestRandomnessCUDA::test_feature_dropout_randomness_error_batched_input_last_dim_3_cuda, test/functorch/test_vmap.py::TestRandomnessCUDA::test_feature_dropout_randomness_error_batched_input_none_dim_2_cuda, test/functorch/test_vmap.py::TestRandomnessCUDA::test_feature_dropout_randomness_error_batched_input_none_dim_3_cuda, test/functorch/test_vmap.py::TestRandomnessCUDA::test_feature_dropout_randomness_same_batched_input_first_dim_2_cuda, test/functorch/test_vmap.py::TestRandomnessCUDA::test_feature_dropout_randomness_same_batched_input_first_dim_3_cuda, test/functorch/test_vmap.py::TestRandomnessCUDA::test_feature_dropout_randomness_same_batched_input_last_dim_2_cuda, test/functorch/test_vmap.py::TestRandomnessCUDA::test_feature_dropout_randomness_same_batched_input_last_dim_3_cuda, test/functorch/test_vmap.py::TestRandomnessCUDA::test_feature_dropout_randomness_same_batched_input_none_dim_2_cuda, test/functorch/test_vmap.py::TestRandomnessCUDA::test_feature_dropout_randomness_same_batched_input_none_dim_3_cuda, test/functorch/test_vmap.py::TestRandomnessCUDA::test_jacfwd_with_random_cuda, test/functorch/test_vmap.py::TestRandomnessCUDA::test_like_functions_randomness_different_batched_input_first_cuda, test/functorch/test_vmap.py::TestRandomnessCUDA::test_like_functions_randomness_different_batched_input_last_cuda, test/functorch/test_vmap.py::TestRandomnessCUDA::test_like_functions_randomness_different_batched_input_none_cuda, test/functorch/test_vmap.py::TestRandomnessCUDA::test_like_functions_randomness_error_batched_input_first_cuda, test/functorch/test_vmap.py::TestRandomnessCUDA::test_like_functions_randomness_error_batched_input_last_cuda, test/functorch/test_vmap.py::TestRandomnessCUDA::test_like_functions_randomness_error_batched_input_none_cuda, test/functorch/test_vmap.py::TestRandomnessCUDA::test_like_functions_randomness_same_batched_input_first_cuda, test/functorch/test_vmap.py::TestRandomnessCUDA::test_like_functions_randomness_same_batched_input_last_cuda, test/functorch/test_vmap.py::TestRandomnessCUDA::test_like_functions_randomness_same_batched_input_none_cuda, test/functorch/test_vmap.py::TestRandomnessCUDA::test_multinomial_use_generator_False_randomness_different_batched_call_False_batched_input_first_cuda, test/functorch/test_vmap.py::TestRandomnessCUDA::test_multinomial_use_generator_False_randomness_different_batched_call_False_batched_input_last_cuda, test/functorch/test_vmap.py::TestRandomnessCUDA::test_multinomial_use_generator_False_randomness_different_batched_call_False_batched_input_none_cuda, test/functorch/test_vmap.py::TestRandomnessCUDA::test_multinomial_use_generator_False_randomness_different_batched_call_True_batched_input_first_cuda, test/functorch/test_vmap.py::TestRandomnessCUDA::test_multinomial_use_generator_False_randomness_different_batched_call_True_batched_input_last_cuda, test/functorch/test_vmap.py::TestRandomnessCUDA::test_multinomial_use_generator_False_randomness_different_batched_call_True_batched_input_none_cuda, test/functorch/test_vmap.py::TestRandomnessCUDA::test_multinomial_use_generator_False_randomness_error_batched_call_False_batched_input_first_cuda, test/functorch/test_vmap.py::TestRandomnessCUDA::test_multinomial_use_generator_False_randomness_error_batched_call_False_batched_input_last_cuda, test/functorch/test_vmap.py::TestRandomnessCUDA::test_multinomial_use_generator_False_randomness_error_batched_call_False_batched_input_none_cuda, test/functorch/test_vmap.py::TestRandomnessCUDA::test_multinomial_use_generator_False_randomness_error_batched_call_True_batched_input_first_cuda, test/functorch/test_vmap.py::TestRandomnessCUDA::test_multinomial_use_generator_False_randomness_error_batched_call_True_batched_input_last_cuda, test/functorch/test_vmap.py::TestRandomnessCUDA::test_multinomial_use_generator_False_randomness_error_batched_call_True_batched_input_none_cuda, test/functorch/test_vmap.py::TestRandomnessCUDA::test_multinomial_use_generator_False_randomness_same_batched_call_False_batched_input_first_cuda, test/functorch/test_vmap.py::TestRandomnessCUDA::test_multinomial_use_generator_False_randomness_same_batched_call_False_batched_input_last_cuda, test/functorch/test_vmap.py::TestRandomnessCUDA::test_multinomial_use_generator_False_randomness_same_batched_call_False_batched_input_none_cuda, test/functorch/test_vmap.py::TestRandomnessCUDA::test_multinomial_use_generator_False_randomness_same_batched_call_True_batched_input_first_cuda, test/functorch/test_vmap.py::TestRandomnessCUDA::test_multinomial_use_generator_False_randomness_same_batched_call_True_batched_input_last_cuda, test/functorch/test_vmap.py::TestRandomnessCUDA::test_multinomial_use_generator_False_randomness_same_batched_call_True_batched_input_none_cuda, test/functorch/test_vmap.py::TestRandomnessCUDA::test_multinomial_use_generator_True_randomness_different_batched_call_False_batched_input_first_cuda, test/functorch/test_vmap.py::TestRandomnessCUDA::test_multinomial_use_generator_True_randomness_different_batched_call_False_batched_input_last_cuda, test/functorch/test_vmap.py::TestRandomnessCUDA::test_multinomial_use_generator_True_randomness_different_batched_call_False_batched_input_none_cuda, test/functorch/test_vmap.py::TestRandomnessCUDA::test_multinomial_use_generator_True_randomness_different_batched_call_True_batched_input_first_cuda, test/functorch/test_vmap.py::TestRandomnessCUDA::test_multinomial_use_generator_True_randomness_different_batched_call_True_batched_input_last_cuda, test/functorch/test_vmap.py::TestRandomnessCUDA::test_multinomial_use_generator_True_randomness_different_batched_call_True_batched_input_none_cuda, test/functorch/test_vmap.py::TestRandomnessCUDA::test_multinomial_use_generator_True_randomness_error_batched_call_False_batched_input_first_cuda, test/functorch/test_vmap.py::TestRandomnessCUDA::test_multinomial_use_generator_True_randomness_error_batched_call_False_batched_input_last_cuda, test/functorch/test_vmap.py::TestRandomnessCUDA::test_multinomial_use_generator_True_randomness_error_batched_call_False_batched_input_none_cuda, test/functorch/test_vmap.py::TestRandomnessCUDA::test_multinomial_use_generator_True_randomness_error_batched_call_True_batched_input_first_cuda, test/functorch/test_vmap.py::TestRandomnessCUDA::test_multinomial_use_generator_True_randomness_error_batched_call_True_batched_input_last_cuda, test/functorch/test_vmap.py::TestRandomnessCUDA::test_multinomial_use_generator_True_randomness_error_batched_call_True_batched_input_none_cuda, test/functorch/test_vmap.py::TestRandomnessCUDA::test_multinomial_use_generator_True_randomness_same_batched_call_False_batched_input_first_cuda, test/functorch/test_vmap.py::TestRandomnessCUDA::test_multinomial_use_generator_True_randomness_same_batched_call_False_batched_input_last_cuda, test/functorch/test_vmap.py::TestRandomnessCUDA::test_multinomial_use_generator_True_randomness_same_batched_call_False_batched_input_none_cuda, test/functorch/test_vmap.py::TestRandomnessCUDA::test_multinomial_use_generator_True_randomness_same_batched_call_True_batched_input_first_cuda, test/functorch/test_vmap.py::TestRandomnessCUDA::test_multinomial_use_generator_True_randomness_same_batched_call_True_batched_input_last_cuda, test/functorch/test_vmap.py::TestRandomnessCUDA::test_multinomial_use_generator_True_randomness_same_batched_call_True_batched_input_none_cuda, test/functorch/test_vmap.py::TestRandomnessCUDA::test_random_binary_out_of_place_use_generator_False_randomness_different_batched_input_first_batched_other_first_cuda, test/functorch/test_vmap.py::TestRandomnessCUDA::test_random_binary_out_of_place_use_generator_False_randomness_different_batched_input_first_batched_other_last_cuda, test/functorch/test_vmap.py::TestRandomnessCUDA::test_random_binary_out_of_place_use_generator_False_randomness_different_batched_input_first_batched_other_none_cuda, test/functorch/test_vmap.py::TestRandomnessCUDA::test_random_binary_out_of_place_use_generator_False_randomness_different_batched_input_last_batched_other_first_cuda, test/functorch/test_vmap.py::TestRandomnessCUDA::test_random_binary_out_of_place_use_generator_False_randomness_different_batched_input_last_batched_other_last_cuda, test/functorch/test_vmap.py::TestRandomnessCUDA::test_random_binary_out_of_place_use_generator_False_randomness_different_batched_input_last_batched_other_none_cuda, test/functorch/test_vmap.py::TestRandomnessCUDA::test_random_binary_out_of_place_use_generator_False_randomness_different_batched_input_none_batched_other_first_cuda, test/functorch/test_vmap.py::TestRandomnessCUDA::test_random_binary_out_of_place_use_generator_False_randomness_different_batched_input_none_batched_other_last_cuda, test/functorch/test_vmap.py::TestRandomnessCUDA::test_random_binary_out_of_place_use_generator_False_randomness_different_batched_input_none_batched_other_none_cuda, test/functorch/test_vmap.py::TestRandomnessCUDA::test_random_binary_out_of_place_use_generator_False_randomness_error_batched_input_first_batched_other_first_cuda, test/functorch/test_vmap.py::TestRandomnessCUDA::test_random_binary_out_of_place_use_generator_False_randomness_error_batched_input_first_batched_other_last_cuda, test/functorch/test_vmap.py::TestRandomnessCUDA::test_random_binary_out_of_place_use_generator_False_randomness_error_batched_input_first_batched_other_none_cuda, test/functorch/test_vmap.py::TestRandomnessCUDA::test_random_binary_out_of_place_use_generator_False_randomness_error_batched_input_last_batched_other_first_cuda, test/functorch/test_vmap.py::TestRandomnessCUDA::test_random_binary_out_of_place_use_generator_False_randomness_error_batched_input_last_batched_other_last_cuda, test/functorch/test_vmap.py::TestRandomnessCUDA::test_random_binary_out_of_place_use_generator_False_randomness_error_batched_input_last_batched_other_none_cuda, test/functorch/test_vmap.py::TestRandomnessCUDA::test_random_binary_out_of_place_use_generator_False_randomness_error_batched_input_none_batched_other_first_cuda, test/functorch/test_vmap.py::TestRandomnessCUDA::test_random_binary_out_of_place_use_generator_False_randomness_error_batched_input_none_batched_other_last_cuda, test/functorch/test_vmap.py::TestRandomnessCUDA::test_random_binary_out_of_place_use_generator_False_randomness_error_batched_input_none_batched_other_none_cuda, test/functorch/test_vmap.py::TestRandomnessCUDA::test_random_binary_out_of_place_use_generator_False_randomness_same_batched_input_first_batched_other_first_cuda, test/functorch/test_vmap.py::TestRandomnessCUDA::test_random_binary_out_of_place_use_generator_False_randomness_same_batched_input_first_batched_other_last_cuda, test/functorch/test_vmap.py::TestRandomnessCUDA::test_random_binary_out_of_place_use_generator_False_randomness_same_batched_input_first_batched_other_none_cuda, test/functorch/test_vmap.py::TestRandomnessCUDA::test_random_binary_out_of_place_use_generator_False_randomness_same_batched_input_last_batched_other_first_cuda, test/functorch/test_vmap.py::TestRandomnessCUDA::test_random_binary_out_of_place_use_generator_False_randomness_same_batched_input_last_batched_other_last_cuda, test/functorch/test_vmap.py::TestRandomnessCUDA::test_random_binary_out_of_place_use_generator_False_randomness_same_batched_input_last_batched_other_none_cuda, test/functorch/test_vmap.py::TestRandomnessCUDA::test_random_binary_out_of_place_use_generator_False_randomness_same_batched_input_none_batched_other_first_cuda, test/functorch/test_vmap.py::TestRandomnessCUDA::test_random_binary_out_of_place_use_generator_False_randomness_same_batched_input_none_batched_other_last_cuda, test/functorch/test_vmap.py::TestRandomnessCUDA::test_random_binary_out_of_place_use_generator_False_randomness_same_batched_input_none_batched_other_none_cuda, test/functorch/test_vmap.py::TestRandomnessCUDA::test_random_binary_out_of_place_use_generator_True_randomness_different_batched_input_first_batched_other_first_cuda, test/functorch/test_vmap.py::TestRandomnessCUDA::test_random_binary_out_of_place_use_generator_True_randomness_different_batched_input_first_batched_other_last_cuda, test/functorch/test_vmap.py::TestRandomnessCUDA::test_random_binary_out_of_place_use_generator_True_randomness_different_batched_input_first_batched_other_none_cuda, test/functorch/test_vmap.py::TestRandomnessCUDA::test_random_binary_out_of_place_use_generator_True_randomness_different_batched_input_last_batched_other_first_cuda, test/functorch/test_vmap.py::TestRandomnessCUDA::test_random_binary_out_of_place_use_generator_True_randomness_different_batched_input_last_batched_other_last_cuda, test/functorch/test_vmap.py::TestRandomnessCUDA::test_random_binary_out_of_place_use_generator_True_randomness_different_batched_input_last_batched_other_none_cuda, test/functorch/test_vmap.py::TestRandomnessCUDA::test_random_binary_out_of_place_use_generator_True_randomness_different_batched_input_none_batched_other_first_cuda, test/functorch/test_vmap.py::TestRandomnessCUDA::test_random_binary_out_of_place_use_generator_True_randomness_different_batched_input_none_batched_other_last_cuda, test/functorch/test_vmap.py::TestRandomnessCUDA::test_random_binary_out_of_place_use_generator_True_randomness_different_batched_input_none_batched_other_none_cuda, test/functorch/test_vmap.py::TestRandomnessCUDA::test_random_binary_out_of_place_use_generator_True_randomness_error_batched_input_first_batched_other_first_cuda, test/functorch/test_vmap.py::TestRandomnessCUDA::test_random_binary_out_of_place_use_generator_True_randomness_error_batched_input_first_batched_other_last_cuda, test/functorch/test_vmap.py::TestRandomnessCUDA::test_random_binary_out_of_place_use_generator_True_randomness_error_batched_input_first_batched_other_none_cuda, test/functorch/test_vmap.py::TestRandomnessCUDA::test_random_binary_out_of_place_use_generator_True_randomness_error_batched_input_last_batched_other_first_cuda, test/functorch/test_vmap.py::TestRandomnessCUDA::test_random_binary_out_of_place_use_generator_True_randomness_error_batched_input_last_batched_other_last_cuda, test/functorch/test_vmap.py::TestRandomnessCUDA::test_random_binary_out_of_place_use_generator_True_randomness_error_batched_input_last_batched_other_none_cuda, test/functorch/test_vmap.py::TestRandomnessCUDA::test_random_binary_out_of_place_use_generator_True_randomness_error_batched_input_none_batched_other_first_cuda, test/functorch/test_vmap.py::TestRandomnessCUDA::test_random_binary_out_of_place_use_generator_True_randomness_error_batched_input_none_batched_other_last_cuda, test/functorch/test_vmap.py::TestRandomnessCUDA::test_random_binary_out_of_place_use_generator_True_randomness_error_batched_input_none_batched_other_none_cuda, test/functorch/test_vmap.py::TestRandomnessCUDA::test_random_binary_out_of_place_use_generator_True_randomness_same_batched_input_first_batched_other_first_cuda, test/functorch/test_vmap.py::TestRandomnessCUDA::test_random_binary_out_of_place_use_generator_True_randomness_same_batched_input_first_batched_other_last_cuda, test/functorch/test_vmap.py::TestRandomnessCUDA::test_random_binary_out_of_place_use_generator_True_randomness_same_batched_input_first_batched_other_none_cuda, test/functorch/test_vmap.py::TestRandomnessCUDA::test_random_binary_out_of_place_use_generator_True_randomness_same_batched_input_last_batched_other_first_cuda, test/functorch/test_vmap.py::TestRandomnessCUDA::test_random_binary_out_of_place_use_generator_True_randomness_same_batched_input_last_batched_other_last_cuda, test/functorch/test_vmap.py::TestRandomnessCUDA::test_random_binary_out_of_place_use_generator_True_randomness_same_batched_input_last_batched_other_none_cuda, test/functorch/test_vmap.py::TestRandomnessCUDA::test_random_binary_out_of_place_use_generator_True_randomness_same_batched_input_none_batched_other_first_cuda, test/functorch/test_vmap.py::TestRandomnessCUDA::test_random_binary_out_of_place_use_generator_True_randomness_same_batched_input_none_batched_other_last_cuda, test/functorch/test_vmap.py::TestRandomnessCUDA::test_random_binary_out_of_place_use_generator_True_randomness_same_batched_input_none_batched_other_none_cuda, test/functorch/test_vmap.py::TestRandomnessCUDA::test_random_unary_inplace_use_generator_False_randomness_different_batched_input_first_cuda, test/functorch/test_vmap.py::TestRandomnessCUDA::test_random_unary_inplace_use_generator_False_randomness_different_batched_input_last_cuda, test/functorch/test_vmap.py::TestRandomnessCUDA::test_random_unary_inplace_use_generator_False_randomness_different_batched_input_none_cuda, test/functorch/test_vmap.py::TestRandomnessCUDA::test_random_unary_inplace_use_generator_False_randomness_error_batched_input_first_cuda, test/functorch/test_vmap.py::TestRandomnessCUDA::test_random_unary_inplace_use_generator_False_randomness_error_batched_input_last_cuda, test/functorch/test_vmap.py::TestRandomnessCUDA::test_random_unary_inplace_use_generator_False_randomness_error_batched_input_none_cuda, test/functorch/test_vmap.py::TestRandomnessCUDA::test_random_unary_inplace_use_generator_False_randomness_same_batched_input_first_cuda, test/functorch/test_vmap.py::TestRandomnessCUDA::test_random_unary_inplace_use_generator_False_randomness_same_batched_input_last_cuda, test/functorch/test_vmap.py::TestRandomnessCUDA::test_random_unary_inplace_use_generator_False_randomness_same_batched_input_none_cuda, test/functorch/test_vmap.py::TestRandomnessCUDA::test_random_unary_inplace_use_generator_True_randomness_different_batched_input_first_cuda, test/functorch/test_vmap.py::TestRandomnessCUDA::test_random_unary_inplace_use_generator_True_randomness_different_batched_input_last_cuda, test/functorch/test_vmap.py::TestRandomnessCUDA::test_random_unary_inplace_use_generator_True_randomness_different_batched_input_none_cuda, test/functorch/test_vmap.py::TestRandomnessCUDA::test_random_unary_inplace_use_generator_True_randomness_error_batched_input_first_cuda, test/functorch/test_vmap.py::TestRandomnessCUDA::test_random_unary_inplace_use_generator_True_randomness_error_batched_input_last_cuda, test/functorch/test_vmap.py::TestRandomnessCUDA::test_random_unary_inplace_use_generator_True_randomness_error_batched_input_none_cuda, test/functorch/test_vmap.py::TestRandomnessCUDA::test_random_unary_inplace_use_generator_True_randomness_same_batched_input_first_cuda, test/functorch/test_vmap.py::TestRandomnessCUDA::test_random_unary_inplace_use_generator_True_randomness_same_batched_input_last_cuda, test/functorch/test_vmap.py::TestRandomnessCUDA::test_random_unary_inplace_use_generator_True_randomness_same_batched_input_none_cuda, test/functorch/test_vmap.py::TestRandomnessCUDA::test_random_unary_out_of_place_use_generator_False_randomness_different_batched_input_first_cuda, test/functorch/test_vmap.py::TestRandomnessCUDA::test_random_unary_out_of_place_use_generator_False_randomness_different_batched_input_last_cuda, test/functorch/test_vmap.py::TestRandomnessCUDA::test_random_unary_out_of_place_use_generator_False_randomness_different_batched_input_none_cuda, test/functorch/test_vmap.py::TestRandomnessCUDA::test_random_unary_out_of_place_use_generator_False_randomness_error_batched_input_first_cuda, test/functorch/test_vmap.py::TestRandomnessCUDA::test_random_unary_out_of_place_use_generator_False_randomness_error_batched_input_last_cuda, test/functorch/test_vmap.py::TestRandomnessCUDA::test_random_unary_out_of_place_use_generator_False_randomness_error_batched_input_none_cuda, test/functorch/test_vmap.py::TestRandomnessCUDA::test_random_unary_out_of_place_use_generator_False_randomness_same_batched_input_first_cuda, test/functorch/test_vmap.py::TestRandomnessCUDA::test_random_unary_out_of_place_use_generator_False_randomness_same_batched_input_last_cuda, test/functorch/test_vmap.py::TestRandomnessCUDA::test_random_unary_out_of_place_use_generator_False_randomness_same_batched_input_none_cuda, test/functorch/test_vmap.py::TestRandomnessCUDA::test_random_unary_out_of_place_use_generator_True_randomness_different_batched_input_first_cuda, test/functorch/test_vmap.py::TestRandomnessCUDA::test_random_unary_out_of_place_use_generator_True_randomness_different_batched_input_last_cuda, test/functorch/test_vmap.py::TestRandomnessCUDA::test_random_unary_out_of_place_use_generator_True_randomness_different_batched_input_none_cuda, test/functorch/test_vmap.py::TestRandomnessCUDA::test_random_unary_out_of_place_use_generator_True_randomness_error_batched_input_first_cuda, test/functorch/test_vmap.py::TestRandomnessCUDA::test_random_unary_out_of_place_use_generator_True_randomness_error_batched_input_last_cuda, test/functorch/test_vmap.py::TestRandomnessCUDA::test_random_unary_out_of_place_use_generator_True_randomness_error_batched_input_none_cuda, test/functorch/test_vmap.py::TestRandomnessCUDA::test_random_unary_out_of_place_use_generator_True_randomness_same_batched_input_first_cuda, test/functorch/test_vmap.py::TestRandomnessCUDA::test_random_unary_out_of_place_use_generator_True_randomness_same_batched_input_last_cuda, test/functorch/test_vmap.py::TestRandomnessCUDA::test_random_unary_out_of_place_use_generator_True_randomness_same_batched_input_none_cuda, test/functorch/test_vmap.py::TestRandomnessCUDA::test_randperm_randomness_different_use_generator_False_cuda, test/functorch/test_vmap.py::TestRandomnessCUDA::test_randperm_randomness_different_use_generator_True_cuda, test/functorch/test_vmap.py::TestRandomnessCUDA::test_randperm_randomness_error_use_generator_False_cuda, test/functorch/test_vmap.py::TestRandomnessCUDA::test_randperm_randomness_error_use_generator_True_cuda, test/functorch/test_vmap.py::TestRandomnessCUDA::test_randperm_randomness_same_use_generator_False_cuda, test/functorch/test_vmap.py::TestRandomnessCUDA::test_randperm_randomness_same_use_generator_True_cuda, test/functorch/test_vmap.py::TestRandomnessCUDA::test_unsupported_random_cuda, test/functorch/test_vmap.py::TestRandomnessCUDA::test_vmap_chunksize_in_dim_0_out_dim_0_cuda, test/functorch/test_vmap.py::TestRandomnessCUDA::test_vmap_chunksize_in_dim_0_out_dim_1_cuda, test/functorch/test_vmap.py::TestRandomnessCUDA::test_vmap_chunksize_in_dim_0_out_dim_2_cuda, test/functorch/test_vmap.py::TestRandomnessCUDA::test_vmap_chunksize_in_dim_1_out_dim_0_cuda, test/functorch/test_vmap.py::TestRandomnessCUDA::test_vmap_chunksize_in_dim_1_out_dim_1_cuda, test/functorch/test_vmap.py::TestRandomnessCUDA::test_vmap_chunksize_in_dim_1_out_dim_2_cuda, test/functorch/test_vmap.py::TestRandomnessCUDA::test_vmap_chunksize_in_dim_2_out_dim_0_cuda, test/functorch/test_vmap.py::TestRandomnessCUDA::test_vmap_chunksize_in_dim_2_out_dim_1_cuda, test/functorch/test_vmap.py::TestRandomnessCUDA::test_vmap_chunksize_in_dim_2_out_dim_2_cuda, test/functorch/test_vmap.py::TestVmapDeviceTypeCUDA::test__is_all_true_cuda, test/functorch/test_vmap.py::TestVmapDeviceTypeCUDA::test__is_any_true_cuda, test/functorch/test_vmap.py::TestVmapDeviceTypeCUDA::test_check_tensor_cuda, test/functorch/test_vmap.py::TestVmapDeviceTypeCUDA::test_vmap_fallback_check, test/functorch/test_vmap.py::TestVmapDeviceTypeCUDA::test_vmap_fallback_check_ok, test/functorch/test_vmap.py::TestVmapNestedTensorCUDA::test_cat_batching_rule_cuda, test/functorch/test_vmap.py::TestVmapNestedTensorCUDA::test_fallback_binary_cuda, test/functorch/test_vmap.py::TestVmapNestedTensorCUDA::test_fallback_binary_nt_and_batched_dense_cuda, test/functorch/test_vmap.py::TestVmapNestedTensorCUDA::test_fallback_binary_nt_and_unbatched_dense_cuda, test/functorch/test_vmap.py::TestVmapNestedTensorCUDA::test_fallback_unary_cuda, test/functorch/test_vmap.py::TestVmapNestedTensorCUDA::test_fallback_with_nt_and_batched_dense_with_nonzero_bdim_raises_cuda, test/functorch/test_vmap.py::TestVmapNestedTensorCUDA::test_multilevel_vmap_raises_cuda, test/functorch/test_vmap.py::TestVmapNestedTensorCUDA::test_nt_acts_as_dense_in_vmap_cuda, test/functorch/test_vmap.py::TestVmapNestedTensorCUDA::test_nt_with_nonzero_in_dim_raises_cuda, test/functorch/test_vmap.py::TestVmapNestedTensorCUDA::test_nt_with_nonzero_out_dim_raises_cuda, test/functorch/test_vmap.py::TestVmapNestedTensorCUDA::test_shape_call_cuda, test/functorch/test_vmap.py::TestVmapNestedTensorCUDA::test_vmap_fallback_check, test/functorch/test_vmap.py::TestVmapNestedTensorCUDA::test_vmap_fallback_check_ok 2025-12-04T13:04:29.5815793Z 2025-12-04T13:04:29.5816026Z Finished functorch/test_vmap 1/1 ... [2025-12-04 13:04:29.450596][12317.392810468], took 6.79min 2025-12-04T13:04:29.5816742Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/functorch.test_vmap/functorch.test_vmap-e514136507c8340c.xml 2025-12-04T13:04:29.6579662Z Running inductor/test_smoke 1/1 ... [2025-12-04 13:04:29.657694][12317.5999115] 2025-12-04T13:04:29.6580130Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2025-12-04T13:04:29.6582846Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'inductor/test_smoke.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-12-04 13:04:29.658050] 2025-12-04T13:04:35.1574752Z 2025-12-04T13:04:35.1575598Z inductor/test_smoke 1/1 was successful, full logs can be found in artifacts with path test/test-reports/inductor.test_smoke_1.1_beac475249fa3092_.log 2025-12-04T13:04:35.1576206Z 2025-12-04T13:04:35.1576471Z Finished inductor/test_smoke 1/1 ... [2025-12-04 13:04:35.157248][12323.099463047], took 0.09min 2025-12-04T13:04:35.1840739Z Running dynamo/test_after_aot 1/1 ... [2025-12-04 13:04:35.183865][12323.126083988] 2025-12-04T13:04:35.1841166Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2025-12-04T13:04:35.1844278Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'dynamo/test_after_aot.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-12-04 13:04:35.184196] 2025-12-04T13:04:43.2122820Z 2025-12-04T13:04:43.2123670Z dynamo/test_after_aot 1/1 was successful, full logs can be found in artifacts with path test/test-reports/dynamo.test_after_aot_1.1_dfa0ff058fa49cd8_.log 2025-12-04T13:04:43.2125103Z Running 2 items in this shard: test/dynamo/test_after_aot.py::TestAfterAot::test_dump_tensor, test/dynamo/test_after_aot.py::TestAfterAot::test_save_graph_repro 2025-12-04T13:04:43.2125699Z 2025-12-04T13:04:43.2125959Z Finished dynamo/test_after_aot 1/1 ... [2025-12-04 13:04:43.212046][12331.154262584], took 0.13min 2025-12-04T13:04:43.2397701Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/dynamo.test_after_aot/dynamo.test_after_aot-392fab221b48829f.xml 2025-12-04T13:04:43.3115094Z Running inductor/test_snode_runtime 1/1 ... [2025-12-04 13:04:43.311272][12331.253490306] 2025-12-04T13:04:43.3115551Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2025-12-04T13:04:43.3118213Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'inductor/test_snode_runtime.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-12-04 13:04:43.311575] 2025-12-04T13:04:57.0496013Z 2025-12-04T13:04:57.0496924Z inductor/test_snode_runtime 1/1 was successful, full logs can be found in artifacts with path test/test-reports/inductor.test_snode_runtime_1.1_25b6d9448d0f8a56_.log 2025-12-04T13:04:57.0503701Z Running 22 items in this shard: test/inductor/test_snode_runtime.py::UnsupportedTests::test_no_cuda, test/inductor/test_snode_runtime.py::UnsupportedTests::test_no_op, test/inductor/test_snode_runtime.py::ComputeBoundedTests::test_addmm, test/inductor/test_snode_runtime.py::ComputeBoundedTests::test_bmm, test/inductor/test_snode_runtime.py::ComputeBoundedTests::test_conv1d, test/inductor/test_snode_runtime.py::ComputeBoundedTests::test_conv2d, test/inductor/test_snode_runtime.py::ComputeBoundedTests::test_conv2d_transpose, test/inductor/test_snode_runtime.py::ComputeBoundedTests::test_conv3d, test/inductor/test_snode_runtime.py::ComputeBoundedTests::test_mm, test/inductor/test_snode_runtime.py::MemoryBoundedTests::test_dynamic, test/inductor/test_snode_runtime.py::MemoryBoundedTests::test_horizontal_reduction_pointwise, test/inductor/test_snode_runtime.py::MemoryBoundedTests::test_pointwise, test/inductor/test_snode_runtime.py::MemoryBoundedTests::test_relu, test/inductor/test_snode_runtime.py::TestCommAnalysis::test_all_gather_into_tensor, test/inductor/test_snode_runtime.py::TestCommAnalysis::test_all_gather_into_tensor_coalesced, test/inductor/test_snode_runtime.py::TestCommAnalysis::test_all_reduce, test/inductor/test_snode_runtime.py::TestCommAnalysis::test_all_reduce_coalesced, test/inductor/test_snode_runtime.py::TestCommAnalysis::test_legacy_all_gather_into_tensor_coalesced, test/inductor/test_snode_runtime.py::TestCommAnalysis::test_legacy_all_reduce, test/inductor/test_snode_runtime.py::TestCommAnalysis::test_legacy_all_reduce_coalesced, test/inductor/test_snode_runtime.py::TestCommAnalysis::test_reduce_scatter_tensor, test/inductor/test_snode_runtime.py::TestCommAnalysis::test_reduce_scatter_tensor_coalesced 2025-12-04T13:04:57.0508898Z 2025-12-04T13:04:57.0509125Z Finished inductor/test_snode_runtime 1/1 ... [2025-12-04 13:04:57.049364][12344.991580421], took 0.23min 2025-12-04T13:04:57.0768057Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/inductor.test_snode_runtime/inductor.test_snode_runtime-a0450ad8c50e7ed1.xml 2025-12-04T13:04:57.1838515Z Running inductor/test_minifier 1/1 ... [2025-12-04 13:04:57.183608][12345.125826466] 2025-12-04T13:04:57.1838971Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2025-12-04T13:04:57.1841838Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'inductor/test_minifier.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-12-04 13:04:57.183922] 2025-12-04T13:06:18.3392828Z 2025-12-04T13:06:18.3393741Z inductor/test_minifier 1/1 was successful, full logs can be found in artifacts with path test/test-reports/inductor.test_minifier_1.1_e91d133d4d8ff89c_.log 2025-12-04T13:06:18.3398335Z Running 14 items in this shard: test/inductor/test_minifier.py::MinifierTests::test_accuracy_vs_strict_accuracy, test/inductor/test_minifier.py::MinifierTests::test_after_aot_cpu_accuracy_error, test/inductor/test_minifier.py::MinifierTests::test_after_aot_cpu_compile_error, test/inductor/test_minifier.py::MinifierTests::test_after_aot_gpu_accuracy_error, test/inductor/test_minifier.py::MinifierTests::test_after_aot_gpu_compile_error, test/inductor/test_minifier.py::MinifierTests::test_aoti_cpu_accuracy_error, test/inductor/test_minifier.py::MinifierTests::test_aoti_cpu_compile_error, test/inductor/test_minifier.py::MinifierTests::test_aoti_cpu_compile_error_unflatten, test/inductor/test_minifier.py::MinifierTests::test_aoti_gpu_accuracy_error, test/inductor/test_minifier.py::MinifierTests::test_aoti_gpu_compile_error, test/inductor/test_minifier.py::MinifierTests::test_aoti_gpu_compile_error_unflatten, test/inductor/test_minifier.py::MinifierTests::test_constant_in_graph, test/inductor/test_minifier.py::MinifierTests::test_offload_to_disk, test/inductor/test_minifier.py::MinifierTests::test_rmse_improves_over_atol 2025-12-04T13:06:18.3401643Z 2025-12-04T13:06:18.3401860Z Finished inductor/test_minifier 1/1 ... [2025-12-04 13:06:18.339026][12426.281243359], took 1.35min 2025-12-04T13:06:18.3663847Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/inductor.test_minifier/inductor.test_minifier-02a4962c79ea03ff.xml 2025-12-04T13:06:18.5183462Z Running inductor/test_compiled_autograd 1/2 ... [2025-12-04 13:06:18.518036][12426.460251555] 2025-12-04T13:06:18.5184218Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2025-12-04T13:06:18.5186189Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'inductor/test_compiled_autograd.py', '--shard-id=1', '--num-shards=2', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-12-04 13:06:18.518374] 2025-12-04T13:13:36.8257229Z 2025-12-04T13:13:36.8258258Z inductor/test_compiled_autograd 1/2 was successful, full logs can be found in artifacts with path test/test-reports/inductor.test_compiled_autograd_1.2_5de33680d1df73db_.log 2025-12-04T13:13:36.8473992Z Running 438 items in this shard: test/inductor/test_compiled_autograd.py::TestCompiledAutograd::test_accumulate_grad_polyfill_case_1_1, test/inductor/test_compiled_autograd.py::TestCompiledAutograd::test_accumulate_grad_polyfill_case_1_2, test/inductor/test_compiled_autograd.py::TestCompiledAutograd::test_accumulate_grad_polyfill_case_1_3, test/inductor/test_compiled_autograd.py::TestCompiledAutograd::test_accumulate_grad_polyfill_case_1_5_2, test/inductor/test_compiled_autograd.py::TestCompiledAutograd::test_accumulate_grad_polyfill_case_3_1, test/inductor/test_compiled_autograd.py::TestCompiledAutograd::test_accumulate_grad_polyfill_case_3_2, test/inductor/test_compiled_autograd.py::TestCompiledAutograd::test_anomaly_mode_already_nan, test/inductor/test_compiled_autograd.py::TestCompiledAutograd::test_anomaly_mode_backward, test/inductor/test_compiled_autograd.py::TestCompiledAutograd::test_anomaly_mode_grad, test/inductor/test_compiled_autograd.py::TestCompiledAutograd::test_autograd_cpp_node_basic_is_traceable_True, test/inductor/test_compiled_autograd.py::TestCompiledAutograd::test_autograd_cpp_node_data_dependent_is_traceable_True, test/inductor/test_compiled_autograd.py::TestCompiledAutograd::test_autograd_cpp_node_id_is_traceable_True, test/inductor/test_compiled_autograd.py::TestCompiledAutograd::test_autograd_cpp_node_non_traceable, test/inductor/test_compiled_autograd.py::TestCompiledAutograd::test_autograd_cpp_node_saved_dynamic_is_traceable_True, test/inductor/test_compiled_autograd.py::TestCompiledAutograd::test_autograd_cpp_node_saved_float_is_traceable_True, test/inductor/test_compiled_autograd.py::TestCompiledAutograd::test_autograd_cpp_node_saved_int_is_traceable_False, test/inductor/test_compiled_autograd.py::TestCompiledAutograd::test_autograd_cpp_node_saved_int_is_traceable_True, test/inductor/test_compiled_autograd.py::TestCompiledAutograd::test_backward_hook_relative_ordering_partial, test/inductor/test_compiled_autograd.py::TestCompiledAutograd::test_cache_hit, test/inductor/test_compiled_autograd.py::TestCompiledAutograd::test_checkpointing_sac, test/inductor/test_compiled_autograd.py::TestCompiledAutograd::test_checkpointing_simple_reentrant_False, test/inductor/test_compiled_autograd.py::TestCompiledAutograd::test_checkpointing_simple_reentrant_True, test/inductor/test_compiled_autograd.py::TestCompiledAutograd::test_compile_api_api_compile_backend_aot_eager, test/inductor/test_compiled_autograd.py::TestCompiledAutograd::test_compile_api_api_compile_backend_eager, test/inductor/test_compiled_autograd.py::TestCompiledAutograd::test_compile_api_api_compile_backend_inductor, test/inductor/test_compiled_autograd.py::TestCompiledAutograd::test_compile_api_api_optimize_backend_aot_eager, test/inductor/test_compiled_autograd.py::TestCompiledAutograd::test_compile_api_disable_api_compile_backend_eager, test/inductor/test_compiled_autograd.py::TestCompiledAutograd::test_compile_api_disable_api_compile_backend_inductor, test/inductor/test_compiled_autograd.py::TestCompiledAutograd::test_compiled_autograd_does_not_specialize_on_bw_symints, test/inductor/test_compiled_autograd.py::TestCompiledAutograd::test_cpu_offloading, test/inductor/test_compiled_autograd.py::TestCompiledAutograd::test_cudagraphs_cpu_graph, test/inductor/test_compiled_autograd.py::TestCompiledAutograd::test_cudagraphs_cpu_scalar_used_in_cpp_custom_op, test/inductor/test_compiled_autograd.py::TestCompiledAutograd::test_cudagraphs_cpu_scalar_used_in_python_custom_op, test/inductor/test_compiled_autograd.py::TestCompiledAutograd::test_cudagraphs_sdpa, test/inductor/test_compiled_autograd.py::TestCompiledAutograd::test_custom_fn_bw_graph_break, test/inductor/test_compiled_autograd.py::TestCompiledAutograd::test_custom_fn_compiled_fw_bw_graph_break, test/inductor/test_compiled_autograd.py::TestCompiledAutograd::test_custom_fn_dynamically_defined_class, test/inductor/test_compiled_autograd.py::TestCompiledAutograd::test_custom_fn_multiple_grads, test/inductor/test_compiled_autograd.py::TestCompiledAutograd::test_custom_fn_saved_attr, test/inductor/test_compiled_autograd.py::TestCompiledAutograd::test_custom_fn_saved_multiple_tensors, test/inductor/test_compiled_autograd.py::TestCompiledAutograd::test_custom_fn_saved_tensors, test/inductor/test_compiled_autograd.py::TestCompiledAutograd::test_ddp_cpp_reducer_error, test/inductor/test_compiled_autograd.py::TestCompiledAutograd::test_ddp_python_reducer, test/inductor/test_compiled_autograd.py::TestCompiledAutograd::test_disk_offloading, test/inductor/test_compiled_autograd.py::TestCompiledAutograd::test_dynamic_shapes_annotations, test/inductor/test_compiled_autograd.py::TestCompiledAutograd::test_dynamic_shapes_eager_node, test/inductor/test_compiled_autograd.py::TestCompiledAutograd::test_dynamo_boxed, test/inductor/test_compiled_autograd.py::TestCompiledAutograd::test_flex_attention, test/inductor/test_compiled_autograd.py::TestCompiledAutograd::test_free_activation_memory_subclass, test/inductor/test_compiled_autograd.py::TestCompiledAutograd::test_higher_order_gradients, test/inductor/test_compiled_autograd.py::TestCompiledAutograd::test_hipify_not_loaded_with_import_cpp_extension, test/inductor/test_compiled_autograd.py::TestCompiledAutograd::test_hipify_not_loaded_with_import_torch, test/inductor/test_compiled_autograd.py::TestCompiledAutograd::test_inplace_grad_update, test/inductor/test_compiled_autograd.py::TestCompiledAutograd::test_inputs_aliasing_bytecode_stack_restore, test/inductor/test_compiled_autograd.py::TestCompiledAutograd::test_issue106555, test/inductor/test_compiled_autograd.py::TestCompiledAutograd::test_keep_graph_usage_after_compiled, test/inductor/test_compiled_autograd.py::TestCompiledAutograd::test_logging_tensor_flaky, test/inductor/test_compiled_autograd.py::TestCompiledAutograd::test_optimize_assert_backend_aot_eager, test/inductor/test_compiled_autograd.py::TestCompiledAutograd::test_optimize_assert_backend_eager, test/inductor/test_compiled_autograd.py::TestCompiledAutograd::test_optimize_assert_backend_inductor, test/inductor/test_compiled_autograd.py::TestCompiledAutograd::test_output_nodes_all_leaves, test/inductor/test_compiled_autograd.py::TestCompiledAutograd::test_reorder_multi_pre_hooks, test/inductor/test_compiled_autograd.py::TestCompiledAutograd::test_reorder_multi_tensor_pre_hooks, test/inductor/test_compiled_autograd.py::TestCompiledAutograd::test_reset, test/inductor/test_compiled_autograd.py::TestCompiledAutograd::test_saved_tensor_unpack_hook_ordering, test/inductor/test_compiled_autograd.py::TestCompiledAutograd::test_tensor_grad_hook1, test/inductor/test_compiled_autograd.py::TestCompiledAutograd::test_tensor_grad_hook2, test/inductor/test_compiled_autograd.py::TestCompiledAutograd::test_torch_compile_only_backward_call, test/inductor/test_compiled_autograd.py::TestCompiledAutograd::test_torch_function_mode, test/inductor/test_compiled_autograd.py::TestCompiledAutograd::test_trace_run_with_rng_state, test/inductor/test_compiled_autograd.py::TestCompiledAutograd::test_verbose_logs_aot_dispatcher_nodes, test/inductor/test_compiled_autograd.py::TestCompiledAutograd::test_verbose_logs_aot_dispatcher_nodes_hop, test/inductor/test_compiled_autograd.py::TestCompiledAutograd::test_verbose_logs_cpp, test/inductor/test_compiled_autograd.py::TestCompiledAutograd::test_verbose_logs_dynamic_shapes, test/inductor/test_compiled_autograd.py::TestCompiledAutograd::test_verbose_logs_snapshot, test/inductor/test_compiled_autograd.py::TestAutogradWithCompiledAutograd::test_access_saved_tensor_twice_without_recomputation_works, test/inductor/test_compiled_autograd.py::TestAutogradWithCompiledAutograd::test_accumulate_grad, test/inductor/test_compiled_autograd.py::TestAutogradWithCompiledAutograd::test_accumulate_grad_posthooks_can_observe_tensor_prehook, test/inductor/test_compiled_autograd.py::TestAutogradWithCompiledAutograd::test_accumulate_grad_posthooks_should_not_execute, test/inductor/test_compiled_autograd.py::TestAutogradWithCompiledAutograd::test_accumulate_grad_with_zero_numel_grad, test/inductor/test_compiled_autograd.py::TestAutogradWithCompiledAutograd::test_anomaly_assign_parent_cleanup, test/inductor/test_compiled_autograd.py::TestAutogradWithCompiledAutograd::test_anomaly_detect_nan, test/inductor/test_compiled_autograd.py::TestAutogradWithCompiledAutograd::test_anomaly_mode_no_check_nan, test/inductor/test_compiled_autograd.py::TestAutogradWithCompiledAutograd::test_autograd_inplace_view_of_view, test/inductor/test_compiled_autograd.py::TestAutogradWithCompiledAutograd::test_autograd_inplace_views_creation_meta, test/inductor/test_compiled_autograd.py::TestAutogradWithCompiledAutograd::test_autograd_inplace_views_cross_dtype, test/inductor/test_compiled_autograd.py::TestAutogradWithCompiledAutograd::test_autograd_multiple_views_python, test/inductor/test_compiled_autograd.py::TestAutogradWithCompiledAutograd::test_autograd_simple_views_python, test/inductor/test_compiled_autograd.py::TestAutogradWithCompiledAutograd::test_autograd_views_codegen, test/inductor/test_compiled_autograd.py::TestAutogradWithCompiledAutograd::test_backward_badcalls, test/inductor/test_compiled_autograd.py::TestAutogradWithCompiledAutograd::test_backward_copy, test/inductor/test_compiled_autograd.py::TestAutogradWithCompiledAutograd::test_backward_create_graph_warns, test/inductor/test_compiled_autograd.py::TestAutogradWithCompiledAutograd::test_backward_hook_relative_ordering, test/inductor/test_compiled_autograd.py::TestAutogradWithCompiledAutograd::test_backward_no_grad, test/inductor/test_compiled_autograd.py::TestAutogradWithCompiledAutograd::test_backward_twice_retained_graph_with_saved_values, test/inductor/test_compiled_autograd.py::TestAutogradWithCompiledAutograd::test_backward_twice_with_saved_values, test/inductor/test_compiled_autograd.py::TestAutogradWithCompiledAutograd::test_backward_with_inputs, test/inductor/test_compiled_autograd.py::TestAutogradWithCompiledAutograd::test_calculate_shape_util, test/inductor/test_compiled_autograd.py::TestAutogradWithCompiledAutograd::test_callback_adds_callback, test/inductor/test_compiled_autograd.py::TestAutogradWithCompiledAutograd::test_cant_create_saved_tensors, test/inductor/test_compiled_autograd.py::TestAutogradWithCompiledAutograd::test_checkpoint_detects_non_determinism, test/inductor/test_compiled_autograd.py::TestAutogradWithCompiledAutograd::test_checkpoint_graph_execution_group, test/inductor/test_compiled_autograd.py::TestAutogradWithCompiledAutograd::test_checkpoint_valid_reset_on_error, test/inductor/test_compiled_autograd.py::TestAutogradWithCompiledAutograd::test_checkpointing, test/inductor/test_compiled_autograd.py::TestAutogradWithCompiledAutograd::test_checkpointing_without_reentrant_correct_grad, test/inductor/test_compiled_autograd.py::TestAutogradWithCompiledAutograd::test_checkpointing_without_reentrant_custom_function_works, test/inductor/test_compiled_autograd.py::TestAutogradWithCompiledAutograd::test_checkpointing_without_reentrant_dataparallel, test/inductor/test_compiled_autograd.py::TestAutogradWithCompiledAutograd::test_checkpointing_without_reentrant_detached_tensor_use_reentrant_False, test/inductor/test_compiled_autograd.py::TestAutogradWithCompiledAutograd::test_checkpointing_without_reentrant_input_requires_grad_False, test/inductor/test_compiled_autograd.py::TestAutogradWithCompiledAutograd::test_checkpointing_without_reentrant_input_requires_grad_True, test/inductor/test_compiled_autograd.py::TestAutogradWithCompiledAutograd::test_checkpointing_without_reentrant_memory_savings, test/inductor/test_compiled_autograd.py::TestAutogradWithCompiledAutograd::test_create_graph_and_full_backward_hook_cycle, test/inductor/test_compiled_autograd.py::TestAutogradWithCompiledAutograd::test_current_graph_task_execution_order, test/inductor/test_compiled_autograd.py::TestAutogradWithCompiledAutograd::test_custom_autograd_ac_early_stop, test/inductor/test_compiled_autograd.py::TestAutogradWithCompiledAutograd::test_custom_autograd_no_early_free, test/inductor/test_compiled_autograd.py::TestAutogradWithCompiledAutograd::test_custom_autograd_repeated_grad_grad, test/inductor/test_compiled_autograd.py::TestAutogradWithCompiledAutograd::test_custom_function_cycle, test/inductor/test_compiled_autograd.py::TestAutogradWithCompiledAutograd::test_custom_function_error, test/inductor/test_compiled_autograd.py::TestAutogradWithCompiledAutograd::test_custom_function_exception, test/inductor/test_compiled_autograd.py::TestAutogradWithCompiledAutograd::test_custom_function_forward_mode_non_differentiable, test/inductor/test_compiled_autograd.py::TestAutogradWithCompiledAutograd::test_custom_function_forward_mode_non_tensor_before_tensor_args, test/inductor/test_compiled_autograd.py::TestAutogradWithCompiledAutograd::test_custom_function_forward_mode_wrong_formula, test/inductor/test_compiled_autograd.py::TestAutogradWithCompiledAutograd::test_custom_function_mark_dirty_not_differentiable, test/inductor/test_compiled_autograd.py::TestAutogradWithCompiledAutograd::test_custom_function_preserve_torch_function_when_return_as_is, test/inductor/test_compiled_autograd.py::TestAutogradWithCompiledAutograd::test_custom_function_saved_tensors, test/inductor/test_compiled_autograd.py::TestAutogradWithCompiledAutograd::test_custom_function_saving_mutated_view_no_leak, test/inductor/test_compiled_autograd.py::TestAutogradWithCompiledAutograd::test_custom_function_setup_context_simple, test/inductor/test_compiled_autograd.py::TestAutogradWithCompiledAutograd::test_custom_function_vmap_defaults, test/inductor/test_compiled_autograd.py::TestAutogradWithCompiledAutograd::test_deep_reentrant, test/inductor/test_compiled_autograd.py::TestAutogradWithCompiledAutograd::test_dep_nograd, test/inductor/test_compiled_autograd.py::TestAutogradWithCompiledAutograd::test_dependent_backward, test/inductor/test_compiled_autograd.py::TestAutogradWithCompiledAutograd::test_detach_base, test/inductor/test_compiled_autograd.py::TestAutogradWithCompiledAutograd::test_detach_then_inplace_raises_in_autograd, test/inductor/test_compiled_autograd.py::TestAutogradWithCompiledAutograd::test_disabling_saved_tensor_hooks, test/inductor/test_compiled_autograd.py::TestAutogradWithCompiledAutograd::test_disabling_saved_tensor_hooks_nested, test/inductor/test_compiled_autograd.py::TestAutogradWithCompiledAutograd::test_duplicate_backward_root, test/inductor/test_compiled_autograd.py::TestAutogradWithCompiledAutograd::test_enable_grad_decorator_no_paren, test/inductor/test_compiled_autograd.py::TestAutogradWithCompiledAutograd::test_first_grad_fn_access_in_no_grad_mode, test/inductor/test_compiled_autograd.py::TestAutogradWithCompiledAutograd::test_free_deep_graph_complicated, test/inductor/test_compiled_autograd.py::TestAutogradWithCompiledAutograd::test_free_deep_graph_pyfunction, test/inductor/test_compiled_autograd.py::TestAutogradWithCompiledAutograd::test_function, test/inductor/test_compiled_autograd.py::TestAutogradWithCompiledAutograd::test_get_data_and_hooks_from_raw_saved_variable, test/inductor/test_compiled_autograd.py::TestAutogradWithCompiledAutograd::test_grad, test/inductor/test_compiled_autograd.py::TestAutogradWithCompiledAutograd::test_grad_batched_grad, test/inductor/test_compiled_autograd.py::TestAutogradWithCompiledAutograd::test_grad_empty_inputs, test/inductor/test_compiled_autograd.py::TestAutogradWithCompiledAutograd::test_grad_fn_badcalls, test/inductor/test_compiled_autograd.py::TestAutogradWithCompiledAutograd::test_grad_fn_input_metadata, test/inductor/test_compiled_autograd.py::TestAutogradWithCompiledAutograd::test_grad_fn_prehooks, test/inductor/test_compiled_autograd.py::TestAutogradWithCompiledAutograd::test_grad_fn_prehooks_multiple_outputs, test/inductor/test_compiled_autograd.py::TestAutogradWithCompiledAutograd::test_grad_nonleaf, test/inductor/test_compiled_autograd.py::TestAutogradWithCompiledAutograd::test_grad_nonleaf_register_hook, test/inductor/test_compiled_autograd.py::TestAutogradWithCompiledAutograd::test_grad_thread_safety, test/inductor/test_compiled_autograd.py::TestAutogradWithCompiledAutograd::test_grad_to_node, test/inductor/test_compiled_autograd.py::TestAutogradWithCompiledAutograd::test_grad_to_node_inplace, test/inductor/test_compiled_autograd.py::TestAutogradWithCompiledAutograd::test_grad_to_node_materialize, test/inductor/test_compiled_autograd.py::TestAutogradWithCompiledAutograd::test_grad_unreachable_discovery, test/inductor/test_compiled_autograd.py::TestAutogradWithCompiledAutograd::test_gradcheck_check_batched_grad, test/inductor/test_compiled_autograd.py::TestAutogradWithCompiledAutograd::test_gradcheck_check_forward_or_backward_only, test/inductor/test_compiled_autograd.py::TestAutogradWithCompiledAutograd::test_gradcheck_complex_non_complex_outputs, test/inductor/test_compiled_autograd.py::TestAutogradWithCompiledAutograd::test_gradcheck_custom_error, test/inductor/test_compiled_autograd.py::TestAutogradWithCompiledAutograd::test_gradcheck_dense_and_sparse_inputs, test/inductor/test_compiled_autograd.py::TestAutogradWithCompiledAutograd::test_gradcheck_forward_ad, test/inductor/test_compiled_autograd.py::TestAutogradWithCompiledAutograd::test_gradcheck_forward_ad_respects_requires_grad, test/inductor/test_compiled_autograd.py::TestAutogradWithCompiledAutograd::test_gradcheck_forward_ad_runs_with_no_requires_grad, test/inductor/test_compiled_autograd.py::TestAutogradWithCompiledAutograd::test_gradcheck_input_layout2, test/inductor/test_compiled_autograd.py::TestAutogradWithCompiledAutograd::test_gradcheck_input_layout4, test/inductor/test_compiled_autograd.py::TestAutogradWithCompiledAutograd::test_gradcheck_output_shape_or_dtype_depend_on_values, test/inductor/test_compiled_autograd.py::TestAutogradWithCompiledAutograd::test_gradcheck_test_outputs, test/inductor/test_compiled_autograd.py::TestAutogradWithCompiledAutograd::test_gradcheck_validates_inputs, test/inductor/test_compiled_autograd.py::TestAutogradWithCompiledAutograd::test_graph_save_on_cpu, test/inductor/test_compiled_autograd.py::TestAutogradWithCompiledAutograd::test_hook_edge_case_when_called_with_grad, test/inductor/test_compiled_autograd.py::TestAutogradWithCompiledAutograd::test_hook_none, test/inductor/test_compiled_autograd.py::TestAutogradWithCompiledAutograd::test_hooks_cpp, test/inductor/test_compiled_autograd.py::TestAutogradWithCompiledAutograd::test_indexing, test/inductor/test_compiled_autograd.py::TestAutogradWithCompiledAutograd::test_inplace_not_requires_grad, test/inductor/test_compiled_autograd.py::TestAutogradWithCompiledAutograd::test_inplace_on_view_backward, test/inductor/test_compiled_autograd.py::TestAutogradWithCompiledAutograd::test_inplace_on_view_leaf_errors, test/inductor/test_compiled_autograd.py::TestAutogradWithCompiledAutograd::test_inplace_on_view_weak_grad_fn, test/inductor/test_compiled_autograd.py::TestAutogradWithCompiledAutograd::test_integer_outputs, test/inductor/test_compiled_autograd.py::TestAutogradWithCompiledAutograd::test_legacy_function_deprecation_exception, test/inductor/test_compiled_autograd.py::TestAutogradWithCompiledAutograd::test_lobpcg, test/inductor/test_compiled_autograd.py::TestAutogradWithCompiledAutograd::test_mark_non_differentiable, test/inductor/test_compiled_autograd.py::TestAutogradWithCompiledAutograd::test_materialize_grads, test/inductor/test_compiled_autograd.py::TestAutogradWithCompiledAutograd::test_multi_backward, test/inductor/test_compiled_autograd.py::TestAutogradWithCompiledAutograd::test_multi_backward_no_grad, test/inductor/test_compiled_autograd.py::TestAutogradWithCompiledAutograd::test_named_tensor_for_complex_views, test/inductor/test_compiled_autograd.py::TestAutogradWithCompiledAutograd::test_naughty_anomaly_access, test/inductor/test_compiled_autograd.py::TestAutogradWithCompiledAutograd::test_naughty_autograd_function_stashing_ctx, test/inductor/test_compiled_autograd.py::TestAutogradWithCompiledAutograd::test_nested_anomaly_printstack_cleanup, test/inductor/test_compiled_autograd.py::TestAutogradWithCompiledAutograd::test_next_functions, test/inductor/test_compiled_autograd.py::TestAutogradWithCompiledAutograd::test_no_grad_python_function, test/inductor/test_compiled_autograd.py::TestAutogradWithCompiledAutograd::test_no_requires_grad_inplace, test/inductor/test_compiled_autograd.py::TestAutogradWithCompiledAutograd::test_no_unnecessary_save, test/inductor/test_compiled_autograd.py::TestAutogradWithCompiledAutograd::test_not_implemented_fwad, test/inductor/test_compiled_autograd.py::TestAutogradWithCompiledAutograd::test_pickle, test/inductor/test_compiled_autograd.py::TestAutogradWithCompiledAutograd::test_post_accumulate_grad_hook_gets_cleaned_up, test/inductor/test_compiled_autograd.py::TestAutogradWithCompiledAutograd::test_post_accumulate_grad_hook_returns_not_None, test/inductor/test_compiled_autograd.py::TestAutogradWithCompiledAutograd::test_pow_zero_tensor_gradient, test/inductor/test_compiled_autograd.py::TestAutogradWithCompiledAutograd::test_power_function, test/inductor/test_compiled_autograd.py::TestAutogradWithCompiledAutograd::test_prehook_ordering, test/inductor/test_compiled_autograd.py::TestAutogradWithCompiledAutograd::test_profiler, test/inductor/test_compiled_autograd.py::TestAutogradWithCompiledAutograd::test_profiler_aggregation_table, test/inductor/test_compiled_autograd.py::TestAutogradWithCompiledAutograd::test_profiler_function_event_avg, test/inductor/test_compiled_autograd.py::TestAutogradWithCompiledAutograd::test_profiler_seq_nr, test/inductor/test_compiled_autograd.py::TestAutogradWithCompiledAutograd::test_profiler_shapes, test/inductor/test_compiled_autograd.py::TestAutogradWithCompiledAutograd::test_record_function, test/inductor/test_compiled_autograd.py::TestAutogradWithCompiledAutograd::test_reentrant_child_error, test/inductor/test_compiled_autograd.py::TestAutogradWithCompiledAutograd::test_reentrant_with_callbacks_depth_0, test/inductor/test_compiled_autograd.py::TestAutogradWithCompiledAutograd::test_reentrant_with_leaf_variable_hook, test/inductor/test_compiled_autograd.py::TestAutogradWithCompiledAutograd::test_requires_grad_, test/inductor/test_compiled_autograd.py::TestAutogradWithCompiledAutograd::test_retain_grad, test/inductor/test_compiled_autograd.py::TestAutogradWithCompiledAutograd::test_retain_grad_cycle, test/inductor/test_compiled_autograd.py::TestAutogradWithCompiledAutograd::test_retains_grad_inplace_multiple_outputs, test/inductor/test_compiled_autograd.py::TestAutogradWithCompiledAutograd::test_return_duplicate, test/inductor/test_compiled_autograd.py::TestAutogradWithCompiledAutograd::test_return_duplicate_inplace, test/inductor/test_compiled_autograd.py::TestAutogradWithCompiledAutograd::test_return_leaf, test/inductor/test_compiled_autograd.py::TestAutogradWithCompiledAutograd::test_save_none_for_backward, test/inductor/test_compiled_autograd.py::TestAutogradWithCompiledAutograd::test_save_on_cpu_and_checkpoint, test/inductor/test_compiled_autograd.py::TestAutogradWithCompiledAutograd::test_save_output_nr, test/inductor/test_compiled_autograd.py::TestAutogradWithCompiledAutograd::test_saved_tensor_hooks_custom_function_intermediates, test/inductor/test_compiled_autograd.py::TestAutogradWithCompiledAutograd::test_saved_tensor_hooks_extra_enter_during_bw_no_leak, test/inductor/test_compiled_autograd.py::TestAutogradWithCompiledAutograd::test_saved_variable_packing_unpacking_did_not_save_original_with_hooks, test/inductor/test_compiled_autograd.py::TestAutogradWithCompiledAutograd::test_saved_variable_packing_unpacking_saved_original_with_default_hooks, test/inductor/test_compiled_autograd.py::TestAutogradWithCompiledAutograd::test_saved_variable_version_counter, test/inductor/test_compiled_autograd.py::TestAutogradWithCompiledAutograd::test_scalar_grad_mixed_device, test/inductor/test_compiled_autograd.py::TestAutogradWithCompiledAutograd::test_select_expanded_v, test/inductor/test_compiled_autograd.py::TestAutogradWithCompiledAutograd::test_set_data_tensorimpl_type, test/inductor/test_compiled_autograd.py::TestAutogradWithCompiledAutograd::test_set_grad_coroutines_benign_exceptions, test/inductor/test_compiled_autograd.py::TestAutogradWithCompiledAutograd::test_set_grad_enabled_wraps, test/inductor/test_compiled_autograd.py::TestAutogradWithCompiledAutograd::test_set_grad_generator_functions, test/inductor/test_compiled_autograd.py::TestAutogradWithCompiledAutograd::test_set_materialize_non_diff_grads, test/inductor/test_compiled_autograd.py::TestAutogradWithCompiledAutograd::test_shape, test/inductor/test_compiled_autograd.py::TestAutogradWithCompiledAutograd::test_sharded_grad, test/inductor/test_compiled_autograd.py::TestAutogradWithCompiledAutograd::test_sparse_gather_both_scalar, test/inductor/test_compiled_autograd.py::TestAutogradWithCompiledAutograd::test_sparse_gather_dim_neg, test/inductor/test_compiled_autograd.py::TestAutogradWithCompiledAutograd::test_sparse_gather_ind_scalar, test/inductor/test_compiled_autograd.py::TestAutogradWithCompiledAutograd::test_tensor_grad_warnings, test/inductor/test_compiled_autograd.py::TestAutogradWithCompiledAutograd::test_tensor_hooks_inplace_multiple_outputs, test/inductor/test_compiled_autograd.py::TestAutogradWithCompiledAutograd::test_thread_shutdown, test/inductor/test_compiled_autograd.py::TestAutogradWithCompiledAutograd::test_too_many_grads, test/inductor/test_compiled_autograd.py::TestAutogradWithCompiledAutograd::test_unrelated_inputs, test/inductor/test_compiled_autograd.py::TestAutogradWithCompiledAutograd::test_unused_output, test/inductor/test_compiled_autograd.py::TestAutogradWithCompiledAutograd::test_var_mean_differentiable, test/inductor/test_compiled_autograd.py::TestAutogradWithCompiledAutograd::test_version_counter, test/inductor/test_compiled_autograd.py::TestAutogradWithCompiledAutograd::test_view_func_replay_with_modified_state, test/inductor/test_compiled_autograd.py::TestAutogradWithCompiledAutograd::test_volatile_deprecated, test/inductor/test_compiled_autograd.py::TestAutogradWithCompiledAutograd::test_will_engine_execute_node, test/inductor/test_compiled_autograd.py::TestNestedCheckpointWithCompiledAutograd::test_nested_checkpoint_early_stop_False, test/inductor/test_compiled_autograd.py::TestNestedCheckpointWithCompiledAutograd::test_nested_checkpoint_early_stop_True, test/inductor/test_compiled_autograd.py::TestNestedCheckpointWithCompiledAutograd::test_nested_checkpoint_kwargs_early_stop_True, test/inductor/test_compiled_autograd.py::TestNestedCheckpointWithCompiledAutograd::test_nested_checkpoint_non_tensor_inputs_and_outputs_early_stop_True, test/inductor/test_compiled_autograd.py::TestNestedCheckpointWithCompiledAutograd::test_nested_checkpoint_reentrant_backwards_early_stop_False, test/inductor/test_compiled_autograd.py::TestNestedCheckpointWithCompiledAutograd::test_nested_checkpoint_reentrant_backwards_early_stop_True, test/inductor/test_compiled_autograd.py::TestNestedCheckpointWithCompiledAutograd::test_nested_checkpoint_same_graph_early_stop_True, test/inductor/test_compiled_autograd.py::TestNestedCheckpointWithCompiledAutograd::test_nested_checkpoint_two_children_early_stop_False, test/inductor/test_compiled_autograd.py::TestNestedCheckpointWithCompiledAutograd::test_nested_checkpoint_two_children_early_stop_True, test/inductor/test_compiled_autograd.py::TestCustomOpWithCompiledAutograd::test_abstract_impl_on_existing_op, test/inductor/test_compiled_autograd.py::TestCustomOpWithCompiledAutograd::test_abstract_impl_on_existing_op_with_CompositeExplicitAutograd, test/inductor/test_compiled_autograd.py::TestCustomOpWithCompiledAutograd::test_backward_dict_grad_for_nontensor, test/inductor/test_compiled_autograd.py::TestCustomOpWithCompiledAutograd::test_backward_impl_on_existing_op_incorrect_schema_mutable, test/inductor/test_compiled_autograd.py::TestCustomOpWithCompiledAutograd::test_backward_impl_on_existing_op_incorrect_schema_no_output, test/inductor/test_compiled_autograd.py::TestCustomOpWithCompiledAutograd::test_backward_impl_on_existing_op_with_key_key_AutogradCUDA, test/inductor/test_compiled_autograd.py::TestCustomOpWithCompiledAutograd::test_backward_output_differentiability_tensorlist, test/inductor/test_compiled_autograd.py::TestCustomOpWithCompiledAutograd::test_backward_tensorlist_input_requires_list_grads_with_same_numel, test/inductor/test_compiled_autograd.py::TestCustomOpWithCompiledAutograd::test_basic_make_fx, test/inductor/test_compiled_autograd.py::TestCustomOpWithCompiledAutograd::test_data_dependent_basic, test/inductor/test_compiled_autograd.py::TestCustomOpWithCompiledAutograd::test_data_dependent_nms_dynamic_compile, test/inductor/test_compiled_autograd.py::TestCustomOpWithCompiledAutograd::test_defined_in_python, test/inductor/test_compiled_autograd.py::TestCustomOpWithCompiledAutograd::test_duplicate_impl, test/inductor/test_compiled_autograd.py::TestCustomOpWithCompiledAutograd::test_impl_abstract_overload, test/inductor/test_compiled_autograd.py::TestCustomOpWithCompiledAutograd::test_impl_device_cpu, test/inductor/test_compiled_autograd.py::TestCustomOpWithCompiledAutograd::test_impl_invalid_devices, test/inductor/test_compiled_autograd.py::TestCustomOpWithCompiledAutograd::test_impl_multiple, test/inductor/test_compiled_autograd.py::TestCustomOpWithCompiledAutograd::test_impl_on_existing_op_with_cpu_registration_key_CPU, test/inductor/test_compiled_autograd.py::TestCustomOpWithCompiledAutograd::test_impl_on_existing_op_with_cpu_registration_key_CompositeImplicitAutograd, test/inductor/test_compiled_autograd.py::TestCustomOpWithCompiledAutograd::test_impl_separate, test/inductor/test_compiled_autograd.py::TestCustomOpWithCompiledAutograd::test_infer_schema_supported, test/inductor/test_compiled_autograd.py::TestCustomOpWithCompiledAutograd::test_infer_schema_unsupported, test/inductor/test_compiled_autograd.py::TestCustomOpWithCompiledAutograd::test_invalid_qualname, test/inductor/test_compiled_autograd.py::TestCustomOpWithCompiledAutograd::test_invalid_schemas, test/inductor/test_compiled_autograd.py::TestCustomOpWithCompiledAutograd::test_is_functional_schema, test/inductor/test_compiled_autograd.py::TestCustomOpWithCompiledAutograd::test_is_tensorlist_like_type, test/inductor/test_compiled_autograd.py::TestCustomOpWithCompiledAutograd::test_legacy_define, test/inductor/test_compiled_autograd.py::TestCustomOpWithCompiledAutograd::test_legacy_impl, test/inductor/test_compiled_autograd.py::TestCustomOpWithCompiledAutograd::test_meta_for_data_dependent_shape_operation, test/inductor/test_compiled_autograd.py::TestCustomOpWithCompiledAutograd::test_name_must_match, test/inductor/test_compiled_autograd.py::TestCustomOpWithCompiledAutograd::test_new_data_dependent_symint, test/inductor/test_compiled_autograd.py::TestCustomOpWithCompiledAutograd::test_override_impl, test/inductor/test_compiled_autograd.py::TestCustomOpWithCompiledAutograd::test_override_meta, test/inductor/test_compiled_autograd.py::TestCustomOpWithCompiledAutograd::test_private_ctor, test/inductor/test_compiled_autograd.py::TestCustomOpWithCompiledAutograd::test_supported_param_types, test/inductor/test_compiled_autograd.py::TestCustomOpWithCompiledAutograd::test_symints, test/inductor/test_compiled_autograd.py::TestCustomOpWithCompiledAutograd::test_unsupported_schemas, test/inductor/test_compiled_autograd.py::HigherOrderOpTestsWithCompiledAutograd::test_allow_python_side_effects_utility, test/inductor/test_compiled_autograd.py::HigherOrderOpTestsWithCompiledAutograd::test_capture_constants, test/inductor/test_compiled_autograd.py::HigherOrderOpTestsWithCompiledAutograd::test_capture_input_num, test/inductor/test_compiled_autograd.py::HigherOrderOpTestsWithCompiledAutograd::test_capture_numpy_number, test/inductor/test_compiled_autograd.py::HigherOrderOpTestsWithCompiledAutograd::test_capture_tracked, test/inductor/test_compiled_autograd.py::HigherOrderOpTestsWithCompiledAutograd::test_capture_untracked_global_nested, test/inductor/test_compiled_autograd.py::HigherOrderOpTestsWithCompiledAutograd::test_cond_branches_no_arguments, test/inductor/test_compiled_autograd.py::HigherOrderOpTestsWithCompiledAutograd::test_cond_free_variable_in_both_branches, test/inductor/test_compiled_autograd.py::HigherOrderOpTestsWithCompiledAutograd::test_cond_graph_break_in_one_branch, test/inductor/test_compiled_autograd.py::HigherOrderOpTestsWithCompiledAutograd::test_cond_pytree_operands, test/inductor/test_compiled_autograd.py::HigherOrderOpTestsWithCompiledAutograd::test_cond_side_effect_in_one_branches, test/inductor/test_compiled_autograd.py::HigherOrderOpTestsWithCompiledAutograd::test_cond_source_fn_stack, test/inductor/test_compiled_autograd.py::HigherOrderOpTestsWithCompiledAutograd::test_cond_with_constant_pred, test/inductor/test_compiled_autograd.py::HigherOrderOpTestsWithCompiledAutograd::test_fallback_on_graph_break_simple, test/inductor/test_compiled_autograd.py::HigherOrderOpTestsWithCompiledAutograd::test_freevars_as_inputs_to_wrap, test/inductor/test_compiled_autograd.py::HigherOrderOpTestsWithCompiledAutograd::test_grad_source_fn_stack, test/inductor/test_compiled_autograd.py::HigherOrderOpTestsWithCompiledAutograd::test_hints_wrapper_no_hints, test/inductor/test_compiled_autograd.py::HigherOrderOpTestsWithCompiledAutograd::test_hopify_generic_wrap, test/inductor/test_compiled_autograd.py::HigherOrderOpTestsWithCompiledAutograd::test_internal_nonlocal, test/inductor/test_compiled_autograd.py::HigherOrderOpTestsWithCompiledAutograd::test_lift_tensors_with_compound_expressions, test/inductor/test_compiled_autograd.py::HigherOrderOpTestsWithCompiledAutograd::test_map_kwargs, test/inductor/test_compiled_autograd.py::HigherOrderOpTestsWithCompiledAutograd::test_map_lowers_to_graph, test/inductor/test_compiled_autograd.py::HigherOrderOpTestsWithCompiledAutograd::test_map_multi_return, test/inductor/test_compiled_autograd.py::HigherOrderOpTestsWithCompiledAutograd::test_map_pytree_return, test/inductor/test_compiled_autograd.py::HigherOrderOpTestsWithCompiledAutograd::test_map_source_fn_stack, test/inductor/test_compiled_autograd.py::HigherOrderOpTestsWithCompiledAutograd::test_map_subgraph_name_is_valid, test/inductor/test_compiled_autograd.py::HigherOrderOpTestsWithCompiledAutograd::test_nested_tuple_output, test/inductor/test_compiled_autograd.py::HigherOrderOpTestsWithCompiledAutograd::test_nested_wrap, test/inductor/test_compiled_autograd.py::HigherOrderOpTestsWithCompiledAutograd::test_no_freevars, test/inductor/test_compiled_autograd.py::HigherOrderOpTestsWithCompiledAutograd::test_output_with_dict, test/inductor/test_compiled_autograd.py::HigherOrderOpTestsWithCompiledAutograd::test_register_subclass, test/inductor/test_compiled_autograd.py::HigherOrderOpTestsWithCompiledAutograd::test_return_captured_var, test/inductor/test_compiled_autograd.py::HigherOrderOpTestsWithCompiledAutograd::test_return_captured_var_used_multiple_times, test/inductor/test_compiled_autograd.py::HigherOrderOpTestsWithCompiledAutograd::test_return_captured_vars, test/inductor/test_compiled_autograd.py::HigherOrderOpTestsWithCompiledAutograd::test_side_effect_del_existing_attr_global_obj, test/inductor/test_compiled_autograd.py::HigherOrderOpTestsWithCompiledAutograd::test_side_effect_del_existing_attr_nonlocal_obj, test/inductor/test_compiled_autograd.py::HigherOrderOpTestsWithCompiledAutograd::test_side_effect_local_list_append_no_graph_break, test/inductor/test_compiled_autograd.py::HigherOrderOpTestsWithCompiledAutograd::test_side_effect_mutate_global_list, test/inductor/test_compiled_autograd.py::HigherOrderOpTestsWithCompiledAutograd::test_side_effect_mutate_global_num, test/inductor/test_compiled_autograd.py::HigherOrderOpTestsWithCompiledAutograd::test_side_effect_mutate_global_num_builtin, test/inductor/test_compiled_autograd.py::HigherOrderOpTestsWithCompiledAutograd::test_side_effect_mutate_global_tensor, test/inductor/test_compiled_autograd.py::HigherOrderOpTestsWithCompiledAutograd::test_side_effect_mutate_nonlocal_num, test/inductor/test_compiled_autograd.py::HigherOrderOpTestsWithCompiledAutograd::test_side_effect_mutate_nonlocal_num_builtin, test/inductor/test_compiled_autograd.py::HigherOrderOpTestsWithCompiledAutograd::test_side_effect_mutate_nonlocal_tensor_builtin, test/inductor/test_compiled_autograd.py::HigherOrderOpTestsWithCompiledAutograd::test_side_effect_nested_nonlocal_list_append_graph_break, test/inductor/test_compiled_autograd.py::HigherOrderOpTestsWithCompiledAutograd::test_side_effect_nonlocal_list_append_graph_break, test/inductor/test_compiled_autograd.py::HigherOrderOpTestsWithCompiledAutograd::test_side_effect_set_existing_attr_global_module, test/inductor/test_compiled_autograd.py::HigherOrderOpTestsWithCompiledAutograd::test_side_effect_set_existing_attr_global_obj, test/inductor/test_compiled_autograd.py::HigherOrderOpTestsWithCompiledAutograd::test_side_effect_set_existing_attr_nonlocal_module, test/inductor/test_compiled_autograd.py::HigherOrderOpTestsWithCompiledAutograd::test_side_effect_set_new_attr_global_module, test/inductor/test_compiled_autograd.py::HigherOrderOpTestsWithCompiledAutograd::test_symint_in_slice, test/inductor/test_compiled_autograd.py::HigherOrderOpTestsWithCompiledAutograd::test_unbacked_symbol_closure, test/inductor/test_compiled_autograd.py::HigherOrderOpTestsWithCompiledAutograd::test_vmap_multiply_scalar, test/inductor/test_compiled_autograd.py::HigherOrderOpTestsWithCompiledAutograd::test_vmap_source_fn_stack, test/inductor/test_compiled_autograd.py::HigherOrderOpTestsWithCompiledAutograd::test_wrap_allow_local_assign_in_body_fn, test/inductor/test_compiled_autograd.py::HigherOrderOpTestsWithCompiledAutograd::test_wrap_inductor_compiled_regions_option, test/inductor/test_compiled_autograd.py::HigherOrderOpTestsWithCompiledAutograd::test_wrap_kwarg, test/inductor/test_compiled_autograd.py::HigherOrderOpTestsWithCompiledAutograd::test_wrap_kwarg_default_else_branch, test/inductor/test_compiled_autograd.py::HigherOrderOpTestsWithCompiledAutograd::test_wrap_kwarg_only, test/inductor/test_compiled_autograd.py::HigherOrderOpTestsWithCompiledAutograd::test_wrap_kwarg_recompile, test/inductor/test_compiled_autograd.py::HigherOrderOpTestsWithCompiledAutograd::test_wrap_pytree_kwargs, test/inductor/test_compiled_autograd.py::HigherOrderOpTestsWithCompiledAutograd::test_wrap_source_fn_stack, test/inductor/test_compiled_autograd.py::FuncTorchHigherOrderOpTestsWithCompiledAutograd::test_functional_call_sequential_params_and_buffers, test/inductor/test_compiled_autograd.py::FuncTorchHigherOrderOpTestsWithCompiledAutograd::test_grad_call_compiled_backward_fn, test/inductor/test_compiled_autograd.py::FuncTorchHigherOrderOpTestsWithCompiledAutograd::test_grad_call_torch_compile_fn, test/inductor/test_compiled_autograd.py::FuncTorchHigherOrderOpTestsWithCompiledAutograd::test_grad_fn_with_kwargs, test/inductor/test_compiled_autograd.py::FuncTorchHigherOrderOpTestsWithCompiledAutograd::test_grad_freevar_python_scalar, test/inductor/test_compiled_autograd.py::FuncTorchHigherOrderOpTestsWithCompiledAutograd::test_grad_freevar_tensor, test/inductor/test_compiled_autograd.py::FuncTorchHigherOrderOpTestsWithCompiledAutograd::test_grad_has_aux, test/inductor/test_compiled_autograd.py::FuncTorchHigherOrderOpTestsWithCompiledAutograd::test_grad_pytree, test/inductor/test_compiled_autograd.py::FuncTorchHigherOrderOpTestsWithCompiledAutograd::test_grad_recompile, test/inductor/test_compiled_autograd.py::FuncTorchHigherOrderOpTestsWithCompiledAutograd::test_grad_with_graph_break, test/inductor/test_compiled_autograd.py::FuncTorchHigherOrderOpTestsWithCompiledAutograd::test_grad_with_side_effect, test/inductor/test_compiled_autograd.py::FuncTorchHigherOrderOpTestsWithCompiledAutograd::test_hessian, test/inductor/test_compiled_autograd.py::FuncTorchHigherOrderOpTestsWithCompiledAutograd::test_hessian_argnums, test/inductor/test_compiled_autograd.py::FuncTorchHigherOrderOpTestsWithCompiledAutograd::test_jacfwd, test/inductor/test_compiled_autograd.py::FuncTorchHigherOrderOpTestsWithCompiledAutograd::test_jacfwd_has_aux, test/inductor/test_compiled_autograd.py::FuncTorchHigherOrderOpTestsWithCompiledAutograd::test_jacrev_has_aux, test/inductor/test_compiled_autograd.py::FuncTorchHigherOrderOpTestsWithCompiledAutograd::test_jacrev_two_tensors_argnums, test/inductor/test_compiled_autograd.py::FuncTorchHigherOrderOpTestsWithCompiledAutograd::test_jvp_call_torch_compile_fn, test/inductor/test_compiled_autograd.py::FuncTorchHigherOrderOpTestsWithCompiledAutograd::test_jvp_freevar_tensor, test/inductor/test_compiled_autograd.py::FuncTorchHigherOrderOpTestsWithCompiledAutograd::test_jvp_has_aux, test/inductor/test_compiled_autograd.py::FuncTorchHigherOrderOpTestsWithCompiledAutograd::test_jvp_simple, test/inductor/test_compiled_autograd.py::FuncTorchHigherOrderOpTestsWithCompiledAutograd::test_jvp_two_tensors_has_aux, test/inductor/test_compiled_autograd.py::FuncTorchHigherOrderOpTestsWithCompiledAutograd::test_teardown_resets_nested_graph_breaks, test/inductor/test_compiled_autograd.py::FuncTorchHigherOrderOpTestsWithCompiledAutograd::test_vjp_call_compiled_backward_fn, test/inductor/test_compiled_autograd.py::FuncTorchHigherOrderOpTestsWithCompiledAutograd::test_vjp_multiple_outputs, test/inductor/test_compiled_autograd.py::FuncTorchHigherOrderOpTestsWithCompiledAutograd::test_vjp_multiple_outputs_python_struct, test/inductor/test_compiled_autograd.py::FuncTorchHigherOrderOpTestsWithCompiledAutograd::test_vmap_call_torch_compile_fn, test/inductor/test_compiled_autograd.py::FuncTorchHigherOrderOpTestsWithCompiledAutograd::test_vmap_free_const, test/inductor/test_compiled_autograd.py::FuncTorchHigherOrderOpTestsWithCompiledAutograd::test_vmap_multiple_invocation_in_dims, test/inductor/test_compiled_autograd.py::FuncTorchHigherOrderOpTestsWithCompiledAutograd::test_vmap_multiple_invocation_out_dims, test/inductor/test_compiled_autograd.py::FuncTorchHigherOrderOpTestsWithCompiledAutograd::test_vmap_multiple_outputs_diff_dims, test/inductor/test_compiled_autograd.py::FuncTorchHigherOrderOpTestsWithCompiledAutograd::test_vmap_over_vmap_captured, test/inductor/test_compiled_autograd.py::FuncTorchHigherOrderOpTestsWithCompiledAutograd::test_vmap_pytree_inputs, test/inductor/test_compiled_autograd.py::FuncTorchHigherOrderOpTestsWithCompiledAutograd::test_vmap_recompile, test/inductor/test_compiled_autograd.py::FuncTorchHigherOrderOpTestsWithCompiledAutograd::test_vmap_recompile_different_config, test/inductor/test_compiled_autograd.py::FuncTorchHigherOrderOpTestsWithCompiledAutograd::test_vmap_recompile_same_config, test/inductor/test_compiled_autograd.py::FuncTorchHigherOrderOpTestsWithCompiledAutograd::test_vmap_side_effects, test/inductor/test_compiled_autograd.py::FuncTorchHigherOrderOpTestsWithCompiledAutograd::test_vmap_side_effects_append_input, test/inductor/test_compiled_autograd.py::FuncTorchHigherOrderOpTestsWithCompiledAutograd::test_vmap_two_inputs, test/inductor/test_compiled_autograd.py::FuncTorchHigherOrderOpTestsWithCompiledAutograd::test_vmap_two_inputs_tuple_in_dims, test/inductor/test_compiled_autograd.py::FuncTorchHigherOrderOpTestsWithCompiledAutograd::test_vmap_with_conditional_graph_break, test/inductor/test_compiled_autograd.py::FuncTorchHigherOrderOpTestsWithCompiledAutograd::test_vmap_with_graph_break, test/inductor/test_compiled_autograd.py::ActivationCheckpointingTestsWithCompiledAutograd::test_cond_with_invalid_kwargs, test/inductor/test_compiled_autograd.py::ActivationCheckpointingTestsWithCompiledAutograd::test_dropout_inductor, test/inductor/test_compiled_autograd.py::ActivationCheckpointingTestsWithCompiledAutograd::test_flop_counter_for_cond, test/inductor/test_compiled_autograd.py::ActivationCheckpointingTestsWithCompiledAutograd::test_flop_counter_for_cond_unbalanced_branches, test/inductor/test_compiled_autograd.py::ActivationCheckpointingTestsWithCompiledAutograd::test_function, test/inductor/test_compiled_autograd.py::ActivationCheckpointingTestsWithCompiledAutograd::test_module, test/inductor/test_compiled_autograd.py::ActivationCheckpointingTestsWithCompiledAutograd::test_non_aliasing_util, test/inductor/test_compiled_autograd.py::TestDTensorCompileWithCompiledAutograd::test_device_mesh_compile, test/inductor/test_compiled_autograd.py::TestDTensorCompileWithCompiledAutograd::test_dtensor_basic_export, test/inductor/test_compiled_autograd.py::TestDTensorCompileWithCompiledAutograd::test_dtensor_constructor_w_dynamo_disable, test/inductor/test_compiled_autograd.py::TestDTensorCompileWithCompiledAutograd::test_dtensor_constructor_w_graph_break, test/inductor/test_compiled_autograd.py::TestDTensorCompileWithCompiledAutograd::test_dtensor_different_gradient_placement, test/inductor/test_compiled_autograd.py::TestDTensorCompileWithCompiledAutograd::test_dtensor_dont_recompile_on_same_placement_devicemesh, test/inductor/test_compiled_autograd.py::TestDTensorCompileWithCompiledAutograd::test_dtensor_dynamic, test/inductor/test_compiled_autograd.py::TestDTensorCompileWithCompiledAutograd::test_dtensor_dynamic_loss_parallel_log_softmax, test/inductor/test_compiled_autograd.py::TestDTensorCompileWithCompiledAutograd::test_dtensor_dynamic_slice, test/inductor/test_compiled_autograd.py::TestDTensorCompileWithCompiledAutograd::test_dtensor_dynamo_device_mesh_attrs, test/inductor/test_compiled_autograd.py::TestDTensorCompileWithCompiledAutograd::test_dtensor_partial_placement_graph_output, test/inductor/test_compiled_autograd.py::TestDTensorCompileWithCompiledAutograd::test_dtensor_partial_placement_redistribute_unbalanced_correct_strides, test/inductor/test_compiled_autograd.py::TestDTensorCompileWithCompiledAutograd::test_dtensor_requires_grad_recompile, test/inductor/test_compiled_autograd.py::TestDTensorCompileWithCompiledAutograd::test_dynamo_dtensor, test/inductor/test_compiled_autograd.py::TestDTensorCompileWithCompiledAutograd::test_dynamo_dtensor_from_local_dynamic_shapes, test/inductor/test_compiled_autograd.py::TestDTensorCompileWithCompiledAutograd::test_dynamo_dtensor_from_local_redistribute, test/inductor/test_compiled_autograd.py::TestDTensorCompileWithCompiledAutograd::test_dynamo_dtensor_from_local_redistribute_async, test/inductor/test_compiled_autograd.py::TestDTensorCompileWithCompiledAutograd::test_dynamo_dtensor_recompile, test/inductor/test_compiled_autograd.py::TestDTensorCompileWithCompiledAutograd::test_dynamo_from_local_grad_placements_sequence_intermediate, test/inductor/test_compiled_autograd.py::TestDTensorCompileWithCompiledAutograd::test_dynamo_from_local_grad_placements_sequence_intermediate_as_args, test/inductor/test_compiled_autograd.py::TestDTensorCompileWithCompiledAutograd::test_dynamo_to_local_grad_placements_sequence, test/inductor/test_compiled_autograd.py::TestDTensorCompileWithCompiledAutograd::test_dynamo_to_local_grad_placements_sequence_intermediate, test/inductor/test_compiled_autograd.py::TestDTensorCompileWithCompiledAutograd::test_dynamo_to_local_kwargs, test/inductor/test_compiled_autograd.py::TestDTensorCompileWithCompiledAutograd::test_dynamo_to_local_kwargs_forward_hook, test/inductor/test_compiled_autograd.py::TestDTensorCompileWithCompiledAutograd::test_fakify_dtensor, test/inductor/test_compiled_autograd.py::TestDTensorCompileWithCompiledAutograd::test_graph_input_is_async, test/inductor/test_compiled_autograd.py::TestDTensorCompileWithCompiledAutograd::test_placement_compile, test/inductor/test_compiled_autograd.py::TestDTensorCompileWithCompiledAutograd::test_unwrap_async_collective_tensor_tangent, test/inductor/test_compiled_autograd.py::TestCompiledAutogradOpInfoCUDA::test_hops_in_bwd_cond_simple_cuda_float32, test/inductor/test_compiled_autograd.py::TestCompiledAutogradOpInfoCUDA::test_hops_in_bwd_invoke_quant_packed_simple_cuda_float32, test/inductor/test_compiled_autograd.py::TestCompiledAutogradOpInfoCUDA::test_hops_in_bwd_invoke_subgraph_simple_cuda_float32, test/inductor/test_compiled_autograd.py::TestCompiledAutogradOpInfoCUDA::test_hops_in_bwd_map_nested_cuda_float32, test/inductor/test_compiled_autograd.py::TestCompiledAutogradOpInfoCUDA::test_hops_in_bwd_map_simple_cuda_float32, test/inductor/test_compiled_autograd.py::TestCompiledAutogradOpInfoCUDA::test_hops_in_bwd_while_loop_simple_cuda_float32 2025-12-04T13:13:36.8617187Z 2025-12-04T13:13:36.8617451Z Finished inductor/test_compiled_autograd 1/2 ... [2025-12-04 13:13:36.826272][12864.768485153], took 7.31min 2025-12-04T13:13:36.8618423Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/inductor.test_compiled_autograd/inductor.test_compiled_autograd-4b5e4b19f368f980.xml 2025-12-04T13:13:37.7569471Z Uploading artifacts took 0.81 seconds 2025-12-04T13:13:37.7572399Z Running inductor/test_torchinductor_strided_blocks 1/1 ... [2025-12-04 13:13:37.757046][12865.699261024] 2025-12-04T13:13:37.7572926Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2025-12-04T13:13:37.7576439Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'inductor/test_torchinductor_strided_blocks.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-12-04 13:13:37.757399] 2025-12-04T13:14:25.0029894Z 2025-12-04T13:14:25.0031338Z inductor/test_torchinductor_strided_blocks 1/1 was successful, full logs can be found in artifacts with path test/test-reports/inductor.test_torchinductor_strided_blocks_1.1_a1d3464a2f8c57ea_.log 2025-12-04T13:14:25.0168333Z Running 302 items in this shard: test/inductor/test_torchinductor_strided_blocks.py::TritonBlockPointerTestCPU::test_2d_reduction_multi_kernel_cpu, test/inductor/test_torchinductor_strided_blocks.py::TritonBlockPointerTestCPU::test_2d_reduction_odd_shapes_view_size0_num_block_pointers_1_num_triton_kernels_1_reduction_op0_cpu, test/inductor/test_torchinductor_strided_blocks.py::TritonBlockPointerTestCPU::test_2d_reduction_odd_shapes_view_size1_num_block_pointers_3_num_triton_kernels_2_reduction_op1_cpu, test/inductor/test_torchinductor_strided_blocks.py::TritonBlockPointerTestCPU::test_2d_reduction_odd_shapes_view_size2_num_block_pointers_1_num_triton_kernels_1_reduction_op2_cpu, test/inductor/test_torchinductor_strided_blocks.py::TritonBlockPointerTestCPU::test_2d_reduction_odd_shapes_view_size3_num_block_pointers_1_num_triton_kernels_1_reduction_op3_cpu, test/inductor/test_torchinductor_strided_blocks.py::TritonBlockPointerTestCPU::test_2d_reduction_odd_shapes_view_size4_num_block_pointers_1_num_triton_kernels_1_reduction_op4_cpu, test/inductor/test_torchinductor_strided_blocks.py::TritonBlockPointerTestCPU::test_2d_reductions_mixed_indexing_reduction_op0_cpu, test/inductor/test_torchinductor_strided_blocks.py::TritonBlockPointerTestCPU::test_2d_reductions_mixed_indexing_reduction_op1_cpu, test/inductor/test_torchinductor_strided_blocks.py::TritonBlockPointerTestCPU::test_2d_welford_reduction_size0_expected_num_block_pointers_1_expected_num_triton_kernels_1_expect_fallback_True_cpu, test/inductor/test_torchinductor_strided_blocks.py::TritonBlockPointerTestCPU::test_2d_welford_reduction_size1_expected_num_block_pointers_7_expected_num_triton_kernels_2_expect_fallback_False_cpu, test/inductor/test_torchinductor_strided_blocks.py::TritonBlockPointerTestCPU::test_3d_permute_tiling_cpu, test/inductor/test_torchinductor_strided_blocks.py::TritonBlockPointerTestCPU::test_boundary_check_block_multiple_False_ynumel_exceed_ygrid_size_False_include_z_True_cpu, test/inductor/test_torchinductor_strided_blocks.py::TritonBlockPointerTestCPU::test_boundary_check_block_multiple_True_ynumel_exceed_ygrid_size_False_include_z_True_cpu, test/inductor/test_torchinductor_strided_blocks.py::TritonBlockPointerTestCPU::test_boundary_check_block_multiple_True_ynumel_exceed_ygrid_size_True_include_z_False_cpu, test/inductor/test_torchinductor_strided_blocks.py::TritonBlockPointerTestCPU::test_broadcast_prefer_nd_tiling_False_x_size0_y_size0_cpu, test/inductor/test_torchinductor_strided_blocks.py::TritonBlockPointerTestCPU::test_broadcast_prefer_nd_tiling_False_x_size1_y_size1_cpu, test/inductor/test_torchinductor_strided_blocks.py::TritonBlockPointerTestCPU::test_broadcast_prefer_nd_tiling_False_x_size2_y_size2_cpu, test/inductor/test_torchinductor_strided_blocks.py::TritonBlockPointerTestCPU::test_broadcast_prefer_nd_tiling_False_x_size3_y_size3_cpu, test/inductor/test_torchinductor_strided_blocks.py::TritonBlockPointerTestCPU::test_broadcast_prefer_nd_tiling_True_x_size0_y_size0_cpu, test/inductor/test_torchinductor_strided_blocks.py::TritonBlockPointerTestCPU::test_broadcast_prefer_nd_tiling_True_x_size1_y_size1_cpu, test/inductor/test_torchinductor_strided_blocks.py::TritonBlockPointerTestCPU::test_broadcast_prefer_nd_tiling_True_x_size2_y_size2_cpu, test/inductor/test_torchinductor_strided_blocks.py::TritonBlockPointerTestCPU::test_broadcast_prefer_nd_tiling_True_x_size3_y_size3_cpu, test/inductor/test_torchinductor_strided_blocks.py::TritonBlockPointerTestCPU::test_broadcast_with_singleton_dims_cpu, test/inductor/test_torchinductor_strided_blocks.py::TritonBlockPointerTestCPU::test_complex_reshape_block_ptr_cpu, test/inductor/test_torchinductor_strided_blocks.py::TritonBlockPointerTestCPU::test_dynamic_shapes_pointwise_multiple_max_block_cpu, test/inductor/test_torchinductor_strided_blocks.py::TritonBlockPointerTestCPU::test_dynamic_shapes_pointwise_nd_tiling_False_num_block_pointers_1_cpu, test/inductor/test_torchinductor_strided_blocks.py::TritonBlockPointerTestCPU::test_dynamic_shapes_pointwise_nd_tiling_True_num_block_pointers_2_cpu, test/inductor/test_torchinductor_strided_blocks.py::TritonBlockPointerTestCPU::test_dynamic_shapes_reduction_with_tiling_False_num_block_pointers_0_cpu, test/inductor/test_torchinductor_strided_blocks.py::TritonBlockPointerTestCPU::test_dynamic_shapes_reduction_with_tiling_True_num_block_pointers_1_cpu, test/inductor/test_torchinductor_strided_blocks.py::TritonBlockPointerTestCPU::test_enable_tiled_reductions_tile_reductions_False_cpu, test/inductor/test_torchinductor_strided_blocks.py::TritonBlockPointerTestCPU::test_enable_tiled_reductions_tile_reductions_True_cpu, test/inductor/test_torchinductor_strided_blocks.py::TritonBlockPointerTestCPU::test_ensure_integral_dims_and_strides_cpu, test/inductor/test_torchinductor_strided_blocks.py::TritonBlockPointerTestCPU::test_expand_broadcast_x_size0_y_size0_cpu, test/inductor/test_torchinductor_strided_blocks.py::TritonBlockPointerTestCPU::test_expand_broadcast_x_size1_y_size1_cpu, test/inductor/test_torchinductor_strided_blocks.py::TritonBlockPointerTestCPU::test_expand_broadcast_x_size2_y_size2_cpu, test/inductor/test_torchinductor_strided_blocks.py::TritonBlockPointerTestCPU::test_expand_broadcast_x_size3_y_size3_cpu, test/inductor/test_torchinductor_strided_blocks.py::TritonBlockPointerTestCPU::test_expand_broadcast_x_size4_y_size4_cpu, test/inductor/test_torchinductor_strided_blocks.py::TritonBlockPointerTestCPU::test_expand_broadcast_x_size5_y_size5_cpu, test/inductor/test_torchinductor_strided_blocks.py::TritonBlockPointerTestCPU::test_expand_broadcast_x_size6_y_size6_cpu, test/inductor/test_torchinductor_strided_blocks.py::TritonBlockPointerTestCPU::test_expand_broadcast_x_size7_y_size7_cpu, test/inductor/test_torchinductor_strided_blocks.py::TritonBlockPointerTestCPU::test_expand_broadcast_x_size8_y_size8_cpu, test/inductor/test_torchinductor_strided_blocks.py::TritonBlockPointerTestCPU::test_expand_broadcast_x_size9_y_size9_cpu, test/inductor/test_torchinductor_strided_blocks.py::TritonBlockPointerTestCPU::test_expand_clone_broadcast_cpu, test/inductor/test_torchinductor_strided_blocks.py::TritonBlockPointerTestCPU::test_expected_num_block_pointers_expected_num_block_pointers_3_raises_False_cpu, test/inductor/test_torchinductor_strided_blocks.py::TritonBlockPointerTestCPU::test_expected_num_block_pointers_expected_num_block_pointers_9_raises_True_cpu, test/inductor/test_torchinductor_strided_blocks.py::TritonBlockPointerTestCPU::test_fused_2d_reduction_cpu, test/inductor/test_torchinductor_strided_blocks.py::TritonBlockPointerTestCPU::test_mixed_pointwise_reduction_view_size0_num_block_pointers_2_num_triton_kernels_1_cpu, test/inductor/test_torchinductor_strided_blocks.py::TritonBlockPointerTestCPU::test_mixed_pointwise_reduction_view_size1_num_block_pointers1_num_triton_kernels1_cpu, test/inductor/test_torchinductor_strided_blocks.py::TritonBlockPointerTestCPU::test_mul_broadcast_multi_output_cpu, test/inductor/test_torchinductor_strided_blocks.py::TritonBlockPointerTestCPU::test_multiple_max_block_non_power_of_2_cpu, test/inductor/test_torchinductor_strided_blocks.py::TritonBlockPointerTestCPU::test_nd_tiling_odd_shapes_pointwise_full_size0_view_size0_num_block_pointers_3_num_tiles_1_cpu, test/inductor/test_torchinductor_strided_blocks.py::TritonBlockPointerTestCPU::test_nd_tiling_odd_shapes_pointwise_full_size1_view_size1_num_block_pointers_3_num_tiles_2_cpu, test/inductor/test_torchinductor_strided_blocks.py::TritonBlockPointerTestCPU::test_nd_tiling_odd_shapes_pointwise_full_size2_view_size2_num_block_pointers_3_num_tiles_2_cpu, test/inductor/test_torchinductor_strided_blocks.py::TritonBlockPointerTestCPU::test_nd_tiling_odd_shapes_pointwise_full_size3_view_size3_num_block_pointers_3_num_tiles_2_cpu, test/inductor/test_torchinductor_strided_blocks.py::TritonBlockPointerTestCPU::test_nd_tiling_odd_shapes_pointwise_full_size4_view_size4_num_block_pointers_3_num_tiles_2_cpu, test/inductor/test_torchinductor_strided_blocks.py::TritonBlockPointerTestCPU::test_nd_tiling_odd_shapes_pointwise_full_size5_view_size5_num_block_pointers_1_num_tiles_2_cpu, test/inductor/test_torchinductor_strided_blocks.py::TritonBlockPointerTestCPU::test_negative_strides_cpu, test/inductor/test_torchinductor_strided_blocks.py::TritonBlockPointerTestCPU::test_pointwise_broadcast_nonzero_strides_prefer_nd_tiling_False_cpu, test/inductor/test_torchinductor_strided_blocks.py::TritonBlockPointerTestCPU::test_pointwise_broadcast_nonzero_strides_prefer_nd_tiling_True_cpu, test/inductor/test_torchinductor_strided_blocks.py::TritonBlockPointerTestCPU::test_pointwise_index_order_cpu, test/inductor/test_torchinductor_strided_blocks.py::TritonBlockPointerTestCPU::test_pointwise_prefer_nd_tiling_False_full_size0_view_size0_stride0_offset0_require_block_ptr_True_cpu, test/inductor/test_torchinductor_strided_blocks.py::TritonBlockPointerTestCPU::test_pointwise_prefer_nd_tiling_False_full_size1_view_size1_stride1_offset1_require_block_ptr_True_cpu, test/inductor/test_torchinductor_strided_blocks.py::TritonBlockPointerTestCPU::test_pointwise_prefer_nd_tiling_False_full_size2_view_size2_stride2_offset2_require_block_ptr_True_cpu, test/inductor/test_torchinductor_strided_blocks.py::TritonBlockPointerTestCPU::test_pointwise_prefer_nd_tiling_False_full_size3_view_size3_stride3_offset_10_require_block_ptr_True_cpu, test/inductor/test_torchinductor_strided_blocks.py::TritonBlockPointerTestCPU::test_pointwise_prefer_nd_tiling_False_full_size4_view_size4_stride4_offset4_require_block_ptr_True_cpu, test/inductor/test_torchinductor_strided_blocks.py::TritonBlockPointerTestCPU::test_pointwise_prefer_nd_tiling_False_full_size5_view_size5_stride5_offset5_require_block_ptr_True_cpu, test/inductor/test_torchinductor_strided_blocks.py::TritonBlockPointerTestCPU::test_pointwise_prefer_nd_tiling_False_full_size6_view_size6_stride6_offset6_require_block_ptr_True_cpu, test/inductor/test_torchinductor_strided_blocks.py::TritonBlockPointerTestCPU::test_pointwise_prefer_nd_tiling_False_full_size7_view_size7_stride7_offset7_require_block_ptr_False_cpu, test/inductor/test_torchinductor_strided_blocks.py::TritonBlockPointerTestCPU::test_pointwise_prefer_nd_tiling_False_full_size8_view_size8_stride8_offset8_require_block_ptr_False_cpu, test/inductor/test_torchinductor_strided_blocks.py::TritonBlockPointerTestCPU::test_pointwise_prefer_nd_tiling_False_full_size9_view_size9_stride9_offset9_require_block_ptr_True_cpu, test/inductor/test_torchinductor_strided_blocks.py::TritonBlockPointerTestCPU::test_pointwise_prefer_nd_tiling_True_full_size0_view_size0_stride0_offset0_require_block_ptr_True_cpu, test/inductor/test_torchinductor_strided_blocks.py::TritonBlockPointerTestCPU::test_pointwise_prefer_nd_tiling_True_full_size1_view_size1_stride1_offset1_require_block_ptr_True_cpu, test/inductor/test_torchinductor_strided_blocks.py::TritonBlockPointerTestCPU::test_pointwise_prefer_nd_tiling_True_full_size2_view_size2_stride2_offset2_require_block_ptr_True_cpu, test/inductor/test_torchinductor_strided_blocks.py::TritonBlockPointerTestCPU::test_pointwise_prefer_nd_tiling_True_full_size3_view_size3_stride3_offset_10_require_block_ptr_True_cpu, test/inductor/test_torchinductor_strided_blocks.py::TritonBlockPointerTestCPU::test_pointwise_prefer_nd_tiling_True_full_size4_view_size4_stride4_offset4_require_block_ptr_True_cpu, test/inductor/test_torchinductor_strided_blocks.py::TritonBlockPointerTestCPU::test_pointwise_prefer_nd_tiling_True_full_size5_view_size5_stride5_offset5_require_block_ptr_True_cpu, test/inductor/test_torchinductor_strided_blocks.py::TritonBlockPointerTestCPU::test_pointwise_prefer_nd_tiling_True_full_size6_view_size6_stride6_offset6_require_block_ptr_True_cpu, test/inductor/test_torchinductor_strided_blocks.py::TritonBlockPointerTestCPU::test_pointwise_prefer_nd_tiling_True_full_size7_view_size7_stride7_offset7_require_block_ptr_False_cpu, test/inductor/test_torchinductor_strided_blocks.py::TritonBlockPointerTestCPU::test_pointwise_prefer_nd_tiling_True_full_size8_view_size8_stride8_offset8_require_block_ptr_False_cpu, test/inductor/test_torchinductor_strided_blocks.py::TritonBlockPointerTestCPU::test_pointwise_prefer_nd_tiling_True_full_size9_view_size9_stride9_offset9_require_block_ptr_True_cpu, test/inductor/test_torchinductor_strided_blocks.py::TritonBlockPointerTestCPU::test_reduction_multiple_discontiguous_dims_cpu, test/inductor/test_torchinductor_strided_blocks.py::TritonBlockPointerTestCPU::test_reduction_prefer_nd_tiling_False_view_size0_num_block_pointers_1_num_triton_kernels_1_cpu, test/inductor/test_torchinductor_strided_blocks.py::TritonBlockPointerTestCPU::test_reduction_prefer_nd_tiling_False_view_size1_num_block_pointers_1_num_triton_kernels_1_cpu, test/inductor/test_torchinductor_strided_blocks.py::TritonBlockPointerTestCPU::test_reduction_prefer_nd_tiling_False_view_size2_num_block_pointers_1_num_triton_kernels_1_cpu, test/inductor/test_torchinductor_strided_blocks.py::TritonBlockPointerTestCPU::test_reduction_prefer_nd_tiling_False_view_size3_num_block_pointers3_num_triton_kernels_1_cpu, test/inductor/test_torchinductor_strided_blocks.py::TritonBlockPointerTestCPU::test_reduction_prefer_nd_tiling_False_view_size4_num_block_pointers_3_num_triton_kernels_2_cpu, test/inductor/test_torchinductor_strided_blocks.py::TritonBlockPointerTestCPU::test_reduction_prefer_nd_tiling_False_view_size5_num_block_pointers_2_num_triton_kernels_2_cpu, test/inductor/test_torchinductor_strided_blocks.py::TritonBlockPointerTestCPU::test_reduction_prefer_nd_tiling_False_view_size6_num_block_pointers_3_num_triton_kernels_2_cpu, test/inductor/test_torchinductor_strided_blocks.py::TritonBlockPointerTestCPU::test_reduction_prefer_nd_tiling_True_view_size0_num_block_pointers_1_num_triton_kernels_1_cpu, test/inductor/test_torchinductor_strided_blocks.py::TritonBlockPointerTestCPU::test_reduction_prefer_nd_tiling_True_view_size1_num_block_pointers_1_num_triton_kernels_1_cpu, test/inductor/test_torchinductor_strided_blocks.py::TritonBlockPointerTestCPU::test_reduction_prefer_nd_tiling_True_view_size2_num_block_pointers_1_num_triton_kernels_1_cpu, test/inductor/test_torchinductor_strided_blocks.py::TritonBlockPointerTestCPU::test_reduction_prefer_nd_tiling_True_view_size3_num_block_pointers3_num_triton_kernels_1_cpu, test/inductor/test_torchinductor_strided_blocks.py::TritonBlockPointerTestCPU::test_reduction_prefer_nd_tiling_True_view_size4_num_block_pointers_3_num_triton_kernels_2_cpu, test/inductor/test_torchinductor_strided_blocks.py::TritonBlockPointerTestCPU::test_reduction_prefer_nd_tiling_True_view_size5_num_block_pointers_2_num_triton_kernels_2_cpu, test/inductor/test_torchinductor_strided_blocks.py::TritonBlockPointerTestCPU::test_reduction_prefer_nd_tiling_True_view_size6_num_block_pointers_3_num_triton_kernels_2_cpu, test/inductor/test_torchinductor_strided_blocks.py::TritonBlockPointerTestCPU::test_removed_buffers_cpu, test/inductor/test_torchinductor_strided_blocks.py::TritonBlockPointerTestCPU::test_unbacked_size_on_non_contig_dim_num_tile_candidates_1_cpu, test/inductor/test_torchinductor_strided_blocks.py::TritonBlockPointerTestCPU::test_unbacked_size_on_non_contig_dim_num_tile_candidates_2_cpu, test/inductor/test_torchinductor_strided_blocks.py::TritonBlockPointerTestCPU::test_welford_non_block_pointer_cpu, test/inductor/test_torchinductor_strided_blocks.py::TritonBlockPointerTestGPU::test_2d_reduction_multi_kernel_cuda, test/inductor/test_torchinductor_strided_blocks.py::TritonBlockPointerTestGPU::test_2d_reduction_odd_shapes_view_size0_num_block_pointers_1_num_triton_kernels_1_reduction_op0_cuda, test/inductor/test_torchinductor_strided_blocks.py::TritonBlockPointerTestGPU::test_2d_reduction_odd_shapes_view_size1_num_block_pointers_3_num_triton_kernels_2_reduction_op1_cuda, test/inductor/test_torchinductor_strided_blocks.py::TritonBlockPointerTestGPU::test_2d_reduction_odd_shapes_view_size2_num_block_pointers_1_num_triton_kernels_1_reduction_op2_cuda, test/inductor/test_torchinductor_strided_blocks.py::TritonBlockPointerTestGPU::test_2d_reduction_odd_shapes_view_size3_num_block_pointers_1_num_triton_kernels_1_reduction_op3_cuda, test/inductor/test_torchinductor_strided_blocks.py::TritonBlockPointerTestGPU::test_2d_reduction_odd_shapes_view_size4_num_block_pointers_1_num_triton_kernels_1_reduction_op4_cuda, test/inductor/test_torchinductor_strided_blocks.py::TritonBlockPointerTestGPU::test_2d_reductions_mixed_indexing_reduction_op0_cuda, test/inductor/test_torchinductor_strided_blocks.py::TritonBlockPointerTestGPU::test_2d_reductions_mixed_indexing_reduction_op1_cuda, test/inductor/test_torchinductor_strided_blocks.py::TritonBlockPointerTestGPU::test_2d_welford_reduction_size0_expected_num_block_pointers_1_expected_num_triton_kernels_1_expect_fallback_True_cuda, test/inductor/test_torchinductor_strided_blocks.py::TritonBlockPointerTestGPU::test_2d_welford_reduction_size1_expected_num_block_pointers_7_expected_num_triton_kernels_2_expect_fallback_False_cuda, test/inductor/test_torchinductor_strided_blocks.py::TritonBlockPointerTestGPU::test_3d_permute_tiling_cuda, test/inductor/test_torchinductor_strided_blocks.py::TritonBlockPointerTestGPU::test_boundary_check_block_multiple_False_ynumel_exceed_ygrid_size_False_include_z_True_cuda, test/inductor/test_torchinductor_strided_blocks.py::TritonBlockPointerTestGPU::test_boundary_check_block_multiple_True_ynumel_exceed_ygrid_size_False_include_z_True_cuda, test/inductor/test_torchinductor_strided_blocks.py::TritonBlockPointerTestGPU::test_boundary_check_block_multiple_True_ynumel_exceed_ygrid_size_True_include_z_False_cuda, test/inductor/test_torchinductor_strided_blocks.py::TritonBlockPointerTestGPU::test_broadcast_prefer_nd_tiling_False_x_size0_y_size0_cuda, test/inductor/test_torchinductor_strided_blocks.py::TritonBlockPointerTestGPU::test_broadcast_prefer_nd_tiling_False_x_size1_y_size1_cuda, test/inductor/test_torchinductor_strided_blocks.py::TritonBlockPointerTestGPU::test_broadcast_prefer_nd_tiling_False_x_size2_y_size2_cuda, test/inductor/test_torchinductor_strided_blocks.py::TritonBlockPointerTestGPU::test_broadcast_prefer_nd_tiling_False_x_size3_y_size3_cuda, test/inductor/test_torchinductor_strided_blocks.py::TritonBlockPointerTestGPU::test_broadcast_prefer_nd_tiling_True_x_size0_y_size0_cuda, test/inductor/test_torchinductor_strided_blocks.py::TritonBlockPointerTestGPU::test_broadcast_prefer_nd_tiling_True_x_size1_y_size1_cuda, test/inductor/test_torchinductor_strided_blocks.py::TritonBlockPointerTestGPU::test_broadcast_prefer_nd_tiling_True_x_size2_y_size2_cuda, test/inductor/test_torchinductor_strided_blocks.py::TritonBlockPointerTestGPU::test_broadcast_prefer_nd_tiling_True_x_size3_y_size3_cuda, test/inductor/test_torchinductor_strided_blocks.py::TritonBlockPointerTestGPU::test_broadcast_with_singleton_dims_cuda, test/inductor/test_torchinductor_strided_blocks.py::TritonBlockPointerTestGPU::test_complex_reshape_block_ptr_cuda, test/inductor/test_torchinductor_strided_blocks.py::TritonBlockPointerTestGPU::test_dynamic_shapes_pointwise_multiple_max_block_cuda, test/inductor/test_torchinductor_strided_blocks.py::TritonBlockPointerTestGPU::test_dynamic_shapes_pointwise_nd_tiling_False_num_block_pointers_1_cuda, test/inductor/test_torchinductor_strided_blocks.py::TritonBlockPointerTestGPU::test_dynamic_shapes_pointwise_nd_tiling_True_num_block_pointers_2_cuda, test/inductor/test_torchinductor_strided_blocks.py::TritonBlockPointerTestGPU::test_dynamic_shapes_reduction_with_tiling_False_num_block_pointers_0_cuda, test/inductor/test_torchinductor_strided_blocks.py::TritonBlockPointerTestGPU::test_dynamic_shapes_reduction_with_tiling_True_num_block_pointers_1_cuda, test/inductor/test_torchinductor_strided_blocks.py::TritonBlockPointerTestGPU::test_enable_tiled_reductions_tile_reductions_False_cuda, test/inductor/test_torchinductor_strided_blocks.py::TritonBlockPointerTestGPU::test_enable_tiled_reductions_tile_reductions_True_cuda, test/inductor/test_torchinductor_strided_blocks.py::TritonBlockPointerTestGPU::test_ensure_integral_dims_and_strides_cuda, test/inductor/test_torchinductor_strided_blocks.py::TritonBlockPointerTestGPU::test_expand_broadcast_x_size0_y_size0_cuda, test/inductor/test_torchinductor_strided_blocks.py::TritonBlockPointerTestGPU::test_expand_broadcast_x_size1_y_size1_cuda, test/inductor/test_torchinductor_strided_blocks.py::TritonBlockPointerTestGPU::test_expand_broadcast_x_size2_y_size2_cuda, test/inductor/test_torchinductor_strided_blocks.py::TritonBlockPointerTestGPU::test_expand_broadcast_x_size3_y_size3_cuda, test/inductor/test_torchinductor_strided_blocks.py::TritonBlockPointerTestGPU::test_expand_broadcast_x_size4_y_size4_cuda, test/inductor/test_torchinductor_strided_blocks.py::TritonBlockPointerTestGPU::test_expand_broadcast_x_size5_y_size5_cuda, test/inductor/test_torchinductor_strided_blocks.py::TritonBlockPointerTestGPU::test_expand_broadcast_x_size6_y_size6_cuda, test/inductor/test_torchinductor_strided_blocks.py::TritonBlockPointerTestGPU::test_expand_broadcast_x_size7_y_size7_cuda, test/inductor/test_torchinductor_strided_blocks.py::TritonBlockPointerTestGPU::test_expand_broadcast_x_size8_y_size8_cuda, test/inductor/test_torchinductor_strided_blocks.py::TritonBlockPointerTestGPU::test_expand_broadcast_x_size9_y_size9_cuda, test/inductor/test_torchinductor_strided_blocks.py::TritonBlockPointerTestGPU::test_expand_clone_broadcast_cuda, test/inductor/test_torchinductor_strided_blocks.py::TritonBlockPointerTestGPU::test_expected_num_block_pointers_expected_num_block_pointers_3_raises_False_cuda, test/inductor/test_torchinductor_strided_blocks.py::TritonBlockPointerTestGPU::test_expected_num_block_pointers_expected_num_block_pointers_9_raises_True_cuda, test/inductor/test_torchinductor_strided_blocks.py::TritonBlockPointerTestGPU::test_fused_2d_reduction_cuda, test/inductor/test_torchinductor_strided_blocks.py::TritonBlockPointerTestGPU::test_mixed_pointwise_reduction_view_size0_num_block_pointers_2_num_triton_kernels_1_cuda, test/inductor/test_torchinductor_strided_blocks.py::TritonBlockPointerTestGPU::test_mixed_pointwise_reduction_view_size1_num_block_pointers1_num_triton_kernels1_cuda, test/inductor/test_torchinductor_strided_blocks.py::TritonBlockPointerTestGPU::test_mul_broadcast_multi_output_cuda, test/inductor/test_torchinductor_strided_blocks.py::TritonBlockPointerTestGPU::test_multiple_max_block_non_power_of_2_cuda, test/inductor/test_torchinductor_strided_blocks.py::TritonBlockPointerTestGPU::test_nd_tiling_odd_shapes_pointwise_full_size0_view_size0_num_block_pointers_3_num_tiles_1_cuda, test/inductor/test_torchinductor_strided_blocks.py::TritonBlockPointerTestGPU::test_nd_tiling_odd_shapes_pointwise_full_size1_view_size1_num_block_pointers_3_num_tiles_2_cuda, test/inductor/test_torchinductor_strided_blocks.py::TritonBlockPointerTestGPU::test_nd_tiling_odd_shapes_pointwise_full_size2_view_size2_num_block_pointers_3_num_tiles_2_cuda, test/inductor/test_torchinductor_strided_blocks.py::TritonBlockPointerTestGPU::test_nd_tiling_odd_shapes_pointwise_full_size3_view_size3_num_block_pointers_3_num_tiles_2_cuda, test/inductor/test_torchinductor_strided_blocks.py::TritonBlockPointerTestGPU::test_nd_tiling_odd_shapes_pointwise_full_size4_view_size4_num_block_pointers_3_num_tiles_2_cuda, test/inductor/test_torchinductor_strided_blocks.py::TritonBlockPointerTestGPU::test_nd_tiling_odd_shapes_pointwise_full_size5_view_size5_num_block_pointers_1_num_tiles_2_cuda, test/inductor/test_torchinductor_strided_blocks.py::TritonBlockPointerTestGPU::test_negative_strides_cuda, test/inductor/test_torchinductor_strided_blocks.py::TritonBlockPointerTestGPU::test_pointwise_broadcast_nonzero_strides_prefer_nd_tiling_False_cuda, test/inductor/test_torchinductor_strided_blocks.py::TritonBlockPointerTestGPU::test_pointwise_broadcast_nonzero_strides_prefer_nd_tiling_True_cuda, test/inductor/test_torchinductor_strided_blocks.py::TritonBlockPointerTestGPU::test_pointwise_index_order_cuda, test/inductor/test_torchinductor_strided_blocks.py::TritonBlockPointerTestGPU::test_pointwise_prefer_nd_tiling_False_full_size0_view_size0_stride0_offset0_require_block_ptr_True_cuda, test/inductor/test_torchinductor_strided_blocks.py::TritonBlockPointerTestGPU::test_pointwise_prefer_nd_tiling_False_full_size1_view_size1_stride1_offset1_require_block_ptr_True_cuda, test/inductor/test_torchinductor_strided_blocks.py::TritonBlockPointerTestGPU::test_pointwise_prefer_nd_tiling_False_full_size2_view_size2_stride2_offset2_require_block_ptr_True_cuda, test/inductor/test_torchinductor_strided_blocks.py::TritonBlockPointerTestGPU::test_pointwise_prefer_nd_tiling_False_full_size3_view_size3_stride3_offset_10_require_block_ptr_True_cuda, test/inductor/test_torchinductor_strided_blocks.py::TritonBlockPointerTestGPU::test_pointwise_prefer_nd_tiling_False_full_size4_view_size4_stride4_offset4_require_block_ptr_True_cuda, test/inductor/test_torchinductor_strided_blocks.py::TritonBlockPointerTestGPU::test_pointwise_prefer_nd_tiling_False_full_size5_view_size5_stride5_offset5_require_block_ptr_True_cuda, test/inductor/test_torchinductor_strided_blocks.py::TritonBlockPointerTestGPU::test_pointwise_prefer_nd_tiling_False_full_size6_view_size6_stride6_offset6_require_block_ptr_True_cuda, test/inductor/test_torchinductor_strided_blocks.py::TritonBlockPointerTestGPU::test_pointwise_prefer_nd_tiling_False_full_size7_view_size7_stride7_offset7_require_block_ptr_False_cuda, test/inductor/test_torchinductor_strided_blocks.py::TritonBlockPointerTestGPU::test_pointwise_prefer_nd_tiling_False_full_size8_view_size8_stride8_offset8_require_block_ptr_False_cuda, test/inductor/test_torchinductor_strided_blocks.py::TritonBlockPointerTestGPU::test_pointwise_prefer_nd_tiling_False_full_size9_view_size9_stride9_offset9_require_block_ptr_True_cuda, test/inductor/test_torchinductor_strided_blocks.py::TritonBlockPointerTestGPU::test_pointwise_prefer_nd_tiling_True_full_size0_view_size0_stride0_offset0_require_block_ptr_True_cuda, test/inductor/test_torchinductor_strided_blocks.py::TritonBlockPointerTestGPU::test_pointwise_prefer_nd_tiling_True_full_size1_view_size1_stride1_offset1_require_block_ptr_True_cuda, test/inductor/test_torchinductor_strided_blocks.py::TritonBlockPointerTestGPU::test_pointwise_prefer_nd_tiling_True_full_size2_view_size2_stride2_offset2_require_block_ptr_True_cuda, test/inductor/test_torchinductor_strided_blocks.py::TritonBlockPointerTestGPU::test_pointwise_prefer_nd_tiling_True_full_size3_view_size3_stride3_offset_10_require_block_ptr_True_cuda, test/inductor/test_torchinductor_strided_blocks.py::TritonBlockPointerTestGPU::test_pointwise_prefer_nd_tiling_True_full_size4_view_size4_stride4_offset4_require_block_ptr_True_cuda, test/inductor/test_torchinductor_strided_blocks.py::TritonBlockPointerTestGPU::test_pointwise_prefer_nd_tiling_True_full_size5_view_size5_stride5_offset5_require_block_ptr_True_cuda, test/inductor/test_torchinductor_strided_blocks.py::TritonBlockPointerTestGPU::test_pointwise_prefer_nd_tiling_True_full_size6_view_size6_stride6_offset6_require_block_ptr_True_cuda, test/inductor/test_torchinductor_strided_blocks.py::TritonBlockPointerTestGPU::test_pointwise_prefer_nd_tiling_True_full_size7_view_size7_stride7_offset7_require_block_ptr_False_cuda, test/inductor/test_torchinductor_strided_blocks.py::TritonBlockPointerTestGPU::test_pointwise_prefer_nd_tiling_True_full_size8_view_size8_stride8_offset8_require_block_ptr_False_cuda, test/inductor/test_torchinductor_strided_blocks.py::TritonBlockPointerTestGPU::test_pointwise_prefer_nd_tiling_True_full_size9_view_size9_stride9_offset9_require_block_ptr_True_cuda, test/inductor/test_torchinductor_strided_blocks.py::TritonBlockPointerTestGPU::test_reduction_multiple_discontiguous_dims_cuda, test/inductor/test_torchinductor_strided_blocks.py::TritonBlockPointerTestGPU::test_reduction_prefer_nd_tiling_False_view_size0_num_block_pointers_1_num_triton_kernels_1_cuda, test/inductor/test_torchinductor_strided_blocks.py::TritonBlockPointerTestGPU::test_reduction_prefer_nd_tiling_False_view_size1_num_block_pointers_1_num_triton_kernels_1_cuda, test/inductor/test_torchinductor_strided_blocks.py::TritonBlockPointerTestGPU::test_reduction_prefer_nd_tiling_False_view_size2_num_block_pointers_1_num_triton_kernels_1_cuda, test/inductor/test_torchinductor_strided_blocks.py::TritonBlockPointerTestGPU::test_reduction_prefer_nd_tiling_False_view_size3_num_block_pointers3_num_triton_kernels_1_cuda, test/inductor/test_torchinductor_strided_blocks.py::TritonBlockPointerTestGPU::test_reduction_prefer_nd_tiling_False_view_size4_num_block_pointers_3_num_triton_kernels_2_cuda, test/inductor/test_torchinductor_strided_blocks.py::TritonBlockPointerTestGPU::test_reduction_prefer_nd_tiling_False_view_size5_num_block_pointers_2_num_triton_kernels_2_cuda, test/inductor/test_torchinductor_strided_blocks.py::TritonBlockPointerTestGPU::test_reduction_prefer_nd_tiling_False_view_size6_num_block_pointers_3_num_triton_kernels_2_cuda, test/inductor/test_torchinductor_strided_blocks.py::TritonBlockPointerTestGPU::test_reduction_prefer_nd_tiling_True_view_size0_num_block_pointers_1_num_triton_kernels_1_cuda, test/inductor/test_torchinductor_strided_blocks.py::TritonBlockPointerTestGPU::test_reduction_prefer_nd_tiling_True_view_size1_num_block_pointers_1_num_triton_kernels_1_cuda, test/inductor/test_torchinductor_strided_blocks.py::TritonBlockPointerTestGPU::test_reduction_prefer_nd_tiling_True_view_size2_num_block_pointers_1_num_triton_kernels_1_cuda, test/inductor/test_torchinductor_strided_blocks.py::TritonBlockPointerTestGPU::test_reduction_prefer_nd_tiling_True_view_size3_num_block_pointers3_num_triton_kernels_1_cuda, test/inductor/test_torchinductor_strided_blocks.py::TritonBlockPointerTestGPU::test_reduction_prefer_nd_tiling_True_view_size4_num_block_pointers_3_num_triton_kernels_2_cuda, test/inductor/test_torchinductor_strided_blocks.py::TritonBlockPointerTestGPU::test_reduction_prefer_nd_tiling_True_view_size5_num_block_pointers_2_num_triton_kernels_2_cuda, test/inductor/test_torchinductor_strided_blocks.py::TritonBlockPointerTestGPU::test_reduction_prefer_nd_tiling_True_view_size6_num_block_pointers_3_num_triton_kernels_2_cuda, test/inductor/test_torchinductor_strided_blocks.py::TritonBlockPointerTestGPU::test_removed_buffers_cuda, test/inductor/test_torchinductor_strided_blocks.py::TritonBlockPointerTestGPU::test_unbacked_size_on_non_contig_dim_num_tile_candidates_1_cuda, test/inductor/test_torchinductor_strided_blocks.py::TritonBlockPointerTestGPU::test_unbacked_size_on_non_contig_dim_num_tile_candidates_2_cuda, test/inductor/test_torchinductor_strided_blocks.py::TritonBlockPointerTestGPU::test_welford_non_block_pointer_cuda, test/inductor/test_torchinductor_strided_blocks.py::TritonTensorDescriptorTestCUDA::test_2d_reduction_multi_kernel_cuda, test/inductor/test_torchinductor_strided_blocks.py::TritonTensorDescriptorTestCUDA::test_2d_reduction_odd_shapes_view_size0_num_block_pointers_1_num_triton_kernels_1_reduction_op0_cuda, test/inductor/test_torchinductor_strided_blocks.py::TritonTensorDescriptorTestCUDA::test_2d_reduction_odd_shapes_view_size1_num_block_pointers_3_num_triton_kernels_2_reduction_op1_cuda, test/inductor/test_torchinductor_strided_blocks.py::TritonTensorDescriptorTestCUDA::test_2d_reduction_odd_shapes_view_size2_num_block_pointers_1_num_triton_kernels_1_reduction_op2_cuda, test/inductor/test_torchinductor_strided_blocks.py::TritonTensorDescriptorTestCUDA::test_2d_reduction_odd_shapes_view_size3_num_block_pointers_1_num_triton_kernels_1_reduction_op3_cuda, test/inductor/test_torchinductor_strided_blocks.py::TritonTensorDescriptorTestCUDA::test_2d_reduction_odd_shapes_view_size4_num_block_pointers_1_num_triton_kernels_1_reduction_op4_cuda, test/inductor/test_torchinductor_strided_blocks.py::TritonTensorDescriptorTestCUDA::test_2d_reductions_mixed_indexing_reduction_op0_cuda, test/inductor/test_torchinductor_strided_blocks.py::TritonTensorDescriptorTestCUDA::test_2d_reductions_mixed_indexing_reduction_op1_cuda, test/inductor/test_torchinductor_strided_blocks.py::TritonTensorDescriptorTestCUDA::test_2d_welford_reduction_size0_expected_num_block_pointers_1_expected_num_triton_kernels_1_expect_fallback_True_cuda, test/inductor/test_torchinductor_strided_blocks.py::TritonTensorDescriptorTestCUDA::test_2d_welford_reduction_size1_expected_num_block_pointers_7_expected_num_triton_kernels_2_expect_fallback_False_cuda, test/inductor/test_torchinductor_strided_blocks.py::TritonTensorDescriptorTestCUDA::test_3d_permute_tiling_cuda, test/inductor/test_torchinductor_strided_blocks.py::TritonTensorDescriptorTestCUDA::test_boundary_check_block_multiple_False_ynumel_exceed_ygrid_size_False_include_z_True_cuda, test/inductor/test_torchinductor_strided_blocks.py::TritonTensorDescriptorTestCUDA::test_boundary_check_block_multiple_True_ynumel_exceed_ygrid_size_False_include_z_True_cuda, test/inductor/test_torchinductor_strided_blocks.py::TritonTensorDescriptorTestCUDA::test_boundary_check_block_multiple_True_ynumel_exceed_ygrid_size_True_include_z_False_cuda, test/inductor/test_torchinductor_strided_blocks.py::TritonTensorDescriptorTestCUDA::test_broadcast_prefer_nd_tiling_False_x_size0_y_size0_cuda, test/inductor/test_torchinductor_strided_blocks.py::TritonTensorDescriptorTestCUDA::test_broadcast_prefer_nd_tiling_False_x_size1_y_size1_cuda, test/inductor/test_torchinductor_strided_blocks.py::TritonTensorDescriptorTestCUDA::test_broadcast_prefer_nd_tiling_False_x_size2_y_size2_cuda, test/inductor/test_torchinductor_strided_blocks.py::TritonTensorDescriptorTestCUDA::test_broadcast_prefer_nd_tiling_False_x_size3_y_size3_cuda, test/inductor/test_torchinductor_strided_blocks.py::TritonTensorDescriptorTestCUDA::test_broadcast_prefer_nd_tiling_True_x_size0_y_size0_cuda, test/inductor/test_torchinductor_strided_blocks.py::TritonTensorDescriptorTestCUDA::test_broadcast_prefer_nd_tiling_True_x_size1_y_size1_cuda, test/inductor/test_torchinductor_strided_blocks.py::TritonTensorDescriptorTestCUDA::test_broadcast_prefer_nd_tiling_True_x_size2_y_size2_cuda, test/inductor/test_torchinductor_strided_blocks.py::TritonTensorDescriptorTestCUDA::test_broadcast_prefer_nd_tiling_True_x_size3_y_size3_cuda, test/inductor/test_torchinductor_strided_blocks.py::TritonTensorDescriptorTestCUDA::test_broadcast_with_singleton_dims_cuda, test/inductor/test_torchinductor_strided_blocks.py::TritonTensorDescriptorTestCUDA::test_complex_reshape_block_ptr_cuda, test/inductor/test_torchinductor_strided_blocks.py::TritonTensorDescriptorTestCUDA::test_dynamic_shapes_pointwise_multiple_max_block_cuda, test/inductor/test_torchinductor_strided_blocks.py::TritonTensorDescriptorTestCUDA::test_dynamic_shapes_pointwise_nd_tiling_False_num_block_pointers_1_cuda, test/inductor/test_torchinductor_strided_blocks.py::TritonTensorDescriptorTestCUDA::test_dynamic_shapes_pointwise_nd_tiling_True_num_block_pointers_2_cuda, test/inductor/test_torchinductor_strided_blocks.py::TritonTensorDescriptorTestCUDA::test_dynamic_shapes_reduction_with_tiling_False_num_block_pointers_0_cuda, test/inductor/test_torchinductor_strided_blocks.py::TritonTensorDescriptorTestCUDA::test_dynamic_shapes_reduction_with_tiling_True_num_block_pointers_1_cuda, test/inductor/test_torchinductor_strided_blocks.py::TritonTensorDescriptorTestCUDA::test_enable_tiled_reductions_tile_reductions_False_cuda, test/inductor/test_torchinductor_strided_blocks.py::TritonTensorDescriptorTestCUDA::test_enable_tiled_reductions_tile_reductions_True_cuda, test/inductor/test_torchinductor_strided_blocks.py::TritonTensorDescriptorTestCUDA::test_ensure_integral_dims_and_strides_cuda, test/inductor/test_torchinductor_strided_blocks.py::TritonTensorDescriptorTestCUDA::test_expand_broadcast_x_size0_y_size0_cuda, test/inductor/test_torchinductor_strided_blocks.py::TritonTensorDescriptorTestCUDA::test_expand_broadcast_x_size1_y_size1_cuda, test/inductor/test_torchinductor_strided_blocks.py::TritonTensorDescriptorTestCUDA::test_expand_broadcast_x_size2_y_size2_cuda, test/inductor/test_torchinductor_strided_blocks.py::TritonTensorDescriptorTestCUDA::test_expand_broadcast_x_size3_y_size3_cuda, test/inductor/test_torchinductor_strided_blocks.py::TritonTensorDescriptorTestCUDA::test_expand_broadcast_x_size4_y_size4_cuda, test/inductor/test_torchinductor_strided_blocks.py::TritonTensorDescriptorTestCUDA::test_expand_broadcast_x_size5_y_size5_cuda, test/inductor/test_torchinductor_strided_blocks.py::TritonTensorDescriptorTestCUDA::test_expand_broadcast_x_size6_y_size6_cuda, test/inductor/test_torchinductor_strided_blocks.py::TritonTensorDescriptorTestCUDA::test_expand_broadcast_x_size7_y_size7_cuda, test/inductor/test_torchinductor_strided_blocks.py::TritonTensorDescriptorTestCUDA::test_expand_broadcast_x_size8_y_size8_cuda, test/inductor/test_torchinductor_strided_blocks.py::TritonTensorDescriptorTestCUDA::test_expand_broadcast_x_size9_y_size9_cuda, test/inductor/test_torchinductor_strided_blocks.py::TritonTensorDescriptorTestCUDA::test_expand_clone_broadcast_cuda, test/inductor/test_torchinductor_strided_blocks.py::TritonTensorDescriptorTestCUDA::test_expected_num_block_pointers_expected_num_block_pointers_3_raises_False_cuda, test/inductor/test_torchinductor_strided_blocks.py::TritonTensorDescriptorTestCUDA::test_expected_num_block_pointers_expected_num_block_pointers_9_raises_True_cuda, test/inductor/test_torchinductor_strided_blocks.py::TritonTensorDescriptorTestCUDA::test_fused_2d_reduction_cuda, test/inductor/test_torchinductor_strided_blocks.py::TritonTensorDescriptorTestCUDA::test_match_with_transpose_view_size0_permute_order0_num_tensor_descriptors_3_expect_transpose_False, test/inductor/test_torchinductor_strided_blocks.py::TritonTensorDescriptorTestCUDA::test_match_with_transpose_view_size1_permute_order1_num_tensor_descriptors_3_expect_transpose_False, test/inductor/test_torchinductor_strided_blocks.py::TritonTensorDescriptorTestCUDA::test_match_with_transpose_view_size2_permute_order2_num_tensor_descriptors_3_expect_transpose_True, test/inductor/test_torchinductor_strided_blocks.py::TritonTensorDescriptorTestCUDA::test_match_with_transpose_view_size3_permute_order3_num_tensor_descriptors_3_expect_transpose_True, test/inductor/test_torchinductor_strided_blocks.py::TritonTensorDescriptorTestCUDA::test_match_with_transpose_view_size4_permute_order4_num_tensor_descriptors_3_expect_transpose_True, test/inductor/test_torchinductor_strided_blocks.py::TritonTensorDescriptorTestCUDA::test_mixed_pointwise_reduction_view_size0_num_block_pointers_2_num_triton_kernels_1_cuda, test/inductor/test_torchinductor_strided_blocks.py::TritonTensorDescriptorTestCUDA::test_mixed_pointwise_reduction_view_size1_num_block_pointers1_num_triton_kernels1_cuda, test/inductor/test_torchinductor_strided_blocks.py::TritonTensorDescriptorTestCUDA::test_mul_broadcast_multi_output_cuda, test/inductor/test_torchinductor_strided_blocks.py::TritonTensorDescriptorTestCUDA::test_multiple_max_block_non_power_of_2_cuda, test/inductor/test_torchinductor_strided_blocks.py::TritonTensorDescriptorTestCUDA::test_nd_tiling_odd_shapes_pointwise_full_size0_view_size0_num_block_pointers_3_num_tiles_1_cuda, test/inductor/test_torchinductor_strided_blocks.py::TritonTensorDescriptorTestCUDA::test_nd_tiling_odd_shapes_pointwise_full_size1_view_size1_num_block_pointers_3_num_tiles_2_cuda, test/inductor/test_torchinductor_strided_blocks.py::TritonTensorDescriptorTestCUDA::test_nd_tiling_odd_shapes_pointwise_full_size2_view_size2_num_block_pointers_3_num_tiles_2_cuda, test/inductor/test_torchinductor_strided_blocks.py::TritonTensorDescriptorTestCUDA::test_nd_tiling_odd_shapes_pointwise_full_size3_view_size3_num_block_pointers_3_num_tiles_2_cuda, test/inductor/test_torchinductor_strided_blocks.py::TritonTensorDescriptorTestCUDA::test_nd_tiling_odd_shapes_pointwise_full_size4_view_size4_num_block_pointers_3_num_tiles_2_cuda, test/inductor/test_torchinductor_strided_blocks.py::TritonTensorDescriptorTestCUDA::test_nd_tiling_odd_shapes_pointwise_full_size5_view_size5_num_block_pointers_1_num_tiles_2_cuda, test/inductor/test_torchinductor_strided_blocks.py::TritonTensorDescriptorTestCUDA::test_negative_strides_cuda, test/inductor/test_torchinductor_strided_blocks.py::TritonTensorDescriptorTestCUDA::test_pointwise_broadcast_nonzero_strides_prefer_nd_tiling_False_cuda, test/inductor/test_torchinductor_strided_blocks.py::TritonTensorDescriptorTestCUDA::test_pointwise_broadcast_nonzero_strides_prefer_nd_tiling_True_cuda, test/inductor/test_torchinductor_strided_blocks.py::TritonTensorDescriptorTestCUDA::test_pointwise_index_order_cuda, test/inductor/test_torchinductor_strided_blocks.py::TritonTensorDescriptorTestCUDA::test_pointwise_prefer_nd_tiling_False_full_size0_view_size0_stride0_offset0_require_block_ptr_True_cuda, test/inductor/test_torchinductor_strided_blocks.py::TritonTensorDescriptorTestCUDA::test_pointwise_prefer_nd_tiling_False_full_size1_view_size1_stride1_offset1_require_block_ptr_True_cuda, test/inductor/test_torchinductor_strided_blocks.py::TritonTensorDescriptorTestCUDA::test_pointwise_prefer_nd_tiling_False_full_size2_view_size2_stride2_offset2_require_block_ptr_True_cuda, test/inductor/test_torchinductor_strided_blocks.py::TritonTensorDescriptorTestCUDA::test_pointwise_prefer_nd_tiling_False_full_size3_view_size3_stride3_offset_10_require_block_ptr_True_cuda, test/inductor/test_torchinductor_strided_blocks.py::TritonTensorDescriptorTestCUDA::test_pointwise_prefer_nd_tiling_False_full_size4_view_size4_stride4_offset4_require_block_ptr_True_cuda, test/inductor/test_torchinductor_strided_blocks.py::TritonTensorDescriptorTestCUDA::test_pointwise_prefer_nd_tiling_False_full_size5_view_size5_stride5_offset5_require_block_ptr_True_cuda, test/inductor/test_torchinductor_strided_blocks.py::TritonTensorDescriptorTestCUDA::test_pointwise_prefer_nd_tiling_False_full_size6_view_size6_stride6_offset6_require_block_ptr_True_cuda, test/inductor/test_torchinductor_strided_blocks.py::TritonTensorDescriptorTestCUDA::test_pointwise_prefer_nd_tiling_False_full_size7_view_size7_stride7_offset7_require_block_ptr_False_cuda, test/inductor/test_torchinductor_strided_blocks.py::TritonTensorDescriptorTestCUDA::test_pointwise_prefer_nd_tiling_False_full_size8_view_size8_stride8_offset8_require_block_ptr_False_cuda, test/inductor/test_torchinductor_strided_blocks.py::TritonTensorDescriptorTestCUDA::test_pointwise_prefer_nd_tiling_False_full_size9_view_size9_stride9_offset9_require_block_ptr_True_cuda, test/inductor/test_torchinductor_strided_blocks.py::TritonTensorDescriptorTestCUDA::test_pointwise_prefer_nd_tiling_True_full_size0_view_size0_stride0_offset0_require_block_ptr_True_cuda, test/inductor/test_torchinductor_strided_blocks.py::TritonTensorDescriptorTestCUDA::test_pointwise_prefer_nd_tiling_True_full_size1_view_size1_stride1_offset1_require_block_ptr_True_cuda, test/inductor/test_torchinductor_strided_blocks.py::TritonTensorDescriptorTestCUDA::test_pointwise_prefer_nd_tiling_True_full_size2_view_size2_stride2_offset2_require_block_ptr_True_cuda, test/inductor/test_torchinductor_strided_blocks.py::TritonTensorDescriptorTestCUDA::test_pointwise_prefer_nd_tiling_True_full_size3_view_size3_stride3_offset_10_require_block_ptr_True_cuda, test/inductor/test_torchinductor_strided_blocks.py::TritonTensorDescriptorTestCUDA::test_pointwise_prefer_nd_tiling_True_full_size4_view_size4_stride4_offset4_require_block_ptr_True_cuda, test/inductor/test_torchinductor_strided_blocks.py::TritonTensorDescriptorTestCUDA::test_pointwise_prefer_nd_tiling_True_full_size5_view_size5_stride5_offset5_require_block_ptr_True_cuda, test/inductor/test_torchinductor_strided_blocks.py::TritonTensorDescriptorTestCUDA::test_pointwise_prefer_nd_tiling_True_full_size6_view_size6_stride6_offset6_require_block_ptr_True_cuda, test/inductor/test_torchinductor_strided_blocks.py::TritonTensorDescriptorTestCUDA::test_pointwise_prefer_nd_tiling_True_full_size7_view_size7_stride7_offset7_require_block_ptr_False_cuda, test/inductor/test_torchinductor_strided_blocks.py::TritonTensorDescriptorTestCUDA::test_pointwise_prefer_nd_tiling_True_full_size8_view_size8_stride8_offset8_require_block_ptr_False_cuda, test/inductor/test_torchinductor_strided_blocks.py::TritonTensorDescriptorTestCUDA::test_pointwise_prefer_nd_tiling_True_full_size9_view_size9_stride9_offset9_require_block_ptr_True_cuda, test/inductor/test_torchinductor_strided_blocks.py::TritonTensorDescriptorTestCUDA::test_reduction_multiple_discontiguous_dims_cuda, test/inductor/test_torchinductor_strided_blocks.py::TritonTensorDescriptorTestCUDA::test_reduction_prefer_nd_tiling_False_view_size0_num_block_pointers_1_num_triton_kernels_1_cuda, test/inductor/test_torchinductor_strided_blocks.py::TritonTensorDescriptorTestCUDA::test_reduction_prefer_nd_tiling_False_view_size1_num_block_pointers_1_num_triton_kernels_1_cuda, test/inductor/test_torchinductor_strided_blocks.py::TritonTensorDescriptorTestCUDA::test_reduction_prefer_nd_tiling_False_view_size2_num_block_pointers_1_num_triton_kernels_1_cuda, test/inductor/test_torchinductor_strided_blocks.py::TritonTensorDescriptorTestCUDA::test_reduction_prefer_nd_tiling_False_view_size3_num_block_pointers3_num_triton_kernels_1_cuda, test/inductor/test_torchinductor_strided_blocks.py::TritonTensorDescriptorTestCUDA::test_reduction_prefer_nd_tiling_False_view_size4_num_block_pointers_3_num_triton_kernels_2_cuda, test/inductor/test_torchinductor_strided_blocks.py::TritonTensorDescriptorTestCUDA::test_reduction_prefer_nd_tiling_False_view_size5_num_block_pointers_2_num_triton_kernels_2_cuda, test/inductor/test_torchinductor_strided_blocks.py::TritonTensorDescriptorTestCUDA::test_reduction_prefer_nd_tiling_False_view_size6_num_block_pointers_3_num_triton_kernels_2_cuda, test/inductor/test_torchinductor_strided_blocks.py::TritonTensorDescriptorTestCUDA::test_reduction_prefer_nd_tiling_True_view_size0_num_block_pointers_1_num_triton_kernels_1_cuda, test/inductor/test_torchinductor_strided_blocks.py::TritonTensorDescriptorTestCUDA::test_reduction_prefer_nd_tiling_True_view_size1_num_block_pointers_1_num_triton_kernels_1_cuda, test/inductor/test_torchinductor_strided_blocks.py::TritonTensorDescriptorTestCUDA::test_reduction_prefer_nd_tiling_True_view_size2_num_block_pointers_1_num_triton_kernels_1_cuda, test/inductor/test_torchinductor_strided_blocks.py::TritonTensorDescriptorTestCUDA::test_reduction_prefer_nd_tiling_True_view_size3_num_block_pointers3_num_triton_kernels_1_cuda, test/inductor/test_torchinductor_strided_blocks.py::TritonTensorDescriptorTestCUDA::test_reduction_prefer_nd_tiling_True_view_size4_num_block_pointers_3_num_triton_kernels_2_cuda, test/inductor/test_torchinductor_strided_blocks.py::TritonTensorDescriptorTestCUDA::test_reduction_prefer_nd_tiling_True_view_size5_num_block_pointers_2_num_triton_kernels_2_cuda, test/inductor/test_torchinductor_strided_blocks.py::TritonTensorDescriptorTestCUDA::test_reduction_prefer_nd_tiling_True_view_size6_num_block_pointers_3_num_triton_kernels_2_cuda, test/inductor/test_torchinductor_strided_blocks.py::TritonTensorDescriptorTestCUDA::test_removed_buffers_cuda, test/inductor/test_torchinductor_strided_blocks.py::TritonTensorDescriptorTestCUDA::test_unbacked_size_on_non_contig_dim_num_tile_candidates_1_cuda, test/inductor/test_torchinductor_strided_blocks.py::TritonTensorDescriptorTestCUDA::test_unbacked_size_on_non_contig_dim_num_tile_candidates_2_cuda, test/inductor/test_torchinductor_strided_blocks.py::TritonTensorDescriptorTestCUDA::test_welford_non_block_pointer_cuda 2025-12-04T13:14:25.0301417Z 2025-12-04T13:14:25.0302088Z Finished inductor/test_torchinductor_strided_blocks 1/1 ... [2025-12-04 13:14:25.003542][12912.945757348], took 0.79min 2025-12-04T13:14:25.0317893Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/inductor.test_torchinductor_strided_blocks/inductor.test_torchinductor_strided_blocks-2a8dc82a74796451.xml 2025-12-04T13:14:25.1131602Z Running inductor/test_pad_mm 1/1 ... [2025-12-04 13:14:25.112909][12913.055126811] 2025-12-04T13:14:25.1132363Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2025-12-04T13:14:25.1134933Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'inductor/test_pad_mm.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-12-04 13:14:25.113239] 2025-12-04T13:14:30.7874177Z 2025-12-04T13:14:30.7876142Z inductor/test_pad_mm 1/1 was successful, full logs can be found in artifacts with path test/test-reports/inductor.test_pad_mm_1.1_ee372f763bdd660b_.log 2025-12-04T13:14:30.7880967Z Running 19 items in this shard: test/inductor/test_pad_mm.py::PadMMTest::test_cat_pad_mm_dyn_m, test/inductor/test_pad_mm.py::PadMMTest::test_exclude_cat_padding, test/inductor/test_pad_mm.py::PadMMTest::test_exclude_padding, test/inductor/test_pad_mm.py::PadMMTest::test_no_autocast_in_pad_bmm_joint_graph_pass, test/inductor/test_pad_mm.py::PadMMTest::test_original_aten_preserved_pad_mm, test/inductor/test_pad_mm.py::PadMMTest::test_pad_addmm_2d_bias, test/inductor/test_pad_mm.py::PadMMTest::test_pad_addmm_dyn_m, test/inductor/test_pad_mm.py::PadMMTest::test_pad_addmm_dyn_mn, test/inductor/test_pad_mm.py::PadMMTest::test_pad_batch, test/inductor/test_pad_mm.py::PadMMTest::test_pad_bmm_dyn_b, test/inductor/test_pad_mm.py::PadMMTest::test_pad_bmm_dyn_bm, test/inductor/test_pad_mm.py::PadMMTest::test_pad_bmm_dyn_k, test/inductor/test_pad_mm.py::PadMMTest::test_pad_mm_bf16, test/inductor/test_pad_mm.py::PadMMTest::test_pad_mm_dyn_k, test/inductor/test_pad_mm.py::PadMMTest::test_pad_mm_dyn_m, test/inductor/test_pad_mm.py::PadMMTest::test_pad_mm_dyn_mnk, test/inductor/test_pad_mm.py::PadMMTest::test_pad_mm_dyn_n, test/inductor/test_pad_mm.py::PadMMTest::test_pad_single_cat, test/inductor/test_pad_mm.py::PadMMTest::test_zero_dim 2025-12-04T13:14:30.7884832Z 2025-12-04T13:14:30.7885119Z Finished inductor/test_pad_mm 1/1 ... [2025-12-04 13:14:30.787105][12918.729318685], took 0.09min 2025-12-04T13:14:30.8158488Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/inductor.test_pad_mm/inductor.test_pad_mm-e95ecbf722b780e7.xml 2025-12-04T13:14:30.8464740Z Running inductor/test_triton_syntax 1/1 ... [2025-12-04 13:14:30.846241][12918.788459605] 2025-12-04T13:14:30.8465326Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2025-12-04T13:14:30.8468555Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'inductor/test_triton_syntax.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-12-04 13:14:30.846575] 2025-12-04T13:14:42.3310280Z 2025-12-04T13:14:42.3311879Z inductor/test_triton_syntax 1/1 was successful, full logs can be found in artifacts with path test/test-reports/inductor.test_triton_syntax_1.1_eefa874937932af8_.log 2025-12-04T13:14:42.3313306Z Running 1 items in this shard: test/inductor/test_triton_syntax.py::TestTritonSyntacticallyValid::test_triton_sqrt 2025-12-04T13:14:42.3313859Z 2025-12-04T13:14:42.3314178Z Finished inductor/test_triton_syntax 1/1 ... [2025-12-04 13:14:42.330556][12930.272768895], took 0.19min 2025-12-04T13:14:42.3587958Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/inductor.test_triton_syntax/inductor.test_triton_syntax-445bbe43cdf23550.xml 2025-12-04T13:14:42.4285731Z Running inductor/test_triton_extension_backend 1/1 ... [2025-12-04 13:14:42.428296][12930.370513431] 2025-12-04T13:14:42.4286357Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2025-12-04T13:14:42.4289234Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'inductor/test_triton_extension_backend.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-12-04 13:14:42.428635] 2025-12-04T13:14:48.5381263Z 2025-12-04T13:14:48.5382691Z inductor/test_triton_extension_backend 1/1 was successful, full logs can be found in artifacts with path test/test-reports/inductor.test_triton_extension_backend_1.1_e08968c1ce90c8be_.log 2025-12-04T13:14:48.5383739Z Running 0 items in this shard: 2025-12-04T13:14:48.5383939Z 2025-12-04T13:14:48.5384504Z Finished inductor/test_triton_extension_backend 1/1 ... [2025-12-04 13:14:48.537901][12936.480115826], took 0.10min 2025-12-04T13:14:48.5661230Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/inductor.test_triton_extension_backend/inductor.test_triton_extension_backend-a1753ebb46ebf85e.xml 2025-12-04T13:14:48.6269347Z Running test_sparse_semi_structured 1/1 ... [2025-12-04 13:14:48.626685][12936.568903643] 2025-12-04T13:14:48.6270105Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2025-12-04T13:14:48.6272180Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'test_sparse_semi_structured.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-12-04 13:14:48.626983] 2025-12-04T13:15:15.8955275Z 2025-12-04T13:15:15.8958980Z test_sparse_semi_structured 1/1 was successful, full logs can be found in artifacts with path test/test-reports/test_sparse_semi_structured_1.1_3df7ac6bb8c91876_.log 2025-12-04T13:15:15.9041920Z Running 218 items in this shard: test/test_sparse_semi_structured.py::SparseSemiStructuredTensorCompileTest::test_mlp_contiguous_relu_compile_cusparselt, test/test_sparse_semi_structured.py::SparseSemiStructuredTensorCompileTest::test_mlp_contiguous_relu_compile_cutlass, test/test_sparse_semi_structured.py::SparseSemiStructuredTensorCompileTest::test_sp24_compile, test/test_sparse_semi_structured.py::TestSparseSemiStructuredCUDA::test_indices_backend_cusparselt_cuda, test/test_sparse_semi_structured.py::TestSparseSemiStructuredCUDA::test_indices_backend_cutlass_cuda, test/test_sparse_semi_structured.py::TestSparseSemiStructuredCUDA::test_linear_dense_input_shape0_inference_mode_False_backend_cusparselt_cuda, test/test_sparse_semi_structured.py::TestSparseSemiStructuredCUDA::test_linear_dense_input_shape0_inference_mode_False_backend_cutlass_cuda, test/test_sparse_semi_structured.py::TestSparseSemiStructuredCUDA::test_linear_dense_input_shape0_inference_mode_True_backend_cusparselt_cuda, test/test_sparse_semi_structured.py::TestSparseSemiStructuredCUDA::test_linear_dense_input_shape0_inference_mode_True_backend_cutlass_cuda, test/test_sparse_semi_structured.py::TestSparseSemiStructuredCUDA::test_linear_dense_input_shape1_inference_mode_False_backend_cusparselt_cuda, test/test_sparse_semi_structured.py::TestSparseSemiStructuredCUDA::test_linear_dense_input_shape1_inference_mode_False_backend_cutlass_cuda, test/test_sparse_semi_structured.py::TestSparseSemiStructuredCUDA::test_linear_dense_input_shape1_inference_mode_True_backend_cusparselt_cuda, test/test_sparse_semi_structured.py::TestSparseSemiStructuredCUDA::test_linear_dense_input_shape1_inference_mode_True_backend_cutlass_cuda, test/test_sparse_semi_structured.py::TestSparseSemiStructuredCUDA::test_linear_dense_input_shape2_inference_mode_False_backend_cusparselt_cuda, test/test_sparse_semi_structured.py::TestSparseSemiStructuredCUDA::test_linear_dense_input_shape2_inference_mode_False_backend_cutlass_cuda, test/test_sparse_semi_structured.py::TestSparseSemiStructuredCUDA::test_linear_dense_input_shape2_inference_mode_True_backend_cusparselt_cuda, test/test_sparse_semi_structured.py::TestSparseSemiStructuredCUDA::test_linear_dense_input_shape2_inference_mode_True_backend_cutlass_cuda, test/test_sparse_semi_structured.py::TestSparseSemiStructuredCUDA::test_linear_dense_input_shape3_inference_mode_False_backend_cusparselt_cuda, test/test_sparse_semi_structured.py::TestSparseSemiStructuredCUDA::test_linear_dense_input_shape3_inference_mode_False_backend_cutlass_cuda, test/test_sparse_semi_structured.py::TestSparseSemiStructuredCUDA::test_linear_dense_input_shape3_inference_mode_True_backend_cusparselt_cuda, test/test_sparse_semi_structured.py::TestSparseSemiStructuredCUDA::test_linear_dense_input_shape3_inference_mode_True_backend_cutlass_cuda, test/test_sparse_semi_structured.py::TestSparseSemiStructuredCUDA::test_min_sparse_shape_backend_cusparselt_cuda_bfloat16, test/test_sparse_semi_structured.py::TestSparseSemiStructuredCUDA::test_min_sparse_shape_backend_cusparselt_cuda_float16, test/test_sparse_semi_structured.py::TestSparseSemiStructuredCUDA::test_min_sparse_shape_backend_cusparselt_cuda_int8, test/test_sparse_semi_structured.py::TestSparseSemiStructuredCUDA::test_min_sparse_shape_backend_cutlass_cuda_bfloat16, test/test_sparse_semi_structured.py::TestSparseSemiStructuredCUDA::test_min_sparse_shape_backend_cutlass_cuda_float16, test/test_sparse_semi_structured.py::TestSparseSemiStructuredCUDA::test_min_sparse_shape_backend_cutlass_cuda_int8, test/test_sparse_semi_structured.py::TestSparseSemiStructuredCUDA::test_mlp_dense_input_shape0_backend_cusparselt_cuda, test/test_sparse_semi_structured.py::TestSparseSemiStructuredCUDA::test_mlp_dense_input_shape0_backend_cutlass_cuda, test/test_sparse_semi_structured.py::TestSparseSemiStructuredCUDA::test_mlp_dense_input_shape1_backend_cusparselt_cuda, test/test_sparse_semi_structured.py::TestSparseSemiStructuredCUDA::test_mlp_dense_input_shape1_backend_cutlass_cuda, test/test_sparse_semi_structured.py::TestSparseSemiStructuredCUDA::test_mlp_dense_input_shape2_backend_cusparselt_cuda, test/test_sparse_semi_structured.py::TestSparseSemiStructuredCUDA::test_mlp_dense_input_shape2_backend_cutlass_cuda, test/test_sparse_semi_structured.py::TestSparseSemiStructuredCUDA::test_mlp_dense_input_shape3_backend_cusparselt_cuda, test/test_sparse_semi_structured.py::TestSparseSemiStructuredCUDA::test_mlp_dense_input_shape3_backend_cutlass_cuda, test/test_sparse_semi_structured.py::TestSparseSemiStructuredCUDA::test_mm_sparse_first_NN_backend_cusparselt_dense_input_shape0_cuda_bfloat16, test/test_sparse_semi_structured.py::TestSparseSemiStructuredCUDA::test_mm_sparse_first_NN_backend_cusparselt_dense_input_shape0_cuda_float16, test/test_sparse_semi_structured.py::TestSparseSemiStructuredCUDA::test_mm_sparse_first_NN_backend_cusparselt_dense_input_shape0_cuda_int8, test/test_sparse_semi_structured.py::TestSparseSemiStructuredCUDA::test_mm_sparse_first_NN_backend_cusparselt_dense_input_shape1_cuda_bfloat16, test/test_sparse_semi_structured.py::TestSparseSemiStructuredCUDA::test_mm_sparse_first_NN_backend_cusparselt_dense_input_shape1_cuda_float16, test/test_sparse_semi_structured.py::TestSparseSemiStructuredCUDA::test_mm_sparse_first_NN_backend_cusparselt_dense_input_shape1_cuda_int8, test/test_sparse_semi_structured.py::TestSparseSemiStructuredCUDA::test_mm_sparse_first_NN_backend_cusparselt_dense_input_shape2_cuda_bfloat16, test/test_sparse_semi_structured.py::TestSparseSemiStructuredCUDA::test_mm_sparse_first_NN_backend_cusparselt_dense_input_shape2_cuda_float16, test/test_sparse_semi_structured.py::TestSparseSemiStructuredCUDA::test_mm_sparse_first_NN_backend_cusparselt_dense_input_shape2_cuda_int8, test/test_sparse_semi_structured.py::TestSparseSemiStructuredCUDA::test_mm_sparse_first_NN_backend_cutlass_dense_input_shape0_cuda_bfloat16, test/test_sparse_semi_structured.py::TestSparseSemiStructuredCUDA::test_mm_sparse_first_NN_backend_cutlass_dense_input_shape0_cuda_float16, test/test_sparse_semi_structured.py::TestSparseSemiStructuredCUDA::test_mm_sparse_first_NN_backend_cutlass_dense_input_shape0_cuda_int8, test/test_sparse_semi_structured.py::TestSparseSemiStructuredCUDA::test_mm_sparse_first_NN_backend_cutlass_dense_input_shape1_cuda_bfloat16, test/test_sparse_semi_structured.py::TestSparseSemiStructuredCUDA::test_mm_sparse_first_NN_backend_cutlass_dense_input_shape1_cuda_float16, test/test_sparse_semi_structured.py::TestSparseSemiStructuredCUDA::test_mm_sparse_first_NN_backend_cutlass_dense_input_shape1_cuda_int8, test/test_sparse_semi_structured.py::TestSparseSemiStructuredCUDA::test_mm_sparse_first_NN_backend_cutlass_dense_input_shape2_cuda_bfloat16, test/test_sparse_semi_structured.py::TestSparseSemiStructuredCUDA::test_mm_sparse_first_NN_backend_cutlass_dense_input_shape2_cuda_float16, test/test_sparse_semi_structured.py::TestSparseSemiStructuredCUDA::test_mm_sparse_first_NN_backend_cutlass_dense_input_shape2_cuda_int8, test/test_sparse_semi_structured.py::TestSparseSemiStructuredCUDA::test_mm_sparse_first_NT_backend_cusparselt_dense_input_shape0_cuda_bfloat16, test/test_sparse_semi_structured.py::TestSparseSemiStructuredCUDA::test_mm_sparse_first_NT_backend_cusparselt_dense_input_shape0_cuda_float16, test/test_sparse_semi_structured.py::TestSparseSemiStructuredCUDA::test_mm_sparse_first_NT_backend_cusparselt_dense_input_shape0_cuda_int8, test/test_sparse_semi_structured.py::TestSparseSemiStructuredCUDA::test_mm_sparse_first_NT_backend_cusparselt_dense_input_shape1_cuda_bfloat16, test/test_sparse_semi_structured.py::TestSparseSemiStructuredCUDA::test_mm_sparse_first_NT_backend_cusparselt_dense_input_shape1_cuda_float16, test/test_sparse_semi_structured.py::TestSparseSemiStructuredCUDA::test_mm_sparse_first_NT_backend_cusparselt_dense_input_shape1_cuda_int8, test/test_sparse_semi_structured.py::TestSparseSemiStructuredCUDA::test_mm_sparse_first_NT_backend_cusparselt_dense_input_shape2_cuda_bfloat16, test/test_sparse_semi_structured.py::TestSparseSemiStructuredCUDA::test_mm_sparse_first_NT_backend_cusparselt_dense_input_shape2_cuda_float16, test/test_sparse_semi_structured.py::TestSparseSemiStructuredCUDA::test_mm_sparse_first_NT_backend_cusparselt_dense_input_shape2_cuda_int8, test/test_sparse_semi_structured.py::TestSparseSemiStructuredCUDA::test_mm_sparse_first_NT_backend_cutlass_dense_input_shape0_cuda_bfloat16, test/test_sparse_semi_structured.py::TestSparseSemiStructuredCUDA::test_mm_sparse_first_NT_backend_cutlass_dense_input_shape0_cuda_float16, test/test_sparse_semi_structured.py::TestSparseSemiStructuredCUDA::test_mm_sparse_first_NT_backend_cutlass_dense_input_shape0_cuda_int8, test/test_sparse_semi_structured.py::TestSparseSemiStructuredCUDA::test_mm_sparse_first_NT_backend_cutlass_dense_input_shape1_cuda_bfloat16, test/test_sparse_semi_structured.py::TestSparseSemiStructuredCUDA::test_mm_sparse_first_NT_backend_cutlass_dense_input_shape1_cuda_float16, test/test_sparse_semi_structured.py::TestSparseSemiStructuredCUDA::test_mm_sparse_first_NT_backend_cutlass_dense_input_shape1_cuda_int8, test/test_sparse_semi_structured.py::TestSparseSemiStructuredCUDA::test_mm_sparse_first_NT_backend_cutlass_dense_input_shape2_cuda_bfloat16, test/test_sparse_semi_structured.py::TestSparseSemiStructuredCUDA::test_mm_sparse_first_NT_backend_cutlass_dense_input_shape2_cuda_float16, test/test_sparse_semi_structured.py::TestSparseSemiStructuredCUDA::test_mm_sparse_first_NT_backend_cutlass_dense_input_shape2_cuda_int8, test/test_sparse_semi_structured.py::TestSparseSemiStructuredCUDA::test_mm_sparse_first_TN_dense_input_shape0_backend_cusparselt_cuda_bfloat16, test/test_sparse_semi_structured.py::TestSparseSemiStructuredCUDA::test_mm_sparse_first_TN_dense_input_shape0_backend_cusparselt_cuda_float16, test/test_sparse_semi_structured.py::TestSparseSemiStructuredCUDA::test_mm_sparse_first_TN_dense_input_shape0_backend_cusparselt_cuda_int8, test/test_sparse_semi_structured.py::TestSparseSemiStructuredCUDA::test_mm_sparse_first_TN_dense_input_shape0_backend_cutlass_cuda_bfloat16, test/test_sparse_semi_structured.py::TestSparseSemiStructuredCUDA::test_mm_sparse_first_TN_dense_input_shape0_backend_cutlass_cuda_float16, test/test_sparse_semi_structured.py::TestSparseSemiStructuredCUDA::test_mm_sparse_first_TN_dense_input_shape0_backend_cutlass_cuda_int8, test/test_sparse_semi_structured.py::TestSparseSemiStructuredCUDA::test_mm_sparse_first_TN_dense_input_shape1_backend_cusparselt_cuda_bfloat16, test/test_sparse_semi_structured.py::TestSparseSemiStructuredCUDA::test_mm_sparse_first_TN_dense_input_shape1_backend_cusparselt_cuda_float16, test/test_sparse_semi_structured.py::TestSparseSemiStructuredCUDA::test_mm_sparse_first_TN_dense_input_shape1_backend_cusparselt_cuda_int8, test/test_sparse_semi_structured.py::TestSparseSemiStructuredCUDA::test_mm_sparse_first_TN_dense_input_shape1_backend_cutlass_cuda_bfloat16, test/test_sparse_semi_structured.py::TestSparseSemiStructuredCUDA::test_mm_sparse_first_TN_dense_input_shape1_backend_cutlass_cuda_float16, test/test_sparse_semi_structured.py::TestSparseSemiStructuredCUDA::test_mm_sparse_first_TN_dense_input_shape1_backend_cutlass_cuda_int8, test/test_sparse_semi_structured.py::TestSparseSemiStructuredCUDA::test_mm_sparse_first_TN_dense_input_shape2_backend_cusparselt_cuda_bfloat16, test/test_sparse_semi_structured.py::TestSparseSemiStructuredCUDA::test_mm_sparse_first_TN_dense_input_shape2_backend_cusparselt_cuda_float16, test/test_sparse_semi_structured.py::TestSparseSemiStructuredCUDA::test_mm_sparse_first_TN_dense_input_shape2_backend_cusparselt_cuda_int8, test/test_sparse_semi_structured.py::TestSparseSemiStructuredCUDA::test_mm_sparse_first_TN_dense_input_shape2_backend_cutlass_cuda_bfloat16, test/test_sparse_semi_structured.py::TestSparseSemiStructuredCUDA::test_mm_sparse_first_TN_dense_input_shape2_backend_cutlass_cuda_float16, test/test_sparse_semi_structured.py::TestSparseSemiStructuredCUDA::test_mm_sparse_first_TN_dense_input_shape2_backend_cutlass_cuda_int8, test/test_sparse_semi_structured.py::TestSparseSemiStructuredCUDA::test_mm_sparse_second_NN_dense_input_shape0_backend_cusparselt_cuda_bfloat16, test/test_sparse_semi_structured.py::TestSparseSemiStructuredCUDA::test_mm_sparse_second_NN_dense_input_shape0_backend_cusparselt_cuda_float16, test/test_sparse_semi_structured.py::TestSparseSemiStructuredCUDA::test_mm_sparse_second_NN_dense_input_shape0_backend_cusparselt_cuda_int8, test/test_sparse_semi_structured.py::TestSparseSemiStructuredCUDA::test_mm_sparse_second_NN_dense_input_shape0_backend_cutlass_cuda_bfloat16, test/test_sparse_semi_structured.py::TestSparseSemiStructuredCUDA::test_mm_sparse_second_NN_dense_input_shape0_backend_cutlass_cuda_float16, test/test_sparse_semi_structured.py::TestSparseSemiStructuredCUDA::test_mm_sparse_second_NN_dense_input_shape0_backend_cutlass_cuda_int8, test/test_sparse_semi_structured.py::TestSparseSemiStructuredCUDA::test_mm_sparse_second_NN_dense_input_shape1_backend_cusparselt_cuda_bfloat16, test/test_sparse_semi_structured.py::TestSparseSemiStructuredCUDA::test_mm_sparse_second_NN_dense_input_shape1_backend_cusparselt_cuda_float16, test/test_sparse_semi_structured.py::TestSparseSemiStructuredCUDA::test_mm_sparse_second_NN_dense_input_shape1_backend_cusparselt_cuda_int8, test/test_sparse_semi_structured.py::TestSparseSemiStructuredCUDA::test_mm_sparse_second_NN_dense_input_shape1_backend_cutlass_cuda_bfloat16, test/test_sparse_semi_structured.py::TestSparseSemiStructuredCUDA::test_mm_sparse_second_NN_dense_input_shape1_backend_cutlass_cuda_float16, test/test_sparse_semi_structured.py::TestSparseSemiStructuredCUDA::test_mm_sparse_second_NN_dense_input_shape1_backend_cutlass_cuda_int8, test/test_sparse_semi_structured.py::TestSparseSemiStructuredCUDA::test_mm_sparse_second_NN_dense_input_shape2_backend_cusparselt_cuda_bfloat16, test/test_sparse_semi_structured.py::TestSparseSemiStructuredCUDA::test_mm_sparse_second_NN_dense_input_shape2_backend_cusparselt_cuda_float16, test/test_sparse_semi_structured.py::TestSparseSemiStructuredCUDA::test_mm_sparse_second_NN_dense_input_shape2_backend_cusparselt_cuda_int8, test/test_sparse_semi_structured.py::TestSparseSemiStructuredCUDA::test_mm_sparse_second_NN_dense_input_shape2_backend_cutlass_cuda_bfloat16, test/test_sparse_semi_structured.py::TestSparseSemiStructuredCUDA::test_mm_sparse_second_NN_dense_input_shape2_backend_cutlass_cuda_float16, test/test_sparse_semi_structured.py::TestSparseSemiStructuredCUDA::test_mm_sparse_second_NN_dense_input_shape2_backend_cutlass_cuda_int8, test/test_sparse_semi_structured.py::TestSparseSemiStructuredCUDA::test_mm_sparse_second_NT_dense_input_shape0_backend_cusparselt_cuda_bfloat16, test/test_sparse_semi_structured.py::TestSparseSemiStructuredCUDA::test_mm_sparse_second_NT_dense_input_shape0_backend_cusparselt_cuda_float16, test/test_sparse_semi_structured.py::TestSparseSemiStructuredCUDA::test_mm_sparse_second_NT_dense_input_shape0_backend_cusparselt_cuda_int8, test/test_sparse_semi_structured.py::TestSparseSemiStructuredCUDA::test_mm_sparse_second_NT_dense_input_shape0_backend_cutlass_cuda_bfloat16, test/test_sparse_semi_structured.py::TestSparseSemiStructuredCUDA::test_mm_sparse_second_NT_dense_input_shape0_backend_cutlass_cuda_float16, test/test_sparse_semi_structured.py::TestSparseSemiStructuredCUDA::test_mm_sparse_second_NT_dense_input_shape0_backend_cutlass_cuda_int8, test/test_sparse_semi_structured.py::TestSparseSemiStructuredCUDA::test_mm_sparse_second_NT_dense_input_shape1_backend_cusparselt_cuda_bfloat16, test/test_sparse_semi_structured.py::TestSparseSemiStructuredCUDA::test_mm_sparse_second_NT_dense_input_shape1_backend_cusparselt_cuda_float16, test/test_sparse_semi_structured.py::TestSparseSemiStructuredCUDA::test_mm_sparse_second_NT_dense_input_shape1_backend_cusparselt_cuda_int8, test/test_sparse_semi_structured.py::TestSparseSemiStructuredCUDA::test_mm_sparse_second_NT_dense_input_shape1_backend_cutlass_cuda_bfloat16, test/test_sparse_semi_structured.py::TestSparseSemiStructuredCUDA::test_mm_sparse_second_NT_dense_input_shape1_backend_cutlass_cuda_float16, test/test_sparse_semi_structured.py::TestSparseSemiStructuredCUDA::test_mm_sparse_second_NT_dense_input_shape1_backend_cutlass_cuda_int8, test/test_sparse_semi_structured.py::TestSparseSemiStructuredCUDA::test_mm_sparse_second_NT_dense_input_shape2_backend_cusparselt_cuda_bfloat16, test/test_sparse_semi_structured.py::TestSparseSemiStructuredCUDA::test_mm_sparse_second_NT_dense_input_shape2_backend_cusparselt_cuda_float16, test/test_sparse_semi_structured.py::TestSparseSemiStructuredCUDA::test_mm_sparse_second_NT_dense_input_shape2_backend_cusparselt_cuda_int8, test/test_sparse_semi_structured.py::TestSparseSemiStructuredCUDA::test_mm_sparse_second_NT_dense_input_shape2_backend_cutlass_cuda_bfloat16, test/test_sparse_semi_structured.py::TestSparseSemiStructuredCUDA::test_mm_sparse_second_NT_dense_input_shape2_backend_cutlass_cuda_float16, test/test_sparse_semi_structured.py::TestSparseSemiStructuredCUDA::test_mm_sparse_second_NT_dense_input_shape2_backend_cutlass_cuda_int8, test/test_sparse_semi_structured.py::TestSparseSemiStructuredCUDA::test_to_sparse_semi_structured_backend_cusparselt_cuda_bfloat16, test/test_sparse_semi_structured.py::TestSparseSemiStructuredCUDA::test_to_sparse_semi_structured_backend_cusparselt_cuda_float16, test/test_sparse_semi_structured.py::TestSparseSemiStructuredCUDA::test_to_sparse_semi_structured_backend_cusparselt_cuda_int8, test/test_sparse_semi_structured.py::TestSparseSemiStructuredCUDA::test_to_sparse_semi_structured_backend_cutlass_cuda_bfloat16, test/test_sparse_semi_structured.py::TestSparseSemiStructuredCUDA::test_to_sparse_semi_structured_backend_cutlass_cuda_float16, test/test_sparse_semi_structured.py::TestSparseSemiStructuredCUDA::test_to_sparse_semi_structured_backend_cutlass_cuda_int8, test/test_sparse_semi_structured.py::TestSparseSemiStructuredCUDA::test_unsupported_dim_backend_cusparselt_cuda, test/test_sparse_semi_structured.py::TestSparseSemiStructuredCUDA::test_unsupported_dim_backend_cutlass_cuda, test/test_sparse_semi_structured.py::TestSparseSemiStructuredCUDA::test_unsupported_dtype_backend_cusparselt_cuda_complex128, test/test_sparse_semi_structured.py::TestSparseSemiStructuredCUDA::test_unsupported_dtype_backend_cusparselt_cuda_complex64, test/test_sparse_semi_structured.py::TestSparseSemiStructuredCUDA::test_unsupported_dtype_backend_cusparselt_cuda_float32, test/test_sparse_semi_structured.py::TestSparseSemiStructuredCUDA::test_unsupported_dtype_backend_cusparselt_cuda_float64, test/test_sparse_semi_structured.py::TestSparseSemiStructuredCUDA::test_unsupported_dtype_backend_cusparselt_cuda_int16, test/test_sparse_semi_structured.py::TestSparseSemiStructuredCUDA::test_unsupported_dtype_backend_cusparselt_cuda_int32, test/test_sparse_semi_structured.py::TestSparseSemiStructuredCUDA::test_unsupported_dtype_backend_cusparselt_cuda_int64, test/test_sparse_semi_structured.py::TestSparseSemiStructuredCUDA::test_unsupported_dtype_backend_cusparselt_cuda_int8, test/test_sparse_semi_structured.py::TestSparseSemiStructuredCUDA::test_unsupported_dtype_backend_cusparselt_cuda_uint8, test/test_sparse_semi_structured.py::TestSparseSemiStructuredCUDA::test_unsupported_dtype_backend_cutlass_cuda_complex128, test/test_sparse_semi_structured.py::TestSparseSemiStructuredCUDA::test_unsupported_dtype_backend_cutlass_cuda_complex64, test/test_sparse_semi_structured.py::TestSparseSemiStructuredCUDA::test_unsupported_dtype_backend_cutlass_cuda_float32, test/test_sparse_semi_structured.py::TestSparseSemiStructuredCUDA::test_unsupported_dtype_backend_cutlass_cuda_float64, test/test_sparse_semi_structured.py::TestSparseSemiStructuredCUDA::test_unsupported_dtype_backend_cutlass_cuda_int16, test/test_sparse_semi_structured.py::TestSparseSemiStructuredCUDA::test_unsupported_dtype_backend_cutlass_cuda_int32, test/test_sparse_semi_structured.py::TestSparseSemiStructuredCUDA::test_unsupported_dtype_backend_cutlass_cuda_int64, test/test_sparse_semi_structured.py::TestSparseSemiStructuredCUDA::test_unsupported_dtype_backend_cutlass_cuda_int8, test/test_sparse_semi_structured.py::TestSparseSemiStructuredCUDA::test_unsupported_dtype_backend_cutlass_cuda_uint8, test/test_sparse_semi_structured.py::TestSparseSemiStructuredCUDA::test_unsupported_shape_backend_cusparselt_cuda_bfloat16, test/test_sparse_semi_structured.py::TestSparseSemiStructuredCUDA::test_unsupported_shape_backend_cusparselt_cuda_float16, test/test_sparse_semi_structured.py::TestSparseSemiStructuredCUDA::test_unsupported_shape_backend_cusparselt_cuda_int8, test/test_sparse_semi_structured.py::TestSparseSemiStructuredCUDA::test_unsupported_shape_backend_cutlass_cuda_bfloat16, test/test_sparse_semi_structured.py::TestSparseSemiStructuredCUDA::test_unsupported_shape_backend_cutlass_cuda_float16, test/test_sparse_semi_structured.py::TestSparseSemiStructuredCUDA::test_unsupported_shape_backend_cutlass_cuda_int8, test/test_sparse_semi_structured.py::TestSparseSemiStructuredCUDA::test_values_backend_cusparselt_cuda, test/test_sparse_semi_structured.py::TestSparseSemiStructuredCUDA::test_values_backend_cutlass_cuda, test/test_sparse_semi_structured.py::TestSparseSemiStructuredCUTLASSCUDA::test_conversions_all_patterns_cuda_bfloat16, test/test_sparse_semi_structured.py::TestSparseSemiStructuredCUTLASSCUDA::test_conversions_all_patterns_cuda_float16, test/test_sparse_semi_structured.py::TestSparseSemiStructuredCUTLASSCUDA::test_conversions_all_patterns_cuda_int8, test/test_sparse_semi_structured.py::TestSparseSemiStructuredCUTLASSCUDA::test_conversions_cuda_bfloat16, test/test_sparse_semi_structured.py::TestSparseSemiStructuredCUTLASSCUDA::test_conversions_cuda_float16, test/test_sparse_semi_structured.py::TestSparseSemiStructuredCUTLASSCUDA::test_conversions_cuda_int8, test/test_sparse_semi_structured.py::TestSparseSemiStructuredCUTLASSCUDA::test_linear_cutlass_cuda_bfloat16, test/test_sparse_semi_structured.py::TestSparseSemiStructuredCUTLASSCUDA::test_linear_cutlass_cuda_float16, test/test_sparse_semi_structured.py::TestSparseSemiStructuredCUTLASSCUDA::test_linear_cutlass_cuda_int8, test/test_sparse_semi_structured.py::TestSparseSemiStructuredCUTLASSCUDA::test_sparse_semi_structured_ops_cutlass_backend_cutlass_cuda_bfloat16, test/test_sparse_semi_structured.py::TestSparseSemiStructuredCUTLASSCUDA::test_sparse_semi_structured_ops_cutlass_backend_cutlass_cuda_float16, test/test_sparse_semi_structured.py::TestSparseSemiStructuredCUTLASSCUDA::test_sparse_semi_structured_ops_cutlass_backend_cutlass_cuda_int8, test/test_sparse_semi_structured.py::TestSparseSemiStructuredTrainingCUDA::test_gemm_cuda_bfloat16, test/test_sparse_semi_structured.py::TestSparseSemiStructuredTrainingCUDA::test_gemm_cuda_float16, test/test_sparse_semi_structured.py::TestSparseSemiStructuredTrainingCUDA::test_pack_both_ways_edge_case1_cuda_bfloat16, test/test_sparse_semi_structured.py::TestSparseSemiStructuredTrainingCUDA::test_pack_both_ways_edge_case1_cuda_float16, test/test_sparse_semi_structured.py::TestSparseSemiStructuredTrainingCUDA::test_pack_both_ways_id_cuda_bfloat16, test/test_sparse_semi_structured.py::TestSparseSemiStructuredTrainingCUDA::test_pack_both_ways_id_cuda_float16, test/test_sparse_semi_structured.py::TestSparseSemiStructuredTrainingCUDA::test_pack_both_ways_meta_correctness_backend_cusparselt_cuda_bfloat16, test/test_sparse_semi_structured.py::TestSparseSemiStructuredTrainingCUDA::test_pack_both_ways_meta_correctness_backend_cusparselt_cuda_float16, test/test_sparse_semi_structured.py::TestSparseSemiStructuredTrainingCUDA::test_pack_both_ways_meta_correctness_backend_cutlass_cuda_bfloat16, test/test_sparse_semi_structured.py::TestSparseSemiStructuredTrainingCUDA::test_pack_both_ways_meta_correctness_backend_cutlass_cuda_float16, test/test_sparse_semi_structured.py::TestSparseSemiStructuredTrainingCUDA::test_prune_dense_static_sort_cuda_bfloat16, test/test_sparse_semi_structured.py::TestSparseSemiStructuredTrainingCUDA::test_prune_dense_static_sort_cuda_float16, test/test_sparse_semi_structured.py::TestSparseSemiStructuredTrainingCUDA::test_pruning_algo_largest_abs_values_greedy_backend_cusparselt_cuda_bfloat16, test/test_sparse_semi_structured.py::TestSparseSemiStructuredTrainingCUDA::test_pruning_algo_largest_abs_values_greedy_backend_cusparselt_cuda_float16, test/test_sparse_semi_structured.py::TestSparseSemiStructuredTrainingCUDA::test_pruning_algo_largest_abs_values_greedy_backend_cutlass_cuda_bfloat16, test/test_sparse_semi_structured.py::TestSparseSemiStructuredTrainingCUDA::test_pruning_algo_largest_abs_values_greedy_backend_cutlass_cuda_float16, test/test_sparse_semi_structured.py::TestSparseSemiStructuredTrainingCUDA::test_sp24_apply_cuda_bfloat16, test/test_sparse_semi_structured.py::TestSparseSemiStructuredTrainingCUDA::test_sp24_apply_cuda_float16, test/test_sparse_semi_structured.py::TestSparseSemiStructuredTrainingCUDA::test_sp24_apply_dense_cuda_bfloat16, test/test_sparse_semi_structured.py::TestSparseSemiStructuredTrainingCUDA::test_sp24_apply_dense_cuda_float16, test/test_sparse_semi_structured.py::TestSparseSemiStructuredTrainingCUDA::test_sp24_matmuls_bmm_cuda, test/test_sparse_semi_structured.py::TestSparseSemiStructuredTrainingCUDA::test_sp24_matmuls_cuda_bfloat16, test/test_sparse_semi_structured.py::TestSparseSemiStructuredTrainingCUDA::test_sp24_matmuls_cuda_float16, test/test_sparse_semi_structured.py::TestSparseSemiStructuredTrainingCUDA::test_sp24_matmuls_mat_vec_cuda, test/test_sparse_semi_structured.py::TestSparseSemiStructuredCUSPARSELTCUDA::test_cslt_sparse_mm_alpha_compile_autotune_bfloat16_cuda, test/test_sparse_semi_structured.py::TestSparseSemiStructuredCUSPARSELTCUDA::test_cslt_sparse_mm_alpha_compile_autotune_float16_cuda, test/test_sparse_semi_structured.py::TestSparseSemiStructuredCUSPARSELTCUDA::test_cslt_sparse_mm_alpha_compile_autotune_int32_cuda, test/test_sparse_semi_structured.py::TestSparseSemiStructuredCUSPARSELTCUDA::test_cslt_sparse_mm_alpha_cuda_bfloat16, test/test_sparse_semi_structured.py::TestSparseSemiStructuredCUSPARSELTCUDA::test_cslt_sparse_mm_alpha_cuda_float16, test/test_sparse_semi_structured.py::TestSparseSemiStructuredCUSPARSELTCUDA::test_cslt_sparse_mm_alpha_mixed_dtype_bfloat16_cuda, test/test_sparse_semi_structured.py::TestSparseSemiStructuredCUSPARSELTCUDA::test_cslt_sparse_mm_alpha_mixed_dtype_float16_cuda, test/test_sparse_semi_structured.py::TestSparseSemiStructuredCUSPARSELTCUDA::test_cslt_sparse_mm_alpha_mixed_dtype_int32_cuda, test/test_sparse_semi_structured.py::TestSparseSemiStructuredCUSPARSELTCUDA::test_cslt_sparse_mm_mixed_dtype_bfloat16_dense_input_shape0_cuda, test/test_sparse_semi_structured.py::TestSparseSemiStructuredCUSPARSELTCUDA::test_cslt_sparse_mm_mixed_dtype_float16_dense_input_shape0_cuda, test/test_sparse_semi_structured.py::TestSparseSemiStructuredCUSPARSELTCUDA::test_cslt_sparse_mm_mixed_dtype_int32_dense_input_shape0_cuda, test/test_sparse_semi_structured.py::TestSparseSemiStructuredCUSPARSELTCUDA::test_cslt_sparse_mm_search_cuda_bfloat16, test/test_sparse_semi_structured.py::TestSparseSemiStructuredCUSPARSELTCUDA::test_cslt_sparse_mm_search_cuda_float16, test/test_sparse_semi_structured.py::TestSparseSemiStructuredCUSPARSELTCUDA::test_cslt_sparse_mm_search_cuda_int8, test/test_sparse_semi_structured.py::TestSparseSemiStructuredCUSPARSELTCUDA::test_csrc_cslt_sparse_mm_search_cuda_bfloat16, test/test_sparse_semi_structured.py::TestSparseSemiStructuredCUSPARSELTCUDA::test_csrc_cslt_sparse_mm_search_cuda_float16, test/test_sparse_semi_structured.py::TestSparseSemiStructuredCUSPARSELTCUDA::test_csrc_cslt_sparse_mm_search_cuda_int8, test/test_sparse_semi_structured.py::TestSparseSemiStructuredCUSPARSELTCUDA::test_cusparselt_backend_cuda, test/test_sparse_semi_structured.py::TestSparseSemiStructuredCUSPARSELTCUDA::test_sparse_fp8fp8_mm_dense_input_shape0_cuda, test/test_sparse_semi_structured.py::TestSparseSemiStructuredCUSPARSELTCUDA::test_sparse_semi_structured_scaled_mm_bfloat16_dense_input_shape0_cuda, test/test_sparse_semi_structured.py::TestSparseSemiStructuredCUSPARSELTCUDA::test_sparse_semi_structured_scaled_mm_float16_dense_input_shape0_cuda, test/test_sparse_semi_structured.py::TestSparseSemiStructuredCUSPARSELTCUDA::test_sparse_semi_structured_scaled_mm_float32_dense_input_shape0_cuda, test/test_sparse_semi_structured.py::TestSparseSemiStructuredCUSPARSELTCUDA::test_sparse_semi_structured_scaled_mm_fp8_cuda 2025-12-04T13:15:15.9123501Z 2025-12-04T13:15:15.9123796Z Finished test_sparse_semi_structured 1/1 ... [2025-12-04 13:15:15.895665][12963.837882196], took 0.45min 2025-12-04T13:15:15.9231386Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/test_sparse_semi_structured/test_sparse_semi_structured-8ce56cf98dd64834.xml 2025-12-04T13:15:16.0165556Z Running inductor/test_gpu_cpp_wrapper 2/2 ... [2025-12-04 13:15:16.016297][12963.958515209] 2025-12-04T13:15:16.0166142Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2025-12-04T13:15:16.0168794Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'inductor/test_gpu_cpp_wrapper.py', '--shard-id=2', '--num-shards=2', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-12-04 13:15:16.016599] 2025-12-04T13:20:26.1834459Z 2025-12-04T13:20:26.1837500Z inductor/test_gpu_cpp_wrapper 2/2 was successful, full logs can be found in artifacts with path test/test-reports/inductor.test_gpu_cpp_wrapper_2.2_8a9e88db96a4576a_.log 2025-12-04T13:20:26.1885401Z Running 145 items in this shard: test/inductor/test_gpu_cpp_wrapper.py::TestGpuWrapper::test_add_complex4_cuda_gpu_wrapper, test/inductor/test_gpu_cpp_wrapper.py::TestGpuWrapper::test_add_complex_cuda_gpu_wrapper, test/inductor/test_gpu_cpp_wrapper.py::TestGpuWrapper::test_addmm_gpu_wrapper, test/inductor/test_gpu_cpp_wrapper.py::TestGpuWrapper::test_aoti_debug_printer_works_on_constants, test/inductor/test_gpu_cpp_wrapper.py::TestGpuWrapper::test_as_strided_cuda_gpu_wrapper, test/inductor/test_gpu_cpp_wrapper.py::TestGpuWrapper::test_bernoulli1_cuda_gpu_wrapper, test/inductor/test_gpu_cpp_wrapper.py::TestGpuWrapper::test_bitwise_cuda_gpu_wrapper, test/inductor/test_gpu_cpp_wrapper.py::TestGpuWrapper::test_bmm2_cuda_gpu_wrapper, test/inductor/test_gpu_cpp_wrapper.py::TestGpuWrapper::test_consecutive_split_cumprod_cuda_gpu_wrapper, test/inductor/test_gpu_cpp_wrapper.py::TestGpuWrapper::test_custom_op_2_cuda_gpu_wrapper, test/inductor/test_gpu_cpp_wrapper.py::TestGpuWrapper::test_custom_op_3_cuda_gpu_wrapper, test/inductor/test_gpu_cpp_wrapper.py::TestGpuWrapper::test_dtypeview_bfloat16_bfloat16_cuda_gpu_wrapper, test/inductor/test_gpu_cpp_wrapper.py::TestGpuWrapper::test_dtypeview_bfloat16_float16_cuda_gpu_wrapper, test/inductor/test_gpu_cpp_wrapper.py::TestGpuWrapper::test_dtypeview_bfloat16_float32_cuda_gpu_wrapper, test/inductor/test_gpu_cpp_wrapper.py::TestGpuWrapper::test_dtypeview_bfloat16_int16_cuda_gpu_wrapper, test/inductor/test_gpu_cpp_wrapper.py::TestGpuWrapper::test_dtypeview_bfloat16_int32_cuda_gpu_wrapper, test/inductor/test_gpu_cpp_wrapper.py::TestGpuWrapper::test_dtypeview_bfloat16_int8_cuda_gpu_wrapper, test/inductor/test_gpu_cpp_wrapper.py::TestGpuWrapper::test_dtypeview_float16_bfloat16_cuda_gpu_wrapper, test/inductor/test_gpu_cpp_wrapper.py::TestGpuWrapper::test_dtypeview_float16_float16_cuda_gpu_wrapper, test/inductor/test_gpu_cpp_wrapper.py::TestGpuWrapper::test_dtypeview_float16_int16_cuda_gpu_wrapper, test/inductor/test_gpu_cpp_wrapper.py::TestGpuWrapper::test_dtypeview_float16_int8_cuda_gpu_wrapper, test/inductor/test_gpu_cpp_wrapper.py::TestGpuWrapper::test_dtypeview_float32_float16_cuda_gpu_wrapper, test/inductor/test_gpu_cpp_wrapper.py::TestGpuWrapper::test_dtypeview_float32_float64_cuda_gpu_wrapper, test/inductor/test_gpu_cpp_wrapper.py::TestGpuWrapper::test_dtypeview_float32_int16_cuda_gpu_wrapper, test/inductor/test_gpu_cpp_wrapper.py::TestGpuWrapper::test_dtypeview_float32_int32_cuda_gpu_wrapper, test/inductor/test_gpu_cpp_wrapper.py::TestGpuWrapper::test_dtypeview_float32_uint8_cuda_gpu_wrapper, test/inductor/test_gpu_cpp_wrapper.py::TestGpuWrapper::test_dtypeview_float64_float16_cuda_gpu_wrapper, test/inductor/test_gpu_cpp_wrapper.py::TestGpuWrapper::test_dtypeview_float64_float32_cuda_gpu_wrapper, test/inductor/test_gpu_cpp_wrapper.py::TestGpuWrapper::test_dtypeview_float64_int16_cuda_gpu_wrapper, test/inductor/test_gpu_cpp_wrapper.py::TestGpuWrapper::test_dtypeview_float64_int8_cuda_gpu_wrapper, test/inductor/test_gpu_cpp_wrapper.py::TestGpuWrapper::test_dtypeview_fusion_cuda_gpu_wrapper, test/inductor/test_gpu_cpp_wrapper.py::TestGpuWrapper::test_dtypeview_int16_bfloat16_cuda_gpu_wrapper, test/inductor/test_gpu_cpp_wrapper.py::TestGpuWrapper::test_dtypeview_int16_int32_cuda_gpu_wrapper, test/inductor/test_gpu_cpp_wrapper.py::TestGpuWrapper::test_dtypeview_int16_int64_cuda_gpu_wrapper, test/inductor/test_gpu_cpp_wrapper.py::TestGpuWrapper::test_dtypeview_int16_int8_cuda_gpu_wrapper, test/inductor/test_gpu_cpp_wrapper.py::TestGpuWrapper::test_dtypeview_int32_bfloat16_cuda_gpu_wrapper, test/inductor/test_gpu_cpp_wrapper.py::TestGpuWrapper::test_dtypeview_int32_float16_cuda_gpu_wrapper, test/inductor/test_gpu_cpp_wrapper.py::TestGpuWrapper::test_dtypeview_int32_int16_cuda_gpu_wrapper, test/inductor/test_gpu_cpp_wrapper.py::TestGpuWrapper::test_dtypeview_int32_int32_cuda_gpu_wrapper, test/inductor/test_gpu_cpp_wrapper.py::TestGpuWrapper::test_dtypeview_int32_int8_cuda_gpu_wrapper, test/inductor/test_gpu_cpp_wrapper.py::TestGpuWrapper::test_dtypeview_int32_uint8_cuda_gpu_wrapper, test/inductor/test_gpu_cpp_wrapper.py::TestGpuWrapper::test_dtypeview_int64_float16_cuda_gpu_wrapper, test/inductor/test_gpu_cpp_wrapper.py::TestGpuWrapper::test_dtypeview_int64_int32_cuda_gpu_wrapper, test/inductor/test_gpu_cpp_wrapper.py::TestGpuWrapper::test_dtypeview_int64_int64_cuda_gpu_wrapper, test/inductor/test_gpu_cpp_wrapper.py::TestGpuWrapper::test_dtypeview_int64_int8_cuda_gpu_wrapper, test/inductor/test_gpu_cpp_wrapper.py::TestGpuWrapper::test_dtypeview_int8_float16_cuda_gpu_wrapper, test/inductor/test_gpu_cpp_wrapper.py::TestGpuWrapper::test_dtypeview_int8_int16_cuda_gpu_wrapper, test/inductor/test_gpu_cpp_wrapper.py::TestGpuWrapper::test_dtypeview_int8_int8_cuda_gpu_wrapper, test/inductor/test_gpu_cpp_wrapper.py::TestGpuWrapper::test_dtypeview_uint8_float16_cuda_gpu_wrapper, test/inductor/test_gpu_cpp_wrapper.py::TestGpuWrapper::test_dtypeview_uint8_float64_cuda_gpu_wrapper, test/inductor/test_gpu_cpp_wrapper.py::TestGpuWrapper::test_dtypeview_uint8_int32_cuda_gpu_wrapper, test/inductor/test_gpu_cpp_wrapper.py::TestGpuWrapper::test_dtypeview_uint8_int8_cuda_gpu_wrapper, test/inductor/test_gpu_cpp_wrapper.py::TestGpuWrapper::test_dtypeview_uint8_uint8_cuda_gpu_wrapper, test/inductor/test_gpu_cpp_wrapper.py::TestGpuWrapper::test_enable_dynamic_shapes_cpp_wrapper_cuda_gpu_wrapper, test/inductor/test_gpu_cpp_wrapper.py::TestGpuWrapper::test_foreach_cpp_wrapper_cuda_gpu_wrapper, test/inductor/test_gpu_cpp_wrapper.py::TestGpuWrapper::test_index_put_deterministic_fallback_cuda_gpu_wrapper, test/inductor/test_gpu_cpp_wrapper.py::TestGpuWrapper::test_inductor_layout_optimization_input_mutations_cuda_gpu_wrapper, test/inductor/test_gpu_cpp_wrapper.py::TestGpuWrapper::test_insignificant_strides_cuda_gpu_wrapper, test/inductor/test_gpu_cpp_wrapper.py::TestGpuWrapper::test_linear1_cuda_gpu_wrapper, test/inductor/test_gpu_cpp_wrapper.py::TestGpuWrapper::test_linear2_cuda_gpu_wrapper, test/inductor/test_gpu_cpp_wrapper.py::TestGpuWrapper::test_mm_plus_mm3_gpu_wrapper, test/inductor/test_gpu_cpp_wrapper.py::TestGpuWrapper::test_multi_threading_cuda_gpu_wrapper, test/inductor/test_gpu_cpp_wrapper.py::TestGpuWrapper::test_non_tensor_args_wrapped_on_cpu, test/inductor/test_gpu_cpp_wrapper.py::TestGpuWrapper::test_profiler_mark_wrapper_call_cuda_gpu_wrapper, test/inductor/test_gpu_cpp_wrapper.py::TestGpuWrapper::test_randint_cuda_gpu_wrapper, test/inductor/test_gpu_cpp_wrapper.py::TestGpuWrapper::test_relu_cuda_gpu_wrapper, test/inductor/test_gpu_cpp_wrapper.py::TestGpuWrapper::test_roi_align_cuda_gpu_wrapper, test/inductor/test_gpu_cpp_wrapper.py::TestGpuWrapper::test_silu_cuda_gpu_wrapper, test/inductor/test_gpu_cpp_wrapper.py::TestGpuWrapper::test_sort_cuda_gpu_wrapper, test/inductor/test_gpu_cpp_wrapper.py::TestGpuWrapper::test_sum_dtype_cuda_gpu_wrapper, test/inductor/test_gpu_cpp_wrapper.py::TestGpuWrapper::test_sum_int_cuda_gpu_wrapper, test/inductor/test_gpu_cpp_wrapper.py::TestGpuWrapper::test_transpose_cuda_gpu_wrapper, test/inductor/test_gpu_cpp_wrapper.py::TestGpuWrapper::test_unspec_inputs_bfloat16_cuda_gpu_wrapper, test/inductor/test_gpu_cpp_wrapper.py::TestGpuWrapper::test_unspec_inputs_float16_cuda_gpu_wrapper, test/inductor/test_gpu_cpp_wrapper.py::TestGpuWrapper::test_unspec_inputs_float32_cuda_gpu_wrapper, test/inductor/test_gpu_cpp_wrapper.py::TestGpuWrapper::test_unspec_inputs_int32_cuda_gpu_wrapper, test/inductor/test_gpu_cpp_wrapper.py::TestGpuWrapper::test_unspec_inputs_int8_cuda_gpu_wrapper, test/inductor/test_gpu_cpp_wrapper.py::DynamicShapesGpuWrapperGpuTests::test_add_complex4_cuda_dynamic_shapes_gpu_wrapper, test/inductor/test_gpu_cpp_wrapper.py::DynamicShapesGpuWrapperGpuTests::test_addmm_dynamic_shapes_gpu_wrapper, test/inductor/test_gpu_cpp_wrapper.py::DynamicShapesGpuWrapperGpuTests::test_annotation_training, test/inductor/test_gpu_cpp_wrapper.py::DynamicShapesGpuWrapperGpuTests::test_bmm1_cuda_dynamic_shapes_gpu_wrapper, test/inductor/test_gpu_cpp_wrapper.py::DynamicShapesGpuWrapperGpuTests::test_cat_cuda_dynamic_shapes_gpu_wrapper, test/inductor/test_gpu_cpp_wrapper.py::DynamicShapesGpuWrapperGpuTests::test_conv_backward_cuda_dynamic_shapes_gpu_wrapper, test/inductor/test_gpu_cpp_wrapper.py::DynamicShapesGpuWrapperGpuTests::test_custom_op_1_cuda_dynamic_shapes_gpu_wrapper, test/inductor/test_gpu_cpp_wrapper.py::DynamicShapesGpuWrapperGpuTests::test_dtypeview_bfloat16_float16_cuda_dynamic_shapes_gpu_wrapper, test/inductor/test_gpu_cpp_wrapper.py::DynamicShapesGpuWrapperGpuTests::test_dtypeview_bfloat16_float32_cuda_dynamic_shapes_gpu_wrapper, test/inductor/test_gpu_cpp_wrapper.py::DynamicShapesGpuWrapperGpuTests::test_dtypeview_bfloat16_int64_cuda_dynamic_shapes_gpu_wrapper, test/inductor/test_gpu_cpp_wrapper.py::DynamicShapesGpuWrapperGpuTests::test_dtypeview_bfloat16_int8_cuda_dynamic_shapes_gpu_wrapper, test/inductor/test_gpu_cpp_wrapper.py::DynamicShapesGpuWrapperGpuTests::test_dtypeview_bfloat16_uint8_cuda_dynamic_shapes_gpu_wrapper, test/inductor/test_gpu_cpp_wrapper.py::DynamicShapesGpuWrapperGpuTests::test_dtypeview_float16_float16_cuda_dynamic_shapes_gpu_wrapper, test/inductor/test_gpu_cpp_wrapper.py::DynamicShapesGpuWrapperGpuTests::test_dtypeview_float16_float32_cuda_dynamic_shapes_gpu_wrapper, test/inductor/test_gpu_cpp_wrapper.py::DynamicShapesGpuWrapperGpuTests::test_dtypeview_float16_float64_cuda_dynamic_shapes_gpu_wrapper, test/inductor/test_gpu_cpp_wrapper.py::DynamicShapesGpuWrapperGpuTests::test_dtypeview_float16_int32_cuda_dynamic_shapes_gpu_wrapper, test/inductor/test_gpu_cpp_wrapper.py::DynamicShapesGpuWrapperGpuTests::test_dtypeview_float16_int8_cuda_dynamic_shapes_gpu_wrapper, test/inductor/test_gpu_cpp_wrapper.py::DynamicShapesGpuWrapperGpuTests::test_dtypeview_float16_uint8_cuda_dynamic_shapes_gpu_wrapper, test/inductor/test_gpu_cpp_wrapper.py::DynamicShapesGpuWrapperGpuTests::test_dtypeview_float32_float16_cuda_dynamic_shapes_gpu_wrapper, test/inductor/test_gpu_cpp_wrapper.py::DynamicShapesGpuWrapperGpuTests::test_dtypeview_float32_float32_cuda_dynamic_shapes_gpu_wrapper, test/inductor/test_gpu_cpp_wrapper.py::DynamicShapesGpuWrapperGpuTests::test_dtypeview_float64_float64_cuda_dynamic_shapes_gpu_wrapper, test/inductor/test_gpu_cpp_wrapper.py::DynamicShapesGpuWrapperGpuTests::test_dtypeview_float64_int16_cuda_dynamic_shapes_gpu_wrapper, test/inductor/test_gpu_cpp_wrapper.py::DynamicShapesGpuWrapperGpuTests::test_dtypeview_float64_int32_cuda_dynamic_shapes_gpu_wrapper, test/inductor/test_gpu_cpp_wrapper.py::DynamicShapesGpuWrapperGpuTests::test_dtypeview_float64_int8_cuda_dynamic_shapes_gpu_wrapper, test/inductor/test_gpu_cpp_wrapper.py::DynamicShapesGpuWrapperGpuTests::test_dtypeview_float64_uint8_cuda_dynamic_shapes_gpu_wrapper, test/inductor/test_gpu_cpp_wrapper.py::DynamicShapesGpuWrapperGpuTests::test_dtypeview_int16_bfloat16_cuda_dynamic_shapes_gpu_wrapper, test/inductor/test_gpu_cpp_wrapper.py::DynamicShapesGpuWrapperGpuTests::test_dtypeview_int16_float32_cuda_dynamic_shapes_gpu_wrapper, test/inductor/test_gpu_cpp_wrapper.py::DynamicShapesGpuWrapperGpuTests::test_dtypeview_int16_float64_cuda_dynamic_shapes_gpu_wrapper, test/inductor/test_gpu_cpp_wrapper.py::DynamicShapesGpuWrapperGpuTests::test_dtypeview_int16_int16_cuda_dynamic_shapes_gpu_wrapper, test/inductor/test_gpu_cpp_wrapper.py::DynamicShapesGpuWrapperGpuTests::test_dtypeview_int16_int32_cuda_dynamic_shapes_gpu_wrapper, test/inductor/test_gpu_cpp_wrapper.py::DynamicShapesGpuWrapperGpuTests::test_dtypeview_int16_int8_cuda_dynamic_shapes_gpu_wrapper, test/inductor/test_gpu_cpp_wrapper.py::DynamicShapesGpuWrapperGpuTests::test_dtypeview_int16_uint8_cuda_dynamic_shapes_gpu_wrapper, test/inductor/test_gpu_cpp_wrapper.py::DynamicShapesGpuWrapperGpuTests::test_dtypeview_int32_int32_cuda_dynamic_shapes_gpu_wrapper, test/inductor/test_gpu_cpp_wrapper.py::DynamicShapesGpuWrapperGpuTests::test_dtypeview_int32_uint8_cuda_dynamic_shapes_gpu_wrapper, test/inductor/test_gpu_cpp_wrapper.py::DynamicShapesGpuWrapperGpuTests::test_dtypeview_int64_float64_cuda_dynamic_shapes_gpu_wrapper, test/inductor/test_gpu_cpp_wrapper.py::DynamicShapesGpuWrapperGpuTests::test_dtypeview_int64_int16_cuda_dynamic_shapes_gpu_wrapper, test/inductor/test_gpu_cpp_wrapper.py::DynamicShapesGpuWrapperGpuTests::test_dtypeview_int64_int32_cuda_dynamic_shapes_gpu_wrapper, test/inductor/test_gpu_cpp_wrapper.py::DynamicShapesGpuWrapperGpuTests::test_dtypeview_int64_int8_cuda_dynamic_shapes_gpu_wrapper, test/inductor/test_gpu_cpp_wrapper.py::DynamicShapesGpuWrapperGpuTests::test_dtypeview_int64_uint8_cuda_dynamic_shapes_gpu_wrapper, test/inductor/test_gpu_cpp_wrapper.py::DynamicShapesGpuWrapperGpuTests::test_dtypeview_int8_bfloat16_cuda_dynamic_shapes_gpu_wrapper, test/inductor/test_gpu_cpp_wrapper.py::DynamicShapesGpuWrapperGpuTests::test_dtypeview_int8_int16_cuda_dynamic_shapes_gpu_wrapper, test/inductor/test_gpu_cpp_wrapper.py::DynamicShapesGpuWrapperGpuTests::test_dtypeview_int8_int8_cuda_dynamic_shapes_gpu_wrapper, test/inductor/test_gpu_cpp_wrapper.py::DynamicShapesGpuWrapperGpuTests::test_dtypeview_uint8_float32_cuda_dynamic_shapes_gpu_wrapper, test/inductor/test_gpu_cpp_wrapper.py::DynamicShapesGpuWrapperGpuTests::test_dtypeview_uint8_float64_cuda_dynamic_shapes_gpu_wrapper, test/inductor/test_gpu_cpp_wrapper.py::DynamicShapesGpuWrapperGpuTests::test_dtypeview_uint8_int8_cuda_dynamic_shapes_gpu_wrapper, test/inductor/test_gpu_cpp_wrapper.py::DynamicShapesGpuWrapperGpuTests::test_enable_dynamic_shapes_cpp_wrapper_cuda_dynamic_shapes_gpu_wrapper, test/inductor/test_gpu_cpp_wrapper.py::DynamicShapesGpuWrapperGpuTests::test_foreach_cpp_wrapper_cuda_dynamic_shapes_gpu_wrapper, test/inductor/test_gpu_cpp_wrapper.py::DynamicShapesGpuWrapperGpuTests::test_inductor_layout_optimization_input_mutations_cuda_dynamic_shapes_gpu_wrapper, test/inductor/test_gpu_cpp_wrapper.py::DynamicShapesGpuWrapperGpuTests::test_insignificant_strides_cuda_dynamic_shapes_gpu_wrapper, test/inductor/test_gpu_cpp_wrapper.py::DynamicShapesGpuWrapperGpuTests::test_layer_norm_cuda_dynamic_shapes_gpu_wrapper, test/inductor/test_gpu_cpp_wrapper.py::DynamicShapesGpuWrapperGpuTests::test_linear1_cuda_dynamic_shapes_gpu_wrapper, test/inductor/test_gpu_cpp_wrapper.py::DynamicShapesGpuWrapperGpuTests::test_linear_relu_dynamic_shapes_gpu_wrapper, test/inductor/test_gpu_cpp_wrapper.py::DynamicShapesGpuWrapperGpuTests::test_mm_plus_mm2_dynamic_shapes_gpu_wrapper, test/inductor/test_gpu_cpp_wrapper.py::DynamicShapesGpuWrapperGpuTests::test_mm_plus_mm3_dynamic_shapes_gpu_wrapper, test/inductor/test_gpu_cpp_wrapper.py::DynamicShapesGpuWrapperGpuTests::test_pointwise_hermite_polynomial_he_cuda_dynamic_shapes_gpu_wrapper, test/inductor/test_gpu_cpp_wrapper.py::DynamicShapesGpuWrapperGpuTests::test_pow3_cuda_dynamic_shapes_gpu_wrapper, test/inductor/test_gpu_cpp_wrapper.py::DynamicShapesGpuWrapperGpuTests::test_profiler_mark_wrapper_call_cuda_dynamic_shapes_gpu_wrapper, test/inductor/test_gpu_cpp_wrapper.py::DynamicShapesGpuWrapperGpuTests::test_randint_cuda_dynamic_shapes_gpu_wrapper, test/inductor/test_gpu_cpp_wrapper.py::DynamicShapesGpuWrapperGpuTests::test_repeat_interleave_2_cuda_dynamic_shapes_gpu_wrapper, test/inductor/test_gpu_cpp_wrapper.py::DynamicShapesGpuWrapperGpuTests::test_roi_align_cuda_dynamic_shapes_gpu_wrapper, test/inductor/test_gpu_cpp_wrapper.py::DynamicShapesGpuWrapperGpuTests::test_scaled_dot_product_attention_cuda_dynamic_shapes_gpu_wrapper, test/inductor/test_gpu_cpp_wrapper.py::DynamicShapesGpuWrapperGpuTests::test_silu_cuda_dynamic_shapes_gpu_wrapper, test/inductor/test_gpu_cpp_wrapper.py::DynamicShapesGpuWrapperGpuTests::test_sum_dtype_cuda_dynamic_shapes_gpu_wrapper, test/inductor/test_gpu_cpp_wrapper.py::DynamicShapesGpuWrapperGpuTests::test_sum_int_cuda_dynamic_shapes_gpu_wrapper, test/inductor/test_gpu_cpp_wrapper.py::DynamicShapesGpuWrapperGpuTests::test_unspec_inputs_float32_cuda_dynamic_shapes_gpu_wrapper, test/inductor/test_gpu_cpp_wrapper.py::DynamicShapesGpuWrapperGpuTests::test_unspec_inputs_int16_cuda_dynamic_shapes_gpu_wrapper, test/inductor/test_gpu_cpp_wrapper.py::DynamicShapesGpuWrapperGpuTests::test_unspec_inputs_int32_cuda_dynamic_shapes_gpu_wrapper, test/inductor/test_gpu_cpp_wrapper.py::DynamicShapesGpuWrapperGpuTests::test_unspec_inputs_int8_cuda_dynamic_shapes_gpu_wrapper 2025-12-04T13:20:26.1931734Z 2025-12-04T13:20:26.1932068Z Finished inductor/test_gpu_cpp_wrapper 2/2 ... [2025-12-04 13:20:26.183514][13274.125727867], took 5.17min 2025-12-04T13:20:26.2123331Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/inductor.test_gpu_cpp_wrapper/inductor.test_gpu_cpp_wrapper-5772573b88e52d5b.xml 2025-12-04T13:20:26.2933535Z Running inductor/test_control_deps 1/1 ... [2025-12-04 13:20:26.293102][13274.23531884] 2025-12-04T13:20:26.2934351Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2025-12-04T13:20:26.2936584Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'inductor/test_control_deps.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-12-04 13:20:26.293409] 2025-12-04T13:20:35.6734476Z 2025-12-04T13:20:35.6735697Z inductor/test_control_deps 1/1 was successful, full logs can be found in artifacts with path test/test-reports/inductor.test_control_deps_1.1_3aef2ceb2e6eb739_.log 2025-12-04T13:20:35.6736842Z Running 1 items in this shard: test/inductor/test_control_deps.py::TestControlDeps::test_control_deps_prevents_fusion 2025-12-04T13:20:35.6737614Z 2025-12-04T13:20:35.6738151Z Finished inductor/test_control_deps 1/1 ... [2025-12-04 13:20:35.673016][13283.615228689], took 0.16min 2025-12-04T13:20:35.7016090Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/inductor.test_control_deps/inductor.test_control_deps-2309f470268f6ec0.xml 2025-12-04T13:20:35.7790045Z Running inductor/test_benchmarking 1/1 ... [2025-12-04 13:20:35.778723][13283.720941167] 2025-12-04T13:20:35.7790809Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2025-12-04T13:20:35.7793073Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'inductor/test_benchmarking.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-12-04 13:20:35.779041] 2025-12-04T13:20:42.6049899Z 2025-12-04T13:20:42.6050888Z inductor/test_benchmarking 1/1 was successful, full logs can be found in artifacts with path test/test-reports/inductor.test_benchmarking_1.1_9e4c863097582737_.log 2025-12-04T13:20:42.6056187Z Running 12 items in this shard: test/inductor/test_benchmarking.py::TestBenchmarker::test_benchmark_cpu_smoke_benchmarker_cls0, test/inductor/test_benchmarking.py::TestBenchmarker::test_benchmark_cpu_smoke_benchmarker_cls1, test/inductor/test_benchmarking.py::TestBenchmarker::test_benchmark_gpu_smoke_benchmarker_cls0, test/inductor/test_benchmarking.py::TestBenchmarker::test_benchmark_gpu_smoke_benchmarker_cls1, test/inductor/test_benchmarking.py::TestBenchmarker::test_benchmark_safely_infers_device_many_devices_benchmarker_cls0, test/inductor/test_benchmarking.py::TestBenchmarker::test_benchmark_safely_infers_device_many_devices_benchmarker_cls1, test/inductor/test_benchmarking.py::TestBenchmarker::test_benchmark_safely_infers_device_no_devices_benchmarker_cls0, test/inductor/test_benchmarking.py::TestBenchmarker::test_benchmark_safely_infers_device_no_devices_benchmarker_cls1, test/inductor/test_benchmarking.py::TestBenchmarker::test_benchmark_smoke_benchmarker_cls0_device_cpu, test/inductor/test_benchmarking.py::TestBenchmarker::test_benchmark_smoke_benchmarker_cls0_device_cuda, test/inductor/test_benchmarking.py::TestBenchmarker::test_benchmark_smoke_benchmarker_cls1_device_cpu, test/inductor/test_benchmarking.py::TestBenchmarker::test_benchmark_smoke_benchmarker_cls1_device_cuda 2025-12-04T13:20:42.6060462Z 2025-12-04T13:20:42.6060710Z Finished inductor/test_benchmarking 1/1 ... [2025-12-04 13:20:42.604733][13290.546950577], took 0.11min 2025-12-04T13:20:42.6332043Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/inductor.test_benchmarking/inductor.test_benchmarking-2c8c5f4991e26591.xml 2025-12-04T13:20:42.7117183Z Running inductor/test_best_config 1/1 ... [2025-12-04 13:20:42.711468][13290.653685467] 2025-12-04T13:20:42.7117658Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2025-12-04T13:20:42.7120498Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'inductor/test_best_config.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-12-04 13:20:42.711780] 2025-12-04T13:20:50.2392534Z 2025-12-04T13:20:50.2394308Z inductor/test_best_config 1/1 was successful, full logs can be found in artifacts with path test/test-reports/inductor.test_best_config_1.1_f3ac44d45d5a649c_.log 2025-12-04T13:20:50.2395498Z Running 1 items in this shard: test/inductor/test_best_config.py::TestKernelBestConfig::test_best_config_has_triton_cache_key 2025-12-04T13:20:50.2396010Z 2025-12-04T13:20:50.2396307Z Finished inductor/test_best_config 1/1 ... [2025-12-04 13:20:50.238747][13298.180959052], took 0.13min 2025-12-04T13:20:50.2679538Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/inductor.test_best_config/inductor.test_best_config-0c0fda7542001e9d.xml 2025-12-04T13:20:50.3367823Z Running export/test_tools 1/1 ... [2025-12-04 13:20:50.336507][13298.278725513] 2025-12-04T13:20:50.3368774Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2025-12-04T13:20:50.3371065Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'export/test_tools.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-12-04 13:20:50.336824] 2025-12-04T13:20:54.3083669Z 2025-12-04T13:20:54.3085081Z export/test_tools 1/1 was successful, full logs can be found in artifacts with path test/test-reports/export.test_tools_1.1_b989024045a73d46_.log 2025-12-04T13:20:54.3086348Z Running 2 items in this shard: test/export/test_tools.py::TestExportTools::test_report_exportability_basic, test/export/test_tools.py::TestExportTools::test_report_exportability_with_issues 2025-12-04T13:20:54.3087062Z 2025-12-04T13:20:54.3087336Z Finished export/test_tools 1/1 ... [2025-12-04 13:20:54.308129][13302.250345341], took 0.07min 2025-12-04T13:20:54.3376484Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/export.test_tools/export.test_tools-2cb5c16e1bf3618d.xml 2025-12-04T13:20:54.3679911Z Running inductor/test_compiled_optimizers 1/3 ... [2025-12-04 13:20:54.367752][13302.3099705] 2025-12-04T13:20:54.3680724Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2025-12-04T13:20:54.3684051Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'inductor/test_compiled_optimizers.py', '--shard-id=1', '--num-shards=3', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-12-04 13:20:54.368090] 2025-12-04T13:28:07.2289896Z 2025-12-04T13:28:07.2291121Z inductor/test_compiled_optimizers 1/3 was successful, full logs can be found in artifacts with path test/test-reports/inductor.test_compiled_optimizers_1.3_2096799fbf01ca07_.log 2025-12-04T13:28:07.2383593Z Running 248 items in this shard: test/inductor/test_compiled_optimizers.py::CompiledOptimizerTests::test_adadelta_cpu, test/inductor/test_compiled_optimizers.py::CompiledOptimizerTests::test_adadelta_foreach_cuda, test/inductor/test_compiled_optimizers.py::CompiledOptimizerTests::test_adadelta_maximize_foreach_cuda, test/inductor/test_compiled_optimizers.py::CompiledOptimizerTests::test_adadelta_tensor_lr_capturable_cuda_exponentiallr, test/inductor/test_compiled_optimizers.py::CompiledOptimizerTests::test_adadelta_tensor_lr_capturable_cuda_multiplicativelr, test/inductor/test_compiled_optimizers.py::CompiledOptimizerTests::test_adadelta_tensor_lr_capturable_cuda_polynomiallr, test/inductor/test_compiled_optimizers.py::CompiledOptimizerTests::test_adadelta_tensor_lr_capturable_cuda_reducelronplateau, test/inductor/test_compiled_optimizers.py::CompiledOptimizerTests::test_adadelta_tensor_lr_capturable_cuda_steplr, test/inductor/test_compiled_optimizers.py::CompiledOptimizerTests::test_adadelta_tensor_lr_capturable_foreach_cuda_constantlr, test/inductor/test_compiled_optimizers.py::CompiledOptimizerTests::test_adadelta_tensor_lr_capturable_foreach_cuda_cycliclr, test/inductor/test_compiled_optimizers.py::CompiledOptimizerTests::test_adadelta_tensor_lr_capturable_foreach_cuda_lambdalr, test/inductor/test_compiled_optimizers.py::CompiledOptimizerTests::test_adadelta_tensor_lr_capturable_foreach_cuda_linearlr, test/inductor/test_compiled_optimizers.py::CompiledOptimizerTests::test_adadelta_tensor_lr_capturable_foreach_cuda_multiplicativelr, test/inductor/test_compiled_optimizers.py::CompiledOptimizerTests::test_adadelta_tensor_lr_capturable_foreach_cuda_polynomiallr, test/inductor/test_compiled_optimizers.py::CompiledOptimizerTests::test_adadelta_weight_decay_capturable_foreach_cuda, test/inductor/test_compiled_optimizers.py::CompiledOptimizerTests::test_adadelta_weight_decay_cuda, test/inductor/test_compiled_optimizers.py::CompiledOptimizerTests::test_adagrad_initial_accumulator_value_weight_decay_foreach_cuda, test/inductor/test_compiled_optimizers.py::CompiledOptimizerTests::test_adagrad_lr_decay_weight_decay_cpu, test/inductor/test_compiled_optimizers.py::CompiledOptimizerTests::test_adagrad_tensor_lr_cpu_constantlr, test/inductor/test_compiled_optimizers.py::CompiledOptimizerTests::test_adagrad_tensor_lr_cpu_cosineannealinglr, test/inductor/test_compiled_optimizers.py::CompiledOptimizerTests::test_adagrad_tensor_lr_cpu_cosineannealingwarmrestarts, test/inductor/test_compiled_optimizers.py::CompiledOptimizerTests::test_adagrad_tensor_lr_cpu_cycliclr, test/inductor/test_compiled_optimizers.py::CompiledOptimizerTests::test_adagrad_tensor_lr_cpu_linearlr, test/inductor/test_compiled_optimizers.py::CompiledOptimizerTests::test_adagrad_tensor_lr_cpu_multisteplr, test/inductor/test_compiled_optimizers.py::CompiledOptimizerTests::test_adagrad_tensor_lr_cpu_polynomiallr, test/inductor/test_compiled_optimizers.py::CompiledOptimizerTests::test_adagrad_tensor_lr_cpu_reducelronplateau, test/inductor/test_compiled_optimizers.py::CompiledOptimizerTests::test_adagrad_tensor_lr_cuda_constantlr, test/inductor/test_compiled_optimizers.py::CompiledOptimizerTests::test_adagrad_tensor_lr_cuda_exponentiallr, test/inductor/test_compiled_optimizers.py::CompiledOptimizerTests::test_adagrad_tensor_lr_cuda_multisteplr, test/inductor/test_compiled_optimizers.py::CompiledOptimizerTests::test_adagrad_tensor_lr_cuda_onecyclelr, test/inductor/test_compiled_optimizers.py::CompiledOptimizerTests::test_adagrad_tensor_lr_cuda_reducelronplateau, test/inductor/test_compiled_optimizers.py::CompiledOptimizerTests::test_adagrad_tensor_lr_foreach_cuda_cycliclr, test/inductor/test_compiled_optimizers.py::CompiledOptimizerTests::test_adagrad_tensor_lr_foreach_cuda_exponentiallr, test/inductor/test_compiled_optimizers.py::CompiledOptimizerTests::test_adagrad_tensor_lr_foreach_cuda_reducelronplateau, test/inductor/test_compiled_optimizers.py::CompiledOptimizerTests::test_adagrad_tensor_lr_foreach_cuda_steplr, test/inductor/test_compiled_optimizers.py::CompiledOptimizerTests::test_adagrad_weight_decay_cpu, test/inductor/test_compiled_optimizers.py::CompiledOptimizerTests::test_adam_capturable_foreach_cuda, test/inductor/test_compiled_optimizers.py::CompiledOptimizerTests::test_adam_cpu, test/inductor/test_compiled_optimizers.py::CompiledOptimizerTests::test_adam_cuda, test/inductor/test_compiled_optimizers.py::CompiledOptimizerTests::test_adam_tensor_lr_amsgrad_capturable_cuda_constantlr, test/inductor/test_compiled_optimizers.py::CompiledOptimizerTests::test_adam_tensor_lr_amsgrad_capturable_cuda_cosineannealinglr, test/inductor/test_compiled_optimizers.py::CompiledOptimizerTests::test_adam_tensor_lr_amsgrad_capturable_cuda_cosineannealingwarmrestarts, test/inductor/test_compiled_optimizers.py::CompiledOptimizerTests::test_adam_tensor_lr_amsgrad_capturable_cuda_linearlr, test/inductor/test_compiled_optimizers.py::CompiledOptimizerTests::test_adam_tensor_lr_amsgrad_capturable_cuda_multiplicativelr, test/inductor/test_compiled_optimizers.py::CompiledOptimizerTests::test_adam_tensor_lr_amsgrad_capturable_cuda_multisteplr, test/inductor/test_compiled_optimizers.py::CompiledOptimizerTests::test_adam_tensor_lr_amsgrad_capturable_cuda_onecyclelr, test/inductor/test_compiled_optimizers.py::CompiledOptimizerTests::test_adam_tensor_lr_amsgrad_capturable_cuda_steplr, test/inductor/test_compiled_optimizers.py::CompiledOptimizerTests::test_adam_tensor_lr_amsgrad_capturable_foreach_cuda_cosineannealinglr, test/inductor/test_compiled_optimizers.py::CompiledOptimizerTests::test_adam_tensor_lr_amsgrad_capturable_foreach_cuda_cosineannealingwarmrestarts, test/inductor/test_compiled_optimizers.py::CompiledOptimizerTests::test_adam_tensor_lr_amsgrad_capturable_foreach_cuda_cycliclr, test/inductor/test_compiled_optimizers.py::CompiledOptimizerTests::test_adam_tensor_lr_amsgrad_capturable_foreach_cuda_exponentiallr, test/inductor/test_compiled_optimizers.py::CompiledOptimizerTests::test_adam_tensor_lr_amsgrad_capturable_foreach_cuda_linearlr, test/inductor/test_compiled_optimizers.py::CompiledOptimizerTests::test_adam_tensor_lr_amsgrad_capturable_foreach_cuda_multiplicativelr, test/inductor/test_compiled_optimizers.py::CompiledOptimizerTests::test_adam_tensor_lr_amsgrad_capturable_foreach_cuda_steplr, test/inductor/test_compiled_optimizers.py::CompiledOptimizerTests::test_adam_tensor_lr_tensor_betas_amsgrad_capturable_cuda_constantlr, test/inductor/test_compiled_optimizers.py::CompiledOptimizerTests::test_adam_tensor_lr_tensor_betas_amsgrad_capturable_cuda_cosineannealingwarmrestarts, test/inductor/test_compiled_optimizers.py::CompiledOptimizerTests::test_adam_tensor_lr_tensor_betas_amsgrad_capturable_cuda_exponentiallr, test/inductor/test_compiled_optimizers.py::CompiledOptimizerTests::test_adam_tensor_lr_tensor_betas_amsgrad_capturable_cuda_lambdalr, test/inductor/test_compiled_optimizers.py::CompiledOptimizerTests::test_adam_tensor_lr_tensor_betas_amsgrad_capturable_cuda_linearlr, test/inductor/test_compiled_optimizers.py::CompiledOptimizerTests::test_adam_tensor_lr_tensor_betas_amsgrad_capturable_foreach_cuda_cosineannealingwarmrestarts, test/inductor/test_compiled_optimizers.py::CompiledOptimizerTests::test_adam_tensor_lr_tensor_betas_amsgrad_capturable_foreach_cuda_lambdalr, test/inductor/test_compiled_optimizers.py::CompiledOptimizerTests::test_adam_tensor_lr_tensor_betas_amsgrad_capturable_foreach_cuda_multiplicativelr, test/inductor/test_compiled_optimizers.py::CompiledOptimizerTests::test_adam_tensor_lr_tensor_betas_amsgrad_capturable_foreach_cuda_multisteplr, test/inductor/test_compiled_optimizers.py::CompiledOptimizerTests::test_adam_tensor_lr_tensor_betas_amsgrad_capturable_foreach_cuda_onecyclelr, test/inductor/test_compiled_optimizers.py::CompiledOptimizerTests::test_adam_tensor_lr_tensor_betas_amsgrad_capturable_foreach_cuda_reducelronplateau, test/inductor/test_compiled_optimizers.py::CompiledOptimizerTests::test_adam_tensor_lr_tensor_betas_capturable_cuda_cosineannealinglr, test/inductor/test_compiled_optimizers.py::CompiledOptimizerTests::test_adam_tensor_lr_tensor_betas_capturable_cuda_cosineannealingwarmrestarts, test/inductor/test_compiled_optimizers.py::CompiledOptimizerTests::test_adam_tensor_lr_tensor_betas_capturable_cuda_exponentiallr, test/inductor/test_compiled_optimizers.py::CompiledOptimizerTests::test_adam_tensor_lr_tensor_betas_capturable_cuda_onecyclelr, test/inductor/test_compiled_optimizers.py::CompiledOptimizerTests::test_adam_tensor_lr_tensor_betas_capturable_cuda_reducelronplateau, test/inductor/test_compiled_optimizers.py::CompiledOptimizerTests::test_adam_tensor_lr_tensor_betas_capturable_foreach_cuda_cosineannealinglr, test/inductor/test_compiled_optimizers.py::CompiledOptimizerTests::test_adam_tensor_lr_tensor_betas_capturable_foreach_cuda_exponentiallr, test/inductor/test_compiled_optimizers.py::CompiledOptimizerTests::test_adam_tensor_lr_tensor_betas_capturable_foreach_cuda_polynomiallr, test/inductor/test_compiled_optimizers.py::CompiledOptimizerTests::test_adam_weight_decay_amsgrad_cpu, test/inductor/test_compiled_optimizers.py::CompiledOptimizerTests::test_adam_weight_decay_amsgrad_foreach_cuda, test/inductor/test_compiled_optimizers.py::CompiledOptimizerTests::test_adam_weight_decay_maximize_foreach_cuda, test/inductor/test_compiled_optimizers.py::CompiledOptimizerTests::test_adamax_maximize_foreach_cuda, test/inductor/test_compiled_optimizers.py::CompiledOptimizerTests::test_adamax_tensor_lr_weight_decay_capturable_cuda_cosineannealingwarmrestarts, test/inductor/test_compiled_optimizers.py::CompiledOptimizerTests::test_adamax_tensor_lr_weight_decay_capturable_cuda_multisteplr, test/inductor/test_compiled_optimizers.py::CompiledOptimizerTests::test_adamax_tensor_lr_weight_decay_capturable_cuda_polynomiallr, test/inductor/test_compiled_optimizers.py::CompiledOptimizerTests::test_adamax_tensor_lr_weight_decay_capturable_cuda_steplr, test/inductor/test_compiled_optimizers.py::CompiledOptimizerTests::test_adamax_tensor_lr_weight_decay_capturable_foreach_cuda_cosineannealingwarmrestarts, test/inductor/test_compiled_optimizers.py::CompiledOptimizerTests::test_adamax_tensor_lr_weight_decay_capturable_foreach_cuda_exponentiallr, test/inductor/test_compiled_optimizers.py::CompiledOptimizerTests::test_adamax_tensor_lr_weight_decay_capturable_foreach_cuda_lambdalr, test/inductor/test_compiled_optimizers.py::CompiledOptimizerTests::test_adamax_tensor_lr_weight_decay_capturable_foreach_cuda_linearlr, test/inductor/test_compiled_optimizers.py::CompiledOptimizerTests::test_adamax_tensor_lr_weight_decay_capturable_foreach_cuda_multisteplr, test/inductor/test_compiled_optimizers.py::CompiledOptimizerTests::test_adamax_tensor_lr_weight_decay_capturable_foreach_cuda_steplr, test/inductor/test_compiled_optimizers.py::CompiledOptimizerTests::test_adamax_weight_decay_capturable_cuda, test/inductor/test_compiled_optimizers.py::CompiledOptimizerTests::test_adamax_weight_decay_maximize_cpu, test/inductor/test_compiled_optimizers.py::CompiledOptimizerTests::test_adamw_capturable_foreach_cuda, test/inductor/test_compiled_optimizers.py::CompiledOptimizerTests::test_adamw_cpu, test/inductor/test_compiled_optimizers.py::CompiledOptimizerTests::test_adamw_tensor_lr_amsgrad_capturable_cuda_cosineannealingwarmrestarts, test/inductor/test_compiled_optimizers.py::CompiledOptimizerTests::test_adamw_tensor_lr_amsgrad_capturable_cuda_exponentiallr, test/inductor/test_compiled_optimizers.py::CompiledOptimizerTests::test_adamw_tensor_lr_amsgrad_capturable_cuda_lambdalr, test/inductor/test_compiled_optimizers.py::CompiledOptimizerTests::test_adamw_tensor_lr_amsgrad_capturable_cuda_multisteplr, test/inductor/test_compiled_optimizers.py::CompiledOptimizerTests::test_adamw_tensor_lr_amsgrad_capturable_cuda_reducelronplateau, test/inductor/test_compiled_optimizers.py::CompiledOptimizerTests::test_adamw_tensor_lr_amsgrad_capturable_foreach_cuda_cosineannealinglr, test/inductor/test_compiled_optimizers.py::CompiledOptimizerTests::test_adamw_tensor_lr_amsgrad_capturable_foreach_cuda_cosineannealingwarmrestarts, test/inductor/test_compiled_optimizers.py::CompiledOptimizerTests::test_adamw_tensor_lr_amsgrad_capturable_foreach_cuda_multiplicativelr, test/inductor/test_compiled_optimizers.py::CompiledOptimizerTests::test_adamw_tensor_lr_amsgrad_capturable_foreach_cuda_multisteplr, test/inductor/test_compiled_optimizers.py::CompiledOptimizerTests::test_adamw_tensor_lr_tensor_betas_amsgrad_capturable_cuda_cosineannealinglr, test/inductor/test_compiled_optimizers.py::CompiledOptimizerTests::test_adamw_tensor_lr_tensor_betas_amsgrad_capturable_cuda_cycliclr, test/inductor/test_compiled_optimizers.py::CompiledOptimizerTests::test_adamw_tensor_lr_tensor_betas_amsgrad_capturable_cuda_linearlr, test/inductor/test_compiled_optimizers.py::CompiledOptimizerTests::test_adamw_tensor_lr_tensor_betas_amsgrad_capturable_foreach_cuda_constantlr, test/inductor/test_compiled_optimizers.py::CompiledOptimizerTests::test_adamw_tensor_lr_tensor_betas_amsgrad_capturable_foreach_cuda_cosineannealingwarmrestarts, test/inductor/test_compiled_optimizers.py::CompiledOptimizerTests::test_adamw_tensor_lr_tensor_betas_amsgrad_capturable_foreach_cuda_multiplicativelr, test/inductor/test_compiled_optimizers.py::CompiledOptimizerTests::test_adamw_tensor_lr_tensor_betas_amsgrad_capturable_foreach_cuda_onecyclelr, test/inductor/test_compiled_optimizers.py::CompiledOptimizerTests::test_adamw_tensor_lr_tensor_betas_amsgrad_capturable_foreach_cuda_polynomiallr, test/inductor/test_compiled_optimizers.py::CompiledOptimizerTests::test_adamw_tensor_lr_tensor_betas_amsgrad_capturable_foreach_cuda_reducelronplateau, test/inductor/test_compiled_optimizers.py::CompiledOptimizerTests::test_adamw_tensor_lr_tensor_betas_capturable_cuda_cosineannealinglr, test/inductor/test_compiled_optimizers.py::CompiledOptimizerTests::test_adamw_tensor_lr_tensor_betas_capturable_cuda_cosineannealingwarmrestarts, test/inductor/test_compiled_optimizers.py::CompiledOptimizerTests::test_adamw_tensor_lr_tensor_betas_capturable_cuda_exponentiallr, test/inductor/test_compiled_optimizers.py::CompiledOptimizerTests::test_adamw_tensor_lr_tensor_betas_capturable_cuda_lambdalr, test/inductor/test_compiled_optimizers.py::CompiledOptimizerTests::test_adamw_tensor_lr_tensor_betas_capturable_cuda_linearlr, test/inductor/test_compiled_optimizers.py::CompiledOptimizerTests::test_adamw_tensor_lr_tensor_betas_capturable_cuda_multisteplr, test/inductor/test_compiled_optimizers.py::CompiledOptimizerTests::test_adamw_tensor_lr_tensor_betas_capturable_foreach_cuda_constantlr, test/inductor/test_compiled_optimizers.py::CompiledOptimizerTests::test_adamw_tensor_lr_tensor_betas_capturable_foreach_cuda_cycliclr, test/inductor/test_compiled_optimizers.py::CompiledOptimizerTests::test_adamw_tensor_lr_tensor_betas_capturable_foreach_cuda_lambdalr, test/inductor/test_compiled_optimizers.py::CompiledOptimizerTests::test_adamw_tensor_lr_tensor_betas_capturable_foreach_cuda_linearlr, test/inductor/test_compiled_optimizers.py::CompiledOptimizerTests::test_adamw_tensor_lr_tensor_betas_capturable_foreach_cuda_multisteplr, test/inductor/test_compiled_optimizers.py::CompiledOptimizerTests::test_adamw_tensor_lr_tensor_betas_capturable_foreach_cuda_onecyclelr, test/inductor/test_compiled_optimizers.py::CompiledOptimizerTests::test_adamw_weight_decay_amsgrad_foreach_cuda, test/inductor/test_compiled_optimizers.py::CompiledOptimizerTests::test_adamw_weight_decay_cpu, test/inductor/test_compiled_optimizers.py::CompiledOptimizerTests::test_adamw_weight_decay_maximize_foreach_cuda, test/inductor/test_compiled_optimizers.py::CompiledOptimizerTests::test_asgd_capturable_foreach_cuda, test/inductor/test_compiled_optimizers.py::CompiledOptimizerTests::test_asgd_cpu, test/inductor/test_compiled_optimizers.py::CompiledOptimizerTests::test_asgd_cuda, test/inductor/test_compiled_optimizers.py::CompiledOptimizerTests::test_asgd_foreach_cuda, test/inductor/test_compiled_optimizers.py::CompiledOptimizerTests::test_asgd_maximize_capturable_cuda, test/inductor/test_compiled_optimizers.py::CompiledOptimizerTests::test_asgd_maximize_cpu, test/inductor/test_compiled_optimizers.py::CompiledOptimizerTests::test_asgd_maximize_foreach_cuda, test/inductor/test_compiled_optimizers.py::CompiledOptimizerTests::test_asgd_t0_cuda, test/inductor/test_compiled_optimizers.py::CompiledOptimizerTests::test_asgd_tensor_lr_weight_decay_maximize_capturable_cuda_constantlr, test/inductor/test_compiled_optimizers.py::CompiledOptimizerTests::test_asgd_tensor_lr_weight_decay_maximize_capturable_cuda_cosineannealinglr, test/inductor/test_compiled_optimizers.py::CompiledOptimizerTests::test_asgd_tensor_lr_weight_decay_maximize_capturable_cuda_lambdalr, test/inductor/test_compiled_optimizers.py::CompiledOptimizerTests::test_asgd_tensor_lr_weight_decay_maximize_capturable_cuda_multiplicativelr, test/inductor/test_compiled_optimizers.py::CompiledOptimizerTests::test_asgd_tensor_lr_weight_decay_maximize_capturable_cuda_reducelronplateau, test/inductor/test_compiled_optimizers.py::CompiledOptimizerTests::test_asgd_tensor_lr_weight_decay_maximize_capturable_cuda_steplr, test/inductor/test_compiled_optimizers.py::CompiledOptimizerTests::test_asgd_tensor_lr_weight_decay_maximize_capturable_foreach_cuda_cosineannealinglr, test/inductor/test_compiled_optimizers.py::CompiledOptimizerTests::test_asgd_tensor_lr_weight_decay_maximize_capturable_foreach_cuda_cycliclr, test/inductor/test_compiled_optimizers.py::CompiledOptimizerTests::test_asgd_tensor_lr_weight_decay_maximize_capturable_foreach_cuda_exponentiallr, test/inductor/test_compiled_optimizers.py::CompiledOptimizerTests::test_asgd_tensor_lr_weight_decay_maximize_capturable_foreach_cuda_lambdalr, test/inductor/test_compiled_optimizers.py::CompiledOptimizerTests::test_asgd_tensor_lr_weight_decay_maximize_capturable_foreach_cuda_linearlr, test/inductor/test_compiled_optimizers.py::CompiledOptimizerTests::test_asgd_tensor_lr_weight_decay_maximize_capturable_foreach_cuda_multiplicativelr, test/inductor/test_compiled_optimizers.py::CompiledOptimizerTests::test_asgd_tensor_lr_weight_decay_maximize_capturable_foreach_cuda_multisteplr, test/inductor/test_compiled_optimizers.py::CompiledOptimizerTests::test_asgd_tensor_lr_weight_decay_maximize_capturable_foreach_cuda_onecyclelr, test/inductor/test_compiled_optimizers.py::CompiledOptimizerTests::test_asgd_tensor_lr_weight_decay_maximize_capturable_foreach_cuda_reducelronplateau, test/inductor/test_compiled_optimizers.py::CompiledOptimizerTests::test_asgd_weight_decay_capturable_cuda, test/inductor/test_compiled_optimizers.py::CompiledOptimizerTests::test_asgd_weight_decay_maximize_capturable_cuda, test/inductor/test_compiled_optimizers.py::CompiledOptimizerTests::test_closure_graph_break, test/inductor/test_compiled_optimizers.py::CompiledOptimizerTests::test_foreach_map_adam, test/inductor/test_compiled_optimizers.py::CompiledOptimizerTests::test_nadam_foreach_cuda, test/inductor/test_compiled_optimizers.py::CompiledOptimizerTests::test_nadam_momentum_decay_foreach_cuda, test/inductor/test_compiled_optimizers.py::CompiledOptimizerTests::test_nadam_tensor_lr_weight_decay_momentum_decay_decoupled_weight_decay_capturable_cuda_multiplicativelr, test/inductor/test_compiled_optimizers.py::CompiledOptimizerTests::test_nadam_tensor_lr_weight_decay_momentum_decay_decoupled_weight_decay_capturable_cuda_multisteplr, test/inductor/test_compiled_optimizers.py::CompiledOptimizerTests::test_nadam_tensor_lr_weight_decay_momentum_decay_decoupled_weight_decay_capturable_foreach_cuda_cosineannealinglr, test/inductor/test_compiled_optimizers.py::CompiledOptimizerTests::test_nadam_tensor_lr_weight_decay_momentum_decay_decoupled_weight_decay_capturable_foreach_cuda_cosineannealingwarmrestarts, test/inductor/test_compiled_optimizers.py::CompiledOptimizerTests::test_nadam_tensor_lr_weight_decay_momentum_decay_decoupled_weight_decay_capturable_foreach_cuda_exponentiallr, test/inductor/test_compiled_optimizers.py::CompiledOptimizerTests::test_nadam_weight_decay_cpu, test/inductor/test_compiled_optimizers.py::CompiledOptimizerTests::test_nadam_weight_decay_foreach_cuda, test/inductor/test_compiled_optimizers.py::CompiledOptimizerTests::test_nadam_weight_decay_maximize_cuda, test/inductor/test_compiled_optimizers.py::CompiledOptimizerTests::test_nadam_weight_decay_momentum_decay_decoupled_weight_decay_capturable_cuda, test/inductor/test_compiled_optimizers.py::CompiledOptimizerTests::test_nadam_weight_decay_momentum_decay_decoupled_weight_decay_cpu, test/inductor/test_compiled_optimizers.py::CompiledOptimizerTests::test_nadam_weight_decay_momentum_decay_decoupled_weight_decay_foreach_cuda, test/inductor/test_compiled_optimizers.py::CompiledOptimizerTests::test_radam_capturable_cuda, test/inductor/test_compiled_optimizers.py::CompiledOptimizerTests::test_radam_capturable_foreach_cuda, test/inductor/test_compiled_optimizers.py::CompiledOptimizerTests::test_radam_capturable_weight_decay_cuda, test/inductor/test_compiled_optimizers.py::CompiledOptimizerTests::test_radam_capturable_weight_decay_foreach_cuda, test/inductor/test_compiled_optimizers.py::CompiledOptimizerTests::test_radam_cpu, test/inductor/test_compiled_optimizers.py::CompiledOptimizerTests::test_radam_tensor_lr_capturable_weight_decay_decoupled_weight_decay_cuda_linearlr, test/inductor/test_compiled_optimizers.py::CompiledOptimizerTests::test_radam_tensor_lr_capturable_weight_decay_decoupled_weight_decay_cuda_multisteplr, test/inductor/test_compiled_optimizers.py::CompiledOptimizerTests::test_radam_tensor_lr_capturable_weight_decay_decoupled_weight_decay_foreach_cuda_exponentiallr, test/inductor/test_compiled_optimizers.py::CompiledOptimizerTests::test_radam_tensor_lr_capturable_weight_decay_decoupled_weight_decay_foreach_cuda_lambdalr, test/inductor/test_compiled_optimizers.py::CompiledOptimizerTests::test_radam_tensor_lr_capturable_weight_decay_decoupled_weight_decay_foreach_cuda_multiplicativelr, test/inductor/test_compiled_optimizers.py::CompiledOptimizerTests::test_radam_tensor_lr_capturable_weight_decay_decoupled_weight_decay_foreach_cuda_multisteplr, test/inductor/test_compiled_optimizers.py::CompiledOptimizerTests::test_radam_weight_decay_decoupled_weight_decay_cuda, test/inductor/test_compiled_optimizers.py::CompiledOptimizerTests::test_radam_weight_decay_decoupled_weight_decay_foreach_cuda, test/inductor/test_compiled_optimizers.py::CompiledOptimizerTests::test_radam_weight_decay_maximize_cpu, test/inductor/test_compiled_optimizers.py::CompiledOptimizerTests::test_rmsprop_cuda, test/inductor/test_compiled_optimizers.py::CompiledOptimizerTests::test_rmsprop_maximize_cpu, test/inductor/test_compiled_optimizers.py::CompiledOptimizerTests::test_rmsprop_maximize_foreach_cuda, test/inductor/test_compiled_optimizers.py::CompiledOptimizerTests::test_rmsprop_maximize_weight_decay_cuda, test/inductor/test_compiled_optimizers.py::CompiledOptimizerTests::test_rmsprop_tensor_lr_capturable_cuda_constantlr, test/inductor/test_compiled_optimizers.py::CompiledOptimizerTests::test_rmsprop_tensor_lr_capturable_cuda_exponentiallr, test/inductor/test_compiled_optimizers.py::CompiledOptimizerTests::test_rmsprop_tensor_lr_capturable_cuda_linearlr, test/inductor/test_compiled_optimizers.py::CompiledOptimizerTests::test_rmsprop_tensor_lr_capturable_foreach_cuda_cosineannealinglr, test/inductor/test_compiled_optimizers.py::CompiledOptimizerTests::test_rmsprop_tensor_lr_capturable_foreach_cuda_cycliclr, test/inductor/test_compiled_optimizers.py::CompiledOptimizerTests::test_rmsprop_tensor_lr_capturable_foreach_cuda_polynomiallr, test/inductor/test_compiled_optimizers.py::CompiledOptimizerTests::test_rmsprop_tensor_lr_capturable_foreach_cuda_steplr, test/inductor/test_compiled_optimizers.py::CompiledOptimizerTests::test_rmsprop_weight_decay_centered_foreach_cuda, test/inductor/test_compiled_optimizers.py::CompiledOptimizerTests::test_rmsprop_weight_decay_maximize_capturable_cuda, test/inductor/test_compiled_optimizers.py::CompiledOptimizerTests::test_rmsprop_weight_decay_maximize_capturable_foreach_cuda, test/inductor/test_compiled_optimizers.py::CompiledOptimizerTests::test_rprop_capturable_cuda, test/inductor/test_compiled_optimizers.py::CompiledOptimizerTests::test_rprop_maximize_cpu, test/inductor/test_compiled_optimizers.py::CompiledOptimizerTests::test_rprop_step_sizes_foreach_cuda, test/inductor/test_compiled_optimizers.py::CompiledOptimizerTests::test_rprop_tensor_lr_capturable_cuda_constantlr, test/inductor/test_compiled_optimizers.py::CompiledOptimizerTests::test_rprop_tensor_lr_capturable_cuda_cosineannealinglr, test/inductor/test_compiled_optimizers.py::CompiledOptimizerTests::test_rprop_tensor_lr_capturable_cuda_cycliclr, test/inductor/test_compiled_optimizers.py::CompiledOptimizerTests::test_rprop_tensor_lr_capturable_cuda_linearlr, test/inductor/test_compiled_optimizers.py::CompiledOptimizerTests::test_rprop_tensor_lr_capturable_cuda_multiplicativelr, test/inductor/test_compiled_optimizers.py::CompiledOptimizerTests::test_rprop_tensor_lr_capturable_cuda_polynomiallr, test/inductor/test_compiled_optimizers.py::CompiledOptimizerTests::test_rprop_tensor_lr_capturable_cuda_reducelronplateau, test/inductor/test_compiled_optimizers.py::CompiledOptimizerTests::test_rprop_tensor_lr_capturable_foreach_cuda_cycliclr, test/inductor/test_compiled_optimizers.py::CompiledOptimizerTests::test_rprop_tensor_lr_capturable_foreach_cuda_exponentiallr, test/inductor/test_compiled_optimizers.py::CompiledOptimizerTests::test_rprop_tensor_lr_capturable_foreach_cuda_linearlr, test/inductor/test_compiled_optimizers.py::CompiledOptimizerTests::test_rprop_tensor_lr_capturable_foreach_cuda_multisteplr, test/inductor/test_compiled_optimizers.py::CompiledOptimizerTests::test_rprop_tensor_lr_capturable_foreach_cuda_polynomiallr, test/inductor/test_compiled_optimizers.py::CompiledOptimizerTests::test_rprop_tensor_lr_capturable_foreach_cuda_reducelronplateau, test/inductor/test_compiled_optimizers.py::CompiledOptimizerTests::test_rprop_tensor_lr_capturable_foreach_cuda_steplr, test/inductor/test_compiled_optimizers.py::CompiledOptimizerTests::test_sgd_cuda, test/inductor/test_compiled_optimizers.py::CompiledOptimizerTests::test_sgd_momentum_cpu, test/inductor/test_compiled_optimizers.py::CompiledOptimizerTests::test_sgd_momentum_foreach_cuda, test/inductor/test_compiled_optimizers.py::CompiledOptimizerTests::test_sgd_momentum_weight_decay_cpu, test/inductor/test_compiled_optimizers.py::CompiledOptimizerTests::test_sgd_momentum_weight_decay_cuda, test/inductor/test_compiled_optimizers.py::CompiledOptimizerTests::test_sgd_momentum_weight_decay_foreach_cuda, test/inductor/test_compiled_optimizers.py::CompiledOptimizerTests::test_sgd_recompile_single, test/inductor/test_compiled_optimizers.py::CompiledOptimizerTests::test_sgd_tensor_lr_cpu_constantlr, test/inductor/test_compiled_optimizers.py::CompiledOptimizerTests::test_sgd_tensor_lr_cpu_cosineannealinglr, test/inductor/test_compiled_optimizers.py::CompiledOptimizerTests::test_sgd_tensor_lr_cpu_exponentiallr, test/inductor/test_compiled_optimizers.py::CompiledOptimizerTests::test_sgd_tensor_lr_cpu_multisteplr, test/inductor/test_compiled_optimizers.py::CompiledOptimizerTests::test_sgd_tensor_lr_cpu_polynomiallr, test/inductor/test_compiled_optimizers.py::CompiledOptimizerTests::test_sgd_tensor_lr_cpu_reducelronplateau, test/inductor/test_compiled_optimizers.py::CompiledOptimizerTests::test_sgd_tensor_lr_cpu_steplr, test/inductor/test_compiled_optimizers.py::CompiledOptimizerTests::test_sgd_tensor_lr_cuda_cosineannealingwarmrestarts, test/inductor/test_compiled_optimizers.py::CompiledOptimizerTests::test_sgd_tensor_lr_cuda_lambdalr, test/inductor/test_compiled_optimizers.py::CompiledOptimizerTests::test_sgd_tensor_lr_cuda_multisteplr, test/inductor/test_compiled_optimizers.py::CompiledOptimizerTests::test_sgd_tensor_lr_cuda_polynomiallr, test/inductor/test_compiled_optimizers.py::CompiledOptimizerTests::test_sgd_tensor_lr_cuda_reducelronplateau, test/inductor/test_compiled_optimizers.py::CompiledOptimizerTests::test_sgd_tensor_lr_foreach_cuda_constantlr, test/inductor/test_compiled_optimizers.py::CompiledOptimizerTests::test_sgd_tensor_lr_foreach_cuda_cycliclr, test/inductor/test_compiled_optimizers.py::CompiledOptimizerTests::test_sgd_tensor_lr_foreach_cuda_linearlr, test/inductor/test_compiled_optimizers.py::CompiledOptimizerTests::test_sgd_tensor_lr_foreach_cuda_multisteplr, test/inductor/test_compiled_optimizers.py::CompiledOptimizerTests::test_sgd_tensor_lr_foreach_cuda_onecyclelr, test/inductor/test_compiled_optimizers.py::CompiledOptimizerTests::test_sgd_tensor_lr_foreach_cuda_steplr, test/inductor/test_compiled_optimizers.py::CompiledOptimizerTests::test_sgd_weight_decay_maximize_cpu, test/inductor/test_compiled_optimizers.py::CompiledOptimizerParityTestsCUDA::test_correctness_ASGD_use_closure_True_cuda_float32, test/inductor/test_compiled_optimizers.py::CompiledOptimizerParityTestsCUDA::test_correctness_Adafactor_use_closure_False_cuda_float32, test/inductor/test_compiled_optimizers.py::CompiledOptimizerParityTestsCUDA::test_correctness_Adafactor_use_closure_True_cuda_float32, test/inductor/test_compiled_optimizers.py::CompiledOptimizerParityTestsCUDA::test_correctness_Adagrad_use_closure_True_cuda_float32, test/inductor/test_compiled_optimizers.py::CompiledOptimizerParityTestsCUDA::test_correctness_Adam_use_closure_True_cuda_float32, test/inductor/test_compiled_optimizers.py::CompiledOptimizerParityTestsCUDA::test_correctness_Adamax_use_closure_False_cuda_float32, test/inductor/test_compiled_optimizers.py::CompiledOptimizerParityTestsCUDA::test_correctness_LBFGS_use_closure_False_cuda_float32, test/inductor/test_compiled_optimizers.py::CompiledOptimizerParityTestsCUDA::test_correctness_Muon_use_closure_False_cuda_float32, test/inductor/test_compiled_optimizers.py::CompiledOptimizerParityTestsCUDA::test_correctness_NAdam_use_closure_False_cuda_float32, test/inductor/test_compiled_optimizers.py::CompiledOptimizerParityTestsCUDA::test_correctness_RAdam_use_closure_True_cuda_float32, test/inductor/test_compiled_optimizers.py::CompiledOptimizerParityTestsCUDA::test_correctness_RMSprop_use_closure_True_cuda_float32, test/inductor/test_compiled_optimizers.py::CompiledOptimizerParityTestsCUDA::test_correctness_Rprop_use_closure_False_cuda_float32, test/inductor/test_compiled_optimizers.py::CompiledOptimizerParityTestsCUDA::test_correctness_SparseAdam_use_closure_False_cuda_float32 2025-12-04T13:28:07.2471437Z 2025-12-04T13:28:07.2471696Z Finished inductor/test_compiled_optimizers 1/3 ... [2025-12-04 13:28:07.229291][13735.171504463], took 7.21min 2025-12-04T13:28:07.2581867Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/inductor.test_compiled_optimizers/inductor.test_compiled_optimizers-c666cafe979acadb.xml 2025-12-04T13:28:07.3496607Z Running inductor/test_control_flow 5/5 ... [2025-12-04 13:28:07.349400][13735.291617333] 2025-12-04T13:28:07.3497088Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2025-12-04T13:28:07.3499914Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'inductor/test_control_flow.py', '--shard-id=5', '--num-shards=5', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-12-04 13:28:07.349713] 2025-12-04T13:37:37.6278250Z 2025-12-04T13:37:37.6279123Z inductor/test_control_flow 5/5 was successful, full logs can be found in artifacts with path test/test-reports/inductor.test_control_flow_5.5_8fba914687b14422_.log 2025-12-04T13:37:37.6382366Z Running 162 items in this shard: test/inductor/test_control_flow.py::CondTests::test_cond_functional_call_device_cuda_dynamic_True, test/inductor/test_control_flow.py::CondTests::test_cond_multiple_outputs_device_cuda_dynamic_False, test/inductor/test_control_flow.py::CondTests::test_cond_nested_control_flow_device_cpu_dynamic_True, test/inductor/test_control_flow.py::CondTests::test_cond_nested_control_flow_device_cuda_dynamic_True, test/inductor/test_control_flow.py::CondTests::test_cond_non_tensor_predicates_device_cpu_dynamic_True, test/inductor/test_control_flow.py::CondTests::test_cond_outer_code_before_after_device_cuda_dynamic_True, test/inductor/test_control_flow.py::CondTests::test_cond_reintepret_view_inputs_outputs, test/inductor/test_control_flow.py::CondTests::test_cond_select_with_input_idx_device_cpu_dynamic_False, test/inductor/test_control_flow.py::CondTests::test_cond_simple_control_flow_device_cpu_dynamic_False, test/inductor/test_control_flow.py::CondTests::test_cond_subgraphs_with_parameters_device_cpu_dynamic_True, test/inductor/test_control_flow.py::CondTests::test_cond_subgraphs_with_parameters_device_cuda_dynamic_True, test/inductor/test_control_flow.py::CondTests::test_cond_unbacked_symint_inner_to_outer_device_cpu, test/inductor/test_control_flow.py::CondTests::test_cond_use_buffers_from_outer_scope, test/inductor/test_control_flow.py::CondTests::test_output_on_different_device, test/inductor/test_control_flow.py::WhileLoopTests::test_while_loop_nested_control_flow_device_cpu_dynamic_False_autograd_False, test/inductor/test_control_flow.py::WhileLoopTests::test_while_loop_nested_control_flow_device_cpu_dynamic_True_autograd_False, test/inductor/test_control_flow.py::WhileLoopTests::test_while_loop_nested_control_flow_device_cuda_dynamic_False_autograd_True, test/inductor/test_control_flow.py::WhileLoopTests::test_while_loop_simple_control_flow_device_cpu_dynamic_False_autograd_False, test/inductor/test_control_flow.py::WhileLoopTests::test_while_loop_simple_control_flow_device_cpu_dynamic_True_autograd_True, test/inductor/test_control_flow.py::WhileLoopTests::test_while_loop_stack_output_simple_device_cuda_dynamic_False, test/inductor/test_control_flow.py::WhileLoopTests::test_while_loop_with_conv_device_cpu_dynamic_True_autograd_False, test/inductor/test_control_flow.py::WhileLoopTests::test_while_loop_with_conv_device_cuda_dynamic_True_autograd_True, test/inductor/test_control_flow.py::WhileLoopTests::test_while_loop_with_data_dependent_in_out_device_cpu_dynamic_True_autograd_False, test/inductor/test_control_flow.py::WhileLoopTests::test_while_loop_with_data_dependent_in_out_device_cuda_dynamic_True_autograd_False, test/inductor/test_control_flow.py::WhileLoopTests::test_while_loop_with_data_dependent_ops_device_cpu_dynamic_False_autograd_False, test/inductor/test_control_flow.py::WhileLoopTests::test_while_loop_with_outer_buffers_device_cuda_dynamic_False_autograd_False, test/inductor/test_control_flow.py::WhileLoopTests::test_while_loop_with_pytree_inputs_device_cpu_dynamic_False_autograd_False, test/inductor/test_control_flow.py::WhileLoopTests::test_while_loop_with_pytree_inputs_device_cuda_dynamic_False_autograd_False, test/inductor/test_control_flow.py::WhileLoopTests::test_while_loop_with_pytree_inputs_device_cuda_dynamic_True_autograd_False, test/inductor/test_control_flow.py::WhileLoopTests::test_while_loop_with_sym_expr_cond_device_cuda_dynamic_False_autograd_False, test/inductor/test_control_flow.py::WhileLoopTests::test_while_loop_with_unbacked_symint_closure_device_cpu_dynamic_False_autograd_True, test/inductor/test_control_flow.py::WhileLoopTests::test_while_loop_with_unbacked_symint_closure_device_cpu_dynamic_True_autograd_False, test/inductor/test_control_flow.py::WhileLoopTests::test_while_loop_with_unbacked_symint_closure_device_cuda_dynamic_True_autograd_True, test/inductor/test_control_flow.py::ScanTests::test_cond_in_scan_device_cpu_dynamic_False_reverse_False_dim_0_scan_length_1_autograd_True, test/inductor/test_control_flow.py::ScanTests::test_cond_in_scan_device_cpu_dynamic_False_reverse_False_dim_1_scan_length_5_autograd_True, test/inductor/test_control_flow.py::ScanTests::test_cond_in_scan_device_cpu_dynamic_False_reverse_False_dim_3_scan_length_5_autograd_False, test/inductor/test_control_flow.py::ScanTests::test_cond_in_scan_device_cpu_dynamic_False_reverse_True_dim_0_scan_length_1_autograd_True, test/inductor/test_control_flow.py::ScanTests::test_cond_in_scan_device_cpu_dynamic_False_reverse_True_dim_1_scan_length_1_autograd_True, test/inductor/test_control_flow.py::ScanTests::test_cond_in_scan_device_cpu_dynamic_False_reverse_True_dim_1_scan_length_5_autograd_False, test/inductor/test_control_flow.py::ScanTests::test_cond_in_scan_device_cpu_dynamic_True_reverse_False_dim_3_scan_length_5_autograd_True, test/inductor/test_control_flow.py::ScanTests::test_cond_in_scan_device_cpu_dynamic_True_reverse_True_dim_1_scan_length_5_autograd_False, test/inductor/test_control_flow.py::ScanTests::test_cond_in_scan_device_cuda_dynamic_False_reverse_False_dim_1_scan_length_5_autograd_False, test/inductor/test_control_flow.py::ScanTests::test_cond_in_scan_device_cuda_dynamic_False_reverse_False_dim_3_scan_length_5_autograd_False, test/inductor/test_control_flow.py::ScanTests::test_cond_in_scan_device_cuda_dynamic_True_reverse_False_dim_0_scan_length_1_autograd_False, test/inductor/test_control_flow.py::ScanTests::test_cond_in_scan_device_cuda_dynamic_True_reverse_False_dim_0_scan_length_5_autograd_False, test/inductor/test_control_flow.py::ScanTests::test_cond_in_scan_device_cuda_dynamic_True_reverse_False_dim_3_scan_length_1_autograd_False, test/inductor/test_control_flow.py::ScanTests::test_cond_in_scan_device_cuda_dynamic_True_reverse_False_dim_3_scan_length_5_autograd_True, test/inductor/test_control_flow.py::ScanTests::test_cond_in_scan_device_cuda_dynamic_True_reverse_True_dim_0_scan_length_1_autograd_True, test/inductor/test_control_flow.py::ScanTests::test_cond_in_scan_device_cuda_dynamic_True_reverse_True_dim_0_scan_length_5_autograd_True, test/inductor/test_control_flow.py::ScanTests::test_cond_in_scan_device_cuda_dynamic_True_reverse_True_dim_1_scan_length_1_autograd_False, test/inductor/test_control_flow.py::ScanTests::test_scan_chunked_ce_device_cuda_dynamic_False_autograd_False, test/inductor/test_control_flow.py::ScanTests::test_scan_chunked_ce_device_cuda_dynamic_False_autograd_True, test/inductor/test_control_flow.py::ScanTests::test_scan_chunked_ce_device_cuda_dynamic_True_autograd_False, test/inductor/test_control_flow.py::ScanTests::test_scan_conv_device_cpu_dynamic_False_reverse_False_dim_1_scan_length_5_autograd_False, test/inductor/test_control_flow.py::ScanTests::test_scan_conv_device_cpu_dynamic_False_reverse_False_dim_3_scan_length_5_autograd_True, test/inductor/test_control_flow.py::ScanTests::test_scan_conv_device_cpu_dynamic_False_reverse_True_dim_0_scan_length_5_autograd_True, test/inductor/test_control_flow.py::ScanTests::test_scan_conv_device_cpu_dynamic_False_reverse_True_dim_3_scan_length_5_autograd_False, test/inductor/test_control_flow.py::ScanTests::test_scan_conv_device_cpu_dynamic_True_reverse_False_dim_0_scan_length_1_autograd_False, test/inductor/test_control_flow.py::ScanTests::test_scan_conv_device_cpu_dynamic_True_reverse_False_dim_0_scan_length_1_autograd_True, test/inductor/test_control_flow.py::ScanTests::test_scan_conv_device_cpu_dynamic_True_reverse_False_dim_0_scan_length_5_autograd_True, test/inductor/test_control_flow.py::ScanTests::test_scan_conv_device_cpu_dynamic_True_reverse_False_dim_1_scan_length_1_autograd_False, test/inductor/test_control_flow.py::ScanTests::test_scan_conv_device_cpu_dynamic_True_reverse_True_dim_0_scan_length_1_autograd_False, test/inductor/test_control_flow.py::ScanTests::test_scan_conv_device_cpu_dynamic_True_reverse_True_dim_0_scan_length_5_autograd_False, test/inductor/test_control_flow.py::ScanTests::test_scan_conv_device_cpu_dynamic_True_reverse_True_dim_0_scan_length_5_autograd_True, test/inductor/test_control_flow.py::ScanTests::test_scan_conv_device_cpu_dynamic_True_reverse_True_dim_1_scan_length_5_autograd_False, test/inductor/test_control_flow.py::ScanTests::test_scan_conv_device_cpu_dynamic_True_reverse_True_dim_3_scan_length_1_autograd_True, test/inductor/test_control_flow.py::ScanTests::test_scan_conv_device_cuda_dynamic_False_reverse_False_dim_0_scan_length_1_autograd_False, test/inductor/test_control_flow.py::ScanTests::test_scan_conv_device_cuda_dynamic_False_reverse_False_dim_0_scan_length_5_autograd_True, test/inductor/test_control_flow.py::ScanTests::test_scan_conv_device_cuda_dynamic_False_reverse_False_dim_3_scan_length_1_autograd_True, test/inductor/test_control_flow.py::ScanTests::test_scan_conv_device_cuda_dynamic_False_reverse_True_dim_0_scan_length_1_autograd_True, test/inductor/test_control_flow.py::ScanTests::test_scan_conv_device_cuda_dynamic_False_reverse_True_dim_0_scan_length_5_autograd_False, test/inductor/test_control_flow.py::ScanTests::test_scan_conv_device_cuda_dynamic_False_reverse_True_dim_3_scan_length_1_autograd_True, test/inductor/test_control_flow.py::ScanTests::test_scan_conv_device_cuda_dynamic_True_reverse_False_dim_0_scan_length_1_autograd_False, test/inductor/test_control_flow.py::ScanTests::test_scan_conv_device_cuda_dynamic_True_reverse_False_dim_0_scan_length_5_autograd_True, test/inductor/test_control_flow.py::ScanTests::test_scan_conv_device_cuda_dynamic_True_reverse_False_dim_3_scan_length_1_autograd_False, test/inductor/test_control_flow.py::ScanTests::test_scan_conv_device_cuda_dynamic_True_reverse_False_dim_3_scan_length_1_autograd_True, test/inductor/test_control_flow.py::ScanTests::test_scan_in_cond_device_cpu_dynamic_False_reverse_False_dim_0_pred_False_scan_length_1_autograd_True, test/inductor/test_control_flow.py::ScanTests::test_scan_in_cond_device_cpu_dynamic_False_reverse_False_dim_0_pred_True_scan_length_1_autograd_True, test/inductor/test_control_flow.py::ScanTests::test_scan_in_cond_device_cpu_dynamic_False_reverse_False_dim_0_pred_True_scan_length_5_autograd_False, test/inductor/test_control_flow.py::ScanTests::test_scan_in_cond_device_cpu_dynamic_False_reverse_False_dim_1_pred_False_scan_length_5_autograd_False, test/inductor/test_control_flow.py::ScanTests::test_scan_in_cond_device_cpu_dynamic_False_reverse_False_dim_1_pred_False_scan_length_5_autograd_True, test/inductor/test_control_flow.py::ScanTests::test_scan_in_cond_device_cpu_dynamic_False_reverse_False_dim_3_pred_False_scan_length_1_autograd_True, test/inductor/test_control_flow.py::ScanTests::test_scan_in_cond_device_cpu_dynamic_False_reverse_False_dim_3_pred_True_scan_length_5_autograd_True, test/inductor/test_control_flow.py::ScanTests::test_scan_in_cond_device_cpu_dynamic_False_reverse_True_dim_0_pred_False_scan_length_1_autograd_False, test/inductor/test_control_flow.py::ScanTests::test_scan_in_cond_device_cpu_dynamic_False_reverse_True_dim_0_pred_False_scan_length_5_autograd_False, test/inductor/test_control_flow.py::ScanTests::test_scan_in_cond_device_cpu_dynamic_False_reverse_True_dim_3_pred_True_scan_length_1_autograd_False, test/inductor/test_control_flow.py::ScanTests::test_scan_in_cond_device_cpu_dynamic_True_reverse_False_dim_0_pred_False_scan_length_1_autograd_False, test/inductor/test_control_flow.py::ScanTests::test_scan_in_cond_device_cpu_dynamic_True_reverse_False_dim_0_pred_False_scan_length_5_autograd_False, test/inductor/test_control_flow.py::ScanTests::test_scan_in_cond_device_cpu_dynamic_True_reverse_False_dim_0_pred_False_scan_length_5_autograd_True, test/inductor/test_control_flow.py::ScanTests::test_scan_in_cond_device_cpu_dynamic_True_reverse_False_dim_1_pred_True_scan_length_1_autograd_False, test/inductor/test_control_flow.py::ScanTests::test_scan_in_cond_device_cpu_dynamic_True_reverse_False_dim_1_pred_True_scan_length_5_autograd_True, test/inductor/test_control_flow.py::ScanTests::test_scan_in_cond_device_cpu_dynamic_True_reverse_False_dim_3_pred_True_scan_length_5_autograd_True, test/inductor/test_control_flow.py::ScanTests::test_scan_in_cond_device_cpu_dynamic_True_reverse_True_dim_0_pred_False_scan_length_1_autograd_False, test/inductor/test_control_flow.py::ScanTests::test_scan_in_cond_device_cpu_dynamic_True_reverse_True_dim_0_pred_False_scan_length_5_autograd_False, test/inductor/test_control_flow.py::ScanTests::test_scan_in_cond_device_cpu_dynamic_True_reverse_True_dim_0_pred_True_scan_length_5_autograd_True, test/inductor/test_control_flow.py::ScanTests::test_scan_in_cond_device_cpu_dynamic_True_reverse_True_dim_3_pred_False_scan_length_1_autograd_False, test/inductor/test_control_flow.py::ScanTests::test_scan_in_cond_device_cuda_dynamic_False_reverse_False_dim_0_pred_True_scan_length_5_autograd_True, test/inductor/test_control_flow.py::ScanTests::test_scan_in_cond_device_cuda_dynamic_False_reverse_False_dim_1_pred_False_scan_length_1_autograd_False, test/inductor/test_control_flow.py::ScanTests::test_scan_in_cond_device_cuda_dynamic_False_reverse_False_dim_1_pred_False_scan_length_5_autograd_True, test/inductor/test_control_flow.py::ScanTests::test_scan_in_cond_device_cuda_dynamic_False_reverse_False_dim_1_pred_True_scan_length_5_autograd_False, test/inductor/test_control_flow.py::ScanTests::test_scan_in_cond_device_cuda_dynamic_False_reverse_False_dim_3_pred_False_scan_length_1_autograd_True, test/inductor/test_control_flow.py::ScanTests::test_scan_in_cond_device_cuda_dynamic_False_reverse_False_dim_3_pred_True_scan_length_5_autograd_False, test/inductor/test_control_flow.py::ScanTests::test_scan_in_cond_device_cuda_dynamic_False_reverse_False_dim_3_pred_True_scan_length_5_autograd_True, test/inductor/test_control_flow.py::ScanTests::test_scan_in_cond_device_cuda_dynamic_False_reverse_True_dim_0_pred_False_scan_length_5_autograd_True, test/inductor/test_control_flow.py::ScanTests::test_scan_in_cond_device_cuda_dynamic_False_reverse_True_dim_0_pred_True_scan_length_5_autograd_False, test/inductor/test_control_flow.py::ScanTests::test_scan_in_cond_device_cuda_dynamic_False_reverse_True_dim_1_pred_False_scan_length_1_autograd_False, test/inductor/test_control_flow.py::ScanTests::test_scan_in_cond_device_cuda_dynamic_False_reverse_True_dim_1_pred_False_scan_length_1_autograd_True, test/inductor/test_control_flow.py::ScanTests::test_scan_in_cond_device_cuda_dynamic_False_reverse_True_dim_3_pred_False_scan_length_1_autograd_True, test/inductor/test_control_flow.py::ScanTests::test_scan_in_cond_device_cuda_dynamic_False_reverse_True_dim_3_pred_True_scan_length_1_autograd_False, test/inductor/test_control_flow.py::ScanTests::test_scan_in_cond_device_cuda_dynamic_True_reverse_False_dim_0_pred_False_scan_length_5_autograd_True, test/inductor/test_control_flow.py::ScanTests::test_scan_in_cond_device_cuda_dynamic_True_reverse_False_dim_0_pred_True_scan_length_1_autograd_False, test/inductor/test_control_flow.py::ScanTests::test_scan_in_cond_device_cuda_dynamic_True_reverse_False_dim_1_pred_True_scan_length_1_autograd_True, test/inductor/test_control_flow.py::ScanTests::test_scan_in_cond_device_cuda_dynamic_True_reverse_False_dim_3_pred_True_scan_length_5_autograd_False, test/inductor/test_control_flow.py::ScanTests::test_scan_in_cond_device_cuda_dynamic_True_reverse_True_dim_1_pred_False_scan_length_5_autograd_False, test/inductor/test_control_flow.py::ScanTests::test_scan_in_cond_device_cuda_dynamic_True_reverse_True_dim_1_pred_True_scan_length_5_autograd_True, test/inductor/test_control_flow.py::ScanTests::test_scan_nn_modules_device_cpu_dynamic_False_reverse_False_dim_0_scan_length_1_autograd_True, test/inductor/test_control_flow.py::ScanTests::test_scan_nn_modules_device_cpu_dynamic_False_reverse_False_dim_1_scan_length_1_autograd_False, test/inductor/test_control_flow.py::ScanTests::test_scan_nn_modules_device_cpu_dynamic_False_reverse_True_dim_0_scan_length_1_autograd_True, test/inductor/test_control_flow.py::ScanTests::test_scan_nn_modules_device_cpu_dynamic_False_reverse_True_dim_1_scan_length_1_autograd_False, test/inductor/test_control_flow.py::ScanTests::test_scan_nn_modules_device_cpu_dynamic_False_reverse_True_dim_1_scan_length_5_autograd_True, test/inductor/test_control_flow.py::ScanTests::test_scan_nn_modules_device_cpu_dynamic_True_reverse_False_dim_3_scan_length_1_autograd_False, test/inductor/test_control_flow.py::ScanTests::test_scan_nn_modules_device_cpu_dynamic_True_reverse_True_dim_0_scan_length_1_autograd_False, test/inductor/test_control_flow.py::ScanTests::test_scan_nn_modules_device_cpu_dynamic_True_reverse_True_dim_0_scan_length_5_autograd_False, test/inductor/test_control_flow.py::ScanTests::test_scan_nn_modules_device_cpu_dynamic_True_reverse_True_dim_3_scan_length_1_autograd_True, test/inductor/test_control_flow.py::ScanTests::test_scan_nn_modules_device_cpu_dynamic_True_reverse_True_dim_3_scan_length_5_autograd_False, test/inductor/test_control_flow.py::ScanTests::test_scan_nn_modules_device_cuda_dynamic_False_reverse_False_dim_3_scan_length_1_autograd_True, test/inductor/test_control_flow.py::ScanTests::test_scan_nn_modules_device_cuda_dynamic_False_reverse_True_dim_1_scan_length_1_autograd_False, test/inductor/test_control_flow.py::ScanTests::test_scan_nn_modules_device_cuda_dynamic_False_reverse_True_dim_1_scan_length_1_autograd_True, test/inductor/test_control_flow.py::ScanTests::test_scan_nn_modules_device_cuda_dynamic_False_reverse_True_dim_3_scan_length_5_autograd_True, test/inductor/test_control_flow.py::ScanTests::test_scan_nn_modules_device_cuda_dynamic_True_reverse_False_dim_1_scan_length_1_autograd_False, test/inductor/test_control_flow.py::ScanTests::test_scan_nn_modules_device_cuda_dynamic_True_reverse_False_dim_1_scan_length_1_autograd_True, test/inductor/test_control_flow.py::ScanTests::test_scan_nn_modules_device_cuda_dynamic_True_reverse_False_dim_1_scan_length_5_autograd_True, test/inductor/test_control_flow.py::ScanTests::test_scan_nn_modules_device_cuda_dynamic_True_reverse_True_dim_0_scan_length_1_autograd_False, test/inductor/test_control_flow.py::ScanTests::test_scan_nn_modules_device_cuda_dynamic_True_reverse_True_dim_3_scan_length_1_autograd_False, test/inductor/test_control_flow.py::ScanTests::test_scan_nn_modules_device_cuda_dynamic_True_reverse_True_dim_3_scan_length_5_autograd_True, test/inductor/test_control_flow.py::ScanTests::test_scan_pytree_in_out_device_cpu_dynamic_False_reverse_False_dim_0_autograd_False, test/inductor/test_control_flow.py::ScanTests::test_scan_pytree_in_out_device_cpu_dynamic_False_reverse_False_dim_0_autograd_True, test/inductor/test_control_flow.py::ScanTests::test_scan_pytree_in_out_device_cpu_dynamic_False_reverse_False_dim_1_autograd_True, test/inductor/test_control_flow.py::ScanTests::test_scan_pytree_in_out_device_cpu_dynamic_False_reverse_True_dim_1_autograd_False, test/inductor/test_control_flow.py::ScanTests::test_scan_pytree_in_out_device_cpu_dynamic_False_reverse_True_dim_2_autograd_True, test/inductor/test_control_flow.py::ScanTests::test_scan_pytree_in_out_device_cpu_dynamic_True_reverse_False_dim_2_autograd_False, test/inductor/test_control_flow.py::ScanTests::test_scan_pytree_in_out_device_cpu_dynamic_True_reverse_True_dim_0_autograd_False, test/inductor/test_control_flow.py::ScanTests::test_scan_pytree_in_out_device_cpu_dynamic_True_reverse_True_dim_0_autograd_True, test/inductor/test_control_flow.py::ScanTests::test_scan_pytree_in_out_device_cpu_dynamic_True_reverse_True_dim_2_autograd_False, test/inductor/test_control_flow.py::ScanTests::test_scan_pytree_in_out_device_cuda_dynamic_False_reverse_False_dim_0_autograd_False, test/inductor/test_control_flow.py::ScanTests::test_scan_pytree_in_out_device_cuda_dynamic_False_reverse_False_dim_2_autograd_True, test/inductor/test_control_flow.py::ScanTests::test_scan_pytree_in_out_device_cuda_dynamic_False_reverse_True_dim_1_autograd_True, test/inductor/test_control_flow.py::ScanTests::test_scan_pytree_in_out_device_cuda_dynamic_True_reverse_False_dim_0_autograd_False, test/inductor/test_control_flow.py::ScanTests::test_scan_pytree_in_out_device_cuda_dynamic_True_reverse_False_dim_0_autograd_True, test/inductor/test_control_flow.py::ScanTests::test_scan_pytree_in_out_device_cuda_dynamic_True_reverse_False_dim_1_autograd_True, test/inductor/test_control_flow.py::ScanTests::test_scan_pytree_in_out_device_cuda_dynamic_True_reverse_False_dim_2_autograd_False, test/inductor/test_control_flow.py::ScanTests::test_scan_pytree_in_out_device_cuda_dynamic_True_reverse_False_dim_2_autograd_True, test/inductor/test_control_flow.py::ScanTests::test_scan_pytree_in_out_device_cuda_dynamic_True_reverse_True_dim_1_autograd_False, test/inductor/test_control_flow.py::ScanTests::test_scan_with_clamp_device_cpu_dynamic_True_autograd_False, test/inductor/test_control_flow.py::ScanTests::test_scan_with_clamp_device_cuda_dynamic_False_autograd_False, test/inductor/test_control_flow.py::MapTests::test_map_nested_with_cond_device_cuda_dynamic_False_autograd_False, test/inductor/test_control_flow.py::MapTests::test_map_nested_with_cond_device_cuda_dynamic_False_autograd_True, test/inductor/test_control_flow.py::MapTests::test_map_pytree_in_out_device_cpu_dynamic_True_autograd_False, test/inductor/test_control_flow.py::MapTests::test_map_pytree_in_out_device_cuda_dynamic_True_autograd_False, test/inductor/test_control_flow.py::MapTests::test_map_simple_linear_with_view_device_cpu_dynamic_False_autograd_False, test/inductor/test_control_flow.py::MapTests::test_map_simple_linear_with_view_device_cpu_dynamic_True_autograd_False, test/inductor/test_control_flow.py::MapTests::test_map_simple_linear_with_view_device_cuda_dynamic_True_autograd_False 2025-12-04T13:37:37.6482118Z 2025-12-04T13:37:37.6488894Z Finished inductor/test_control_flow 5/5 ... [2025-12-04 13:37:37.648685][14305.590895586], took 9.50min 2025-12-04T13:37:37.6786894Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/inductor.test_control_flow/inductor.test_control_flow-7c7af97e129282ca.xml 2025-12-04T13:37:38.7728124Z Uploading artifacts took 1.02 seconds 2025-12-04T13:37:38.7730774Z Running inductor/test_pallas 1/1 ... [2025-12-04 13:37:38.772867][14306.71508247] 2025-12-04T13:37:38.7734375Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2025-12-04T13:37:38.7735186Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'inductor/test_pallas.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-12-04 13:37:38.773260] 2025-12-04T13:37:44.7203321Z 2025-12-04T13:37:44.7204154Z inductor/test_pallas 1/1 was successful, full logs can be found in artifacts with path test/test-reports/inductor.test_pallas_1.1_29615ac8ce961259_.log 2025-12-04T13:37:44.7204786Z 2025-12-04T13:37:44.7205080Z Finished inductor/test_pallas 1/1 ... [2025-12-04 13:37:44.720082][14312.662299045], took 0.10min 2025-12-04T13:37:44.7495158Z Running test_per_overload_api 1/1 ... [2025-12-04 13:37:44.749288][14312.691507226] 2025-12-04T13:37:44.7495603Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2025-12-04T13:37:44.7498923Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'test_per_overload_api.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-12-04 13:37:44.749652] 2025-12-04T13:37:48.0198803Z 2025-12-04T13:37:48.0199809Z test_per_overload_api 1/1 was successful, full logs can be found in artifacts with path test/test-reports/test_per_overload_api_1.1_fd510f1512712c78_.log 2025-12-04T13:37:48.0201266Z Running 3 items in this shard: test/test_per_overload_api.py::TestPerOverloadAPI::test_basics_opoverload, test/test_per_overload_api.py::TestPerOverloadAPI::test_basics_opoverloadpacket, test/test_per_overload_api.py::TestPerOverloadAPI::test_decompose 2025-12-04T13:37:48.0202382Z 2025-12-04T13:37:48.0202649Z Finished test_per_overload_api 1/1 ... [2025-12-04 13:37:48.019529][14315.961742201], took 0.05min 2025-12-04T13:37:48.0489449Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/test_per_overload_api/test_per_overload_api-380dd5ceb2a22fbd.xml 2025-12-04T13:37:48.1313697Z Running test_multiprocessing 1/1 ... [2025-12-04 13:37:48.131132][14316.073350426] 2025-12-04T13:37:48.1314149Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2025-12-04T13:37:48.1316995Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'test_multiprocessing.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-12-04 13:37:48.131472] 2025-12-04T13:39:36.0888798Z 2025-12-04T13:39:36.0893175Z test_multiprocessing 1/1 was successful, full logs can be found in artifacts with path test/test-reports/test_multiprocessing_1.1_889420b6a5edd168_.log 2025-12-04T13:39:36.0902982Z Running 42 items in this shard: test/test_multiprocessing.py::TestMultiprocessing::test_autograd_errors, test/test_multiprocessing.py::TestMultiprocessing::test_autograd_fine_with_spawn, test/test_multiprocessing.py::TestMultiprocessing::test_cuda_bad_call, test/test_multiprocessing.py::TestMultiprocessing::test_cuda_ipc_deadlock, test/test_multiprocessing.py::TestMultiprocessing::test_cuda_memory_allocation, test/test_multiprocessing.py::TestMultiprocessing::test_cuda_parameter_sharing, test/test_multiprocessing.py::TestMultiprocessing::test_cuda_send_many, test/test_multiprocessing.py::TestMultiprocessing::test_cuda_simple, test/test_multiprocessing.py::TestMultiprocessing::test_cuda_small_tensors, test/test_multiprocessing.py::TestMultiprocessing::test_cuda_variable_sharing, test/test_multiprocessing.py::TestMultiprocessing::test_empty_shared, test/test_multiprocessing.py::TestMultiprocessing::test_empty_tensor_sharing, test/test_multiprocessing.py::TestMultiprocessing::test_empty_tensor_sharing_cuda, test/test_multiprocessing.py::TestMultiprocessing::test_empty_tensor_sharing_meta, test/test_multiprocessing.py::TestMultiprocessing::test_event, test/test_multiprocessing.py::TestMultiprocessing::test_event_handle_exporter, test/test_multiprocessing.py::TestMultiprocessing::test_event_handle_importer, test/test_multiprocessing.py::TestMultiprocessing::test_event_handle_multi_gpu, test/test_multiprocessing.py::TestMultiprocessing::test_event_multiprocess, test/test_multiprocessing.py::TestMultiprocessing::test_fd_pool, test/test_multiprocessing.py::TestMultiprocessing::test_fd_preserve_sharing, test/test_multiprocessing.py::TestMultiprocessing::test_fd_sharing, test/test_multiprocessing.py::TestMultiprocessing::test_fs, test/test_multiprocessing.py::TestMultiprocessing::test_fs_is_shared, test/test_multiprocessing.py::TestMultiprocessing::test_fs_pool, test/test_multiprocessing.py::TestMultiprocessing::test_fs_preserve_sharing, test/test_multiprocessing.py::TestMultiprocessing::test_fs_sharing, test/test_multiprocessing.py::TestMultiprocessing::test_inherit_tensor, test/test_multiprocessing.py::TestMultiprocessing::test_integer_parameter_serialization_cpu, test/test_multiprocessing.py::TestMultiprocessing::test_integer_parameter_serialization_cuda, test/test_multiprocessing.py::TestMultiprocessing::test_is_shared, test/test_multiprocessing.py::TestMultiprocessing::test_is_shared_cuda, test/test_multiprocessing.py::TestMultiprocessing::test_leaf_variable_sharing, test/test_multiprocessing.py::TestMultiprocessing::test_meta_simple, test/test_multiprocessing.py::TestMultiprocessing::test_mixed_types_cuda_sharing, test/test_multiprocessing.py::TestMultiprocessing::test_non_leaf_variable_sharing, test/test_multiprocessing.py::TestMultiprocessing::test_parameter_sharing, test/test_multiprocessing.py::TestMultiprocessing::test_rebuild_cuda_tensor, test/test_multiprocessing.py::TestMultiprocessing::test_set_thread_name, test/test_multiprocessing.py::TestMultiprocessing::test_tensor_sharing_meta, test/test_multiprocessing.py::TestMultiprocessing::test_variable_sharing, test/test_multiprocessing.py::TestMultiprocessing::test_wrong_cuda_fork 2025-12-04T13:39:36.0912435Z 2025-12-04T13:39:36.0912671Z Finished test_multiprocessing 1/1 ... [2025-12-04 13:39:36.088452][14424.030665477], took 1.80min 2025-12-04T13:39:36.1190993Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/test_multiprocessing/test_multiprocessing-1ba78cf82b9f2c08.xml 2025-12-04T13:39:36.2039446Z Running functorch/test_control_flow 2/3 ... [2025-12-04 13:39:36.203707][14424.145925006] 2025-12-04T13:39:36.2039917Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2025-12-04T13:39:36.2042560Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'functorch/test_control_flow.py', '--shard-id=2', '--num-shards=3', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-12-04 13:39:36.204011] 2025-12-04T13:50:18.7021510Z 2025-12-04T13:50:18.7022499Z functorch/test_control_flow 2/3 was successful, full logs can be found in artifacts with path test/test-reports/functorch.test_control_flow_2.3_c6fd620740f70764_.log 2025-12-04T13:50:18.7341710Z Running 672 items in this shard: test/functorch/test_control_flow.py::TestControlFlow::test_cond_autograd_different_pytree_output, test/functorch/test_control_flow.py::TestControlFlow::test_cond_autograd_gpu, test/functorch/test_control_flow.py::TestControlFlow::test_cond_autograd_simple, test/functorch/test_control_flow.py::TestControlFlow::test_cond_autograd_user_nn_module, test/functorch/test_control_flow.py::TestControlFlow::test_cond_no_trace, test/functorch/test_control_flow.py::TestControlFlow::test_map_autograd_simple, test/functorch/test_control_flow.py::TestControlFlow::test_map_dict_in_out, test/functorch/test_control_flow.py::TestControlFlow::test_scan_associative_scan, test/functorch/test_control_flow.py::TestControlFlow::test_scan_binary_operator_reverse_False_compile_mode_none_cuda_autograd_False, test/functorch/test_control_flow.py::TestControlFlow::test_scan_binary_operator_reverse_True_compile_mode_eager_cuda_autograd_True, test/functorch/test_control_flow.py::TestControlFlow::test_scan_binary_operator_reverse_True_compile_mode_none_cpu_autograd_False, test/functorch/test_control_flow.py::TestControlFlow::test_scan_binary_operator_reverse_True_compile_mode_none_cpu_autograd_True, test/functorch/test_control_flow.py::TestControlFlow::test_scan_binary_operator_reverse_True_compile_mode_none_cuda_autograd_True, test/functorch/test_control_flow.py::TestControlFlow::test_scan_carry_carry_alias, test/functorch/test_control_flow.py::TestControlFlow::test_scan_closure_RNN_compile_mode_eager_autograd_True, test/functorch/test_control_flow.py::TestControlFlow::test_scan_closure_RNN_compile_mode_none_autograd_True, test/functorch/test_control_flow.py::TestControlFlow::test_scan_closure_RNN_partial_autograd_reverse_False_compile_mode_eager_partial_grad_complex_cpu, test/functorch/test_control_flow.py::TestControlFlow::test_scan_closure_RNN_partial_autograd_reverse_False_compile_mode_eager_partial_grad_random_cpu, test/functorch/test_control_flow.py::TestControlFlow::test_scan_closure_RNN_partial_autograd_reverse_False_compile_mode_eager_partial_grad_xs_cuda, test/functorch/test_control_flow.py::TestControlFlow::test_scan_closure_RNN_partial_autograd_reverse_False_compile_mode_none_partial_grad_additional_inputs_cuda, test/functorch/test_control_flow.py::TestControlFlow::test_scan_closure_RNN_partial_autograd_reverse_False_compile_mode_none_partial_grad_complex_cpu, test/functorch/test_control_flow.py::TestControlFlow::test_scan_closure_RNN_partial_autograd_reverse_False_compile_mode_none_partial_grad_complex_cuda, test/functorch/test_control_flow.py::TestControlFlow::test_scan_closure_RNN_partial_autograd_reverse_False_compile_mode_none_partial_grad_init_cuda, test/functorch/test_control_flow.py::TestControlFlow::test_scan_closure_RNN_partial_autograd_reverse_False_compile_mode_none_partial_grad_xs_cuda, test/functorch/test_control_flow.py::TestControlFlow::test_scan_closure_RNN_partial_autograd_reverse_True_compile_mode_eager_partial_grad_additional_inputs_cpu, test/functorch/test_control_flow.py::TestControlFlow::test_scan_closure_RNN_partial_autograd_reverse_True_compile_mode_eager_partial_grad_init_cuda, test/functorch/test_control_flow.py::TestControlFlow::test_scan_closure_RNN_partial_autograd_reverse_True_compile_mode_none_partial_grad_init_cpu, test/functorch/test_control_flow.py::TestControlFlow::test_scan_closure_RNN_partial_autograd_reverse_True_compile_mode_none_partial_grad_random_cuda, test/functorch/test_control_flow.py::TestControlFlow::test_scan_closure_combine_fn_carries_ys_same_grad_reverse_False_compile_mode_eager_cuda_autograd_False, test/functorch/test_control_flow.py::TestControlFlow::test_scan_closure_combine_fn_carries_ys_same_grad_reverse_False_compile_mode_none_cpu_autograd_False, test/functorch/test_control_flow.py::TestControlFlow::test_scan_closure_combine_fn_carries_ys_same_grad_reverse_False_compile_mode_none_cpu_autograd_True, test/functorch/test_control_flow.py::TestControlFlow::test_scan_closure_combine_fn_carries_ys_same_grad_reverse_False_compile_mode_none_cuda_autograd_False, test/functorch/test_control_flow.py::TestControlFlow::test_scan_closure_combine_fn_carries_ys_same_grad_reverse_False_compile_mode_none_cuda_autograd_True, test/functorch/test_control_flow.py::TestControlFlow::test_scan_closure_combine_fn_carries_ys_same_grad_reverse_True_compile_mode_eager_cuda_autograd_True, test/functorch/test_control_flow.py::TestControlFlow::test_scan_closure_combine_fn_carries_ys_same_grad_reverse_True_compile_mode_none_cpu_autograd_False, test/functorch/test_control_flow.py::TestControlFlow::test_scan_closure_combine_fn_carries_ys_same_grad_reverse_True_compile_mode_none_cuda_autograd_True, test/functorch/test_control_flow.py::TestControlFlow::test_scan_closure_combine_fn_with_no_grad_additional_inputs_all_reverse_False_compile_mode_eager_cpu_autograd_False, test/functorch/test_control_flow.py::TestControlFlow::test_scan_closure_combine_fn_with_no_grad_additional_inputs_all_reverse_False_compile_mode_eager_cuda_autograd_False, test/functorch/test_control_flow.py::TestControlFlow::test_scan_closure_combine_fn_with_no_grad_additional_inputs_all_reverse_False_compile_mode_none_cpu_autograd_False, test/functorch/test_control_flow.py::TestControlFlow::test_scan_closure_combine_fn_with_no_grad_additional_inputs_all_reverse_True_compile_mode_eager_cpu_autograd_False, test/functorch/test_control_flow.py::TestControlFlow::test_scan_closure_combine_fn_with_no_grad_additional_inputs_all_reverse_True_compile_mode_none_cpu_autograd_False, test/functorch/test_control_flow.py::TestControlFlow::test_scan_closure_combine_fn_with_no_grad_additional_inputs_all_reverse_True_compile_mode_none_cpu_autograd_True, test/functorch/test_control_flow.py::TestControlFlow::test_scan_closure_combine_fn_with_no_grad_additional_inputs_all_reverse_True_compile_mode_none_cuda_autograd_False, test/functorch/test_control_flow.py::TestControlFlow::test_scan_closure_combine_fn_with_no_grad_additional_inputs_partial_reverse_False_compile_mode_eager_cuda_autograd_False, test/functorch/test_control_flow.py::TestControlFlow::test_scan_closure_combine_fn_with_no_grad_additional_inputs_partial_reverse_False_compile_mode_eager_cuda_autograd_True, test/functorch/test_control_flow.py::TestControlFlow::test_scan_closure_combine_fn_with_no_grad_additional_inputs_partial_reverse_False_compile_mode_none_cuda_autograd_False, test/functorch/test_control_flow.py::TestControlFlow::test_scan_closure_combine_fn_with_no_grad_additional_inputs_partial_reverse_False_compile_mode_none_cuda_autograd_True, test/functorch/test_control_flow.py::TestControlFlow::test_scan_closure_combine_fn_with_no_grad_additional_inputs_partial_reverse_True_compile_mode_eager_cpu_autograd_False, test/functorch/test_control_flow.py::TestControlFlow::test_scan_closure_combine_fn_with_no_grad_for_out_reverse_False_compile_mode_eager_cpu_autograd_False, test/functorch/test_control_flow.py::TestControlFlow::test_scan_closure_combine_fn_with_no_grad_for_out_reverse_False_compile_mode_none_cpu_autograd_False, test/functorch/test_control_flow.py::TestControlFlow::test_scan_closure_combine_fn_with_no_grad_for_out_reverse_False_compile_mode_none_cpu_autograd_True, test/functorch/test_control_flow.py::TestControlFlow::test_scan_closure_combine_fn_with_no_grad_for_out_reverse_False_compile_mode_none_cuda_autograd_True, test/functorch/test_control_flow.py::TestControlFlow::test_scan_closure_combine_fn_with_no_grad_for_out_reverse_True_compile_mode_none_cpu_autograd_False, test/functorch/test_control_flow.py::TestControlFlow::test_scan_closure_combine_fn_with_no_grad_for_out_reverse_True_compile_mode_none_cuda_autograd_True, test/functorch/test_control_flow.py::TestControlFlow::test_scan_closure_combine_fn_with_no_grad_init_carries_equal_grad_reverse_False_compile_mode_none_cpu_autograd_False, test/functorch/test_control_flow.py::TestControlFlow::test_scan_closure_combine_fn_with_no_grad_init_carries_equal_grad_reverse_False_compile_mode_none_cuda_autograd_False, test/functorch/test_control_flow.py::TestControlFlow::test_scan_closure_combine_fn_with_no_grad_init_carries_equal_grad_reverse_True_compile_mode_eager_cpu_autograd_False, test/functorch/test_control_flow.py::TestControlFlow::test_scan_closure_combine_fn_with_no_grad_init_carries_equal_grad_reverse_True_compile_mode_none_cpu_autograd_False, test/functorch/test_control_flow.py::TestControlFlow::test_scan_closure_combine_fn_with_no_grad_init_carries_equal_grad_reverse_True_compile_mode_none_cuda_autograd_False, test/functorch/test_control_flow.py::TestControlFlow::test_scan_closure_combine_fn_with_no_grad_init_carries_unequal_grad_reverse_False_compile_mode_eager_cpu_autograd_False, test/functorch/test_control_flow.py::TestControlFlow::test_scan_closure_combine_fn_with_no_grad_init_carries_unequal_grad_reverse_False_compile_mode_eager_cpu_autograd_True, test/functorch/test_control_flow.py::TestControlFlow::test_scan_closure_combine_fn_with_no_grad_init_carries_unequal_grad_reverse_False_compile_mode_none_cpu_autograd_True, test/functorch/test_control_flow.py::TestControlFlow::test_scan_closure_combine_fn_with_no_grad_init_carries_unequal_grad_reverse_False_compile_mode_none_cuda_autograd_False, test/functorch/test_control_flow.py::TestControlFlow::test_scan_closure_combine_fn_with_no_grad_init_carries_unequal_grad_reverse_True_compile_mode_eager_cpu_autograd_True, test/functorch/test_control_flow.py::TestControlFlow::test_scan_closure_combine_fn_with_no_grad_init_carries_unequal_grad_reverse_True_compile_mode_eager_cuda_autograd_False, test/functorch/test_control_flow.py::TestControlFlow::test_scan_closure_combine_fn_with_no_grad_init_carries_unequal_grad_reverse_True_compile_mode_none_cpu_autograd_True, test/functorch/test_control_flow.py::TestControlFlow::test_scan_closure_combine_fn_with_no_grad_init_carries_unequal_grad_reverse_True_compile_mode_none_cuda_autograd_False, test/functorch/test_control_flow.py::TestControlFlow::test_scan_closure_nested_reverse_False_compile_mode_eager_cpu_autograd_True, test/functorch/test_control_flow.py::TestControlFlow::test_scan_closure_nested_reverse_False_compile_mode_none_cpu_autograd_True, test/functorch/test_control_flow.py::TestControlFlow::test_scan_closure_nested_reverse_False_compile_mode_none_cuda_autograd_True, test/functorch/test_control_flow.py::TestControlFlow::test_scan_closure_nested_reverse_True_compile_mode_eager_cpu_autograd_False, test/functorch/test_control_flow.py::TestControlFlow::test_scan_compile_cnt_reverse_False_cuda, test/functorch/test_control_flow.py::TestControlFlow::test_scan_compile_cnt_reverse_True_cuda, test/functorch/test_control_flow.py::TestControlFlow::test_scan_compile_reverse_False_compile_mode_eager_cuda_autograd_False, test/functorch/test_control_flow.py::TestControlFlow::test_scan_compile_reverse_False_compile_mode_none_cpu_autograd_True, test/functorch/test_control_flow.py::TestControlFlow::test_scan_compile_reverse_False_compile_mode_none_cuda_autograd_False, test/functorch/test_control_flow.py::TestControlFlow::test_scan_compile_reverse_True_compile_mode_eager_cpu_autograd_True, test/functorch/test_control_flow.py::TestControlFlow::test_scan_compile_reverse_True_compile_mode_none_cpu_autograd_True, test/functorch/test_control_flow.py::TestControlFlow::test_scan_compile_reverse_True_compile_mode_none_cuda_autograd_True, test/functorch/test_control_flow.py::TestControlFlow::test_scan_complex_pytree_reverse_False_compile_mode_eager_cpu_autograd_False, test/functorch/test_control_flow.py::TestControlFlow::test_scan_complex_pytree_reverse_False_compile_mode_none_cuda_autograd_False, test/functorch/test_control_flow.py::TestControlFlow::test_scan_complex_pytree_reverse_True_compile_mode_eager_cuda_autograd_True, test/functorch/test_control_flow.py::TestControlFlow::test_scan_complex_pytree_reverse_True_compile_mode_none_cpu_autograd_True, test/functorch/test_control_flow.py::TestControlFlow::test_scan_complex_pytree_reverse_True_compile_mode_none_cuda_autograd_True, test/functorch/test_control_flow.py::TestControlFlow::test_scan_dim_reverse_False_compile_mode_eager_cpu_autograd_False, test/functorch/test_control_flow.py::TestControlFlow::test_scan_dim_reverse_False_compile_mode_none_cuda_autograd_False, test/functorch/test_control_flow.py::TestControlFlow::test_scan_dim_reverse_True_compile_mode_eager_cuda_autograd_False, test/functorch/test_control_flow.py::TestControlFlow::test_scan_dim_reverse_True_compile_mode_eager_cuda_autograd_True, test/functorch/test_control_flow.py::TestControlFlow::test_scan_dim_reverse_True_compile_mode_none_cuda_autograd_False, test/functorch/test_control_flow.py::TestControlFlow::test_scan_downstream_scan_matmul_compile_mode_eager_reverse_False_cpu_autograd_False, test/functorch/test_control_flow.py::TestControlFlow::test_scan_downstream_scan_matmul_compile_mode_eager_reverse_False_cuda_autograd_True, test/functorch/test_control_flow.py::TestControlFlow::test_scan_downstream_scan_matmul_compile_mode_eager_reverse_True_cpu_autograd_False, test/functorch/test_control_flow.py::TestControlFlow::test_scan_downstream_scan_matmul_compile_mode_eager_reverse_True_cuda_autograd_False, test/functorch/test_control_flow.py::TestControlFlow::test_scan_downstream_scan_matmul_compile_mode_none_reverse_True_cpu_autograd_False, test/functorch/test_control_flow.py::TestControlFlow::test_scan_downstream_scan_scan_dim_compile_mode_eager_reverse_False_cuda_autograd_False, test/functorch/test_control_flow.py::TestControlFlow::test_scan_downstream_scan_scan_dim_compile_mode_eager_reverse_False_cuda_autograd_True, test/functorch/test_control_flow.py::TestControlFlow::test_scan_downstream_scan_scan_dim_compile_mode_eager_reverse_True_cpu_autograd_False, test/functorch/test_control_flow.py::TestControlFlow::test_scan_downstream_scan_scan_dim_compile_mode_eager_reverse_True_cpu_autograd_True, test/functorch/test_control_flow.py::TestControlFlow::test_scan_downstream_scan_scan_dim_compile_mode_none_reverse_False_cpu_autograd_False, test/functorch/test_control_flow.py::TestControlFlow::test_scan_downstream_scan_scan_dim_compile_mode_none_reverse_False_cuda_autograd_False, test/functorch/test_control_flow.py::TestControlFlow::test_scan_downstream_scan_scan_dim_compile_mode_none_reverse_False_cuda_autograd_True, test/functorch/test_control_flow.py::TestControlFlow::test_scan_downstream_scan_scan_dim_compile_mode_none_reverse_True_cpu_autograd_False, test/functorch/test_control_flow.py::TestControlFlow::test_scan_downstream_scan_scan_dim_compile_mode_none_reverse_True_cuda_autograd_False, test/functorch/test_control_flow.py::TestControlFlow::test_scan_dtype_reverse_False_compile_mode_eager_cpu_float16, test/functorch/test_control_flow.py::TestControlFlow::test_scan_dtype_reverse_False_compile_mode_eager_cpu_float32, test/functorch/test_control_flow.py::TestControlFlow::test_scan_dtype_reverse_False_compile_mode_eager_cpu_int64, test/functorch/test_control_flow.py::TestControlFlow::test_scan_dtype_reverse_False_compile_mode_none_cpu_float16, test/functorch/test_control_flow.py::TestControlFlow::test_scan_dtype_reverse_False_compile_mode_none_cpu_int64, test/functorch/test_control_flow.py::TestControlFlow::test_scan_dtype_reverse_False_compile_mode_none_cuda_float16, test/functorch/test_control_flow.py::TestControlFlow::test_scan_dtype_reverse_False_compile_mode_none_cuda_int64, test/functorch/test_control_flow.py::TestControlFlow::test_scan_dtype_reverse_True_compile_mode_eager_cpu_complex64, test/functorch/test_control_flow.py::TestControlFlow::test_scan_dtype_reverse_True_compile_mode_eager_cpu_float16, test/functorch/test_control_flow.py::TestControlFlow::test_scan_dtype_reverse_True_compile_mode_eager_cuda_int32, test/functorch/test_control_flow.py::TestControlFlow::test_scan_dtype_reverse_True_compile_mode_none_cpu_float16, test/functorch/test_control_flow.py::TestControlFlow::test_scan_dtype_reverse_True_compile_mode_none_cpu_float32, test/functorch/test_control_flow.py::TestControlFlow::test_scan_dtype_reverse_True_compile_mode_none_cpu_int32, test/functorch/test_control_flow.py::TestControlFlow::test_scan_dtype_reverse_True_compile_mode_none_cpu_int64, test/functorch/test_control_flow.py::TestControlFlow::test_scan_dtype_reverse_True_compile_mode_none_cuda_complex64, test/functorch/test_control_flow.py::TestControlFlow::test_scan_dtype_reverse_True_compile_mode_none_cuda_float16, test/functorch/test_control_flow.py::TestControlFlow::test_scan_dtype_reverse_True_compile_mode_none_cuda_int32, test/functorch/test_control_flow.py::TestControlFlow::test_scan_init_non_tensor, test/functorch/test_control_flow.py::TestControlFlow::test_scan_init_pytree_complex_reverse_False_compile_mode_eager_cuda_autograd_False, test/functorch/test_control_flow.py::TestControlFlow::test_scan_init_pytree_complex_reverse_False_compile_mode_none_cpu_autograd_False, test/functorch/test_control_flow.py::TestControlFlow::test_scan_init_pytree_complex_reverse_False_compile_mode_none_cpu_autograd_True, test/functorch/test_control_flow.py::TestControlFlow::test_scan_init_pytree_complex_reverse_False_compile_mode_none_cuda_autograd_True, test/functorch/test_control_flow.py::TestControlFlow::test_scan_init_pytree_complex_reverse_True_compile_mode_eager_cpu_autograd_True, test/functorch/test_control_flow.py::TestControlFlow::test_scan_init_pytree_complex_reverse_True_compile_mode_none_cpu_autograd_False, test/functorch/test_control_flow.py::TestControlFlow::test_scan_init_pytree_complex_reverse_True_compile_mode_none_cpu_autograd_True, test/functorch/test_control_flow.py::TestControlFlow::test_scan_init_pytree_complex_reverse_True_compile_mode_none_cuda_autograd_True, test/functorch/test_control_flow.py::TestControlFlow::test_scan_init_reverse_False_compile_mode_eager_cpu_autograd_True, test/functorch/test_control_flow.py::TestControlFlow::test_scan_init_reverse_False_compile_mode_eager_cuda_autograd_True, test/functorch/test_control_flow.py::TestControlFlow::test_scan_init_reverse_True_compile_mode_eager_cpu_autograd_True, test/functorch/test_control_flow.py::TestControlFlow::test_scan_init_reverse_True_compile_mode_none_cpu_autograd_False, test/functorch/test_control_flow.py::TestControlFlow::test_scan_init_reverse_True_compile_mode_none_cpu_autograd_True, test/functorch/test_control_flow.py::TestControlFlow::test_scan_init_scanned_0, test/functorch/test_control_flow.py::TestControlFlow::test_scan_init_wrong_pytree_complex_reverse_False_cpu, test/functorch/test_control_flow.py::TestControlFlow::test_scan_init_wrong_pytree_complex_reverse_False_cuda, test/functorch/test_control_flow.py::TestControlFlow::test_scan_init_wrong_pytree_complex_reverse_True_cpu, test/functorch/test_control_flow.py::TestControlFlow::test_scan_init_wrong_shape, test/functorch/test_control_flow.py::TestControlFlow::test_scan_input_mutation, test/functorch/test_control_flow.py::TestControlFlow::test_scan_multiple_layers_gradient_layers_3_device_cpu, test/functorch/test_control_flow.py::TestControlFlow::test_scan_multiple_layers_gradient_layers_3_device_cuda, test/functorch/test_control_flow.py::TestControlFlow::test_scan_non_pointwise_reverse_False_compile_mode_eager_cpu_autograd_True, test/functorch/test_control_flow.py::TestControlFlow::test_scan_non_pointwise_reverse_True_compile_mode_none_cpu_autograd_False, test/functorch/test_control_flow.py::TestControlFlow::test_scan_non_pointwise_reverse_True_compile_mode_none_cpu_autograd_True, test/functorch/test_control_flow.py::TestControlFlow::test_scan_tuple_reverse_False_compile_mode_eager_cuda_autograd_True, test/functorch/test_control_flow.py::TestControlFlow::test_scan_tuple_reverse_False_compile_mode_none_cpu_autograd_False, test/functorch/test_control_flow.py::TestControlFlow::test_scan_tuple_reverse_False_compile_mode_none_cpu_autograd_True, test/functorch/test_control_flow.py::TestControlFlow::test_scan_tuple_reverse_False_compile_mode_none_cuda_autograd_False, test/functorch/test_control_flow.py::TestControlFlow::test_scan_tuple_reverse_True_compile_mode_eager_cuda_autograd_False, test/functorch/test_control_flow.py::TestControlFlow::test_scan_tuple_reverse_True_compile_mode_eager_cuda_autograd_True, test/functorch/test_control_flow.py::TestControlFlow::test_scan_tuple_reverse_True_compile_mode_none_cuda_autograd_False, test/functorch/test_control_flow.py::TestControlFlow::test_scan_tuple_reverse_True_compile_mode_none_cuda_autograd_True, test/functorch/test_control_flow.py::TestControlFlow::test_while_loop_gpu, test/functorch/test_control_flow.py::AssociativeScanTests::test_associative_scan_binary_operator_compile_mode_compile_combine_mode_generic_reverse_False_cuda_autograd_False, test/functorch/test_control_flow.py::AssociativeScanTests::test_associative_scan_binary_operator_compile_mode_compile_combine_mode_generic_reverse_True_cpu_autograd_False, test/functorch/test_control_flow.py::AssociativeScanTests::test_associative_scan_binary_operator_compile_mode_compile_combine_mode_generic_reverse_True_cuda_autograd_False, test/functorch/test_control_flow.py::AssociativeScanTests::test_associative_scan_binary_operator_compile_mode_compile_combine_mode_pointwise_reverse_False_cuda_autograd_False, test/functorch/test_control_flow.py::AssociativeScanTests::test_associative_scan_binary_operator_compile_mode_compile_combine_mode_pointwise_reverse_True_cpu_autograd_True, test/functorch/test_control_flow.py::AssociativeScanTests::test_associative_scan_binary_operator_compile_mode_compile_combine_mode_pointwise_reverse_True_cuda_autograd_False, test/functorch/test_control_flow.py::AssociativeScanTests::test_associative_scan_binary_operator_compile_mode_compile_dynamic_shape_combine_mode_generic_reverse_False_cpu_autograd_True, test/functorch/test_control_flow.py::AssociativeScanTests::test_associative_scan_binary_operator_compile_mode_compile_dynamic_shape_combine_mode_generic_reverse_False_cuda_autograd_False, test/functorch/test_control_flow.py::AssociativeScanTests::test_associative_scan_binary_operator_compile_mode_compile_dynamic_shape_combine_mode_generic_reverse_False_cuda_autograd_True, test/functorch/test_control_flow.py::AssociativeScanTests::test_associative_scan_binary_operator_compile_mode_compile_dynamic_shape_combine_mode_generic_reverse_True_cpu_autograd_True, test/functorch/test_control_flow.py::AssociativeScanTests::test_associative_scan_binary_operator_compile_mode_compile_dynamic_shape_combine_mode_generic_reverse_True_cuda_autograd_False, test/functorch/test_control_flow.py::AssociativeScanTests::test_associative_scan_binary_operator_compile_mode_compile_dynamic_shape_combine_mode_pointwise_reverse_False_cpu_autograd_True, test/functorch/test_control_flow.py::AssociativeScanTests::test_associative_scan_binary_operator_compile_mode_compile_dynamic_shape_combine_mode_pointwise_reverse_False_cuda_autograd_False, test/functorch/test_control_flow.py::AssociativeScanTests::test_associative_scan_binary_operator_compile_mode_compile_dynamic_shape_combine_mode_pointwise_reverse_True_cpu_autograd_False, test/functorch/test_control_flow.py::AssociativeScanTests::test_associative_scan_binary_operator_compile_mode_eager_combine_mode_generic_reverse_True_cpu_autograd_True, test/functorch/test_control_flow.py::AssociativeScanTests::test_associative_scan_binary_operator_compile_mode_eager_combine_mode_pointwise_reverse_False_cpu_autograd_False, test/functorch/test_control_flow.py::AssociativeScanTests::test_associative_scan_binary_operator_compile_mode_eager_combine_mode_pointwise_reverse_False_cpu_autograd_True, test/functorch/test_control_flow.py::AssociativeScanTests::test_associative_scan_binary_operator_compile_mode_eager_combine_mode_pointwise_reverse_True_cuda_autograd_True, test/functorch/test_control_flow.py::AssociativeScanTests::test_associative_scan_binary_operator_compile_mode_none_combine_mode_pointwise_reverse_True_cpu_autograd_True, test/functorch/test_control_flow.py::AssociativeScanTests::test_associative_scan_binary_operator_compile_mode_none_combine_mode_pointwise_reverse_True_cuda_autograd_True, test/functorch/test_control_flow.py::AssociativeScanTests::test_associative_scan_combine_fn_wrong_meta_in_combine_fn, test/functorch/test_control_flow.py::AssociativeScanTests::test_associative_scan_compile_reverse_False_compile_mode_compile_combine_mode_generic_cuda_autograd_False, test/functorch/test_control_flow.py::AssociativeScanTests::test_associative_scan_compile_reverse_False_compile_mode_compile_combine_mode_pointwise_cpu_autograd_False, test/functorch/test_control_flow.py::AssociativeScanTests::test_associative_scan_compile_reverse_False_compile_mode_compile_combine_mode_pointwise_cuda_autograd_True, test/functorch/test_control_flow.py::AssociativeScanTests::test_associative_scan_compile_reverse_False_compile_mode_compile_dynamic_shape_combine_mode_generic_cpu_autograd_True, test/functorch/test_control_flow.py::AssociativeScanTests::test_associative_scan_compile_reverse_False_compile_mode_compile_dynamic_shape_combine_mode_pointwise_cuda_autograd_False, test/functorch/test_control_flow.py::AssociativeScanTests::test_associative_scan_compile_reverse_False_compile_mode_eager_combine_mode_generic_cpu_autograd_False, test/functorch/test_control_flow.py::AssociativeScanTests::test_associative_scan_compile_reverse_False_compile_mode_eager_combine_mode_generic_cuda_autograd_True, test/functorch/test_control_flow.py::AssociativeScanTests::test_associative_scan_compile_reverse_False_compile_mode_eager_combine_mode_pointwise_cuda_autograd_False, test/functorch/test_control_flow.py::AssociativeScanTests::test_associative_scan_compile_reverse_False_compile_mode_eager_combine_mode_pointwise_cuda_autograd_True, test/functorch/test_control_flow.py::AssociativeScanTests::test_associative_scan_compile_reverse_False_compile_mode_none_combine_mode_pointwise_cpu_autograd_False, test/functorch/test_control_flow.py::AssociativeScanTests::test_associative_scan_compile_reverse_False_compile_mode_none_combine_mode_pointwise_cpu_autograd_True, test/functorch/test_control_flow.py::AssociativeScanTests::test_associative_scan_compile_reverse_True_compile_mode_compile_combine_mode_generic_cuda_autograd_False, test/functorch/test_control_flow.py::AssociativeScanTests::test_associative_scan_compile_reverse_True_compile_mode_compile_combine_mode_pointwise_cuda_autograd_False, test/functorch/test_control_flow.py::AssociativeScanTests::test_associative_scan_compile_reverse_True_compile_mode_compile_dynamic_shape_combine_mode_generic_cpu_autograd_True, test/functorch/test_control_flow.py::AssociativeScanTests::test_associative_scan_compile_reverse_True_compile_mode_compile_dynamic_shape_combine_mode_pointwise_cpu_autograd_True, test/functorch/test_control_flow.py::AssociativeScanTests::test_associative_scan_compile_reverse_True_compile_mode_eager_combine_mode_pointwise_cuda_autograd_False, test/functorch/test_control_flow.py::AssociativeScanTests::test_associative_scan_compile_reverse_True_compile_mode_eager_combine_mode_pointwise_cuda_autograd_True, test/functorch/test_control_flow.py::AssociativeScanTests::test_associative_scan_compile_reverse_True_compile_mode_none_combine_mode_generic_cuda_autograd_False, test/functorch/test_control_flow.py::AssociativeScanTests::test_associative_scan_compile_reverse_True_compile_mode_none_combine_mode_generic_cuda_autograd_True, test/functorch/test_control_flow.py::AssociativeScanTests::test_associative_scan_complex_pytree_compile_mode_compile_combine_mode_generic_reverse_False_cpu_autograd_True, test/functorch/test_control_flow.py::AssociativeScanTests::test_associative_scan_complex_pytree_compile_mode_compile_combine_mode_generic_reverse_False_cuda_autograd_False, test/functorch/test_control_flow.py::AssociativeScanTests::test_associative_scan_complex_pytree_compile_mode_compile_combine_mode_generic_reverse_True_cuda_autograd_False, test/functorch/test_control_flow.py::AssociativeScanTests::test_associative_scan_complex_pytree_compile_mode_compile_combine_mode_pointwise_reverse_True_cpu_autograd_False, test/functorch/test_control_flow.py::AssociativeScanTests::test_associative_scan_complex_pytree_compile_mode_compile_combine_mode_pointwise_reverse_True_cuda_autograd_False, test/functorch/test_control_flow.py::AssociativeScanTests::test_associative_scan_complex_pytree_compile_mode_compile_dynamic_shape_combine_mode_generic_reverse_True_cpu_autograd_False, test/functorch/test_control_flow.py::AssociativeScanTests::test_associative_scan_complex_pytree_compile_mode_compile_dynamic_shape_combine_mode_generic_reverse_True_cuda_autograd_True, test/functorch/test_control_flow.py::AssociativeScanTests::test_associative_scan_complex_pytree_compile_mode_compile_dynamic_shape_combine_mode_pointwise_reverse_False_cuda_autograd_False, test/functorch/test_control_flow.py::AssociativeScanTests::test_associative_scan_complex_pytree_compile_mode_compile_dynamic_shape_combine_mode_pointwise_reverse_True_cpu_autograd_False, test/functorch/test_control_flow.py::AssociativeScanTests::test_associative_scan_complex_pytree_compile_mode_compile_dynamic_shape_combine_mode_pointwise_reverse_True_cuda_autograd_False, test/functorch/test_control_flow.py::AssociativeScanTests::test_associative_scan_complex_pytree_compile_mode_eager_combine_mode_generic_reverse_True_cpu_autograd_True, test/functorch/test_control_flow.py::AssociativeScanTests::test_associative_scan_complex_pytree_compile_mode_eager_combine_mode_pointwise_reverse_False_cpu_autograd_True, test/functorch/test_control_flow.py::AssociativeScanTests::test_associative_scan_complex_pytree_compile_mode_eager_combine_mode_pointwise_reverse_True_cpu_autograd_False, test/functorch/test_control_flow.py::AssociativeScanTests::test_associative_scan_complex_pytree_compile_mode_eager_combine_mode_pointwise_reverse_True_cuda_autograd_False, test/functorch/test_control_flow.py::AssociativeScanTests::test_associative_scan_complex_pytree_compile_mode_none_combine_mode_generic_reverse_False_cpu_autograd_False, test/functorch/test_control_flow.py::AssociativeScanTests::test_associative_scan_complex_pytree_compile_mode_none_combine_mode_generic_reverse_True_cpu_autograd_False, test/functorch/test_control_flow.py::AssociativeScanTests::test_associative_scan_complex_pytree_compile_mode_none_combine_mode_generic_reverse_True_cpu_autograd_True, test/functorch/test_control_flow.py::AssociativeScanTests::test_associative_scan_complex_pytree_compile_mode_none_combine_mode_generic_reverse_True_cuda_autograd_True, test/functorch/test_control_flow.py::AssociativeScanTests::test_associative_scan_complex_pytree_compile_mode_none_combine_mode_pointwise_reverse_True_cuda_autograd_True, test/functorch/test_control_flow.py::AssociativeScanTests::test_associative_scan_cond_in_combine_fn_compile_mode_compile_dynamic_shape_reverse_False_cpu_autograd_False, test/functorch/test_control_flow.py::AssociativeScanTests::test_associative_scan_cond_in_combine_fn_compile_mode_compile_reverse_True_cpu_autograd_False, test/functorch/test_control_flow.py::AssociativeScanTests::test_associative_scan_cond_in_combine_fn_compile_mode_compile_reverse_True_cuda_autograd_False, test/functorch/test_control_flow.py::AssociativeScanTests::test_associative_scan_cond_in_combine_fn_compile_mode_eager_reverse_False_cpu_autograd_True, test/functorch/test_control_flow.py::AssociativeScanTests::test_associative_scan_cond_in_combine_fn_compile_mode_eager_reverse_True_cpu_autograd_False, test/functorch/test_control_flow.py::AssociativeScanTests::test_associative_scan_cond_in_combine_fn_compile_mode_eager_reverse_True_cpu_autograd_True, test/functorch/test_control_flow.py::AssociativeScanTests::test_associative_scan_cond_in_combine_fn_compile_mode_eager_reverse_True_cuda_autograd_False, test/functorch/test_control_flow.py::AssociativeScanTests::test_associative_scan_cond_in_combine_fn_compile_mode_none_reverse_False_cpu_autograd_True, test/functorch/test_control_flow.py::AssociativeScanTests::test_associative_scan_cond_in_combine_fn_compile_mode_none_reverse_False_cuda_autograd_True, test/functorch/test_control_flow.py::AssociativeScanTests::test_associative_scan_cond_in_combine_fn_compile_mode_none_reverse_True_cuda_autograd_False, test/functorch/test_control_flow.py::AssociativeScanTests::test_associative_scan_different_input_size_compile_mode_compile_dynamic_shape_reverse_True_cuda, test/functorch/test_control_flow.py::AssociativeScanTests::test_associative_scan_different_input_size_compile_mode_compile_reverse_False_cpu, test/functorch/test_control_flow.py::AssociativeScanTests::test_associative_scan_different_input_size_compile_mode_eager_reverse_False_cuda, test/functorch/test_control_flow.py::AssociativeScanTests::test_associative_scan_different_input_size_compile_mode_eager_reverse_True_cpu, test/functorch/test_control_flow.py::AssociativeScanTests::test_associative_scan_dim_reverse_False_compile_mode_compile_combine_mode_pointwise_cuda_autograd_True, test/functorch/test_control_flow.py::AssociativeScanTests::test_associative_scan_dim_reverse_False_compile_mode_compile_dynamic_shape_combine_mode_generic_cuda_autograd_False, test/functorch/test_control_flow.py::AssociativeScanTests::test_associative_scan_dim_reverse_False_compile_mode_compile_dynamic_shape_combine_mode_generic_cuda_autograd_True, test/functorch/test_control_flow.py::AssociativeScanTests::test_associative_scan_dim_reverse_False_compile_mode_compile_dynamic_shape_combine_mode_pointwise_cpu_autograd_True, test/functorch/test_control_flow.py::AssociativeScanTests::test_associative_scan_dim_reverse_False_compile_mode_compile_dynamic_shape_combine_mode_pointwise_cuda_autograd_False, test/functorch/test_control_flow.py::AssociativeScanTests::test_associative_scan_dim_reverse_False_compile_mode_eager_combine_mode_generic_cpu_autograd_False, test/functorch/test_control_flow.py::AssociativeScanTests::test_associative_scan_dim_reverse_False_compile_mode_eager_combine_mode_generic_cpu_autograd_True, test/functorch/test_control_flow.py::AssociativeScanTests::test_associative_scan_dim_reverse_False_compile_mode_eager_combine_mode_generic_cuda_autograd_True, test/functorch/test_control_flow.py::AssociativeScanTests::test_associative_scan_dim_reverse_False_compile_mode_eager_combine_mode_pointwise_cpu_autograd_True, test/functorch/test_control_flow.py::AssociativeScanTests::test_associative_scan_dim_reverse_False_compile_mode_eager_combine_mode_pointwise_cuda_autograd_False, test/functorch/test_control_flow.py::AssociativeScanTests::test_associative_scan_dim_reverse_False_compile_mode_none_combine_mode_generic_cpu_autograd_False, test/functorch/test_control_flow.py::AssociativeScanTests::test_associative_scan_dim_reverse_False_compile_mode_none_combine_mode_generic_cpu_autograd_True, test/functorch/test_control_flow.py::AssociativeScanTests::test_associative_scan_dim_reverse_False_compile_mode_none_combine_mode_generic_cuda_autograd_True, test/functorch/test_control_flow.py::AssociativeScanTests::test_associative_scan_dim_reverse_False_compile_mode_none_combine_mode_pointwise_cuda_autograd_False, test/functorch/test_control_flow.py::AssociativeScanTests::test_associative_scan_dim_reverse_True_compile_mode_compile_combine_mode_generic_cuda_autograd_False, test/functorch/test_control_flow.py::AssociativeScanTests::test_associative_scan_dim_reverse_True_compile_mode_compile_combine_mode_pointwise_cpu_autograd_False, test/functorch/test_control_flow.py::AssociativeScanTests::test_associative_scan_dim_reverse_True_compile_mode_compile_dynamic_shape_combine_mode_pointwise_cpu_autograd_False, test/functorch/test_control_flow.py::AssociativeScanTests::test_associative_scan_dim_reverse_True_compile_mode_compile_dynamic_shape_combine_mode_pointwise_cuda_autograd_True, test/functorch/test_control_flow.py::AssociativeScanTests::test_associative_scan_dim_reverse_True_compile_mode_eager_combine_mode_generic_cpu_autograd_True, test/functorch/test_control_flow.py::AssociativeScanTests::test_associative_scan_dim_reverse_True_compile_mode_eager_combine_mode_pointwise_cuda_autograd_False, test/functorch/test_control_flow.py::AssociativeScanTests::test_associative_scan_dim_reverse_True_compile_mode_none_combine_mode_generic_cuda_autograd_False, test/functorch/test_control_flow.py::AssociativeScanTests::test_associative_scan_dim_reverse_True_compile_mode_none_combine_mode_pointwise_cpu_autograd_False, test/functorch/test_control_flow.py::AssociativeScanTests::test_associative_scan_dim_reverse_True_compile_mode_none_combine_mode_pointwise_cuda_autograd_True, test/functorch/test_control_flow.py::AssociativeScanTests::test_associative_scan_downstream_scan_matmul_combine_mode_generic_compile_mode_compile_dynamic_shape_reverse_False_cpu_autograd_False, test/functorch/test_control_flow.py::AssociativeScanTests::test_associative_scan_downstream_scan_matmul_combine_mode_generic_compile_mode_compile_dynamic_shape_reverse_False_cuda_autograd_False, test/functorch/test_control_flow.py::AssociativeScanTests::test_associative_scan_downstream_scan_matmul_combine_mode_generic_compile_mode_compile_dynamic_shape_reverse_True_cuda_autograd_True, test/functorch/test_control_flow.py::AssociativeScanTests::test_associative_scan_downstream_scan_matmul_combine_mode_generic_compile_mode_compile_reverse_False_cpu_autograd_False, test/functorch/test_control_flow.py::AssociativeScanTests::test_associative_scan_downstream_scan_matmul_combine_mode_generic_compile_mode_compile_reverse_True_cpu_autograd_False, test/functorch/test_control_flow.py::AssociativeScanTests::test_associative_scan_downstream_scan_matmul_combine_mode_generic_compile_mode_compile_reverse_True_cuda_autograd_False, test/functorch/test_control_flow.py::AssociativeScanTests::test_associative_scan_downstream_scan_matmul_combine_mode_generic_compile_mode_eager_reverse_False_cpu_autograd_False, test/functorch/test_control_flow.py::AssociativeScanTests::test_associative_scan_downstream_scan_matmul_combine_mode_generic_compile_mode_eager_reverse_False_cuda_autograd_False, test/functorch/test_control_flow.py::AssociativeScanTests::test_associative_scan_downstream_scan_matmul_combine_mode_generic_compile_mode_eager_reverse_True_cpu_autograd_False, test/functorch/test_control_flow.py::AssociativeScanTests::test_associative_scan_downstream_scan_matmul_combine_mode_generic_compile_mode_eager_reverse_True_cuda_autograd_False, test/functorch/test_control_flow.py::AssociativeScanTests::test_associative_scan_downstream_scan_matmul_combine_mode_generic_compile_mode_eager_reverse_True_cuda_autograd_True, test/functorch/test_control_flow.py::AssociativeScanTests::test_associative_scan_downstream_scan_matmul_combine_mode_generic_compile_mode_none_reverse_True_cpu_autograd_True, test/functorch/test_control_flow.py::AssociativeScanTests::test_associative_scan_downstream_scan_matmul_combine_mode_pointwise_compile_mode_compile_dynamic_shape_reverse_False_cuda_autograd_False, test/functorch/test_control_flow.py::AssociativeScanTests::test_associative_scan_downstream_scan_matmul_combine_mode_pointwise_compile_mode_compile_dynamic_shape_reverse_False_cuda_autograd_True, test/functorch/test_control_flow.py::AssociativeScanTests::test_associative_scan_downstream_scan_matmul_combine_mode_pointwise_compile_mode_compile_dynamic_shape_reverse_True_cpu_autograd_True, test/functorch/test_control_flow.py::AssociativeScanTests::test_associative_scan_downstream_scan_matmul_combine_mode_pointwise_compile_mode_compile_reverse_False_cpu_autograd_False, test/functorch/test_control_flow.py::AssociativeScanTests::test_associative_scan_downstream_scan_matmul_combine_mode_pointwise_compile_mode_compile_reverse_False_cuda_autograd_False, test/functorch/test_control_flow.py::AssociativeScanTests::test_associative_scan_downstream_scan_matmul_combine_mode_pointwise_compile_mode_compile_reverse_False_cuda_autograd_True, test/functorch/test_control_flow.py::AssociativeScanTests::test_associative_scan_downstream_scan_matmul_combine_mode_pointwise_compile_mode_eager_reverse_False_cpu_autograd_True, test/functorch/test_control_flow.py::AssociativeScanTests::test_associative_scan_downstream_scan_matmul_combine_mode_pointwise_compile_mode_eager_reverse_False_cuda_autograd_True, test/functorch/test_control_flow.py::AssociativeScanTests::test_associative_scan_downstream_scan_matmul_combine_mode_pointwise_compile_mode_eager_reverse_True_cuda_autograd_True, test/functorch/test_control_flow.py::AssociativeScanTests::test_associative_scan_downstream_scan_matmul_combine_mode_pointwise_compile_mode_none_reverse_False_cpu_autograd_True, test/functorch/test_control_flow.py::AssociativeScanTests::test_associative_scan_downstream_scan_matmul_combine_mode_pointwise_compile_mode_none_reverse_True_cuda_autograd_False, test/functorch/test_control_flow.py::AssociativeScanTests::test_associative_scan_downstream_scan_matmul_combine_mode_pointwise_compile_mode_none_reverse_True_cuda_autograd_True, test/functorch/test_control_flow.py::AssociativeScanTests::test_associative_scan_downstream_scan_scan_combine_mode_generic_compile_mode_compile_dynamic_shape_reverse_False_cpu_autograd_False, test/functorch/test_control_flow.py::AssociativeScanTests::test_associative_scan_downstream_scan_scan_combine_mode_generic_compile_mode_compile_dynamic_shape_reverse_True_cpu_autograd_False, test/functorch/test_control_flow.py::AssociativeScanTests::test_associative_scan_downstream_scan_scan_combine_mode_generic_compile_mode_compile_reverse_False_cpu_autograd_True, test/functorch/test_control_flow.py::AssociativeScanTests::test_associative_scan_downstream_scan_scan_combine_mode_generic_compile_mode_compile_reverse_False_cuda_autograd_False, test/functorch/test_control_flow.py::AssociativeScanTests::test_associative_scan_downstream_scan_scan_combine_mode_generic_compile_mode_compile_reverse_True_cpu_autograd_True, test/functorch/test_control_flow.py::AssociativeScanTests::test_associative_scan_downstream_scan_scan_combine_mode_generic_compile_mode_eager_reverse_False_cpu_autograd_False, test/functorch/test_control_flow.py::AssociativeScanTests::test_associative_scan_downstream_scan_scan_combine_mode_generic_compile_mode_eager_reverse_False_cuda_autograd_False, test/functorch/test_control_flow.py::AssociativeScanTests::test_associative_scan_downstream_scan_scan_combine_mode_generic_compile_mode_none_reverse_False_cpu_autograd_False, test/functorch/test_control_flow.py::AssociativeScanTests::test_associative_scan_downstream_scan_scan_combine_mode_generic_compile_mode_none_reverse_False_cpu_autograd_True, test/functorch/test_control_flow.py::AssociativeScanTests::test_associative_scan_downstream_scan_scan_combine_mode_pointwise_compile_mode_compile_dynamic_shape_reverse_False_cpu_autograd_False, test/functorch/test_control_flow.py::AssociativeScanTests::test_associative_scan_downstream_scan_scan_combine_mode_pointwise_compile_mode_compile_dynamic_shape_reverse_True_cpu_autograd_False, test/functorch/test_control_flow.py::AssociativeScanTests::test_associative_scan_downstream_scan_scan_combine_mode_pointwise_compile_mode_compile_dynamic_shape_reverse_True_cpu_autograd_True, test/functorch/test_control_flow.py::AssociativeScanTests::test_associative_scan_downstream_scan_scan_combine_mode_pointwise_compile_mode_compile_dynamic_shape_reverse_True_cuda_autograd_True, test/functorch/test_control_flow.py::AssociativeScanTests::test_associative_scan_downstream_scan_scan_combine_mode_pointwise_compile_mode_compile_reverse_False_cpu_autograd_True, test/functorch/test_control_flow.py::AssociativeScanTests::test_associative_scan_downstream_scan_scan_combine_mode_pointwise_compile_mode_compile_reverse_True_cuda_autograd_False, test/functorch/test_control_flow.py::AssociativeScanTests::test_associative_scan_downstream_scan_scan_combine_mode_pointwise_compile_mode_compile_reverse_True_cuda_autograd_True, test/functorch/test_control_flow.py::AssociativeScanTests::test_associative_scan_downstream_scan_scan_combine_mode_pointwise_compile_mode_eager_reverse_False_cpu_autograd_False, test/functorch/test_control_flow.py::AssociativeScanTests::test_associative_scan_downstream_scan_scan_combine_mode_pointwise_compile_mode_eager_reverse_False_cuda_autograd_True, test/functorch/test_control_flow.py::AssociativeScanTests::test_associative_scan_downstream_scan_scan_different_dim_combine_mode_generic_compile_mode_compile_dynamic_shape_reverse_first_False_same_direction_False_cpu_autograd_False, test/functorch/test_control_flow.py::AssociativeScanTests::test_associative_scan_downstream_scan_scan_different_dim_combine_mode_generic_compile_mode_compile_dynamic_shape_reverse_first_False_same_direction_True_cpu_autograd_False, test/functorch/test_control_flow.py::AssociativeScanTests::test_associative_scan_downstream_scan_scan_different_dim_combine_mode_generic_compile_mode_compile_dynamic_shape_reverse_first_True_same_direction_True_cpu_autograd_False, test/functorch/test_control_flow.py::AssociativeScanTests::test_associative_scan_downstream_scan_scan_different_dim_combine_mode_generic_compile_mode_compile_reverse_first_False_same_direction_False_cuda_autograd_True, test/functorch/test_control_flow.py::AssociativeScanTests::test_associative_scan_downstream_scan_scan_different_dim_combine_mode_generic_compile_mode_compile_reverse_first_False_same_direction_True_cuda_autograd_False, test/functorch/test_control_flow.py::AssociativeScanTests::test_associative_scan_downstream_scan_scan_different_dim_combine_mode_generic_compile_mode_compile_reverse_first_True_same_direction_False_cuda_autograd_False, test/functorch/test_control_flow.py::AssociativeScanTests::test_associative_scan_downstream_scan_scan_different_dim_combine_mode_generic_compile_mode_compile_reverse_first_True_same_direction_True_cpu_autograd_False, test/functorch/test_control_flow.py::AssociativeScanTests::test_associative_scan_downstream_scan_scan_different_dim_combine_mode_generic_compile_mode_compile_reverse_first_True_same_direction_True_cuda_autograd_False, test/functorch/test_control_flow.py::AssociativeScanTests::test_associative_scan_downstream_scan_scan_different_dim_combine_mode_generic_compile_mode_compile_reverse_first_True_same_direction_True_cuda_autograd_True, test/functorch/test_control_flow.py::AssociativeScanTests::test_associative_scan_downstream_scan_scan_different_dim_combine_mode_generic_compile_mode_eager_reverse_first_False_same_direction_True_cpu_autograd_False, test/functorch/test_control_flow.py::AssociativeScanTests::test_associative_scan_downstream_scan_scan_different_dim_combine_mode_generic_compile_mode_eager_reverse_first_False_same_direction_True_cpu_autograd_True, test/functorch/test_control_flow.py::AssociativeScanTests::test_associative_scan_downstream_scan_scan_different_dim_combine_mode_generic_compile_mode_eager_reverse_first_True_same_direction_True_cpu_autograd_False, test/functorch/test_control_flow.py::AssociativeScanTests::test_associative_scan_downstream_scan_scan_different_dim_combine_mode_generic_compile_mode_none_reverse_first_False_same_direction_False_cpu_autograd_True, test/functorch/test_control_flow.py::AssociativeScanTests::test_associative_scan_downstream_scan_scan_different_dim_combine_mode_generic_compile_mode_none_reverse_first_False_same_direction_False_cuda_autograd_False, test/functorch/test_control_flow.py::AssociativeScanTests::test_associative_scan_downstream_scan_scan_different_dim_combine_mode_generic_compile_mode_none_reverse_first_False_same_direction_True_cuda_autograd_False, test/functorch/test_control_flow.py::AssociativeScanTests::test_associative_scan_downstream_scan_scan_different_dim_combine_mode_generic_compile_mode_none_reverse_first_True_same_direction_False_cuda_autograd_False, test/functorch/test_control_flow.py::AssociativeScanTests::test_associative_scan_downstream_scan_scan_different_dim_combine_mode_pointwise_compile_mode_compile_dynamic_shape_reverse_first_False_same_direction_False_cuda_autograd_False, test/functorch/test_control_flow.py::AssociativeScanTests::test_associative_scan_downstream_scan_scan_different_dim_combine_mode_pointwise_compile_mode_compile_dynamic_shape_reverse_first_False_same_direction_False_cuda_autograd_True, test/functorch/test_control_flow.py::AssociativeScanTests::test_associative_scan_downstream_scan_scan_different_dim_combine_mode_pointwise_compile_mode_compile_dynamic_shape_reverse_first_True_same_direction_False_cuda_autograd_False, test/functorch/test_control_flow.py::AssociativeScanTests::test_associative_scan_downstream_scan_scan_different_dim_combine_mode_pointwise_compile_mode_compile_dynamic_shape_reverse_first_True_same_direction_True_cuda_autograd_False, test/functorch/test_control_flow.py::AssociativeScanTests::test_associative_scan_downstream_scan_scan_different_dim_combine_mode_pointwise_compile_mode_compile_reverse_first_False_same_direction_False_cuda_autograd_False, test/functorch/test_control_flow.py::AssociativeScanTests::test_associative_scan_downstream_scan_scan_different_dim_combine_mode_pointwise_compile_mode_compile_reverse_first_False_same_direction_False_cuda_autograd_True, test/functorch/test_control_flow.py::AssociativeScanTests::test_associative_scan_downstream_scan_scan_different_dim_combine_mode_pointwise_compile_mode_compile_reverse_first_False_same_direction_True_cpu_autograd_True, test/functorch/test_control_flow.py::AssociativeScanTests::test_associative_scan_downstream_scan_scan_different_dim_combine_mode_pointwise_compile_mode_compile_reverse_first_False_same_direction_True_cuda_autograd_False, test/functorch/test_control_flow.py::AssociativeScanTests::test_associative_scan_downstream_scan_scan_different_dim_combine_mode_pointwise_compile_mode_compile_reverse_first_True_same_direction_True_cpu_autograd_True, test/functorch/test_control_flow.py::AssociativeScanTests::test_associative_scan_downstream_scan_scan_different_dim_combine_mode_pointwise_compile_mode_eager_reverse_first_False_same_direction_False_cpu_autograd_False, test/functorch/test_control_flow.py::AssociativeScanTests::test_associative_scan_downstream_scan_scan_different_dim_combine_mode_pointwise_compile_mode_eager_reverse_first_False_same_direction_True_cuda_autograd_False, test/functorch/test_control_flow.py::AssociativeScanTests::test_associative_scan_downstream_scan_scan_different_dim_combine_mode_pointwise_compile_mode_eager_reverse_first_False_same_direction_True_cuda_autograd_True, test/functorch/test_control_flow.py::AssociativeScanTests::test_associative_scan_downstream_scan_scan_different_dim_combine_mode_pointwise_compile_mode_eager_reverse_first_True_same_direction_False_cpu_autograd_False, test/functorch/test_control_flow.py::AssociativeScanTests::test_associative_scan_downstream_scan_scan_different_dim_combine_mode_pointwise_compile_mode_eager_reverse_first_True_same_direction_True_cpu_autograd_False, test/functorch/test_control_flow.py::AssociativeScanTests::test_associative_scan_downstream_scan_scan_different_dim_combine_mode_pointwise_compile_mode_eager_reverse_first_True_same_direction_True_cpu_autograd_True, test/functorch/test_control_flow.py::AssociativeScanTests::test_associative_scan_downstream_scan_scan_different_dim_combine_mode_pointwise_compile_mode_none_reverse_first_False_same_direction_False_cpu_autograd_False, test/functorch/test_control_flow.py::AssociativeScanTests::test_associative_scan_downstream_scan_scan_different_dim_combine_mode_pointwise_compile_mode_none_reverse_first_False_same_direction_False_cuda_autograd_False, test/functorch/test_control_flow.py::AssociativeScanTests::test_associative_scan_downstream_scan_scan_different_dim_combine_mode_pointwise_compile_mode_none_reverse_first_False_same_direction_True_cpu_autograd_False, test/functorch/test_control_flow.py::AssociativeScanTests::test_associative_scan_downstream_scan_scan_different_dim_combine_mode_pointwise_compile_mode_none_reverse_first_True_same_direction_False_cuda_autograd_False, test/functorch/test_control_flow.py::AssociativeScanTests::test_associative_scan_downstream_scan_scan_different_dim_combine_mode_pointwise_compile_mode_none_reverse_first_True_same_direction_False_cuda_autograd_True, test/functorch/test_control_flow.py::AssociativeScanTests::test_associative_scan_downstream_scan_scan_different_dim_combine_mode_pointwise_compile_mode_none_reverse_first_True_same_direction_True_cpu_autograd_True, test/functorch/test_control_flow.py::AssociativeScanTests::test_associative_scan_expand_in_combine_fn_compile_mode_compile_dynamic_shape_reverse_True_cpu_autograd_True, test/functorch/test_control_flow.py::AssociativeScanTests::test_associative_scan_expand_in_combine_fn_compile_mode_compile_dynamic_shape_reverse_True_cuda_autograd_False, test/functorch/test_control_flow.py::AssociativeScanTests::test_associative_scan_expand_in_combine_fn_compile_mode_compile_reverse_False_cpu_autograd_False, test/functorch/test_control_flow.py::AssociativeScanTests::test_associative_scan_expand_in_combine_fn_compile_mode_compile_reverse_False_cpu_autograd_True, test/functorch/test_control_flow.py::AssociativeScanTests::test_associative_scan_expand_in_combine_fn_compile_mode_compile_reverse_True_cpu_autograd_True, test/functorch/test_control_flow.py::AssociativeScanTests::test_associative_scan_expand_in_combine_fn_compile_mode_compile_reverse_True_cuda_autograd_False, test/functorch/test_control_flow.py::AssociativeScanTests::test_associative_scan_expand_in_combine_fn_compile_mode_eager_reverse_False_cuda_autograd_True, test/functorch/test_control_flow.py::AssociativeScanTests::test_associative_scan_expand_in_combine_fn_compile_mode_eager_reverse_True_cuda_autograd_False, test/functorch/test_control_flow.py::AssociativeScanTests::test_associative_scan_expand_in_combine_fn_compile_mode_none_reverse_False_cpu_autograd_False, test/functorch/test_control_flow.py::AssociativeScanTests::test_associative_scan_expand_in_combine_fn_compile_mode_none_reverse_False_cpu_autograd_True, test/functorch/test_control_flow.py::AssociativeScanTests::test_associative_scan_expand_in_combine_fn_compile_mode_none_reverse_False_cuda_autograd_True, test/functorch/test_control_flow.py::AssociativeScanTests::test_associative_scan_expand_in_combine_fn_compile_mode_none_reverse_True_cpu_autograd_False, test/functorch/test_control_flow.py::AssociativeScanTests::test_associative_scan_expand_in_combine_fn_compile_mode_none_reverse_True_cpu_autograd_True, test/functorch/test_control_flow.py::AssociativeScanTests::test_associative_scan_expand_in_combine_fn_compile_mode_none_reverse_True_cuda_autograd_False, test/functorch/test_control_flow.py::AssociativeScanTests::test_associative_scan_freevars_fct_compile_mode_compile_combine_mode_generic_reverse_False_cuda_autograd_True, test/functorch/test_control_flow.py::AssociativeScanTests::test_associative_scan_freevars_fct_compile_mode_compile_combine_mode_pointwise_reverse_False_cpu_autograd_False, test/functorch/test_control_flow.py::AssociativeScanTests::test_associative_scan_freevars_fct_compile_mode_compile_combine_mode_pointwise_reverse_False_cuda_autograd_False, test/functorch/test_control_flow.py::AssociativeScanTests::test_associative_scan_freevars_fct_compile_mode_compile_combine_mode_pointwise_reverse_True_cuda_autograd_True, test/functorch/test_control_flow.py::AssociativeScanTests::test_associative_scan_freevars_fct_compile_mode_compile_dynamic_shape_combine_mode_generic_reverse_False_cpu_autograd_False, test/functorch/test_control_flow.py::AssociativeScanTests::test_associative_scan_freevars_fct_compile_mode_compile_dynamic_shape_combine_mode_generic_reverse_False_cuda_autograd_True, test/functorch/test_control_flow.py::AssociativeScanTests::test_associative_scan_freevars_fct_compile_mode_compile_dynamic_shape_combine_mode_pointwise_reverse_False_cpu_autograd_True, test/functorch/test_control_flow.py::AssociativeScanTests::test_associative_scan_freevars_fct_compile_mode_compile_dynamic_shape_combine_mode_pointwise_reverse_True_cuda_autograd_False, test/functorch/test_control_flow.py::AssociativeScanTests::test_associative_scan_freevars_fct_compile_mode_compile_dynamic_shape_combine_mode_pointwise_reverse_True_cuda_autograd_True, test/functorch/test_control_flow.py::AssociativeScanTests::test_associative_scan_freevars_fct_compile_mode_eager_combine_mode_generic_reverse_False_cpu_autograd_False, test/functorch/test_control_flow.py::AssociativeScanTests::test_associative_scan_freevars_fct_compile_mode_eager_combine_mode_generic_reverse_False_cpu_autograd_True, test/functorch/test_control_flow.py::AssociativeScanTests::test_associative_scan_freevars_fct_compile_mode_eager_combine_mode_generic_reverse_False_cuda_autograd_False, test/functorch/test_control_flow.py::AssociativeScanTests::test_associative_scan_freevars_fct_compile_mode_eager_combine_mode_generic_reverse_True_cuda_autograd_False, test/functorch/test_control_flow.py::AssociativeScanTests::test_associative_scan_freevars_fct_compile_mode_eager_combine_mode_generic_reverse_True_cuda_autograd_True, test/functorch/test_control_flow.py::AssociativeScanTests::test_associative_scan_freevars_fct_compile_mode_eager_combine_mode_pointwise_reverse_True_cpu_autograd_True, test/functorch/test_control_flow.py::AssociativeScanTests::test_associative_scan_freevars_fct_compile_mode_eager_combine_mode_pointwise_reverse_True_cuda_autograd_True, test/functorch/test_control_flow.py::AssociativeScanTests::test_associative_scan_freevars_fct_compile_mode_none_combine_mode_pointwise_reverse_False_cpu_autograd_False, test/functorch/test_control_flow.py::AssociativeScanTests::test_associative_scan_freevars_fct_compile_mode_none_combine_mode_pointwise_reverse_False_cuda_autograd_False, test/functorch/test_control_flow.py::AssociativeScanTests::test_associative_scan_freevars_fct_compile_mode_none_combine_mode_pointwise_reverse_True_cpu_autograd_False, test/functorch/test_control_flow.py::AssociativeScanTests::test_associative_scan_freevars_fct_compile_mode_none_combine_mode_pointwise_reverse_True_cuda_autograd_True, test/functorch/test_control_flow.py::AssociativeScanTests::test_associative_scan_freevars_fct_generic_compile_mode_compile_dynamic_shape_reverse_True_cpu_autograd_False, test/functorch/test_control_flow.py::AssociativeScanTests::test_associative_scan_freevars_fct_generic_compile_mode_compile_reverse_False_cpu_autograd_True, test/functorch/test_control_flow.py::AssociativeScanTests::test_associative_scan_freevars_fct_generic_compile_mode_compile_reverse_True_cuda_autograd_False, test/functorch/test_control_flow.py::AssociativeScanTests::test_associative_scan_freevars_fct_generic_compile_mode_eager_reverse_False_cuda_autograd_False, test/functorch/test_control_flow.py::AssociativeScanTests::test_associative_scan_freevars_fct_generic_compile_mode_eager_reverse_False_cuda_autograd_True, test/functorch/test_control_flow.py::AssociativeScanTests::test_associative_scan_freevars_fct_generic_compile_mode_eager_reverse_True_cpu_autograd_True, test/functorch/test_control_flow.py::AssociativeScanTests::test_associative_scan_freevars_fct_generic_compile_mode_eager_reverse_True_cuda_autograd_False, test/functorch/test_control_flow.py::AssociativeScanTests::test_associative_scan_freevars_fct_generic_compile_mode_none_reverse_False_cuda_autograd_False, test/functorch/test_control_flow.py::AssociativeScanTests::test_associative_scan_freevars_nested_compile_mode_compile_combine_mode_generic_reverse_False_cuda_autograd_True, test/functorch/test_control_flow.py::AssociativeScanTests::test_associative_scan_freevars_nested_compile_mode_compile_combine_mode_pointwise_reverse_False_cuda_autograd_False, test/functorch/test_control_flow.py::AssociativeScanTests::test_associative_scan_freevars_nested_compile_mode_compile_combine_mode_pointwise_reverse_False_cuda_autograd_True, test/functorch/test_control_flow.py::AssociativeScanTests::test_associative_scan_freevars_nested_compile_mode_compile_combine_mode_pointwise_reverse_True_cpu_autograd_False, test/functorch/test_control_flow.py::AssociativeScanTests::test_associative_scan_freevars_nested_compile_mode_compile_combine_mode_pointwise_reverse_True_cuda_autograd_False, test/functorch/test_control_flow.py::AssociativeScanTests::test_associative_scan_freevars_nested_compile_mode_compile_dynamic_shape_combine_mode_generic_reverse_False_cuda_autograd_True, test/functorch/test_control_flow.py::AssociativeScanTests::test_associative_scan_freevars_nested_compile_mode_compile_dynamic_shape_combine_mode_generic_reverse_True_cuda_autograd_False, test/functorch/test_control_flow.py::AssociativeScanTests::test_associative_scan_freevars_nested_compile_mode_compile_dynamic_shape_combine_mode_pointwise_reverse_False_cuda_autograd_False, test/functorch/test_control_flow.py::AssociativeScanTests::test_associative_scan_freevars_nested_compile_mode_compile_dynamic_shape_combine_mode_pointwise_reverse_False_cuda_autograd_True, test/functorch/test_control_flow.py::AssociativeScanTests::test_associative_scan_freevars_nested_compile_mode_eager_combine_mode_generic_reverse_False_cpu_autograd_False, test/functorch/test_control_flow.py::AssociativeScanTests::test_associative_scan_freevars_nested_compile_mode_eager_combine_mode_generic_reverse_False_cpu_autograd_True, test/functorch/test_control_flow.py::AssociativeScanTests::test_associative_scan_freevars_nested_compile_mode_eager_combine_mode_generic_reverse_True_cpu_autograd_False, test/functorch/test_control_flow.py::AssociativeScanTests::test_associative_scan_freevars_nested_compile_mode_eager_combine_mode_pointwise_reverse_False_cpu_autograd_True, test/functorch/test_control_flow.py::AssociativeScanTests::test_associative_scan_freevars_nested_compile_mode_eager_combine_mode_pointwise_reverse_False_cuda_autograd_True, test/functorch/test_control_flow.py::AssociativeScanTests::test_associative_scan_freevars_nested_compile_mode_eager_combine_mode_pointwise_reverse_True_cpu_autograd_True, test/functorch/test_control_flow.py::AssociativeScanTests::test_associative_scan_freevars_nested_compile_mode_none_combine_mode_generic_reverse_False_cpu_autograd_False, test/functorch/test_control_flow.py::AssociativeScanTests::test_associative_scan_freevars_nested_compile_mode_none_combine_mode_generic_reverse_False_cpu_autograd_True, test/functorch/test_control_flow.py::AssociativeScanTests::test_associative_scan_freevars_nested_compile_mode_none_combine_mode_generic_reverse_True_cpu_autograd_False, test/functorch/test_control_flow.py::AssociativeScanTests::test_associative_scan_freevars_nested_compile_mode_none_combine_mode_generic_reverse_True_cpu_autograd_True, test/functorch/test_control_flow.py::AssociativeScanTests::test_associative_scan_freevars_nested_compile_mode_none_combine_mode_generic_reverse_True_cuda_autograd_True, test/functorch/test_control_flow.py::AssociativeScanTests::test_associative_scan_freevars_nested_compile_mode_none_combine_mode_pointwise_reverse_False_cpu_autograd_True, test/functorch/test_control_flow.py::AssociativeScanTests::test_associative_scan_freevars_nested_compile_mode_none_combine_mode_pointwise_reverse_False_cuda_autograd_True, test/functorch/test_control_flow.py::AssociativeScanTests::test_associative_scan_freevars_nested_compile_mode_none_combine_mode_pointwise_reverse_True_cuda_autograd_False, test/functorch/test_control_flow.py::AssociativeScanTests::test_associative_scan_freevars_pytree_compile_mode_compile_dynamic_shape_reverse_False_cpu_combine_mode_generic_autograd_False, test/functorch/test_control_flow.py::AssociativeScanTests::test_associative_scan_freevars_pytree_compile_mode_compile_dynamic_shape_reverse_False_cuda_combine_mode_generic_autograd_False, test/functorch/test_control_flow.py::AssociativeScanTests::test_associative_scan_freevars_pytree_compile_mode_compile_dynamic_shape_reverse_False_cuda_combine_mode_pointwise_autograd_True, test/functorch/test_control_flow.py::AssociativeScanTests::test_associative_scan_freevars_pytree_compile_mode_compile_dynamic_shape_reverse_True_cuda_combine_mode_generic_autograd_False, test/functorch/test_control_flow.py::AssociativeScanTests::test_associative_scan_freevars_pytree_compile_mode_compile_dynamic_shape_reverse_True_cuda_combine_mode_pointwise_autograd_True, test/functorch/test_control_flow.py::AssociativeScanTests::test_associative_scan_freevars_pytree_compile_mode_compile_reverse_False_cpu_combine_mode_generic_autograd_False, test/functorch/test_control_flow.py::AssociativeScanTests::test_associative_scan_freevars_pytree_compile_mode_compile_reverse_False_cpu_combine_mode_generic_autograd_True, test/functorch/test_control_flow.py::AssociativeScanTests::test_associative_scan_freevars_pytree_compile_mode_compile_reverse_False_cpu_combine_mode_pointwise_autograd_True, test/functorch/test_control_flow.py::AssociativeScanTests::test_associative_scan_freevars_pytree_compile_mode_compile_reverse_False_cuda_combine_mode_pointwise_autograd_True, test/functorch/test_control_flow.py::AssociativeScanTests::test_associative_scan_freevars_pytree_compile_mode_compile_reverse_True_cpu_combine_mode_generic_autograd_False, test/functorch/test_control_flow.py::AssociativeScanTests::test_associative_scan_freevars_pytree_compile_mode_compile_reverse_True_cpu_combine_mode_pointwise_autograd_False, test/functorch/test_control_flow.py::AssociativeScanTests::test_associative_scan_freevars_pytree_compile_mode_compile_reverse_True_cuda_combine_mode_generic_autograd_False, test/functorch/test_control_flow.py::AssociativeScanTests::test_associative_scan_freevars_pytree_compile_mode_compile_reverse_True_cuda_combine_mode_generic_autograd_True, test/functorch/test_control_flow.py::AssociativeScanTests::test_associative_scan_freevars_pytree_compile_mode_compile_reverse_True_cuda_combine_mode_pointwise_autograd_True, test/functorch/test_control_flow.py::AssociativeScanTests::test_associative_scan_freevars_pytree_compile_mode_eager_reverse_False_cpu_combine_mode_generic_autograd_False, test/functorch/test_control_flow.py::AssociativeScanTests::test_associative_scan_freevars_pytree_compile_mode_eager_reverse_False_cpu_combine_mode_pointwise_autograd_False, test/functorch/test_control_flow.py::AssociativeScanTests::test_associative_scan_freevars_pytree_compile_mode_eager_reverse_False_cpu_combine_mode_pointwise_autograd_True, test/functorch/test_control_flow.py::AssociativeScanTests::test_associative_scan_freevars_pytree_compile_mode_eager_reverse_False_cuda_combine_mode_generic_autograd_True, test/functorch/test_control_flow.py::AssociativeScanTests::test_associative_scan_freevars_pytree_compile_mode_eager_reverse_False_cuda_combine_mode_pointwise_autograd_False, test/functorch/test_control_flow.py::AssociativeScanTests::test_associative_scan_freevars_pytree_compile_mode_eager_reverse_False_cuda_combine_mode_pointwise_autograd_True, test/functorch/test_control_flow.py::AssociativeScanTests::test_associative_scan_freevars_pytree_compile_mode_eager_reverse_True_cpu_combine_mode_generic_autograd_False, test/functorch/test_control_flow.py::AssociativeScanTests::test_associative_scan_freevars_pytree_compile_mode_eager_reverse_True_cpu_combine_mode_generic_autograd_True, test/functorch/test_control_flow.py::AssociativeScanTests::test_associative_scan_freevars_pytree_compile_mode_eager_reverse_True_cpu_combine_mode_pointwise_autograd_False, test/functorch/test_control_flow.py::AssociativeScanTests::test_associative_scan_freevars_pytree_compile_mode_eager_reverse_True_cpu_combine_mode_pointwise_autograd_True, test/functorch/test_control_flow.py::AssociativeScanTests::test_associative_scan_freevars_pytree_compile_mode_eager_reverse_True_cuda_combine_mode_pointwise_autograd_False, test/functorch/test_control_flow.py::AssociativeScanTests::test_associative_scan_freevars_pytree_compile_mode_none_reverse_False_cpu_combine_mode_pointwise_autograd_True, test/functorch/test_control_flow.py::AssociativeScanTests::test_associative_scan_freevars_pytree_compile_mode_none_reverse_True_cpu_combine_mode_pointwise_autograd_True, test/functorch/test_control_flow.py::AssociativeScanTests::test_associative_scan_freevars_pytree_compile_mode_none_reverse_True_cuda_combine_mode_generic_autograd_False, test/functorch/test_control_flow.py::AssociativeScanTests::test_associative_scan_freevars_pytree_compile_mode_none_reverse_True_cuda_combine_mode_pointwise_autograd_False, test/functorch/test_control_flow.py::AssociativeScanTests::test_associative_scan_freevars_shape_check_compile_mode_compile_combine_mode_generic_reverse_True_cpu_autograd_False, test/functorch/test_control_flow.py::AssociativeScanTests::test_associative_scan_freevars_shape_check_compile_mode_compile_combine_mode_pointwise_reverse_False_cpu_autograd_False, test/functorch/test_control_flow.py::AssociativeScanTests::test_associative_scan_freevars_shape_check_compile_mode_compile_combine_mode_pointwise_reverse_False_cuda_autograd_False, test/functorch/test_control_flow.py::AssociativeScanTests::test_associative_scan_freevars_shape_check_compile_mode_compile_combine_mode_pointwise_reverse_False_cuda_autograd_True, test/functorch/test_control_flow.py::AssociativeScanTests::test_associative_scan_freevars_shape_check_compile_mode_compile_combine_mode_pointwise_reverse_True_cuda_autograd_False, test/functorch/test_control_flow.py::AssociativeScanTests::test_associative_scan_freevars_shape_check_compile_mode_compile_dynamic_shape_combine_mode_generic_reverse_False_cpu_autograd_True, test/functorch/test_control_flow.py::AssociativeScanTests::test_associative_scan_freevars_shape_check_compile_mode_compile_dynamic_shape_combine_mode_generic_reverse_False_cuda_autograd_False, test/functorch/test_control_flow.py::AssociativeScanTests::test_associative_scan_freevars_shape_check_compile_mode_compile_dynamic_shape_combine_mode_generic_reverse_True_cuda_autograd_False, test/functorch/test_control_flow.py::AssociativeScanTests::test_associative_scan_freevars_shape_check_compile_mode_compile_dynamic_shape_combine_mode_pointwise_reverse_False_cpu_autograd_False, test/functorch/test_control_flow.py::AssociativeScanTests::test_associative_scan_freevars_shape_check_compile_mode_compile_dynamic_shape_combine_mode_pointwise_reverse_False_cpu_autograd_True, test/functorch/test_control_flow.py::AssociativeScanTests::test_associative_scan_freevars_shape_check_compile_mode_compile_dynamic_shape_combine_mode_pointwise_reverse_True_cpu_autograd_False, test/functorch/test_control_flow.py::AssociativeScanTests::test_associative_scan_freevars_shape_check_compile_mode_compile_dynamic_shape_combine_mode_pointwise_reverse_True_cpu_autograd_True, test/functorch/test_control_flow.py::AssociativeScanTests::test_associative_scan_freevars_shape_check_compile_mode_eager_combine_mode_generic_reverse_False_cuda_autograd_True, test/functorch/test_control_flow.py::AssociativeScanTests::test_associative_scan_freevars_shape_check_compile_mode_eager_combine_mode_pointwise_reverse_False_cpu_autograd_False, test/functorch/test_control_flow.py::AssociativeScanTests::test_associative_scan_freevars_shape_check_compile_mode_eager_combine_mode_pointwise_reverse_False_cuda_autograd_True, test/functorch/test_control_flow.py::AssociativeScanTests::test_associative_scan_freevars_shape_check_compile_mode_none_combine_mode_generic_reverse_False_cuda_autograd_False, test/functorch/test_control_flow.py::AssociativeScanTests::test_associative_scan_freevars_shape_check_compile_mode_none_combine_mode_generic_reverse_True_cuda_autograd_False, test/functorch/test_control_flow.py::AssociativeScanTests::test_associative_scan_freevars_shape_check_compile_mode_none_combine_mode_generic_reverse_True_cuda_autograd_True, test/functorch/test_control_flow.py::AssociativeScanTests::test_associative_scan_freevars_shape_check_compile_mode_none_combine_mode_pointwise_reverse_False_cpu_autograd_True, test/functorch/test_control_flow.py::AssociativeScanTests::test_associative_scan_freevars_simple_compile_mode_compile_combine_mode_generic_reverse_True_cpu_autograd_False, test/functorch/test_control_flow.py::AssociativeScanTests::test_associative_scan_freevars_simple_compile_mode_compile_combine_mode_generic_reverse_True_cuda_autograd_True, test/functorch/test_control_flow.py::AssociativeScanTests::test_associative_scan_freevars_simple_compile_mode_compile_combine_mode_pointwise_reverse_True_cpu_autograd_False, test/functorch/test_control_flow.py::AssociativeScanTests::test_associative_scan_freevars_simple_compile_mode_compile_combine_mode_pointwise_reverse_True_cuda_autograd_False, test/functorch/test_control_flow.py::AssociativeScanTests::test_associative_scan_freevars_simple_compile_mode_compile_combine_mode_pointwise_reverse_True_cuda_autograd_True, test/functorch/test_control_flow.py::AssociativeScanTests::test_associative_scan_freevars_simple_compile_mode_compile_dynamic_shape_combine_mode_generic_reverse_True_cpu_autograd_False, test/functorch/test_control_flow.py::AssociativeScanTests::test_associative_scan_freevars_simple_compile_mode_compile_dynamic_shape_combine_mode_generic_reverse_True_cpu_autograd_True, test/functorch/test_control_flow.py::AssociativeScanTests::test_associative_scan_freevars_simple_compile_mode_compile_dynamic_shape_combine_mode_pointwise_reverse_False_cuda_autograd_True, test/functorch/test_control_flow.py::AssociativeScanTests::test_associative_scan_freevars_simple_compile_mode_compile_dynamic_shape_combine_mode_pointwise_reverse_True_cuda_autograd_True, test/functorch/test_control_flow.py::AssociativeScanTests::test_associative_scan_freevars_simple_compile_mode_eager_combine_mode_pointwise_reverse_False_cpu_autograd_True, test/functorch/test_control_flow.py::AssociativeScanTests::test_associative_scan_freevars_simple_compile_mode_eager_combine_mode_pointwise_reverse_False_cuda_autograd_False, test/functorch/test_control_flow.py::AssociativeScanTests::test_associative_scan_freevars_simple_compile_mode_none_combine_mode_generic_reverse_False_cuda_autograd_False, test/functorch/test_control_flow.py::AssociativeScanTests::test_associative_scan_freevars_simple_compile_mode_none_combine_mode_generic_reverse_False_cuda_autograd_True, test/functorch/test_control_flow.py::AssociativeScanTests::test_associative_scan_freevars_simple_compile_mode_none_combine_mode_generic_reverse_True_cuda_autograd_False, test/functorch/test_control_flow.py::AssociativeScanTests::test_associative_scan_freevars_simple_compile_mode_none_combine_mode_pointwise_reverse_False_cpu_autograd_True, test/functorch/test_control_flow.py::AssociativeScanTests::test_associative_scan_freevars_simple_compile_mode_none_combine_mode_pointwise_reverse_False_cuda_autograd_True, test/functorch/test_control_flow.py::AssociativeScanTests::test_associative_scan_input_mutation, test/functorch/test_control_flow.py::AssociativeScanTests::test_associative_scan_input_output_alias, test/functorch/test_control_flow.py::AssociativeScanTests::test_associative_scan_loop_in_combine_fn_compile_mode_compile_dynamic_shape_loop_type_for_reverse_False_cpu_autograd_True, test/functorch/test_control_flow.py::AssociativeScanTests::test_associative_scan_loop_in_combine_fn_compile_mode_compile_dynamic_shape_loop_type_for_reverse_False_cuda_autograd_False, test/functorch/test_control_flow.py::AssociativeScanTests::test_associative_scan_loop_in_combine_fn_compile_mode_compile_dynamic_shape_loop_type_for_reverse_False_cuda_autograd_True, test/functorch/test_control_flow.py::AssociativeScanTests::test_associative_scan_loop_in_combine_fn_compile_mode_compile_dynamic_shape_loop_type_for_reverse_True_cpu_autograd_True, test/functorch/test_control_flow.py::AssociativeScanTests::test_associative_scan_loop_in_combine_fn_compile_mode_compile_dynamic_shape_loop_type_for_reverse_True_cuda_autograd_False, test/functorch/test_control_flow.py::AssociativeScanTests::test_associative_scan_loop_in_combine_fn_compile_mode_compile_loop_type_for_reverse_True_cpu_autograd_False, test/functorch/test_control_flow.py::AssociativeScanTests::test_associative_scan_loop_in_combine_fn_compile_mode_compile_loop_type_for_reverse_True_cpu_autograd_True, test/functorch/test_control_flow.py::AssociativeScanTests::test_associative_scan_loop_in_combine_fn_compile_mode_eager_loop_type_for_reverse_True_cpu_autograd_False, test/functorch/test_control_flow.py::AssociativeScanTests::test_associative_scan_loop_in_combine_fn_compile_mode_eager_loop_type_for_reverse_True_cuda_autograd_False, test/functorch/test_control_flow.py::AssociativeScanTests::test_associative_scan_loop_in_combine_fn_compile_mode_none_loop_type_for_reverse_False_cuda_autograd_False, test/functorch/test_control_flow.py::AssociativeScanTests::test_associative_scan_loop_in_combine_fn_compile_mode_none_loop_type_for_reverse_True_cpu_autograd_False, test/functorch/test_control_flow.py::AssociativeScanTests::test_associative_scan_loop_in_combine_fn_failure, test/functorch/test_control_flow.py::AssociativeScanTests::test_associative_scan_non_contiguous_tensor_compile_mode_compile_dynamic_shape_reverse_False_cpu_autograd_False, test/functorch/test_control_flow.py::AssociativeScanTests::test_associative_scan_non_contiguous_tensor_compile_mode_compile_dynamic_shape_reverse_False_cuda_autograd_True, test/functorch/test_control_flow.py::AssociativeScanTests::test_associative_scan_non_contiguous_tensor_compile_mode_compile_dynamic_shape_reverse_True_cpu_autograd_True, test/functorch/test_control_flow.py::AssociativeScanTests::test_associative_scan_non_contiguous_tensor_compile_mode_compile_dynamic_shape_reverse_True_cuda_autograd_False, test/functorch/test_control_flow.py::AssociativeScanTests::test_associative_scan_non_contiguous_tensor_compile_mode_compile_dynamic_shape_reverse_True_cuda_autograd_True, test/functorch/test_control_flow.py::AssociativeScanTests::test_associative_scan_non_contiguous_tensor_compile_mode_compile_reverse_False_cpu_autograd_False, test/functorch/test_control_flow.py::AssociativeScanTests::test_associative_scan_non_contiguous_tensor_compile_mode_eager_reverse_False_cpu_autograd_False, test/functorch/test_control_flow.py::AssociativeScanTests::test_associative_scan_non_contiguous_tensor_compile_mode_eager_reverse_False_cuda_autograd_False, test/functorch/test_control_flow.py::AssociativeScanTests::test_associative_scan_non_contiguous_tensor_compile_mode_eager_reverse_True_cpu_autograd_False, test/functorch/test_control_flow.py::AssociativeScanTests::test_associative_scan_non_contiguous_tensor_compile_mode_eager_reverse_True_cpu_autograd_True, test/functorch/test_control_flow.py::AssociativeScanTests::test_associative_scan_non_contiguous_tensor_compile_mode_none_reverse_False_cpu_autograd_True, test/functorch/test_control_flow.py::AssociativeScanTests::test_associative_scan_non_contiguous_tensor_compile_mode_none_reverse_False_cuda_autograd_True, test/functorch/test_control_flow.py::AssociativeScanTests::test_associative_scan_non_contiguous_tensor_compile_mode_none_reverse_True_cpu_autograd_True, test/functorch/test_control_flow.py::AssociativeScanTests::test_associative_scan_non_pointwise_generic_reverse_False_compile_mode_compile_cpu_autograd_True, test/functorch/test_control_flow.py::AssociativeScanTests::test_associative_scan_non_pointwise_generic_reverse_False_compile_mode_compile_dynamic_shape_cuda_autograd_False, test/functorch/test_control_flow.py::AssociativeScanTests::test_associative_scan_non_pointwise_generic_reverse_False_compile_mode_eager_cuda_autograd_False, test/functorch/test_control_flow.py::AssociativeScanTests::test_associative_scan_non_pointwise_generic_reverse_False_compile_mode_none_cpu_autograd_True, test/functorch/test_control_flow.py::AssociativeScanTests::test_associative_scan_non_pointwise_generic_reverse_False_compile_mode_none_cuda_autograd_False, test/functorch/test_control_flow.py::AssociativeScanTests::test_associative_scan_non_pointwise_generic_reverse_True_compile_mode_compile_cuda_autograd_False, test/functorch/test_control_flow.py::AssociativeScanTests::test_associative_scan_non_pointwise_generic_reverse_True_compile_mode_eager_cpu_autograd_True, test/functorch/test_control_flow.py::AssociativeScanTests::test_associative_scan_non_pointwise_generic_reverse_True_compile_mode_eager_cuda_autograd_True, test/functorch/test_control_flow.py::AssociativeScanTests::test_associative_scan_non_pointwise_generic_reverse_True_compile_mode_none_cpu_autograd_True, test/functorch/test_control_flow.py::AssociativeScanTests::test_associative_scan_output_output_alias, test/functorch/test_control_flow.py::AssociativeScanTests::test_associative_scan_partial_grad_combine_mode_generic_compile_mode_compile_dynamic_shape_reverse_True_cpu, test/functorch/test_control_flow.py::AssociativeScanTests::test_associative_scan_partial_grad_combine_mode_generic_compile_mode_compile_dynamic_shape_reverse_True_cuda, test/functorch/test_control_flow.py::AssociativeScanTests::test_associative_scan_partial_grad_combine_mode_generic_compile_mode_compile_reverse_False_cuda, test/functorch/test_control_flow.py::AssociativeScanTests::test_associative_scan_partial_grad_combine_mode_generic_compile_mode_compile_reverse_True_cuda, test/functorch/test_control_flow.py::AssociativeScanTests::test_associative_scan_partial_grad_combine_mode_generic_compile_mode_eager_reverse_False_cpu, test/functorch/test_control_flow.py::AssociativeScanTests::test_associative_scan_partial_grad_combine_mode_generic_compile_mode_eager_reverse_True_cuda, test/functorch/test_control_flow.py::AssociativeScanTests::test_associative_scan_partial_grad_combine_mode_generic_compile_mode_none_reverse_True_cpu, test/functorch/test_control_flow.py::AssociativeScanTests::test_associative_scan_partial_grad_combine_mode_pointwise_compile_mode_compile_dynamic_shape_reverse_True_cpu, test/functorch/test_control_flow.py::AssociativeScanTests::test_associative_scan_partial_grad_combine_mode_pointwise_compile_mode_compile_dynamic_shape_reverse_True_cuda, test/functorch/test_control_flow.py::AssociativeScanTests::test_associative_scan_partial_grad_combine_mode_pointwise_compile_mode_none_reverse_False_cuda, test/functorch/test_control_flow.py::AssociativeScanTests::test_associative_scan_partial_grad_combine_mode_pointwise_compile_mode_none_reverse_True_cpu, test/functorch/test_control_flow.py::AssociativeScanTests::test_associative_scan_partial_grad_combine_mode_pointwise_compile_mode_none_reverse_True_cuda, test/functorch/test_control_flow.py::AssociativeScanTests::test_associative_scan_partial_grad_no_grad_combine_mode_generic_compile_mode_compile_reverse_True_cpu, test/functorch/test_control_flow.py::AssociativeScanTests::test_associative_scan_partial_grad_no_grad_combine_mode_generic_compile_mode_eager_reverse_True_cpu, test/functorch/test_control_flow.py::AssociativeScanTests::test_associative_scan_partial_grad_no_grad_combine_mode_generic_compile_mode_none_reverse_True_cpu, test/functorch/test_control_flow.py::AssociativeScanTests::test_associative_scan_partial_grad_no_grad_combine_mode_pointwise_compile_mode_compile_dynamic_shape_reverse_False_cuda, test/functorch/test_control_flow.py::AssociativeScanTests::test_associative_scan_partial_grad_no_grad_combine_mode_pointwise_compile_mode_compile_dynamic_shape_reverse_True_cpu, test/functorch/test_control_flow.py::AssociativeScanTests::test_associative_scan_partial_grad_no_grad_combine_mode_pointwise_compile_mode_compile_dynamic_shape_reverse_True_cuda, test/functorch/test_control_flow.py::AssociativeScanTests::test_associative_scan_partial_grad_no_grad_combine_mode_pointwise_compile_mode_compile_reverse_False_cpu, test/functorch/test_control_flow.py::AssociativeScanTests::test_associative_scan_partial_grad_no_grad_combine_mode_pointwise_compile_mode_compile_reverse_False_cuda, test/functorch/test_control_flow.py::AssociativeScanTests::test_associative_scan_partial_grad_no_grad_combine_mode_pointwise_compile_mode_eager_reverse_True_cuda, test/functorch/test_control_flow.py::AssociativeScanTests::test_associative_scan_partial_grad_no_grad_combine_mode_pointwise_compile_mode_none_reverse_False_cpu, test/functorch/test_control_flow.py::AssociativeScanTests::test_associative_scan_pytree_output, test/functorch/test_control_flow.py::AssociativeScanTests::test_associative_scan_sparse_tensor, test/functorch/test_control_flow.py::AssociativeScanTests::test_associative_scan_tuple_compile_mode_compile_combine_mode_generic_reverse_False_cpu_autograd_False, test/functorch/test_control_flow.py::AssociativeScanTests::test_associative_scan_tuple_compile_mode_compile_combine_mode_generic_reverse_False_cpu_autograd_True, test/functorch/test_control_flow.py::AssociativeScanTests::test_associative_scan_tuple_compile_mode_compile_combine_mode_generic_reverse_False_cuda_autograd_True, test/functorch/test_control_flow.py::AssociativeScanTests::test_associative_scan_tuple_compile_mode_compile_combine_mode_generic_reverse_True_cpu_autograd_False, test/functorch/test_control_flow.py::AssociativeScanTests::test_associative_scan_tuple_compile_mode_compile_combine_mode_generic_reverse_True_cpu_autograd_True, test/functorch/test_control_flow.py::AssociativeScanTests::test_associative_scan_tuple_compile_mode_compile_combine_mode_generic_reverse_True_cuda_autograd_True, test/functorch/test_control_flow.py::AssociativeScanTests::test_associative_scan_tuple_compile_mode_compile_combine_mode_pointwise_reverse_False_cpu_autograd_True, test/functorch/test_control_flow.py::AssociativeScanTests::test_associative_scan_tuple_compile_mode_compile_combine_mode_pointwise_reverse_False_cuda_autograd_False, test/functorch/test_control_flow.py::AssociativeScanTests::test_associative_scan_tuple_compile_mode_compile_dynamic_shape_combine_mode_generic_reverse_False_cpu_autograd_False, test/functorch/test_control_flow.py::AssociativeScanTests::test_associative_scan_tuple_compile_mode_compile_dynamic_shape_combine_mode_generic_reverse_False_cpu_autograd_True, test/functorch/test_control_flow.py::AssociativeScanTests::test_associative_scan_tuple_compile_mode_compile_dynamic_shape_combine_mode_pointwise_reverse_False_cuda_autograd_True, test/functorch/test_control_flow.py::AssociativeScanTests::test_associative_scan_tuple_compile_mode_compile_dynamic_shape_combine_mode_pointwise_reverse_True_cuda_autograd_True, test/functorch/test_control_flow.py::AssociativeScanTests::test_associative_scan_tuple_compile_mode_eager_combine_mode_generic_reverse_False_cpu_autograd_False, test/functorch/test_control_flow.py::AssociativeScanTests::test_associative_scan_tuple_compile_mode_eager_combine_mode_generic_reverse_False_cpu_autograd_True, test/functorch/test_control_flow.py::AssociativeScanTests::test_associative_scan_tuple_compile_mode_eager_combine_mode_generic_reverse_False_cuda_autograd_True, test/functorch/test_control_flow.py::AssociativeScanTests::test_associative_scan_tuple_compile_mode_eager_combine_mode_generic_reverse_True_cpu_autograd_True, test/functorch/test_control_flow.py::AssociativeScanTests::test_associative_scan_tuple_compile_mode_eager_combine_mode_pointwise_reverse_True_cpu_autograd_False, test/functorch/test_control_flow.py::AssociativeScanTests::test_associative_scan_tuple_compile_mode_eager_combine_mode_pointwise_reverse_True_cpu_autograd_True, test/functorch/test_control_flow.py::AssociativeScanTests::test_associative_scan_tuple_compile_mode_eager_combine_mode_pointwise_reverse_True_cuda_autograd_False, test/functorch/test_control_flow.py::AssociativeScanTests::test_associative_scan_tuple_compile_mode_none_combine_mode_generic_reverse_False_cpu_autograd_True, test/functorch/test_control_flow.py::AssociativeScanTests::test_associative_scan_tuple_compile_mode_none_combine_mode_generic_reverse_True_cpu_autograd_False, test/functorch/test_control_flow.py::AssociativeScanTests::test_associative_scan_tuple_compile_mode_none_combine_mode_generic_reverse_True_cpu_autograd_True, test/functorch/test_control_flow.py::AssociativeScanTests::test_associative_scan_tuple_compile_mode_none_combine_mode_pointwise_reverse_False_cpu_autograd_True, test/functorch/test_control_flow.py::AssociativeScanTests::test_associative_scan_tuple_compile_mode_none_combine_mode_pointwise_reverse_False_cuda_autograd_True, test/functorch/test_control_flow.py::AssociativeScanTests::test_associative_scan_tuple_compile_mode_none_combine_mode_pointwise_reverse_True_cpu_autograd_False, test/functorch/test_control_flow.py::AssociativeScanTests::test_associative_scan_vmap_in_combine_fn_compile_mode_compile_dynamic_shape_reverse_False_cpu_autograd_True, test/functorch/test_control_flow.py::AssociativeScanTests::test_associative_scan_vmap_in_combine_fn_compile_mode_compile_dynamic_shape_reverse_True_cuda_autograd_True, test/functorch/test_control_flow.py::AssociativeScanTests::test_associative_scan_vmap_in_combine_fn_compile_mode_compile_reverse_False_cuda_autograd_False, test/functorch/test_control_flow.py::AssociativeScanTests::test_associative_scan_vmap_in_combine_fn_compile_mode_compile_reverse_True_cpu_autograd_False, test/functorch/test_control_flow.py::AssociativeScanTests::test_associative_scan_vmap_in_combine_fn_compile_mode_compile_reverse_True_cuda_autograd_False, test/functorch/test_control_flow.py::AssociativeScanTests::test_associative_scan_vmap_in_combine_fn_compile_mode_eager_reverse_False_cpu_autograd_False, test/functorch/test_control_flow.py::AssociativeScanTests::test_associative_scan_vmap_in_combine_fn_compile_mode_eager_reverse_True_cpu_autograd_False, test/functorch/test_control_flow.py::AssociativeScanTests::test_associative_scan_vmap_in_combine_fn_compile_mode_eager_reverse_True_cpu_autograd_True, test/functorch/test_control_flow.py::AssociativeScanTests::test_associative_scan_vmap_in_combine_fn_compile_mode_none_reverse_True_cpu_autograd_True, test/functorch/test_control_flow.py::TestControlFlowTraced::test_compile_while_loop_stack_output_dynamic_False_backend_aot_eager, test/functorch/test_control_flow.py::TestControlFlowTraced::test_cond_accepts_torch_function_as_inputs, test/functorch/test_control_flow.py::TestControlFlowTraced::test_cond_functionalized, test/functorch/test_control_flow.py::TestControlFlowTraced::test_cond_functionalized_nested_input_mutation_with_aot_func, test/functorch/test_control_flow.py::TestControlFlowTraced::test_cond_mismatched_branch_output_dynamic_False_backend_aot_eager, test/functorch/test_control_flow.py::TestControlFlowTraced::test_cond_mismatched_branch_output_dynamic_True_backend_aot_eager, test/functorch/test_control_flow.py::TestControlFlowTraced::test_cond_nested_traced_fake_tensor, test/functorch/test_control_flow.py::TestControlFlowTraced::test_cond_nested_traced_multi, test/functorch/test_control_flow.py::TestControlFlowTraced::test_cond_nested_traced_other_inputs_fake_tensor, test/functorch/test_control_flow.py::TestControlFlowTraced::test_cond_symint_closure, test/functorch/test_control_flow.py::TestControlFlowTraced::test_cond_symint_operands_requires_grad_True, test/functorch/test_control_flow.py::TestControlFlowTraced::test_cond_trace_set__and_mutate_input, test/functorch/test_control_flow.py::TestControlFlowTraced::test_cond_trace_set__and_mutate_intermediate, test/functorch/test_control_flow.py::TestControlFlowTraced::test_cond_tracing_with_valid_inputs_predType_boolTensor_innerFnType_function_nOperands_0_nClosure_0_nesting_0, test/functorch/test_control_flow.py::TestControlFlowTraced::test_cond_tracing_with_valid_inputs_predType_boolTensor_innerFnType_function_nOperands_0_nClosure_0_nesting_2, test/functorch/test_control_flow.py::TestControlFlowTraced::test_cond_tracing_with_valid_inputs_predType_boolTensor_innerFnType_function_nOperands_1_nClosure_1_nesting_0, test/functorch/test_control_flow.py::TestControlFlowTraced::test_cond_tracing_with_valid_inputs_predType_boolTensor_innerFnType_function_nOperands_1_nClosure_1_nesting_2, test/functorch/test_control_flow.py::TestControlFlowTraced::test_cond_tracing_with_valid_inputs_predType_boolTensor_innerFnType_module_nOperands_0_nClosure_1_nesting_0, test/functorch/test_control_flow.py::TestControlFlowTraced::test_cond_tracing_with_valid_inputs_predType_boolTensor_innerFnType_module_nOperands_0_nClosure_1_nesting_2, test/functorch/test_control_flow.py::TestControlFlowTraced::test_cond_tracing_with_valid_inputs_predType_boolTensor_innerFnType_module_nOperands_1_nClosure_0_nesting_0, test/functorch/test_control_flow.py::TestControlFlowTraced::test_cond_tracing_with_valid_inputs_predType_boolTensor_innerFnType_module_nOperands_1_nClosure_1_nesting_0, test/functorch/test_control_flow.py::TestControlFlowTraced::test_cond_tracing_with_valid_inputs_predType_boolTensor_innerFnType_object_nOperands_0_nClosure_0_nesting_2, test/functorch/test_control_flow.py::TestControlFlowTraced::test_cond_tracing_with_valid_inputs_predType_boolTensor_innerFnType_object_nOperands_0_nClosure_1_nesting_0, test/functorch/test_control_flow.py::TestControlFlowTraced::test_cond_tracing_with_valid_inputs_predType_boolTensor_innerFnType_object_nOperands_1_nClosure_0_nesting_0, test/functorch/test_control_flow.py::TestControlFlowTraced::test_cond_tracing_with_valid_inputs_predType_boolTensor_innerFnType_object_nOperands_1_nClosure_0_nesting_2, test/functorch/test_control_flow.py::TestControlFlowTraced::test_cond_tracing_with_valid_inputs_predType_boolTensor_innerFnType_object_nOperands_1_nClosure_1_nesting_2, test/functorch/test_control_flow.py::TestControlFlowTraced::test_cond_tracing_with_valid_inputs_predType_bool_innerFnType_function_nOperands_0_nClosure_1_nesting_0, test/functorch/test_control_flow.py::TestControlFlowTraced::test_cond_tracing_with_valid_inputs_predType_bool_innerFnType_function_nOperands_1_nClosure_0_nesting_2, test/functorch/test_control_flow.py::TestControlFlowTraced::test_cond_tracing_with_valid_inputs_predType_bool_innerFnType_function_nOperands_1_nClosure_1_nesting_2, test/functorch/test_control_flow.py::TestControlFlowTraced::test_cond_tracing_with_valid_inputs_predType_bool_innerFnType_module_nOperands_0_nClosure_0_nesting_0, test/functorch/test_control_flow.py::TestControlFlowTraced::test_cond_tracing_with_valid_inputs_predType_bool_innerFnType_module_nOperands_1_nClosure_1_nesting_0, test/functorch/test_control_flow.py::TestControlFlowTraced::test_cond_tracing_with_valid_inputs_predType_bool_innerFnType_module_nOperands_1_nClosure_1_nesting_2, test/functorch/test_control_flow.py::TestControlFlowTraced::test_cond_tracing_with_valid_inputs_predType_bool_innerFnType_object_nOperands_0_nClosure_1_nesting_2, test/functorch/test_control_flow.py::TestControlFlowTraced::test_cond_tracing_with_valid_inputs_predType_bool_innerFnType_object_nOperands_1_nClosure_0_nesting_2, test/functorch/test_control_flow.py::TestControlFlowTraced::test_cond_tracing_with_valid_inputs_predType_bool_innerFnType_object_nOperands_1_nClosure_1_nesting_0, test/functorch/test_control_flow.py::TestControlFlowTraced::test_cond_tracing_with_valid_inputs_predType_floatTensor_innerFnType_function_nOperands_1_nClosure_0_nesting_0, test/functorch/test_control_flow.py::TestControlFlowTraced::test_cond_tracing_with_valid_inputs_predType_floatTensor_innerFnType_function_nOperands_1_nClosure_0_nesting_2, test/functorch/test_control_flow.py::TestControlFlowTraced::test_cond_tracing_with_valid_inputs_predType_floatTensor_innerFnType_object_nOperands_0_nClosure_0_nesting_0, test/functorch/test_control_flow.py::TestControlFlowTraced::test_cond_tracing_with_valid_inputs_predType_floatTensor_innerFnType_object_nOperands_0_nClosure_0_nesting_2, test/functorch/test_control_flow.py::TestControlFlowTraced::test_cond_tracing_with_valid_inputs_predType_floatTensor_innerFnType_object_nOperands_0_nClosure_1_nesting_0, test/functorch/test_control_flow.py::TestControlFlowTraced::test_cond_tracing_with_valid_inputs_predType_floatTensor_innerFnType_object_nOperands_1_nClosure_0_nesting_0, test/functorch/test_control_flow.py::TestControlFlowTraced::test_cond_tracing_with_valid_inputs_predType_intTensor_innerFnType_function_nOperands_0_nClosure_0_nesting_0, test/functorch/test_control_flow.py::TestControlFlowTraced::test_cond_tracing_with_valid_inputs_predType_intTensor_innerFnType_function_nOperands_0_nClosure_1_nesting_2, test/functorch/test_control_flow.py::TestControlFlowTraced::test_cond_tracing_with_valid_inputs_predType_intTensor_innerFnType_module_nOperands_0_nClosure_0_nesting_0, test/functorch/test_control_flow.py::TestControlFlowTraced::test_cond_tracing_with_valid_inputs_predType_intTensor_innerFnType_module_nOperands_0_nClosure_1_nesting_0, test/functorch/test_control_flow.py::TestControlFlowTraced::test_cond_tracing_with_valid_inputs_predType_intTensor_innerFnType_module_nOperands_0_nClosure_1_nesting_2, test/functorch/test_control_flow.py::TestControlFlowTraced::test_cond_tracing_with_valid_inputs_predType_intTensor_innerFnType_module_nOperands_1_nClosure_1_nesting_0, test/functorch/test_control_flow.py::TestControlFlowTraced::test_cond_tracing_with_valid_inputs_predType_intTensor_innerFnType_object_nOperands_0_nClosure_0_nesting_2, test/functorch/test_control_flow.py::TestControlFlowTraced::test_cond_tracing_with_valid_inputs_predType_intTensor_innerFnType_object_nOperands_0_nClosure_1_nesting_0, test/functorch/test_control_flow.py::TestControlFlowTraced::test_cond_tracing_with_valid_inputs_predType_intTensor_innerFnType_object_nOperands_0_nClosure_1_nesting_2, test/functorch/test_control_flow.py::TestControlFlowTraced::test_cond_tracing_with_valid_inputs_predType_intTensor_innerFnType_object_nOperands_1_nClosure_1_nesting_0, test/functorch/test_control_flow.py::TestControlFlowTraced::test_cond_vmap_multiple_outputs_nClosure_0, test/functorch/test_control_flow.py::TestControlFlowTraced::test_cond_vmap_multiple_outputs_nClosure_1, test/functorch/test_control_flow.py::TestControlFlowTraced::test_cond_vmap_predType_boolTensor_innerFnType_function_nOperands_1_nClosure_1_nesting_0, test/functorch/test_control_flow.py::TestControlFlowTraced::test_cond_vmap_predType_boolTensor_innerFnType_function_nOperands_2_nClosure_1_nesting_0, test/functorch/test_control_flow.py::TestControlFlowTraced::test_cond_vmap_predType_boolTensor_innerFnType_module_nOperands_1_nClosure_1_nesting_0, test/functorch/test_control_flow.py::TestControlFlowTraced::test_cond_vmap_predType_boolTensor_innerFnType_module_nOperands_2_nClosure_0_nesting_0, test/functorch/test_control_flow.py::TestControlFlowTraced::test_cond_vmap_predType_boolTensor_innerFnType_object_nOperands_1_nClosure_0_nesting_0, test/functorch/test_control_flow.py::TestControlFlowTraced::test_cond_vmap_predType_boolTensor_innerFnType_object_nOperands_2_nClosure_1_nesting_0, test/functorch/test_control_flow.py::TestControlFlowTraced::test_cond_with_unbacked_sym_pred, test/functorch/test_control_flow.py::TestControlFlowTraced::test_hop_raises_if_not_overriding_call, test/functorch/test_control_flow.py::TestControlFlowTraced::test_map_functionalized, test/functorch/test_control_flow.py::TestControlFlowTraced::test_map_functionalized_aot_func, test/functorch/test_control_flow.py::TestControlFlowTraced::test_map_functionalized_elem_alias, test/functorch/test_control_flow.py::TestControlFlowTraced::test_map_unfunc_boolean_tensor_for_nested_map_cond, test/functorch/test_control_flow.py::TestControlFlowTraced::test_nested_cond_map_cond_symbolic, test/functorch/test_control_flow.py::TestControlFlowTraced::test_nested_map_cond_symbolic, test/functorch/test_control_flow.py::TestControlFlowTraced::test_raise_error_on_mismatch_type_size, test/functorch/test_control_flow.py::TestControlFlowTraced::test_raise_error_on_mismatch_type_size_fake_tensor, test/functorch/test_control_flow.py::TestControlFlowTraced::test_scan_functionalized_elem_alias, test/functorch/test_control_flow.py::TestControlFlowTraced::test_scan_in_vmap_mixed_batch_dims, test/functorch/test_control_flow.py::TestControlFlowTraced::test_scan_in_vmap_simple, test/functorch/test_control_flow.py::TestControlFlowTraced::test_scan_pytree_closure, test/functorch/test_control_flow.py::TestControlFlowTraced::test_scan_vmap_scan_nested, test/functorch/test_control_flow.py::TestControlFlowTraced::test_tracing_map_autograd_symbolic_dict, test/functorch/test_control_flow.py::TestControlFlowTraced::test_tracing_map_symbolic_list, test/functorch/test_control_flow.py::TestControlFlowTraced::test_vmap_closure_weight_error, test/functorch/test_control_flow.py::TestControlFlowTraced::test_vmap_scan_vmap_scan_nested, test/functorch/test_control_flow.py::TestControlFlowTraced::test_vmap_vmap_boolcond_False, test/functorch/test_control_flow.py::TestControlFlowTraced::test_vmap_vmap_boolcond_True, test/functorch/test_control_flow.py::TestControlFlowTraced::test_while_loop_autograd_simple, test/functorch/test_control_flow.py::TestControlFlowTraced::test_while_loop_compile_backend_aot_eager_while_loop_test_nested2, test/functorch/test_control_flow.py::TestControlFlowTraced::test_while_loop_compile_backend_aot_eager_while_loop_test_nested_with_linear, test/functorch/test_control_flow.py::TestControlFlowTraced::test_while_loop_compile_backend_aot_eager_while_loop_test_pytree_int_carry, test/functorch/test_control_flow.py::TestControlFlowTraced::test_while_loop_compile_backend_aot_eager_while_loop_test_simple_with_pytree_carry, test/functorch/test_control_flow.py::TestControlFlowTraced::test_while_loop_compile_backend_eager_while_loop_test_pytree_int_carry, test/functorch/test_control_flow.py::TestControlFlowTraced::test_while_loop_compile_backend_eager_while_loop_test_simple, test/functorch/test_control_flow.py::TestControlFlowTraced::test_while_loop_functionalize_func_type_cpp_while_loop_test_simple_with_mutation, test/functorch/test_control_flow.py::TestControlFlowTraced::test_while_loop_functionalize_func_type_functorch_while_loop_test_simple_with_mutation, test/functorch/test_control_flow.py::TestControlFlowTraced::test_while_loop_functionalize_func_type_no_while_loop_test_simple_with_mutation, test/functorch/test_control_flow.py::TestControlFlowTraced::test_while_loop_functionalize_func_type_python_while_loop_test_simple_with_pytree_carry, test/functorch/test_control_flow.py::TestControlFlowTraced::test_while_loop_op_constant_and_symint_output_compile_dynamic_False_backend_aot_eager, test/functorch/test_control_flow.py::TestControlFlowTraced::test_while_loop_op_constant_and_symint_output_compile_dynamic_True_backend_aot_eager, test/functorch/test_control_flow.py::TestControlFlowTraced::test_while_loop_op_constant_and_symint_output_compile_dynamic_True_backend_eager, test/functorch/test_control_flow.py::TestControlFlowTraced::test_while_loop_op_constant_and_symint_output_export_strict_True_dynamic_False, test/functorch/test_control_flow.py::TestControlFlowTraced::test_while_loop_op_int_carry_compile_dynamic_False_backend_aot_eager, test/functorch/test_control_flow.py::TestControlFlowTraced::test_while_loop_op_int_carry_compile_dynamic_False_backend_eager, test/functorch/test_control_flow.py::TestControlFlowTraced::test_while_loop_op_int_carry_export_strict_False_dynamic_False, test/functorch/test_control_flow.py::TestControlFlowTraced::test_while_loop_op_int_carry_export_strict_False_dynamic_True, test/functorch/test_control_flow.py::TestControlFlowTraced::test_while_loop_op_pytree_int_carry_compile_dynamic_False_backend_aot_eager, test/functorch/test_control_flow.py::TestControlFlowTraced::test_while_loop_op_pytree_int_carry_compile_dynamic_True_backend_aot_eager, test/functorch/test_control_flow.py::TestControlFlowTraced::test_while_loop_op_pytree_int_carry_compile_dynamic_True_backend_eager, test/functorch/test_control_flow.py::TestControlFlowTraced::test_while_loop_op_pytree_int_carry_export_strict_False_dynamic_True, test/functorch/test_control_flow.py::TestControlFlowTraced::test_while_loop_op_pytree_int_carry_export_strict_True_dynamic_True, test/functorch/test_control_flow.py::TestControlFlowTraced::test_while_loop_pytree_carry, test/functorch/test_control_flow.py::TestControlFlowTraced::test_while_loop_simple_functionalize_check_graph_func_type_cpp, test/functorch/test_control_flow.py::TestControlFlowTraced::test_while_loop_simple_functionalize_check_graph_func_type_no, test/functorch/test_control_flow.py::TestControlFlowTraced::test_while_loop_simple_functionalize_check_graph_func_type_python, test/functorch/test_control_flow.py::TestControlFlowTraced::test_while_loop_tracing_while_loop_test_nested2, test/functorch/test_control_flow.py::TestControlFlowTraced::test_while_loop_tracing_while_loop_test_nested_with_linear, test/functorch/test_control_flow.py::TestControlFlowTraced::test_while_loop_tracing_while_loop_test_simple_with_mutation, test/functorch/test_control_flow.py::TestHopSchema::test_associative_scan_gen_schema_tensor_inputs, test/functorch/test_control_flow.py::TestHopSchema::test_function_schema_gen, test/functorch/test_control_flow.py::TestHopSchema::test_list_gen_schema_type_bool, test/functorch/test_control_flow.py::TestHopSchema::test_scan_gen_schema_tensor_inputs, test/functorch/test_control_flow.py::TestHopSchema::test_type_gen_schema_type_Tensor, test/functorch/test_control_flow.py::TestHopSchema::test_type_gen_schema_type_str, test/functorch/test_control_flow.py::TestHopSchema::test_while_loop_gen_schema_with_additional_inputs, test/functorch/test_control_flow.py::TestHopSchema::test_while_loop_gen_schema_with_input_mutation 2025-12-04T13:50:18.7630110Z 2025-12-04T13:50:18.7630365Z Finished functorch/test_control_flow 2/3 ... [2025-12-04 13:50:18.713798][15066.656009714], took 10.71min 2025-12-04T13:50:18.7631164Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/functorch.test_control_flow/functorch.test_control_flow-f3515033a541aea0.xml 2025-12-04T13:50:18.8420493Z Running optim/test_swa_utils 1/1 ... [2025-12-04 13:50:18.841797][15066.784013997] 2025-12-04T13:50:18.8421046Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2025-12-04T13:50:18.8423775Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'optim/test_swa_utils.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-12-04 13:50:18.842109] 2025-12-04T13:50:21.6240325Z 2025-12-04T13:50:21.6241199Z optim/test_swa_utils 1/1 was successful, full logs can be found in artifacts with path test/test-reports/optim.test_swa_utils_1.1_2ca1b92ac8acd203_.log 2025-12-04T13:50:21.6241814Z 2025-12-04T13:50:21.6242077Z Finished optim/test_swa_utils 1/1 ... [2025-12-04 13:50:21.623807][15069.566023589], took 0.05min 2025-12-04T13:50:21.6529338Z Running test_tensorboard 1/1 ... [2025-12-04 13:50:21.652727][15069.594945925] 2025-12-04T13:50:21.6529781Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2025-12-04T13:50:21.6532606Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'test_tensorboard.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-12-04 13:50:21.653044] 2025-12-04T13:50:41.4082196Z 2025-12-04T13:50:41.4083127Z test_tensorboard 1/1 was successful, full logs can be found in artifacts with path test/test-reports/test_tensorboard_1.1_041aa7283ea5c0c5_.log 2025-12-04T13:50:41.4096622Z Running 50 items in this shard: test/test_tensorboard.py::TestTensorBoardPyTorchNumpy::test_pytorch_autograd_np, test/test_tensorboard.py::TestTensorBoardPyTorchNumpy::test_pytorch_histogram, test/test_tensorboard.py::TestTensorBoardPyTorchNumpy::test_pytorch_histogram_raw, test/test_tensorboard.py::TestTensorBoardPyTorchNumpy::test_pytorch_np, test/test_tensorboard.py::TestTensorBoardPyTorchNumpy::test_pytorch_write, test/test_tensorboard.py::TestTensorBoardUtils::test_convert_to_HWC_dtype_remains_same, test/test_tensorboard.py::TestTensorBoardUtils::test_numpy_vid_uint8, test/test_tensorboard.py::TestTensorBoardUtils::test_prepare_video, test/test_tensorboard.py::TestTensorBoardUtils::test_to_HWC, test/test_tensorboard.py::TestTensorBoardWriter::test_writer, test/test_tensorboard.py::TestTensorBoardSummaryWriter::test_pathlib, test/test_tensorboard.py::TestTensorBoardSummaryWriter::test_summary_writer_close, test/test_tensorboard.py::TestTensorBoardSummaryWriter::test_summary_writer_ctx, test/test_tensorboard.py::TestTensorBoardEmbedding::test_embedding, test/test_tensorboard.py::TestTensorBoardEmbedding::test_embedding_64, test/test_tensorboard.py::TestTensorBoardSummary::test_audio, test/test_tensorboard.py::TestTensorBoardSummary::test_custom_scalars, test/test_tensorboard.py::TestTensorBoardSummary::test_empty_input, test/test_tensorboard.py::TestTensorBoardSummary::test_float32_image, test/test_tensorboard.py::TestTensorBoardSummary::test_histogram_auto, test/test_tensorboard.py::TestTensorBoardSummary::test_histogram_doane, test/test_tensorboard.py::TestTensorBoardSummary::test_histogram_fd, test/test_tensorboard.py::TestTensorBoardSummary::test_image_with_3_channel_batched, test/test_tensorboard.py::TestTensorBoardSummary::test_image_with_boxes, test/test_tensorboard.py::TestTensorBoardSummary::test_image_with_one_channel, test/test_tensorboard.py::TestTensorBoardSummary::test_image_with_one_channel_batched, test/test_tensorboard.py::TestTensorBoardSummary::test_image_without_channel, test/test_tensorboard.py::TestTensorBoardSummary::test_list_input, test/test_tensorboard.py::TestTensorBoardSummary::test_mesh, test/test_tensorboard.py::TestTensorBoardSummary::test_scalar_new_style, test/test_tensorboard.py::TestTensorBoardSummary::test_text, test/test_tensorboard.py::TestTensorBoardSummary::test_uint8_image, test/test_tensorboard.py::TestTensorBoardSummary::test_video, test/test_tensorboard.py::TestTensorBoardPytorchGraph::test_mlp_graph, test/test_tensorboard.py::TestTensorBoardPytorchGraph::test_nested_nn_squential, test/test_tensorboard.py::TestTensorBoardPytorchGraph::test_pytorch_graph, test/test_tensorboard.py::TestTensorBoardPytorchGraph::test_pytorch_graph_dict_input, test/test_tensorboard.py::TestTensorBoardPytorchGraph::test_torchvision_smoke, test/test_tensorboard.py::TestTensorBoardPytorchGraph::test_wrong_input_size, test/test_tensorboard.py::TestTensorBoardFigure::test_figure, test/test_tensorboard.py::TestTensorBoardFigure::test_figure_list, test/test_tensorboard.py::TestTensorBoardNumpy::test_pytorch_np_expect_fail, test/test_tensorboard.py::TestTensorBoardNumpy::test_scalar, test/test_tensorboard.py::TestTensorProtoSummary::test_complex_tensor_proto, test/test_tensorboard.py::TestTensorProtoSummary::test_empty_tensor_proto, test/test_tensorboard.py::TestTensorProtoSummary::test_float_tensor_proto, test/test_tensorboard.py::TestTensorProtoSummary::test_half_tensor_proto_bfloat16_proto_type_14, test/test_tensorboard.py::TestTensorProtoSummary::test_half_tensor_proto_float16_proto_type_19, test/test_tensorboard.py::TestTensorProtoSummary::test_int_tensor_proto, test/test_tensorboard.py::TestTensorProtoSummary::test_scalar_tensor_proto 2025-12-04T13:50:41.4107518Z 2025-12-04T13:50:41.4107723Z Finished test_tensorboard 1/1 ... [2025-12-04 13:50:41.407962][15089.350179181], took 0.33min 2025-12-04T13:50:41.4386850Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/test_tensorboard/test_tensorboard-fca5410f597c8a55.xml 2025-12-04T13:50:41.5027796Z Running torch_np/numpy_tests/core/test_indexing 1/1 ... [2025-12-04 13:50:41.502526][15089.444743938] 2025-12-04T13:50:41.5028305Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2025-12-04T13:50:41.5030820Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'torch_np/numpy_tests/core/test_indexing.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-12-04 13:50:41.502830] 2025-12-04T13:50:45.1239747Z 2025-12-04T13:50:45.1241116Z torch_np/numpy_tests/core/test_indexing 1/1 was successful, full logs can be found in artifacts with path test/test-reports/torch_np.numpy_tests.core.test_indexing_1.1_e67fb534e4576208_.log 2025-12-04T13:50:45.1262108Z Running 67 items in this shard: test/torch_np/numpy_tests/core/test_indexing.py::TestIndexing::test_boolean_assignment_value_mismatch, test/torch_np/numpy_tests/core/test_indexing.py::TestIndexing::test_boolean_indexing_list, test/torch_np/numpy_tests/core/test_indexing.py::TestIndexing::test_boolean_indexing_onedim, test/torch_np/numpy_tests/core/test_indexing.py::TestIndexing::test_boolean_indexing_twodim, test/torch_np/numpy_tests/core/test_indexing.py::TestIndexing::test_boolean_shape_mismatch, test/torch_np/numpy_tests/core/test_indexing.py::TestIndexing::test_broaderrors_indexing, test/torch_np/numpy_tests/core/test_indexing.py::TestIndexing::test_broken_sequence_not_nd_index, test/torch_np/numpy_tests/core/test_indexing.py::TestIndexing::test_ellipsis_index, test/torch_np/numpy_tests/core/test_indexing.py::TestIndexing::test_ellipsis_index_2, test/torch_np/numpy_tests/core/test_indexing.py::TestIndexing::test_empty_fancy_index, test/torch_np/numpy_tests/core/test_indexing.py::TestIndexing::test_empty_tuple_index, test/torch_np/numpy_tests/core/test_indexing.py::TestIndexing::test_everything_returns_views, test/torch_np/numpy_tests/core/test_indexing.py::TestIndexing::test_index_no_array_to_index, test/torch_np/numpy_tests/core/test_indexing.py::TestIndexing::test_index_no_floats, test/torch_np/numpy_tests/core/test_indexing.py::TestIndexing::test_indexing_array_negative_strides, test/torch_np/numpy_tests/core/test_indexing.py::TestIndexing::test_indexing_array_weird_strides, test/torch_np/numpy_tests/core/test_indexing.py::TestIndexing::test_memory_order, test/torch_np/numpy_tests/core/test_indexing.py::TestIndexing::test_none_index, test/torch_np/numpy_tests/core/test_indexing.py::TestIndexing::test_nontuple_ndindex, test/torch_np/numpy_tests/core/test_indexing.py::TestIndexing::test_reverse_strides_and_subspace_bufferinit, test/torch_np/numpy_tests/core/test_indexing.py::TestIndexing::test_reversed_strides_result_allocation, test/torch_np/numpy_tests/core/test_indexing.py::TestIndexing::test_same_kind_index_casting, test/torch_np/numpy_tests/core/test_indexing.py::TestIndexing::test_scalar_array_bool, test/torch_np/numpy_tests/core/test_indexing.py::TestIndexing::test_single_bool_index, test/torch_np/numpy_tests/core/test_indexing.py::TestIndexing::test_single_int_index, test/torch_np/numpy_tests/core/test_indexing.py::TestIndexing::test_slicing_no_floats, test/torch_np/numpy_tests/core/test_indexing.py::TestIndexing::test_small_regressions, test/torch_np/numpy_tests/core/test_indexing.py::TestIndexing::test_too_many_advanced_indices_index2_num_32_original_ndim_1, test/torch_np/numpy_tests/core/test_indexing.py::TestIndexing::test_too_many_advanced_indices_index2_num_32_original_ndim_32, test/torch_np/numpy_tests/core/test_indexing.py::TestIndexing::test_too_many_advanced_indices_index2_num_40_original_ndim_1, test/torch_np/numpy_tests/core/test_indexing.py::TestIndexing::test_too_many_advanced_indices_index2_num_40_original_ndim_32, test/torch_np/numpy_tests/core/test_indexing.py::TestIndexing::test_too_many_advanced_indices_index_False_num_32_original_ndim_1, test/torch_np/numpy_tests/core/test_indexing.py::TestIndexing::test_too_many_advanced_indices_index_False_num_32_original_ndim_32, test/torch_np/numpy_tests/core/test_indexing.py::TestIndexing::test_too_many_advanced_indices_index_False_num_40_original_ndim_1, test/torch_np/numpy_tests/core/test_indexing.py::TestIndexing::test_too_many_advanced_indices_index_False_num_40_original_ndim_32, test/torch_np/numpy_tests/core/test_indexing.py::TestIndexing::test_too_many_advanced_indices_index_True_num_32_original_ndim_1, test/torch_np/numpy_tests/core/test_indexing.py::TestIndexing::test_too_many_advanced_indices_index_True_num_32_original_ndim_32, test/torch_np/numpy_tests/core/test_indexing.py::TestIndexing::test_too_many_advanced_indices_index_True_num_40_original_ndim_1, test/torch_np/numpy_tests/core/test_indexing.py::TestIndexing::test_too_many_advanced_indices_index_True_num_40_original_ndim_32, test/torch_np/numpy_tests/core/test_indexing.py::TestIndexing::test_too_many_fancy_indices_special_case, test/torch_np/numpy_tests/core/test_indexing.py::TestIndexing::test_trivial_fancy_not_possible, test/torch_np/numpy_tests/core/test_indexing.py::TestIndexing::test_trivial_fancy_out_of_bounds, test/torch_np/numpy_tests/core/test_indexing.py::TestIndexing::test_tuple_subclass, test/torch_np/numpy_tests/core/test_indexing.py::TestIndexing::test_uncontiguous_subspace_assignment, test/torch_np/numpy_tests/core/test_indexing.py::TestBroadcastedAssignments::test_broadcast_error_reports_correct_shape_index0, test/torch_np/numpy_tests/core/test_indexing.py::TestBroadcastedAssignments::test_broadcast_error_reports_correct_shape_index1, test/torch_np/numpy_tests/core/test_indexing.py::TestBroadcastedAssignments::test_broadcast_error_reports_correct_shape_index2, test/torch_np/numpy_tests/core/test_indexing.py::TestBroadcastedAssignments::test_broadcast_subspace, test/torch_np/numpy_tests/core/test_indexing.py::TestBroadcastedAssignments::test_index_is_larger, test/torch_np/numpy_tests/core/test_indexing.py::TestBroadcastedAssignments::test_prepend_not_one, test/torch_np/numpy_tests/core/test_indexing.py::TestBroadcastedAssignments::test_prepending_ones, test/torch_np/numpy_tests/core/test_indexing.py::TestBroadcastedAssignments::test_simple_broadcasting_errors, test/torch_np/numpy_tests/core/test_indexing.py::TestFancyIndexingCast::test_boolean_index_cast_assign, test/torch_np/numpy_tests/core/test_indexing.py::TestMultiIndexingAutomated::test_1d, test/torch_np/numpy_tests/core/test_indexing.py::TestMultiIndexingAutomated::test_boolean, test/torch_np/numpy_tests/core/test_indexing.py::TestMultiIndexingAutomated::test_multidim, test/torch_np/numpy_tests/core/test_indexing.py::TestFloatNonIntegerArgument::test_non_integer_argument_errors, test/torch_np/numpy_tests/core/test_indexing.py::TestFloatNonIntegerArgument::test_non_integer_sequence_multiplication, test/torch_np/numpy_tests/core/test_indexing.py::TestFloatNonIntegerArgument::test_reduce_axis_float_index, test/torch_np/numpy_tests/core/test_indexing.py::TestFloatNonIntegerArgument::test_valid_indexing, test/torch_np/numpy_tests/core/test_indexing.py::TestFloatNonIntegerArgument::test_valid_slicing, test/torch_np/numpy_tests/core/test_indexing.py::TestBooleanIndexing::test_bool_as_int_argument_errors, test/torch_np/numpy_tests/core/test_indexing.py::TestBooleanIndexing::test_boolean_indexing_fast_path, test/torch_np/numpy_tests/core/test_indexing.py::TestBooleanIndexing::test_boolean_indexing_weirdness, test/torch_np/numpy_tests/core/test_indexing.py::TestArrayToIndexDeprecation::test_array_to_index_error, test/torch_np/numpy_tests/core/test_indexing.py::TestNonIntegerArrayLike::test_basic, test/torch_np/numpy_tests/core/test_indexing.py::TestMultipleEllipsisError::test_basic 2025-12-04T13:50:45.1281762Z 2025-12-04T13:50:45.1282032Z Finished torch_np/numpy_tests/core/test_indexing 1/1 ... [2025-12-04 13:50:45.123860][15093.066071383], took 0.06min 2025-12-04T13:50:45.1540276Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/torch_np.numpy_tests.core.test_indexing/torch_np.numpy_tests.core.test_indexing-55ec670873971da8.xml 2025-12-04T13:50:45.1846346Z Running test_futures 1/1 ... [2025-12-04 13:50:45.184400][15093.126616611] 2025-12-04T13:50:45.1846985Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2025-12-04T13:50:45.1849828Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'test_futures.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-12-04 13:50:45.184695] 2025-12-04T13:50:49.1560248Z 2025-12-04T13:50:49.1561329Z test_futures 1/1 was successful, full logs can be found in artifacts with path test/test-reports/test_futures_1.1_c5e13566c134c7a8_.log 2025-12-04T13:50:49.1566707Z Running 22 items in this shard: test/test_futures.py::TestFuture::test_add_done_callback_error_is_ignored, test/test_futures.py::TestFuture::test_add_done_callback_maintains_callback_order, test/test_futures.py::TestFuture::test_add_done_callback_no_arg_error_is_ignored, test/test_futures.py::TestFuture::test_add_done_callback_simple, test/test_futures.py::TestFuture::test_chained_then, test/test_futures.py::TestFuture::test_collect_all, test/test_futures.py::TestFuture::test_done, test/test_futures.py::TestFuture::test_done_exception, test/test_futures.py::TestFuture::test_interleaving_then_and_add_done_callback_maintains_callback_order, test/test_futures.py::TestFuture::test_interleaving_then_and_add_done_callback_propagates_error, test/test_futures.py::TestFuture::test_mark_future_twice, test/test_futures.py::TestFuture::test_pickle_future, test/test_futures.py::TestFuture::test_set_exception, test/test_futures.py::TestFuture::test_set_exception_multithreading, test/test_futures.py::TestFuture::test_then, test/test_futures.py::TestFuture::test_then_no_arg, test/test_futures.py::TestFuture::test_then_raise, test/test_futures.py::TestFuture::test_then_wrong_arg, test/test_futures.py::TestFuture::test_wait, test/test_futures.py::TestFuture::test_wait_all, test/test_futures.py::TestFuture::test_wait_multi_thread, test/test_futures.py::TestFuture::test_wait_none 2025-12-04T13:50:49.1570853Z 2025-12-04T13:50:49.1571036Z Finished test_futures 1/1 ... [2025-12-04 13:50:49.155798][15097.09800857], took 0.07min 2025-12-04T13:50:49.1862257Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/test_futures/test_futures-365879bca82fb89f.xml 2025-12-04T13:50:49.2174372Z Running test_tensor_creation_ops 1/1 ... [2025-12-04 13:50:49.217189][15097.15940765] 2025-12-04T13:50:49.2175078Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2025-12-04T13:50:49.2177500Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'test_tensor_creation_ops.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-12-04 13:50:49.217480] 2025-12-04T13:52:18.1323863Z 2025-12-04T13:52:18.1325021Z test_tensor_creation_ops 1/1 was successful, full logs can be found in artifacts with path test/test-reports/test_tensor_creation_ops_1.1_21234e58f391600d_.log 2025-12-04T13:52:18.1461917Z Running 536 items in this shard: test/test_tensor_creation_ops.py::TestTensorCreationCUDA::test_arange_cuda, test/test_tensor_creation_ops.py::TestTensorCreationCUDA::test_arange_device_vs_cpu_cuda_float32, test/test_tensor_creation_ops.py::TestTensorCreationCUDA::test_arange_device_vs_cpu_cuda_float64, test/test_tensor_creation_ops.py::TestTensorCreationCUDA::test_arange_device_vs_cpu_cuda_int32, test/test_tensor_creation_ops.py::TestTensorCreationCUDA::test_arange_device_vs_cpu_cuda_int64, test/test_tensor_creation_ops.py::TestTensorCreationCUDA::test_arange_inference_cuda, test/test_tensor_creation_ops.py::TestTensorCreationCUDA::test_arange_lowp_cuda_bfloat16, test/test_tensor_creation_ops.py::TestTensorCreationCUDA::test_arange_lowp_cuda_float16, test/test_tensor_creation_ops.py::TestTensorCreationCUDA::test_as_strided_neg_cuda, test/test_tensor_creation_ops.py::TestTensorCreationCUDA::test_as_tensor_cuda, test/test_tensor_creation_ops.py::TestTensorCreationCUDA::test_block_diag_cuda, test/test_tensor_creation_ops.py::TestTensorCreationCUDA::test_block_diag_scipy_cuda, test/test_tensor_creation_ops.py::TestTensorCreationCUDA::test_cartesian_prod_cuda, test/test_tensor_creation_ops.py::TestTensorCreationCUDA::test_cat2_cuda_float16, test/test_tensor_creation_ops.py::TestTensorCreationCUDA::test_cat2_cuda_float64, test/test_tensor_creation_ops.py::TestTensorCreationCUDA::test_cat2_cuda_int32, test/test_tensor_creation_ops.py::TestTensorCreationCUDA::test_cat_all_dtypes_and_devices_cuda, test/test_tensor_creation_ops.py::TestTensorCreationCUDA::test_cat_big_cuda, test/test_tensor_creation_ops.py::TestTensorCreationCUDA::test_cat_channels_last_large_inputs_cuda, test/test_tensor_creation_ops.py::TestTensorCreationCUDA::test_cat_cuda, test/test_tensor_creation_ops.py::TestTensorCreationCUDA::test_cat_empty_cuda, test/test_tensor_creation_ops.py::TestTensorCreationCUDA::test_cat_empty_legacy_cuda, test/test_tensor_creation_ops.py::TestTensorCreationCUDA::test_cat_in_channels_last_cuda, test/test_tensor_creation_ops.py::TestTensorCreationCUDA::test_cat_large_tensor_cuda_float32, test/test_tensor_creation_ops.py::TestTensorCreationCUDA::test_cat_mem_overlap_cuda, test/test_tensor_creation_ops.py::TestTensorCreationCUDA::test_cat_misaligned_cuda_float32, test/test_tensor_creation_ops.py::TestTensorCreationCUDA::test_cat_multi_batch_cuda_float32, test/test_tensor_creation_ops.py::TestTensorCreationCUDA::test_cat_out_channels_last_cuda, test/test_tensor_creation_ops.py::TestTensorCreationCUDA::test_cat_out_cuda, test/test_tensor_creation_ops.py::TestTensorCreationCUDA::test_cat_out_fast_path_dim0_dim1_cuda_complex128, test/test_tensor_creation_ops.py::TestTensorCreationCUDA::test_cat_out_fast_path_dim0_dim1_cuda_complex64, test/test_tensor_creation_ops.py::TestTensorCreationCUDA::test_cat_out_fast_path_dim0_dim1_cuda_float32, test/test_tensor_creation_ops.py::TestTensorCreationCUDA::test_cat_out_fast_path_dim0_dim1_cuda_float64, test/test_tensor_creation_ops.py::TestTensorCreationCUDA::test_cat_out_fast_path_dim0_dim1_cuda_int16, test/test_tensor_creation_ops.py::TestTensorCreationCUDA::test_cat_out_fast_path_dim0_dim1_cuda_int32, test/test_tensor_creation_ops.py::TestTensorCreationCUDA::test_cat_out_fast_path_dim0_dim1_cuda_int64, test/test_tensor_creation_ops.py::TestTensorCreationCUDA::test_cat_out_fast_path_dim0_dim1_cuda_int8, test/test_tensor_creation_ops.py::TestTensorCreationCUDA::test_cat_out_fast_path_dim0_dim1_cuda_uint16, test/test_tensor_creation_ops.py::TestTensorCreationCUDA::test_cat_out_fast_path_dim0_dim1_cuda_uint32, test/test_tensor_creation_ops.py::TestTensorCreationCUDA::test_cat_out_fast_path_dim0_dim1_cuda_uint64, test/test_tensor_creation_ops.py::TestTensorCreationCUDA::test_cat_out_fast_path_dim0_dim1_cuda_uint8, test/test_tensor_creation_ops.py::TestTensorCreationCUDA::test_cat_out_memory_format_cuda, test/test_tensor_creation_ops.py::TestTensorCreationCUDA::test_cat_preserve_channels_last_cuda, test/test_tensor_creation_ops.py::TestTensorCreationCUDA::test_cat_size1_cuda_float32, test/test_tensor_creation_ops.py::TestTensorCreationCUDA::test_cat_stack_cross_devices_cuda, test/test_tensor_creation_ops.py::TestTensorCreationCUDA::test_cat_trailing_dim_cuda_float32, test/test_tensor_creation_ops.py::TestTensorCreationCUDA::test_combinations_cuda, test/test_tensor_creation_ops.py::TestTensorCreationCUDA::test_complex_type_conversions_cuda, test/test_tensor_creation_ops.py::TestTensorCreationCUDA::test_concat_empty_list_error_cuda, test/test_tensor_creation_ops.py::TestTensorCreationCUDA::test_constructor_device_legacy_cuda, test/test_tensor_creation_ops.py::TestTensorCreationCUDA::test_constructor_dtypes_cuda, test/test_tensor_creation_ops.py::TestTensorCreationCUDA::test_ctor_with_numpy_array_cuda, test/test_tensor_creation_ops.py::TestTensorCreationCUDA::test_device_rounding_cuda_float16, test/test_tensor_creation_ops.py::TestTensorCreationCUDA::test_device_rounding_cuda_float32, test/test_tensor_creation_ops.py::TestTensorCreationCUDA::test_device_rounding_cuda_float64, test/test_tensor_creation_ops.py::TestTensorCreationCUDA::test_diag_embed_cuda_float32, test/test_tensor_creation_ops.py::TestTensorCreationCUDA::test_diagflat_cuda, test/test_tensor_creation_ops.py::TestTensorCreationCUDA::test_dsplit_cuda_complex64, test/test_tensor_creation_ops.py::TestTensorCreationCUDA::test_dsplit_cuda_float32, test/test_tensor_creation_ops.py::TestTensorCreationCUDA::test_dsplit_cuda_int64, test/test_tensor_creation_ops.py::TestTensorCreationCUDA::test_dstack_cuda_complex128, test/test_tensor_creation_ops.py::TestTensorCreationCUDA::test_dstack_cuda_complex64, test/test_tensor_creation_ops.py::TestTensorCreationCUDA::test_dstack_cuda_float16, test/test_tensor_creation_ops.py::TestTensorCreationCUDA::test_dstack_cuda_float32, test/test_tensor_creation_ops.py::TestTensorCreationCUDA::test_dstack_cuda_float64, test/test_tensor_creation_ops.py::TestTensorCreationCUDA::test_dstack_cuda_int16, test/test_tensor_creation_ops.py::TestTensorCreationCUDA::test_dstack_cuda_int32, test/test_tensor_creation_ops.py::TestTensorCreationCUDA::test_dstack_cuda_int64, test/test_tensor_creation_ops.py::TestTensorCreationCUDA::test_dstack_cuda_int8, test/test_tensor_creation_ops.py::TestTensorCreationCUDA::test_dstack_cuda_uint8, test/test_tensor_creation_ops.py::TestTensorCreationCUDA::test_empty_full_cuda, test/test_tensor_creation_ops.py::TestTensorCreationCUDA::test_empty_overflow_cuda, test/test_tensor_creation_ops.py::TestTensorCreationCUDA::test_empty_strided_cuda, test/test_tensor_creation_ops.py::TestTensorCreationCUDA::test_empty_tensor_props_cuda, test/test_tensor_creation_ops.py::TestTensorCreationCUDA::test_eye_cuda, test/test_tensor_creation_ops.py::TestTensorCreationCUDA::test_fill_all_dtypes_and_devices_cuda, test/test_tensor_creation_ops.py::TestTensorCreationCUDA::test_float_to_int_conversion_finite_cuda_bool, test/test_tensor_creation_ops.py::TestTensorCreationCUDA::test_float_to_int_conversion_finite_cuda_int16, test/test_tensor_creation_ops.py::TestTensorCreationCUDA::test_float_to_int_conversion_finite_cuda_int32, test/test_tensor_creation_ops.py::TestTensorCreationCUDA::test_float_to_int_conversion_finite_cuda_int64, test/test_tensor_creation_ops.py::TestTensorCreationCUDA::test_float_to_int_conversion_finite_cuda_int8, test/test_tensor_creation_ops.py::TestTensorCreationCUDA::test_float_to_int_conversion_finite_cuda_uint8, test/test_tensor_creation_ops.py::TestTensorCreationCUDA::test_float_to_int_conversion_nonfinite_cuda_bool, test/test_tensor_creation_ops.py::TestTensorCreationCUDA::test_float_to_int_conversion_nonfinite_cuda_int16, test/test_tensor_creation_ops.py::TestTensorCreationCUDA::test_float_to_int_conversion_nonfinite_cuda_int32, test/test_tensor_creation_ops.py::TestTensorCreationCUDA::test_float_to_int_conversion_nonfinite_cuda_int64, test/test_tensor_creation_ops.py::TestTensorCreationCUDA::test_float_to_int_conversion_nonfinite_cuda_int8, test/test_tensor_creation_ops.py::TestTensorCreationCUDA::test_float_to_int_conversion_nonfinite_cuda_uint8, test/test_tensor_creation_ops.py::TestTensorCreationCUDA::test_from_file_shared_False_cuda, test/test_tensor_creation_ops.py::TestTensorCreationCUDA::test_from_file_shared_True_cuda, test/test_tensor_creation_ops.py::TestTensorCreationCUDA::test_full_inference_cuda_float16, test/test_tensor_creation_ops.py::TestTensorCreationCUDA::test_full_inference_cuda_float32, test/test_tensor_creation_ops.py::TestTensorCreationCUDA::test_full_inference_cuda_float64, test/test_tensor_creation_ops.py::TestTensorCreationCUDA::test_full_out_cuda, test/test_tensor_creation_ops.py::TestTensorCreationCUDA::test_hsplit_cuda_complex64, test/test_tensor_creation_ops.py::TestTensorCreationCUDA::test_hsplit_cuda_float32, test/test_tensor_creation_ops.py::TestTensorCreationCUDA::test_hsplit_cuda_int64, test/test_tensor_creation_ops.py::TestTensorCreationCUDA::test_hstack_column_stack_cuda_complex128, test/test_tensor_creation_ops.py::TestTensorCreationCUDA::test_hstack_column_stack_cuda_complex64, test/test_tensor_creation_ops.py::TestTensorCreationCUDA::test_hstack_column_stack_cuda_float16, test/test_tensor_creation_ops.py::TestTensorCreationCUDA::test_hstack_column_stack_cuda_float32, test/test_tensor_creation_ops.py::TestTensorCreationCUDA::test_hstack_column_stack_cuda_float64, test/test_tensor_creation_ops.py::TestTensorCreationCUDA::test_hstack_column_stack_cuda_int16, test/test_tensor_creation_ops.py::TestTensorCreationCUDA::test_hstack_column_stack_cuda_int32, test/test_tensor_creation_ops.py::TestTensorCreationCUDA::test_hstack_column_stack_cuda_int64, test/test_tensor_creation_ops.py::TestTensorCreationCUDA::test_hstack_column_stack_cuda_int8, test/test_tensor_creation_ops.py::TestTensorCreationCUDA::test_hstack_column_stack_cuda_uint8, test/test_tensor_creation_ops.py::TestTensorCreationCUDA::test_kaiser_cuda_float32, test/test_tensor_creation_ops.py::TestTensorCreationCUDA::test_kaiser_cuda_float64, test/test_tensor_creation_ops.py::TestTensorCreationCUDA::test_kaiser_window_cuda_bfloat16, test/test_tensor_creation_ops.py::TestTensorCreationCUDA::test_kaiser_window_cuda_float16, test/test_tensor_creation_ops.py::TestTensorCreationCUDA::test_kaiser_window_cuda_float32, test/test_tensor_creation_ops.py::TestTensorCreationCUDA::test_kaiser_window_cuda_float64, test/test_tensor_creation_ops.py::TestTensorCreationCUDA::test_kaiser_window_cuda_int64, test/test_tensor_creation_ops.py::TestTensorCreationCUDA::test_large_linspace_cuda_int32, test/test_tensor_creation_ops.py::TestTensorCreationCUDA::test_large_linspace_cuda_int64, test/test_tensor_creation_ops.py::TestTensorCreationCUDA::test_like_fn_stride_proparation_vs_tensoriterator_unary_op_cuda, test/test_tensor_creation_ops.py::TestTensorCreationCUDA::test_linlogspace_mem_overlap_cuda, test/test_tensor_creation_ops.py::TestTensorCreationCUDA::test_linspace_cuda_bfloat16, test/test_tensor_creation_ops.py::TestTensorCreationCUDA::test_linspace_cuda_complex128, test/test_tensor_creation_ops.py::TestTensorCreationCUDA::test_linspace_cuda_complex64, test/test_tensor_creation_ops.py::TestTensorCreationCUDA::test_linspace_cuda_float32, test/test_tensor_creation_ops.py::TestTensorCreationCUDA::test_linspace_cuda_float64, test/test_tensor_creation_ops.py::TestTensorCreationCUDA::test_linspace_cuda_int16, test/test_tensor_creation_ops.py::TestTensorCreationCUDA::test_linspace_cuda_int32, test/test_tensor_creation_ops.py::TestTensorCreationCUDA::test_linspace_cuda_int64, test/test_tensor_creation_ops.py::TestTensorCreationCUDA::test_linspace_cuda_int8, test/test_tensor_creation_ops.py::TestTensorCreationCUDA::test_linspace_cuda_uint8, test/test_tensor_creation_ops.py::TestTensorCreationCUDA::test_linspace_deduction_cuda, test/test_tensor_creation_ops.py::TestTensorCreationCUDA::test_linspace_device_vs_cpu_cuda_bfloat16, test/test_tensor_creation_ops.py::TestTensorCreationCUDA::test_linspace_device_vs_cpu_cuda_complex128, test/test_tensor_creation_ops.py::TestTensorCreationCUDA::test_linspace_device_vs_cpu_cuda_complex64, test/test_tensor_creation_ops.py::TestTensorCreationCUDA::test_linspace_device_vs_cpu_cuda_float16, test/test_tensor_creation_ops.py::TestTensorCreationCUDA::test_linspace_device_vs_cpu_cuda_float32, test/test_tensor_creation_ops.py::TestTensorCreationCUDA::test_linspace_device_vs_cpu_cuda_float64, test/test_tensor_creation_ops.py::TestTensorCreationCUDA::test_linspace_special_steps_cuda_bfloat16, test/test_tensor_creation_ops.py::TestTensorCreationCUDA::test_linspace_special_steps_cuda_complex128, test/test_tensor_creation_ops.py::TestTensorCreationCUDA::test_linspace_special_steps_cuda_complex64, test/test_tensor_creation_ops.py::TestTensorCreationCUDA::test_linspace_special_steps_cuda_float16, test/test_tensor_creation_ops.py::TestTensorCreationCUDA::test_linspace_special_steps_cuda_float32, test/test_tensor_creation_ops.py::TestTensorCreationCUDA::test_linspace_special_steps_cuda_float64, test/test_tensor_creation_ops.py::TestTensorCreationCUDA::test_linspace_vs_numpy_complex_cuda_complex64, test/test_tensor_creation_ops.py::TestTensorCreationCUDA::test_linspace_vs_numpy_cuda_complex128, test/test_tensor_creation_ops.py::TestTensorCreationCUDA::test_linspace_vs_numpy_cuda_complex64, test/test_tensor_creation_ops.py::TestTensorCreationCUDA::test_linspace_vs_numpy_cuda_float32, test/test_tensor_creation_ops.py::TestTensorCreationCUDA::test_linspace_vs_numpy_cuda_float64, test/test_tensor_creation_ops.py::TestTensorCreationCUDA::test_linspace_vs_numpy_integral_cuda_int16, test/test_tensor_creation_ops.py::TestTensorCreationCUDA::test_linspace_vs_numpy_integral_cuda_int32, test/test_tensor_creation_ops.py::TestTensorCreationCUDA::test_linspace_vs_numpy_integral_cuda_int64, test/test_tensor_creation_ops.py::TestTensorCreationCUDA::test_linspace_vs_numpy_integral_cuda_int8, test/test_tensor_creation_ops.py::TestTensorCreationCUDA::test_linspace_vs_numpy_integral_cuda_uint8, test/test_tensor_creation_ops.py::TestTensorCreationCUDA::test_logspace_base2_cuda_float16, test/test_tensor_creation_ops.py::TestTensorCreationCUDA::test_logspace_base2_cuda_float32, test/test_tensor_creation_ops.py::TestTensorCreationCUDA::test_logspace_base2_cuda_float64, test/test_tensor_creation_ops.py::TestTensorCreationCUDA::test_logspace_cuda_bfloat16, test/test_tensor_creation_ops.py::TestTensorCreationCUDA::test_logspace_cuda_float16, test/test_tensor_creation_ops.py::TestTensorCreationCUDA::test_logspace_cuda_float32, test/test_tensor_creation_ops.py::TestTensorCreationCUDA::test_logspace_cuda_float64, test/test_tensor_creation_ops.py::TestTensorCreationCUDA::test_logspace_cuda_int16, test/test_tensor_creation_ops.py::TestTensorCreationCUDA::test_logspace_cuda_int32, test/test_tensor_creation_ops.py::TestTensorCreationCUDA::test_logspace_cuda_int64, test/test_tensor_creation_ops.py::TestTensorCreationCUDA::test_logspace_cuda_int8, test/test_tensor_creation_ops.py::TestTensorCreationCUDA::test_logspace_cuda_uint8, test/test_tensor_creation_ops.py::TestTensorCreationCUDA::test_logspace_deduction_cuda, test/test_tensor_creation_ops.py::TestTensorCreationCUDA::test_logspace_device_vs_cpu_cuda_float16, test/test_tensor_creation_ops.py::TestTensorCreationCUDA::test_logspace_device_vs_cpu_cuda_float32, test/test_tensor_creation_ops.py::TestTensorCreationCUDA::test_logspace_device_vs_cpu_cuda_float64, test/test_tensor_creation_ops.py::TestTensorCreationCUDA::test_logspace_special_steps_cuda_float16, test/test_tensor_creation_ops.py::TestTensorCreationCUDA::test_logspace_special_steps_cuda_float32, test/test_tensor_creation_ops.py::TestTensorCreationCUDA::test_logspace_special_steps_cuda_float64, test/test_tensor_creation_ops.py::TestTensorCreationCUDA::test_logspace_vs_numpy_complex_cuda_complex64, test/test_tensor_creation_ops.py::TestTensorCreationCUDA::test_logspace_vs_numpy_cuda_float32, test/test_tensor_creation_ops.py::TestTensorCreationCUDA::test_logspace_vs_numpy_cuda_float64, test/test_tensor_creation_ops.py::TestTensorCreationCUDA::test_meshgrid_default_indexing_cuda, test/test_tensor_creation_ops.py::TestTensorCreationCUDA::test_meshgrid_empty_cuda, test/test_tensor_creation_ops.py::TestTensorCreationCUDA::test_meshgrid_ij_indexing_cuda, test/test_tensor_creation_ops.py::TestTensorCreationCUDA::test_meshgrid_ij_indexing_is_default_cuda, test/test_tensor_creation_ops.py::TestTensorCreationCUDA::test_meshgrid_inconsistent_device_cuda, test/test_tensor_creation_ops.py::TestTensorCreationCUDA::test_meshgrid_inconsistent_dtype_cuda, test/test_tensor_creation_ops.py::TestTensorCreationCUDA::test_meshgrid_non_1d_tensor_cuda, test/test_tensor_creation_ops.py::TestTensorCreationCUDA::test_meshgrid_unsupported_indexing_cuda, test/test_tensor_creation_ops.py::TestTensorCreationCUDA::test_meshgrid_vs_numpy_cuda, test/test_tensor_creation_ops.py::TestTensorCreationCUDA::test_meshgrid_warns_if_no_indexing_cuda, test/test_tensor_creation_ops.py::TestTensorCreationCUDA::test_meshgrid_xy_indexing_cuda, test/test_tensor_creation_ops.py::TestTensorCreationCUDA::test_new_empty_strided_cuda, test/test_tensor_creation_ops.py::TestTensorCreationCUDA::test_new_methods_requires_grad_cuda, test/test_tensor_creation_ops.py::TestTensorCreationCUDA::test_new_tensor_cuda, test/test_tensor_creation_ops.py::TestTensorCreationCUDA::test_new_tensor_device_cuda, test/test_tensor_creation_ops.py::TestTensorCreationCUDA::test_offset_scalar_cast_cuda, test/test_tensor_creation_ops.py::TestTensorCreationCUDA::test_ones_cuda, test/test_tensor_creation_ops.py::TestTensorCreationCUDA::test_random_bool_cuda, test/test_tensor_creation_ops.py::TestTensorCreationCUDA::test_random_cuda_float32, test/test_tensor_creation_ops.py::TestTensorCreationCUDA::test_random_cuda_float64, test/test_tensor_creation_ops.py::TestTensorCreationCUDA::test_random_cuda_int16, test/test_tensor_creation_ops.py::TestTensorCreationCUDA::test_random_cuda_int32, test/test_tensor_creation_ops.py::TestTensorCreationCUDA::test_random_cuda_int64, test/test_tensor_creation_ops.py::TestTensorCreationCUDA::test_random_cuda_int8, test/test_tensor_creation_ops.py::TestTensorCreationCUDA::test_random_default_cuda_bfloat16, test/test_tensor_creation_ops.py::TestTensorCreationCUDA::test_random_default_cuda_float16, test/test_tensor_creation_ops.py::TestTensorCreationCUDA::test_random_default_cuda_float32, test/test_tensor_creation_ops.py::TestTensorCreationCUDA::test_random_default_cuda_float64, test/test_tensor_creation_ops.py::TestTensorCreationCUDA::test_random_default_cuda_int16, test/test_tensor_creation_ops.py::TestTensorCreationCUDA::test_random_default_cuda_int32, test/test_tensor_creation_ops.py::TestTensorCreationCUDA::test_random_default_cuda_int64, test/test_tensor_creation_ops.py::TestTensorCreationCUDA::test_random_default_cuda_int8, test/test_tensor_creation_ops.py::TestTensorCreationCUDA::test_random_default_cuda_uint8, test/test_tensor_creation_ops.py::TestTensorCreationCUDA::test_random_from_to_bool_cuda, test/test_tensor_creation_ops.py::TestTensorCreationCUDA::test_random_from_to_cuda_bfloat16, test/test_tensor_creation_ops.py::TestTensorCreationCUDA::test_random_from_to_cuda_float16, test/test_tensor_creation_ops.py::TestTensorCreationCUDA::test_random_from_to_cuda_float32, test/test_tensor_creation_ops.py::TestTensorCreationCUDA::test_random_from_to_cuda_float64, test/test_tensor_creation_ops.py::TestTensorCreationCUDA::test_random_from_to_cuda_int16, test/test_tensor_creation_ops.py::TestTensorCreationCUDA::test_random_from_to_cuda_int32, test/test_tensor_creation_ops.py::TestTensorCreationCUDA::test_random_from_to_cuda_int64, test/test_tensor_creation_ops.py::TestTensorCreationCUDA::test_random_from_to_cuda_int8, test/test_tensor_creation_ops.py::TestTensorCreationCUDA::test_random_from_to_cuda_uint16, test/test_tensor_creation_ops.py::TestTensorCreationCUDA::test_random_from_to_cuda_uint32, test/test_tensor_creation_ops.py::TestTensorCreationCUDA::test_random_from_to_cuda_uint8, test/test_tensor_creation_ops.py::TestTensorCreationCUDA::test_random_full_range_cuda_bfloat16, test/test_tensor_creation_ops.py::TestTensorCreationCUDA::test_random_full_range_cuda_float16, test/test_tensor_creation_ops.py::TestTensorCreationCUDA::test_random_full_range_cuda_float32, test/test_tensor_creation_ops.py::TestTensorCreationCUDA::test_random_full_range_cuda_float64, test/test_tensor_creation_ops.py::TestTensorCreationCUDA::test_random_full_range_cuda_int16, test/test_tensor_creation_ops.py::TestTensorCreationCUDA::test_random_full_range_cuda_int32, test/test_tensor_creation_ops.py::TestTensorCreationCUDA::test_random_full_range_cuda_int64, test/test_tensor_creation_ops.py::TestTensorCreationCUDA::test_random_full_range_cuda_int8, test/test_tensor_creation_ops.py::TestTensorCreationCUDA::test_random_full_range_cuda_uint16, test/test_tensor_creation_ops.py::TestTensorCreationCUDA::test_random_full_range_cuda_uint32, test/test_tensor_creation_ops.py::TestTensorCreationCUDA::test_random_full_range_cuda_uint8, test/test_tensor_creation_ops.py::TestTensorCreationCUDA::test_random_to_cuda_bfloat16, test/test_tensor_creation_ops.py::TestTensorCreationCUDA::test_random_to_cuda_float16, test/test_tensor_creation_ops.py::TestTensorCreationCUDA::test_random_to_cuda_float32, test/test_tensor_creation_ops.py::TestTensorCreationCUDA::test_random_to_cuda_float64, test/test_tensor_creation_ops.py::TestTensorCreationCUDA::test_random_to_cuda_int16, test/test_tensor_creation_ops.py::TestTensorCreationCUDA::test_random_to_cuda_int32, test/test_tensor_creation_ops.py::TestTensorCreationCUDA::test_random_to_cuda_int64, test/test_tensor_creation_ops.py::TestTensorCreationCUDA::test_random_to_cuda_int8, test/test_tensor_creation_ops.py::TestTensorCreationCUDA::test_random_to_cuda_uint16, test/test_tensor_creation_ops.py::TestTensorCreationCUDA::test_random_to_cuda_uint32, test/test_tensor_creation_ops.py::TestTensorCreationCUDA::test_random_to_cuda_uint8, test/test_tensor_creation_ops.py::TestTensorCreationCUDA::test_range_cuda_float32, test/test_tensor_creation_ops.py::TestTensorCreationCUDA::test_range_factories_64bit_indexing_cuda, test/test_tensor_creation_ops.py::TestTensorCreationCUDA::test_range_warning_cuda, test/test_tensor_creation_ops.py::TestTensorCreationCUDA::test_refs_tensor_cuda_bfloat16, test/test_tensor_creation_ops.py::TestTensorCreationCUDA::test_refs_tensor_cuda_bool, test/test_tensor_creation_ops.py::TestTensorCreationCUDA::test_refs_tensor_cuda_complex128, test/test_tensor_creation_ops.py::TestTensorCreationCUDA::test_refs_tensor_cuda_complex64, test/test_tensor_creation_ops.py::TestTensorCreationCUDA::test_refs_tensor_cuda_float16, test/test_tensor_creation_ops.py::TestTensorCreationCUDA::test_refs_tensor_cuda_float32, test/test_tensor_creation_ops.py::TestTensorCreationCUDA::test_refs_tensor_cuda_float64, test/test_tensor_creation_ops.py::TestTensorCreationCUDA::test_refs_tensor_cuda_int16, test/test_tensor_creation_ops.py::TestTensorCreationCUDA::test_refs_tensor_cuda_int32, test/test_tensor_creation_ops.py::TestTensorCreationCUDA::test_refs_tensor_cuda_int64, test/test_tensor_creation_ops.py::TestTensorCreationCUDA::test_refs_tensor_cuda_int8, test/test_tensor_creation_ops.py::TestTensorCreationCUDA::test_refs_tensor_cuda_uint8, test/test_tensor_creation_ops.py::TestTensorCreationCUDA::test_repeat_interleave_cuda, test/test_tensor_creation_ops.py::TestTensorCreationCUDA::test_roll_cuda, test/test_tensor_creation_ops.py::TestTensorCreationCUDA::test_signal_window_functions_window_bartlett_cuda_bfloat16, test/test_tensor_creation_ops.py::TestTensorCreationCUDA::test_signal_window_functions_window_bartlett_cuda_float16, test/test_tensor_creation_ops.py::TestTensorCreationCUDA::test_signal_window_functions_window_bartlett_cuda_float32, test/test_tensor_creation_ops.py::TestTensorCreationCUDA::test_signal_window_functions_window_bartlett_cuda_float64, test/test_tensor_creation_ops.py::TestTensorCreationCUDA::test_signal_window_functions_window_bartlett_cuda_int64, test/test_tensor_creation_ops.py::TestTensorCreationCUDA::test_signal_window_functions_window_blackman_cuda_bfloat16, test/test_tensor_creation_ops.py::TestTensorCreationCUDA::test_signal_window_functions_window_blackman_cuda_float16, test/test_tensor_creation_ops.py::TestTensorCreationCUDA::test_signal_window_functions_window_blackman_cuda_float32, test/test_tensor_creation_ops.py::TestTensorCreationCUDA::test_signal_window_functions_window_blackman_cuda_float64, test/test_tensor_creation_ops.py::TestTensorCreationCUDA::test_signal_window_functions_window_blackman_cuda_int64, test/test_tensor_creation_ops.py::TestTensorCreationCUDA::test_signal_window_functions_window_hamming_cuda_bfloat16, test/test_tensor_creation_ops.py::TestTensorCreationCUDA::test_signal_window_functions_window_hamming_cuda_float16, test/test_tensor_creation_ops.py::TestTensorCreationCUDA::test_signal_window_functions_window_hamming_cuda_float32, test/test_tensor_creation_ops.py::TestTensorCreationCUDA::test_signal_window_functions_window_hamming_cuda_float64, test/test_tensor_creation_ops.py::TestTensorCreationCUDA::test_signal_window_functions_window_hamming_cuda_int64, test/test_tensor_creation_ops.py::TestTensorCreationCUDA::test_signal_window_functions_window_hann_cuda_bfloat16, test/test_tensor_creation_ops.py::TestTensorCreationCUDA::test_signal_window_functions_window_hann_cuda_float16, test/test_tensor_creation_ops.py::TestTensorCreationCUDA::test_signal_window_functions_window_hann_cuda_float32, test/test_tensor_creation_ops.py::TestTensorCreationCUDA::test_signal_window_functions_window_hann_cuda_float64, test/test_tensor_creation_ops.py::TestTensorCreationCUDA::test_signal_window_functions_window_hann_cuda_int64, test/test_tensor_creation_ops.py::TestTensorCreationCUDA::test_signal_windows_functions_window_bartlett_cuda_float32, test/test_tensor_creation_ops.py::TestTensorCreationCUDA::test_signal_windows_functions_window_bartlett_cuda_float64, test/test_tensor_creation_ops.py::TestTensorCreationCUDA::test_signal_windows_functions_window_blackman_cuda_float32, test/test_tensor_creation_ops.py::TestTensorCreationCUDA::test_signal_windows_functions_window_blackman_cuda_float64, test/test_tensor_creation_ops.py::TestTensorCreationCUDA::test_signal_windows_functions_window_cosine_cuda_float32, test/test_tensor_creation_ops.py::TestTensorCreationCUDA::test_signal_windows_functions_window_cosine_cuda_float64, test/test_tensor_creation_ops.py::TestTensorCreationCUDA::test_signal_windows_functions_window_hamming_cuda_float32, test/test_tensor_creation_ops.py::TestTensorCreationCUDA::test_signal_windows_functions_window_hamming_cuda_float64, test/test_tensor_creation_ops.py::TestTensorCreationCUDA::test_signal_windows_functions_window_hann_cuda_float32, test/test_tensor_creation_ops.py::TestTensorCreationCUDA::test_signal_windows_functions_window_hann_cuda_float64, test/test_tensor_creation_ops.py::TestTensorCreationCUDA::test_signal_windows_functions_window_nuttall_cuda_float32, test/test_tensor_creation_ops.py::TestTensorCreationCUDA::test_signal_windows_functions_window_nuttall_cuda_float64, test/test_tensor_creation_ops.py::TestTensorCreationCUDA::test_simple_scalar_cast_cuda, test/test_tensor_creation_ops.py::TestTensorCreationCUDA::test_stack_cuda, test/test_tensor_creation_ops.py::TestTensorCreationCUDA::test_stack_out_cuda, test/test_tensor_creation_ops.py::TestTensorCreationCUDA::test_storage_filename_cuda, test/test_tensor_creation_ops.py::TestTensorCreationCUDA::test_strided_mismatched_stride_shape_cuda, test/test_tensor_creation_ops.py::TestTensorCreationCUDA::test_tensor_ctor_device_inference_cuda, test/test_tensor_creation_ops.py::TestTensorCreationCUDA::test_tensor_device_cuda, test/test_tensor_creation_ops.py::TestTensorCreationCUDA::test_tensor_factories_empty_cuda, test/test_tensor_creation_ops.py::TestTensorCreationCUDA::test_tensor_factory_copy_var_cuda, test/test_tensor_creation_ops.py::TestTensorCreationCUDA::test_tensor_factory_cuda, test/test_tensor_creation_ops.py::TestTensorCreationCUDA::test_tensor_factory_gpu_type_cuda, test/test_tensor_creation_ops.py::TestTensorCreationCUDA::test_tensor_factory_gpu_type_inference_cuda, test/test_tensor_creation_ops.py::TestTensorCreationCUDA::test_tensor_factory_type_inference_cuda, test/test_tensor_creation_ops.py::TestTensorCreationCUDA::test_tensor_from_non_writable_numpy_cuda, test/test_tensor_creation_ops.py::TestTensorCreationCUDA::test_tensor_from_sequence_cuda, test/test_tensor_creation_ops.py::TestTensorCreationCUDA::test_torch_complex_cuda_float16, test/test_tensor_creation_ops.py::TestTensorCreationCUDA::test_torch_complex_cuda_float32, test/test_tensor_creation_ops.py::TestTensorCreationCUDA::test_torch_complex_cuda_float64, test/test_tensor_creation_ops.py::TestTensorCreationCUDA::test_torch_complex_floating_dtype_error_cuda_bool, test/test_tensor_creation_ops.py::TestTensorCreationCUDA::test_torch_complex_floating_dtype_error_cuda_complex128, test/test_tensor_creation_ops.py::TestTensorCreationCUDA::test_torch_complex_floating_dtype_error_cuda_complex64, test/test_tensor_creation_ops.py::TestTensorCreationCUDA::test_torch_complex_floating_dtype_error_cuda_int16, test/test_tensor_creation_ops.py::TestTensorCreationCUDA::test_torch_complex_floating_dtype_error_cuda_int32, test/test_tensor_creation_ops.py::TestTensorCreationCUDA::test_torch_complex_floating_dtype_error_cuda_int64, test/test_tensor_creation_ops.py::TestTensorCreationCUDA::test_torch_complex_floating_dtype_error_cuda_int8, test/test_tensor_creation_ops.py::TestTensorCreationCUDA::test_torch_complex_floating_dtype_error_cuda_uint8, test/test_tensor_creation_ops.py::TestTensorCreationCUDA::test_torch_complex_out_dtype_error_cuda_float32, test/test_tensor_creation_ops.py::TestTensorCreationCUDA::test_torch_complex_out_dtype_error_cuda_float64, test/test_tensor_creation_ops.py::TestTensorCreationCUDA::test_torch_complex_same_dtype_error_cuda_float32, test/test_tensor_creation_ops.py::TestTensorCreationCUDA::test_torch_complex_same_dtype_error_cuda_float64, test/test_tensor_creation_ops.py::TestTensorCreationCUDA::test_torch_polar_cuda_float32, test/test_tensor_creation_ops.py::TestTensorCreationCUDA::test_torch_polar_cuda_float64, test/test_tensor_creation_ops.py::TestTensorCreationCUDA::test_unpack_double_cuda_float32, test/test_tensor_creation_ops.py::TestTensorCreationCUDA::test_unpack_double_cuda_float64, test/test_tensor_creation_ops.py::TestTensorCreationCUDA::test_vander_cuda, test/test_tensor_creation_ops.py::TestTensorCreationCUDA::test_vander_types_cuda_bool, test/test_tensor_creation_ops.py::TestTensorCreationCUDA::test_vander_types_cuda_complex128, test/test_tensor_creation_ops.py::TestTensorCreationCUDA::test_vander_types_cuda_complex64, test/test_tensor_creation_ops.py::TestTensorCreationCUDA::test_vander_types_cuda_float32, test/test_tensor_creation_ops.py::TestTensorCreationCUDA::test_vander_types_cuda_float64, test/test_tensor_creation_ops.py::TestTensorCreationCUDA::test_vander_types_cuda_int16, test/test_tensor_creation_ops.py::TestTensorCreationCUDA::test_vander_types_cuda_int32, test/test_tensor_creation_ops.py::TestTensorCreationCUDA::test_vander_types_cuda_int64, test/test_tensor_creation_ops.py::TestTensorCreationCUDA::test_vander_types_cuda_int8, test/test_tensor_creation_ops.py::TestTensorCreationCUDA::test_vander_types_cuda_uint8, test/test_tensor_creation_ops.py::TestTensorCreationCUDA::test_vsplit_cuda_complex64, test/test_tensor_creation_ops.py::TestTensorCreationCUDA::test_vsplit_cuda_float32, test/test_tensor_creation_ops.py::TestTensorCreationCUDA::test_vsplit_cuda_int64, test/test_tensor_creation_ops.py::TestTensorCreationCUDA::test_vstack_row_stack_cuda_complex128, test/test_tensor_creation_ops.py::TestTensorCreationCUDA::test_vstack_row_stack_cuda_complex64, test/test_tensor_creation_ops.py::TestTensorCreationCUDA::test_vstack_row_stack_cuda_float16, test/test_tensor_creation_ops.py::TestTensorCreationCUDA::test_vstack_row_stack_cuda_float32, test/test_tensor_creation_ops.py::TestTensorCreationCUDA::test_vstack_row_stack_cuda_float64, test/test_tensor_creation_ops.py::TestTensorCreationCUDA::test_vstack_row_stack_cuda_int16, test/test_tensor_creation_ops.py::TestTensorCreationCUDA::test_vstack_row_stack_cuda_int32, test/test_tensor_creation_ops.py::TestTensorCreationCUDA::test_vstack_row_stack_cuda_int64, test/test_tensor_creation_ops.py::TestTensorCreationCUDA::test_vstack_row_stack_cuda_int8, test/test_tensor_creation_ops.py::TestTensorCreationCUDA::test_vstack_row_stack_cuda_uint8, test/test_tensor_creation_ops.py::TestTensorCreationCUDA::test_zeros_bounds_checking_cuda, test/test_tensor_creation_ops.py::TestTensorCreationCUDA::test_zeros_cuda, test/test_tensor_creation_ops.py::TestTensorCreationCUDA::test_zeros_dtype_layout_device_match_cuda_bool, test/test_tensor_creation_ops.py::TestTensorCreationCUDA::test_zeros_dtype_layout_device_match_cuda_complex64, test/test_tensor_creation_ops.py::TestTensorCreationCUDA::test_zeros_dtype_layout_device_match_cuda_float16, test/test_tensor_creation_ops.py::TestTensorCreationCUDA::test_zeros_dtype_layout_device_match_cuda_float32, test/test_tensor_creation_ops.py::TestTensorCreationCUDA::test_zeros_dtype_layout_device_match_cuda_int16, test/test_tensor_creation_ops.py::TestTensorCreationCUDA::test_zeros_dtype_layout_device_match_cuda_int64, test/test_tensor_creation_ops.py::TestTensorCreationCUDA::test_zeros_dtype_layout_device_match_cuda_uint8, test/test_tensor_creation_ops.py::TestTensorCreationCUDA::test_zeros_out_cuda, test/test_tensor_creation_ops.py::TestRandomTensorCreationCUDA::test_normal_cuda_float32, test/test_tensor_creation_ops.py::TestRandomTensorCreationCUDA::test_normal_cuda_float64, test/test_tensor_creation_ops.py::TestRandomTensorCreationCUDA::test_normal_std_error_cuda, test/test_tensor_creation_ops.py::TestRandomTensorCreationCUDA::test_rand_cuda_complex128, test/test_tensor_creation_ops.py::TestRandomTensorCreationCUDA::test_rand_cuda_complex32, test/test_tensor_creation_ops.py::TestRandomTensorCreationCUDA::test_rand_cuda_complex64, test/test_tensor_creation_ops.py::TestRandomTensorCreationCUDA::test_rand_cuda_float32, test/test_tensor_creation_ops.py::TestRandomTensorCreationCUDA::test_rand_cuda_float64, test/test_tensor_creation_ops.py::TestRandomTensorCreationCUDA::test_randint_cuda, test/test_tensor_creation_ops.py::TestRandomTensorCreationCUDA::test_randint_distribution_cuda, test/test_tensor_creation_ops.py::TestRandomTensorCreationCUDA::test_randint_inference_cuda, test/test_tensor_creation_ops.py::TestRandomTensorCreationCUDA::test_randn_cuda_bfloat16, test/test_tensor_creation_ops.py::TestRandomTensorCreationCUDA::test_randn_cuda_complex128, test/test_tensor_creation_ops.py::TestRandomTensorCreationCUDA::test_randn_cuda_complex32, test/test_tensor_creation_ops.py::TestRandomTensorCreationCUDA::test_randn_cuda_complex64, test/test_tensor_creation_ops.py::TestRandomTensorCreationCUDA::test_randn_cuda_float16, test/test_tensor_creation_ops.py::TestRandomTensorCreationCUDA::test_randn_cuda_float32, test/test_tensor_creation_ops.py::TestRandomTensorCreationCUDA::test_randn_cuda_float64, test/test_tensor_creation_ops.py::TestRandomTensorCreationCUDA::test_random_neg_values_cuda, test/test_tensor_creation_ops.py::TestRandomTensorCreationCUDA::test_randperm_cuda, test/test_tensor_creation_ops.py::TestRandomTensorCreationCUDA::test_randperm_device_compatibility_cuda, test/test_tensor_creation_ops.py::TestRandomTensorCreationCUDA::test_randperm_large_cuda, test/test_tensor_creation_ops.py::TestRandomTensorCreationCUDA::test_uniform_from_to_cuda_bfloat16, test/test_tensor_creation_ops.py::TestRandomTensorCreationCUDA::test_uniform_from_to_cuda_float16, test/test_tensor_creation_ops.py::TestRandomTensorCreationCUDA::test_uniform_from_to_cuda_float32, test/test_tensor_creation_ops.py::TestRandomTensorCreationCUDA::test_uniform_from_to_cuda_float64, test/test_tensor_creation_ops.py::TestLikeTensorCreationCUDA::test_empty_like_cuda, test/test_tensor_creation_ops.py::TestLikeTensorCreationCUDA::test_full_like_inference_cuda, test/test_tensor_creation_ops.py::TestLikeTensorCreationCUDA::test_ones_like_cuda, test/test_tensor_creation_ops.py::TestLikeTensorCreationCUDA::test_ones_like_multiple_device_cuda, test/test_tensor_creation_ops.py::TestLikeTensorCreationCUDA::test_rand_like_cuda, test/test_tensor_creation_ops.py::TestLikeTensorCreationCUDA::test_randint_like_cuda, test/test_tensor_creation_ops.py::TestLikeTensorCreationCUDA::test_randn_like_cuda, test/test_tensor_creation_ops.py::TestLikeTensorCreationCUDA::test_zeros_like_cuda, test/test_tensor_creation_ops.py::TestLikeTensorCreationCUDA::test_zeros_like_multiple_device_cuda, test/test_tensor_creation_ops.py::TestAsArrayCUDA::test_alias_from_buffer_cuda_bool, test/test_tensor_creation_ops.py::TestAsArrayCUDA::test_alias_from_buffer_cuda_complex128, test/test_tensor_creation_ops.py::TestAsArrayCUDA::test_alias_from_buffer_cuda_complex64, test/test_tensor_creation_ops.py::TestAsArrayCUDA::test_alias_from_buffer_cuda_float16, test/test_tensor_creation_ops.py::TestAsArrayCUDA::test_alias_from_buffer_cuda_float32, test/test_tensor_creation_ops.py::TestAsArrayCUDA::test_alias_from_buffer_cuda_float64, test/test_tensor_creation_ops.py::TestAsArrayCUDA::test_alias_from_buffer_cuda_int16, test/test_tensor_creation_ops.py::TestAsArrayCUDA::test_alias_from_buffer_cuda_int32, test/test_tensor_creation_ops.py::TestAsArrayCUDA::test_alias_from_buffer_cuda_int64, test/test_tensor_creation_ops.py::TestAsArrayCUDA::test_alias_from_buffer_cuda_int8, test/test_tensor_creation_ops.py::TestAsArrayCUDA::test_alias_from_buffer_cuda_uint16, test/test_tensor_creation_ops.py::TestAsArrayCUDA::test_alias_from_buffer_cuda_uint32, test/test_tensor_creation_ops.py::TestAsArrayCUDA::test_alias_from_buffer_cuda_uint64, test/test_tensor_creation_ops.py::TestAsArrayCUDA::test_alias_from_buffer_cuda_uint8, test/test_tensor_creation_ops.py::TestAsArrayCUDA::test_alias_from_dlpack_cuda_bfloat16, test/test_tensor_creation_ops.py::TestAsArrayCUDA::test_alias_from_dlpack_cuda_complex128, test/test_tensor_creation_ops.py::TestAsArrayCUDA::test_alias_from_dlpack_cuda_complex64, test/test_tensor_creation_ops.py::TestAsArrayCUDA::test_alias_from_dlpack_cuda_float16, test/test_tensor_creation_ops.py::TestAsArrayCUDA::test_alias_from_dlpack_cuda_float32, test/test_tensor_creation_ops.py::TestAsArrayCUDA::test_alias_from_dlpack_cuda_float64, test/test_tensor_creation_ops.py::TestAsArrayCUDA::test_alias_from_dlpack_cuda_int16, test/test_tensor_creation_ops.py::TestAsArrayCUDA::test_alias_from_dlpack_cuda_int32, test/test_tensor_creation_ops.py::TestAsArrayCUDA::test_alias_from_dlpack_cuda_int64, test/test_tensor_creation_ops.py::TestAsArrayCUDA::test_alias_from_dlpack_cuda_int8, test/test_tensor_creation_ops.py::TestAsArrayCUDA::test_alias_from_dlpack_cuda_uint8, test/test_tensor_creation_ops.py::TestAsArrayCUDA::test_alias_from_numpy_cuda_bool, test/test_tensor_creation_ops.py::TestAsArrayCUDA::test_alias_from_numpy_cuda_complex128, test/test_tensor_creation_ops.py::TestAsArrayCUDA::test_alias_from_numpy_cuda_complex64, test/test_tensor_creation_ops.py::TestAsArrayCUDA::test_alias_from_numpy_cuda_float16, test/test_tensor_creation_ops.py::TestAsArrayCUDA::test_alias_from_numpy_cuda_float32, test/test_tensor_creation_ops.py::TestAsArrayCUDA::test_alias_from_numpy_cuda_float64, test/test_tensor_creation_ops.py::TestAsArrayCUDA::test_alias_from_numpy_cuda_int16, test/test_tensor_creation_ops.py::TestAsArrayCUDA::test_alias_from_numpy_cuda_int32, test/test_tensor_creation_ops.py::TestAsArrayCUDA::test_alias_from_numpy_cuda_int64, test/test_tensor_creation_ops.py::TestAsArrayCUDA::test_alias_from_numpy_cuda_int8, test/test_tensor_creation_ops.py::TestAsArrayCUDA::test_alias_from_numpy_cuda_uint16, test/test_tensor_creation_ops.py::TestAsArrayCUDA::test_alias_from_numpy_cuda_uint32, test/test_tensor_creation_ops.py::TestAsArrayCUDA::test_alias_from_numpy_cuda_uint64, test/test_tensor_creation_ops.py::TestAsArrayCUDA::test_alias_from_numpy_cuda_uint8, test/test_tensor_creation_ops.py::TestAsArrayCUDA::test_alias_from_tensor_cuda_bfloat16, test/test_tensor_creation_ops.py::TestAsArrayCUDA::test_alias_from_tensor_cuda_bool, test/test_tensor_creation_ops.py::TestAsArrayCUDA::test_alias_from_tensor_cuda_complex128, test/test_tensor_creation_ops.py::TestAsArrayCUDA::test_alias_from_tensor_cuda_complex64, test/test_tensor_creation_ops.py::TestAsArrayCUDA::test_alias_from_tensor_cuda_float16, test/test_tensor_creation_ops.py::TestAsArrayCUDA::test_alias_from_tensor_cuda_float32, test/test_tensor_creation_ops.py::TestAsArrayCUDA::test_alias_from_tensor_cuda_float64, test/test_tensor_creation_ops.py::TestAsArrayCUDA::test_alias_from_tensor_cuda_int16, test/test_tensor_creation_ops.py::TestAsArrayCUDA::test_alias_from_tensor_cuda_int32, test/test_tensor_creation_ops.py::TestAsArrayCUDA::test_alias_from_tensor_cuda_int64, test/test_tensor_creation_ops.py::TestAsArrayCUDA::test_alias_from_tensor_cuda_int8, test/test_tensor_creation_ops.py::TestAsArrayCUDA::test_alias_from_tensor_cuda_uint8, test/test_tensor_creation_ops.py::TestAsArrayCUDA::test_astensor_consistency_cuda, test/test_tensor_creation_ops.py::TestAsArrayCUDA::test_copy_from_buffer_cuda_bool, test/test_tensor_creation_ops.py::TestAsArrayCUDA::test_copy_from_buffer_cuda_complex128, test/test_tensor_creation_ops.py::TestAsArrayCUDA::test_copy_from_buffer_cuda_complex64, test/test_tensor_creation_ops.py::TestAsArrayCUDA::test_copy_from_buffer_cuda_float16, test/test_tensor_creation_ops.py::TestAsArrayCUDA::test_copy_from_buffer_cuda_float32, test/test_tensor_creation_ops.py::TestAsArrayCUDA::test_copy_from_buffer_cuda_float64, test/test_tensor_creation_ops.py::TestAsArrayCUDA::test_copy_from_buffer_cuda_int16, test/test_tensor_creation_ops.py::TestAsArrayCUDA::test_copy_from_buffer_cuda_int32, test/test_tensor_creation_ops.py::TestAsArrayCUDA::test_copy_from_buffer_cuda_int64, test/test_tensor_creation_ops.py::TestAsArrayCUDA::test_copy_from_buffer_cuda_int8, test/test_tensor_creation_ops.py::TestAsArrayCUDA::test_copy_from_buffer_cuda_uint16, test/test_tensor_creation_ops.py::TestAsArrayCUDA::test_copy_from_buffer_cuda_uint32, test/test_tensor_creation_ops.py::TestAsArrayCUDA::test_copy_from_buffer_cuda_uint64, test/test_tensor_creation_ops.py::TestAsArrayCUDA::test_copy_from_buffer_cuda_uint8, test/test_tensor_creation_ops.py::TestAsArrayCUDA::test_copy_from_dlpack_cuda_bfloat16, test/test_tensor_creation_ops.py::TestAsArrayCUDA::test_copy_from_dlpack_cuda_complex128, test/test_tensor_creation_ops.py::TestAsArrayCUDA::test_copy_from_dlpack_cuda_complex64, test/test_tensor_creation_ops.py::TestAsArrayCUDA::test_copy_from_dlpack_cuda_float16, test/test_tensor_creation_ops.py::TestAsArrayCUDA::test_copy_from_dlpack_cuda_float32, test/test_tensor_creation_ops.py::TestAsArrayCUDA::test_copy_from_dlpack_cuda_float64, test/test_tensor_creation_ops.py::TestAsArrayCUDA::test_copy_from_dlpack_cuda_int16, test/test_tensor_creation_ops.py::TestAsArrayCUDA::test_copy_from_dlpack_cuda_int32, test/test_tensor_creation_ops.py::TestAsArrayCUDA::test_copy_from_dlpack_cuda_int64, test/test_tensor_creation_ops.py::TestAsArrayCUDA::test_copy_from_dlpack_cuda_int8, test/test_tensor_creation_ops.py::TestAsArrayCUDA::test_copy_from_dlpack_cuda_uint8, test/test_tensor_creation_ops.py::TestAsArrayCUDA::test_copy_from_dlpack_mult_devices_cuda_bfloat16, test/test_tensor_creation_ops.py::TestAsArrayCUDA::test_copy_from_dlpack_mult_devices_cuda_complex128, test/test_tensor_creation_ops.py::TestAsArrayCUDA::test_copy_from_dlpack_mult_devices_cuda_complex64, test/test_tensor_creation_ops.py::TestAsArrayCUDA::test_copy_from_dlpack_mult_devices_cuda_float16, test/test_tensor_creation_ops.py::TestAsArrayCUDA::test_copy_from_dlpack_mult_devices_cuda_float32, test/test_tensor_creation_ops.py::TestAsArrayCUDA::test_copy_from_dlpack_mult_devices_cuda_float64, test/test_tensor_creation_ops.py::TestAsArrayCUDA::test_copy_from_dlpack_mult_devices_cuda_int16, test/test_tensor_creation_ops.py::TestAsArrayCUDA::test_copy_from_dlpack_mult_devices_cuda_int32, test/test_tensor_creation_ops.py::TestAsArrayCUDA::test_copy_from_dlpack_mult_devices_cuda_int64, test/test_tensor_creation_ops.py::TestAsArrayCUDA::test_copy_from_dlpack_mult_devices_cuda_int8, test/test_tensor_creation_ops.py::TestAsArrayCUDA::test_copy_from_dlpack_mult_devices_cuda_uint8, test/test_tensor_creation_ops.py::TestAsArrayCUDA::test_copy_from_numpy_cuda_bool, test/test_tensor_creation_ops.py::TestAsArrayCUDA::test_copy_from_numpy_cuda_complex128, test/test_tensor_creation_ops.py::TestAsArrayCUDA::test_copy_from_numpy_cuda_complex64, test/test_tensor_creation_ops.py::TestAsArrayCUDA::test_copy_from_numpy_cuda_float16, test/test_tensor_creation_ops.py::TestAsArrayCUDA::test_copy_from_numpy_cuda_float32, test/test_tensor_creation_ops.py::TestAsArrayCUDA::test_copy_from_numpy_cuda_float64, test/test_tensor_creation_ops.py::TestAsArrayCUDA::test_copy_from_numpy_cuda_int16, test/test_tensor_creation_ops.py::TestAsArrayCUDA::test_copy_from_numpy_cuda_int32, test/test_tensor_creation_ops.py::TestAsArrayCUDA::test_copy_from_numpy_cuda_int64, test/test_tensor_creation_ops.py::TestAsArrayCUDA::test_copy_from_numpy_cuda_int8, test/test_tensor_creation_ops.py::TestAsArrayCUDA::test_copy_from_numpy_cuda_uint16, test/test_tensor_creation_ops.py::TestAsArrayCUDA::test_copy_from_numpy_cuda_uint32, test/test_tensor_creation_ops.py::TestAsArrayCUDA::test_copy_from_numpy_cuda_uint64, test/test_tensor_creation_ops.py::TestAsArrayCUDA::test_copy_from_numpy_cuda_uint8, test/test_tensor_creation_ops.py::TestAsArrayCUDA::test_copy_from_tensor_mult_devices_cuda_bfloat16, test/test_tensor_creation_ops.py::TestAsArrayCUDA::test_copy_from_tensor_mult_devices_cuda_complex128, test/test_tensor_creation_ops.py::TestAsArrayCUDA::test_copy_from_tensor_mult_devices_cuda_complex64, test/test_tensor_creation_ops.py::TestAsArrayCUDA::test_copy_from_tensor_mult_devices_cuda_float16, test/test_tensor_creation_ops.py::TestAsArrayCUDA::test_copy_from_tensor_mult_devices_cuda_float32, test/test_tensor_creation_ops.py::TestAsArrayCUDA::test_copy_from_tensor_mult_devices_cuda_float64, test/test_tensor_creation_ops.py::TestAsArrayCUDA::test_copy_from_tensor_mult_devices_cuda_int16, test/test_tensor_creation_ops.py::TestAsArrayCUDA::test_copy_from_tensor_mult_devices_cuda_int32, test/test_tensor_creation_ops.py::TestAsArrayCUDA::test_copy_from_tensor_mult_devices_cuda_int64, test/test_tensor_creation_ops.py::TestAsArrayCUDA::test_copy_from_tensor_mult_devices_cuda_int8, test/test_tensor_creation_ops.py::TestAsArrayCUDA::test_copy_from_tensor_mult_devices_cuda_uint8, test/test_tensor_creation_ops.py::TestAsArrayCUDA::test_copy_list_cuda_bfloat16, test/test_tensor_creation_ops.py::TestAsArrayCUDA::test_copy_list_cuda_bool, test/test_tensor_creation_ops.py::TestAsArrayCUDA::test_copy_list_cuda_complex128, test/test_tensor_creation_ops.py::TestAsArrayCUDA::test_copy_list_cuda_complex64, test/test_tensor_creation_ops.py::TestAsArrayCUDA::test_copy_list_cuda_float16, test/test_tensor_creation_ops.py::TestAsArrayCUDA::test_copy_list_cuda_float32, test/test_tensor_creation_ops.py::TestAsArrayCUDA::test_copy_list_cuda_float64, test/test_tensor_creation_ops.py::TestAsArrayCUDA::test_copy_list_cuda_int16, test/test_tensor_creation_ops.py::TestAsArrayCUDA::test_copy_list_cuda_int32, test/test_tensor_creation_ops.py::TestAsArrayCUDA::test_copy_list_cuda_int64, test/test_tensor_creation_ops.py::TestAsArrayCUDA::test_copy_list_cuda_int8, test/test_tensor_creation_ops.py::TestAsArrayCUDA::test_copy_list_cuda_uint8, test/test_tensor_creation_ops.py::TestAsArrayCUDA::test_copy_tensor_cuda_bfloat16, test/test_tensor_creation_ops.py::TestAsArrayCUDA::test_copy_tensor_cuda_bool, test/test_tensor_creation_ops.py::TestAsArrayCUDA::test_copy_tensor_cuda_complex128, test/test_tensor_creation_ops.py::TestAsArrayCUDA::test_copy_tensor_cuda_complex64, test/test_tensor_creation_ops.py::TestAsArrayCUDA::test_copy_tensor_cuda_float16, test/test_tensor_creation_ops.py::TestAsArrayCUDA::test_copy_tensor_cuda_float32, test/test_tensor_creation_ops.py::TestAsArrayCUDA::test_copy_tensor_cuda_float64, test/test_tensor_creation_ops.py::TestAsArrayCUDA::test_copy_tensor_cuda_int16, test/test_tensor_creation_ops.py::TestAsArrayCUDA::test_copy_tensor_cuda_int32, test/test_tensor_creation_ops.py::TestAsArrayCUDA::test_copy_tensor_cuda_int64, test/test_tensor_creation_ops.py::TestAsArrayCUDA::test_copy_tensor_cuda_int8, test/test_tensor_creation_ops.py::TestAsArrayCUDA::test_copy_tensor_cuda_uint8, test/test_tensor_creation_ops.py::TestAsArrayCUDA::test_default_device_cuda, test/test_tensor_creation_ops.py::TestAsArrayCUDA::test_device_without_index_cuda, test/test_tensor_creation_ops.py::TestAsArrayCUDA::test_numpy_scalars_cuda, test/test_tensor_creation_ops.py::TestAsArrayCUDA::test_retain_autograd_history_cuda_complex64, test/test_tensor_creation_ops.py::TestAsArrayCUDA::test_retain_autograd_history_cuda_float32, test/test_tensor_creation_ops.py::TestAsArrayCUDA::test_unsupported_alias_cuda_float32, test/test_tensor_creation_ops.py::TestAsArrayCUDA::test_unsupported_alias_mult_devices_cuda_float32 2025-12-04T13:52:18.1595397Z 2025-12-04T13:52:18.1595629Z Finished test_tensor_creation_ops 1/1 ... [2025-12-04 13:52:18.132862][15186.075079166], took 1.48min 2025-12-04T13:52:18.1629466Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/test_tensor_creation_ops/test_tensor_creation_ops-3fdf3b4be21ec825.xml 2025-12-04T13:52:18.2656454Z Running torch_np/numpy_tests/linalg/test_linalg 1/1 ... [2025-12-04 13:52:18.265379][15186.207597322] 2025-12-04T13:52:18.2657429Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2025-12-04T13:52:18.2660148Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'torch_np/numpy_tests/linalg/test_linalg.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-12-04 13:52:18.265701] 2025-12-04T13:52:28.8969793Z 2025-12-04T13:52:28.8971496Z torch_np/numpy_tests/linalg/test_linalg 1/1 was successful, full logs can be found in artifacts with path test/test-reports/torch_np.numpy_tests.linalg.test_linalg_1.1_e7d4ead41bf88610_.log 2025-12-04T13:52:28.9036534Z Running 268 items in this shard: test/torch_np/numpy_tests/linalg/test_linalg.py::TestSolve::test_0_size, test/torch_np/numpy_tests/linalg/test_linalg.py::TestSolve::test_0_size_k, test/torch_np/numpy_tests/linalg/test_linalg.py::TestSolve::test_empty_sq_cases, test/torch_np/numpy_tests/linalg/test_linalg.py::TestSolve::test_generalized_empty_sq_cases, test/torch_np/numpy_tests/linalg/test_linalg.py::TestSolve::test_generalized_sq_cases, test/torch_np/numpy_tests/linalg/test_linalg.py::TestSolve::test_sq_cases, test/torch_np/numpy_tests/linalg/test_linalg.py::TestSolve::test_types_dtype0, test/torch_np/numpy_tests/linalg/test_linalg.py::TestSolve::test_types_dtype1, test/torch_np/numpy_tests/linalg/test_linalg.py::TestSolve::test_types_dtype2, test/torch_np/numpy_tests/linalg/test_linalg.py::TestSolve::test_types_dtype3, test/torch_np/numpy_tests/linalg/test_linalg.py::TestInv::test_0_size, test/torch_np/numpy_tests/linalg/test_linalg.py::TestInv::test_empty_sq_cases, test/torch_np/numpy_tests/linalg/test_linalg.py::TestInv::test_generalized_empty_sq_cases, test/torch_np/numpy_tests/linalg/test_linalg.py::TestInv::test_generalized_sq_cases, test/torch_np/numpy_tests/linalg/test_linalg.py::TestInv::test_sq_cases, test/torch_np/numpy_tests/linalg/test_linalg.py::TestInv::test_types_dtype0, test/torch_np/numpy_tests/linalg/test_linalg.py::TestInv::test_types_dtype1, test/torch_np/numpy_tests/linalg/test_linalg.py::TestInv::test_types_dtype2, test/torch_np/numpy_tests/linalg/test_linalg.py::TestInv::test_types_dtype3, test/torch_np/numpy_tests/linalg/test_linalg.py::TestEigvals::test_0_size, test/torch_np/numpy_tests/linalg/test_linalg.py::TestEigvals::test_empty_sq_cases, test/torch_np/numpy_tests/linalg/test_linalg.py::TestEigvals::test_generalized_empty_sq_cases, test/torch_np/numpy_tests/linalg/test_linalg.py::TestEigvals::test_generalized_sq_cases, test/torch_np/numpy_tests/linalg/test_linalg.py::TestEigvals::test_sq_cases, test/torch_np/numpy_tests/linalg/test_linalg.py::TestEigvals::test_types_dtype0, test/torch_np/numpy_tests/linalg/test_linalg.py::TestEigvals::test_types_dtype1, test/torch_np/numpy_tests/linalg/test_linalg.py::TestEigvals::test_types_dtype2, test/torch_np/numpy_tests/linalg/test_linalg.py::TestEigvals::test_types_dtype3, test/torch_np/numpy_tests/linalg/test_linalg.py::TestEig::test_0_size, test/torch_np/numpy_tests/linalg/test_linalg.py::TestEig::test_empty_sq_cases, test/torch_np/numpy_tests/linalg/test_linalg.py::TestEig::test_generalized_empty_sq_cases, test/torch_np/numpy_tests/linalg/test_linalg.py::TestEig::test_generalized_sq_cases, test/torch_np/numpy_tests/linalg/test_linalg.py::TestEig::test_sq_cases, test/torch_np/numpy_tests/linalg/test_linalg.py::TestEig::test_types_dtype0, test/torch_np/numpy_tests/linalg/test_linalg.py::TestEig::test_types_dtype1, test/torch_np/numpy_tests/linalg/test_linalg.py::TestEig::test_types_dtype2, test/torch_np/numpy_tests/linalg/test_linalg.py::TestEig::test_types_dtype3, test/torch_np/numpy_tests/linalg/test_linalg.py::TestSVD::test_empty_identity, test/torch_np/numpy_tests/linalg/test_linalg.py::TestSVD::test_empty_sq_cases, test/torch_np/numpy_tests/linalg/test_linalg.py::TestSVD::test_generalized_empty_sq_cases, test/torch_np/numpy_tests/linalg/test_linalg.py::TestSVD::test_generalized_sq_cases, test/torch_np/numpy_tests/linalg/test_linalg.py::TestSVD::test_sq_cases, test/torch_np/numpy_tests/linalg/test_linalg.py::TestSVD::test_types_dtype0, test/torch_np/numpy_tests/linalg/test_linalg.py::TestSVD::test_types_dtype1, test/torch_np/numpy_tests/linalg/test_linalg.py::TestSVD::test_types_dtype2, test/torch_np/numpy_tests/linalg/test_linalg.py::TestSVD::test_types_dtype3, test/torch_np/numpy_tests/linalg/test_linalg.py::TestSVDHermitian::test_empty_herm_cases, test/torch_np/numpy_tests/linalg/test_linalg.py::TestSVDHermitian::test_generalized_empty_herm_cases, test/torch_np/numpy_tests/linalg/test_linalg.py::TestSVDHermitian::test_generalized_herm_cases, test/torch_np/numpy_tests/linalg/test_linalg.py::TestSVDHermitian::test_herm_cases, test/torch_np/numpy_tests/linalg/test_linalg.py::TestSVDHermitian::test_types_dtype0, test/torch_np/numpy_tests/linalg/test_linalg.py::TestSVDHermitian::test_types_dtype1, test/torch_np/numpy_tests/linalg/test_linalg.py::TestSVDHermitian::test_types_dtype2, test/torch_np/numpy_tests/linalg/test_linalg.py::TestSVDHermitian::test_types_dtype3, test/torch_np/numpy_tests/linalg/test_linalg.py::TestCond::test_basic_nonsvd, test/torch_np/numpy_tests/linalg/test_linalg.py::TestCond::test_empty_sq_cases, test/torch_np/numpy_tests/linalg/test_linalg.py::TestCond::test_generalized_empty_sq_cases, test/torch_np/numpy_tests/linalg/test_linalg.py::TestCond::test_generalized_sq_cases, test/torch_np/numpy_tests/linalg/test_linalg.py::TestCond::test_nan, test/torch_np/numpy_tests/linalg/test_linalg.py::TestCond::test_singular, test/torch_np/numpy_tests/linalg/test_linalg.py::TestCond::test_sq_cases, test/torch_np/numpy_tests/linalg/test_linalg.py::TestCond::test_stacked_singular, test/torch_np/numpy_tests/linalg/test_linalg.py::TestPinv::test_empty_nonsq_cases, test/torch_np/numpy_tests/linalg/test_linalg.py::TestPinv::test_empty_sq_cases, test/torch_np/numpy_tests/linalg/test_linalg.py::TestPinv::test_generalized_empty_nonsq_cases, test/torch_np/numpy_tests/linalg/test_linalg.py::TestPinv::test_generalized_empty_sq_cases, test/torch_np/numpy_tests/linalg/test_linalg.py::TestPinv::test_generalized_nonsq_cases, test/torch_np/numpy_tests/linalg/test_linalg.py::TestPinv::test_generalized_sq_cases, test/torch_np/numpy_tests/linalg/test_linalg.py::TestPinv::test_nonsq_cases, test/torch_np/numpy_tests/linalg/test_linalg.py::TestPinv::test_sq_cases, test/torch_np/numpy_tests/linalg/test_linalg.py::TestPinvHermitian::test_empty_herm_cases, test/torch_np/numpy_tests/linalg/test_linalg.py::TestPinvHermitian::test_generalized_empty_herm_cases, test/torch_np/numpy_tests/linalg/test_linalg.py::TestPinvHermitian::test_generalized_herm_cases, test/torch_np/numpy_tests/linalg/test_linalg.py::TestPinvHermitian::test_herm_cases, test/torch_np/numpy_tests/linalg/test_linalg.py::TestDet::test_0_size, test/torch_np/numpy_tests/linalg/test_linalg.py::TestDet::test_empty_sq_cases, test/torch_np/numpy_tests/linalg/test_linalg.py::TestDet::test_generalized_empty_sq_cases, test/torch_np/numpy_tests/linalg/test_linalg.py::TestDet::test_generalized_sq_cases, test/torch_np/numpy_tests/linalg/test_linalg.py::TestDet::test_sq_cases, test/torch_np/numpy_tests/linalg/test_linalg.py::TestDet::test_types_dtype0, test/torch_np/numpy_tests/linalg/test_linalg.py::TestDet::test_types_dtype1, test/torch_np/numpy_tests/linalg/test_linalg.py::TestDet::test_types_dtype2, test/torch_np/numpy_tests/linalg/test_linalg.py::TestDet::test_types_dtype3, test/torch_np/numpy_tests/linalg/test_linalg.py::TestDet::test_zero, test/torch_np/numpy_tests/linalg/test_linalg.py::TestLstsq::test_empty_a_b_m_0_n_0_n_rhs_0, test/torch_np/numpy_tests/linalg/test_linalg.py::TestLstsq::test_empty_a_b_m_0_n_4_n_rhs_1, test/torch_np/numpy_tests/linalg/test_linalg.py::TestLstsq::test_empty_a_b_m_0_n_4_n_rhs_2, test/torch_np/numpy_tests/linalg/test_linalg.py::TestLstsq::test_empty_a_b_m_4_n_0_n_rhs_1, test/torch_np/numpy_tests/linalg/test_linalg.py::TestLstsq::test_empty_a_b_m_4_n_0_n_rhs_2, test/torch_np/numpy_tests/linalg/test_linalg.py::TestLstsq::test_empty_a_b_m_4_n_2_n_rhs_2, test/torch_np/numpy_tests/linalg/test_linalg.py::TestLstsq::test_empty_nonsq_cases, test/torch_np/numpy_tests/linalg/test_linalg.py::TestLstsq::test_empty_sq_cases, test/torch_np/numpy_tests/linalg/test_linalg.py::TestLstsq::test_future_rcond, test/torch_np/numpy_tests/linalg/test_linalg.py::TestLstsq::test_incompatible_dims, test/torch_np/numpy_tests/linalg/test_linalg.py::TestLstsq::test_nonsq_cases, test/torch_np/numpy_tests/linalg/test_linalg.py::TestLstsq::test_sq_cases, test/torch_np/numpy_tests/linalg/test_linalg.py::TestEigvalshCases::test_generalized_herm_cases, test/torch_np/numpy_tests/linalg/test_linalg.py::TestEigvalshCases::test_generalized_empty_herm_cases, test/torch_np/numpy_tests/linalg/test_linalg.py::TestEigvalshCases::test_herm_cases, test/torch_np/numpy_tests/linalg/test_linalg.py::TestEigvalshCases::test_empty_herm_cases, test/torch_np/numpy_tests/linalg/test_linalg.py::TestEigvalsh::test_0_size, test/torch_np/numpy_tests/linalg/test_linalg.py::TestEigvalsh::test_UPLO, test/torch_np/numpy_tests/linalg/test_linalg.py::TestEigvalsh::test_invalid, test/torch_np/numpy_tests/linalg/test_linalg.py::TestEigvalsh::test_types_dtype0, test/torch_np/numpy_tests/linalg/test_linalg.py::TestEigvalsh::test_types_dtype1, test/torch_np/numpy_tests/linalg/test_linalg.py::TestEigvalsh::test_types_dtype2, test/torch_np/numpy_tests/linalg/test_linalg.py::TestEigvalsh::test_types_dtype3, test/torch_np/numpy_tests/linalg/test_linalg.py::TestEighCases::test_generalized_herm_cases, test/torch_np/numpy_tests/linalg/test_linalg.py::TestEighCases::test_generalized_empty_herm_cases, test/torch_np/numpy_tests/linalg/test_linalg.py::TestEighCases::test_herm_cases, test/torch_np/numpy_tests/linalg/test_linalg.py::TestEighCases::test_empty_herm_cases, test/torch_np/numpy_tests/linalg/test_linalg.py::TestEigh::test_0_size, test/torch_np/numpy_tests/linalg/test_linalg.py::TestEigh::test_UPLO, test/torch_np/numpy_tests/linalg/test_linalg.py::TestEigh::test_invalid, test/torch_np/numpy_tests/linalg/test_linalg.py::TestEigh::test_types_dtype0, test/torch_np/numpy_tests/linalg/test_linalg.py::TestEigh::test_types_dtype1, test/torch_np/numpy_tests/linalg/test_linalg.py::TestEigh::test_types_dtype2, test/torch_np/numpy_tests/linalg/test_linalg.py::TestEigh::test_types_dtype3, test/torch_np/numpy_tests/linalg/test_linalg.py::TestNorm_NonSystematic::test_intmin, test/torch_np/numpy_tests/linalg/test_linalg.py::TestNormDouble::test_axis, test/torch_np/numpy_tests/linalg/test_linalg.py::TestNormDouble::test_bad_args, test/torch_np/numpy_tests/linalg/test_linalg.py::TestNormDouble::test_empty, test/torch_np/numpy_tests/linalg/test_linalg.py::TestNormDouble::test_keepdims, test/torch_np/numpy_tests/linalg/test_linalg.py::TestNormDouble::test_matrix_2x2, test/torch_np/numpy_tests/linalg/test_linalg.py::TestNormDouble::test_matrix_3x3, test/torch_np/numpy_tests/linalg/test_linalg.py::TestNormDouble::test_matrix_empty, test/torch_np/numpy_tests/linalg/test_linalg.py::TestNormDouble::test_matrix_return_type, test/torch_np/numpy_tests/linalg/test_linalg.py::TestNormDouble::test_vector, test/torch_np/numpy_tests/linalg/test_linalg.py::TestNormDouble::test_vector_return_type, test/torch_np/numpy_tests/linalg/test_linalg.py::TestNormSingle::test_axis, test/torch_np/numpy_tests/linalg/test_linalg.py::TestNormSingle::test_bad_args, test/torch_np/numpy_tests/linalg/test_linalg.py::TestNormSingle::test_empty, test/torch_np/numpy_tests/linalg/test_linalg.py::TestNormSingle::test_keepdims, test/torch_np/numpy_tests/linalg/test_linalg.py::TestNormSingle::test_matrix_2x2, test/torch_np/numpy_tests/linalg/test_linalg.py::TestNormSingle::test_matrix_3x3, test/torch_np/numpy_tests/linalg/test_linalg.py::TestNormSingle::test_matrix_empty, test/torch_np/numpy_tests/linalg/test_linalg.py::TestNormSingle::test_matrix_return_type, test/torch_np/numpy_tests/linalg/test_linalg.py::TestNormSingle::test_vector, test/torch_np/numpy_tests/linalg/test_linalg.py::TestNormSingle::test_vector_return_type, test/torch_np/numpy_tests/linalg/test_linalg.py::TestNormInt64::test_axis, test/torch_np/numpy_tests/linalg/test_linalg.py::TestNormInt64::test_bad_args, test/torch_np/numpy_tests/linalg/test_linalg.py::TestNormInt64::test_empty, test/torch_np/numpy_tests/linalg/test_linalg.py::TestNormInt64::test_keepdims, test/torch_np/numpy_tests/linalg/test_linalg.py::TestNormInt64::test_matrix_2x2, test/torch_np/numpy_tests/linalg/test_linalg.py::TestNormInt64::test_matrix_3x3, test/torch_np/numpy_tests/linalg/test_linalg.py::TestNormInt64::test_matrix_empty, test/torch_np/numpy_tests/linalg/test_linalg.py::TestNormInt64::test_matrix_return_type, test/torch_np/numpy_tests/linalg/test_linalg.py::TestNormInt64::test_vector, test/torch_np/numpy_tests/linalg/test_linalg.py::TestNormInt64::test_vector_return_type, test/torch_np/numpy_tests/linalg/test_linalg.py::TestMatrixRank::test_matrix_rank, test/torch_np/numpy_tests/linalg/test_linalg.py::TestMatrixRank::test_reduced_rank, test/torch_np/numpy_tests/linalg/test_linalg.py::TestMatrixRank::test_symmetric_rank, test/torch_np/numpy_tests/linalg/test_linalg.py::TestQR::test_mode_all_but_economic, test/torch_np/numpy_tests/linalg/test_linalg.py::TestQR::test_mode_raw, test/torch_np/numpy_tests/linalg/test_linalg.py::TestQR::test_qr_empty_m_0_n_0, test/torch_np/numpy_tests/linalg/test_linalg.py::TestQR::test_qr_empty_m_0_n_3, test/torch_np/numpy_tests/linalg/test_linalg.py::TestQR::test_qr_empty_m_3_n_0, test/torch_np/numpy_tests/linalg/test_linalg.py::TestQR::test_stacked_inputs_size0_outer_size0_dt0, test/torch_np/numpy_tests/linalg/test_linalg.py::TestQR::test_stacked_inputs_size0_outer_size0_dt1, test/torch_np/numpy_tests/linalg/test_linalg.py::TestQR::test_stacked_inputs_size0_outer_size0_dt2, test/torch_np/numpy_tests/linalg/test_linalg.py::TestQR::test_stacked_inputs_size0_outer_size0_dt3, test/torch_np/numpy_tests/linalg/test_linalg.py::TestQR::test_stacked_inputs_size0_outer_size1_dt0, test/torch_np/numpy_tests/linalg/test_linalg.py::TestQR::test_stacked_inputs_size0_outer_size1_dt1, test/torch_np/numpy_tests/linalg/test_linalg.py::TestQR::test_stacked_inputs_size0_outer_size1_dt2, test/torch_np/numpy_tests/linalg/test_linalg.py::TestQR::test_stacked_inputs_size0_outer_size1_dt3, test/torch_np/numpy_tests/linalg/test_linalg.py::TestQR::test_stacked_inputs_size0_outer_size2_dt0, test/torch_np/numpy_tests/linalg/test_linalg.py::TestQR::test_stacked_inputs_size0_outer_size2_dt1, test/torch_np/numpy_tests/linalg/test_linalg.py::TestQR::test_stacked_inputs_size0_outer_size2_dt2, test/torch_np/numpy_tests/linalg/test_linalg.py::TestQR::test_stacked_inputs_size0_outer_size2_dt3, test/torch_np/numpy_tests/linalg/test_linalg.py::TestQR::test_stacked_inputs_size1_outer_size0_dt0, test/torch_np/numpy_tests/linalg/test_linalg.py::TestQR::test_stacked_inputs_size1_outer_size0_dt1, test/torch_np/numpy_tests/linalg/test_linalg.py::TestQR::test_stacked_inputs_size1_outer_size0_dt2, test/torch_np/numpy_tests/linalg/test_linalg.py::TestQR::test_stacked_inputs_size1_outer_size0_dt3, test/torch_np/numpy_tests/linalg/test_linalg.py::TestQR::test_stacked_inputs_size1_outer_size1_dt0, test/torch_np/numpy_tests/linalg/test_linalg.py::TestQR::test_stacked_inputs_size1_outer_size1_dt1, test/torch_np/numpy_tests/linalg/test_linalg.py::TestQR::test_stacked_inputs_size1_outer_size1_dt2, test/torch_np/numpy_tests/linalg/test_linalg.py::TestQR::test_stacked_inputs_size1_outer_size1_dt3, test/torch_np/numpy_tests/linalg/test_linalg.py::TestQR::test_stacked_inputs_size1_outer_size2_dt0, test/torch_np/numpy_tests/linalg/test_linalg.py::TestQR::test_stacked_inputs_size1_outer_size2_dt1, test/torch_np/numpy_tests/linalg/test_linalg.py::TestQR::test_stacked_inputs_size1_outer_size2_dt2, test/torch_np/numpy_tests/linalg/test_linalg.py::TestQR::test_stacked_inputs_size1_outer_size2_dt3, test/torch_np/numpy_tests/linalg/test_linalg.py::TestQR::test_stacked_inputs_size2_outer_size0_dt0, test/torch_np/numpy_tests/linalg/test_linalg.py::TestQR::test_stacked_inputs_size2_outer_size0_dt1, test/torch_np/numpy_tests/linalg/test_linalg.py::TestQR::test_stacked_inputs_size2_outer_size0_dt2, test/torch_np/numpy_tests/linalg/test_linalg.py::TestQR::test_stacked_inputs_size2_outer_size0_dt3, test/torch_np/numpy_tests/linalg/test_linalg.py::TestQR::test_stacked_inputs_size2_outer_size1_dt0, test/torch_np/numpy_tests/linalg/test_linalg.py::TestQR::test_stacked_inputs_size2_outer_size1_dt1, test/torch_np/numpy_tests/linalg/test_linalg.py::TestQR::test_stacked_inputs_size2_outer_size1_dt2, test/torch_np/numpy_tests/linalg/test_linalg.py::TestQR::test_stacked_inputs_size2_outer_size1_dt3, test/torch_np/numpy_tests/linalg/test_linalg.py::TestQR::test_stacked_inputs_size2_outer_size2_dt0, test/torch_np/numpy_tests/linalg/test_linalg.py::TestQR::test_stacked_inputs_size2_outer_size2_dt1, test/torch_np/numpy_tests/linalg/test_linalg.py::TestQR::test_stacked_inputs_size2_outer_size2_dt2, test/torch_np/numpy_tests/linalg/test_linalg.py::TestQR::test_stacked_inputs_size2_outer_size2_dt3, test/torch_np/numpy_tests/linalg/test_linalg.py::TestQR::test_stacked_inputs_size3_outer_size0_dt0, test/torch_np/numpy_tests/linalg/test_linalg.py::TestQR::test_stacked_inputs_size3_outer_size0_dt1, test/torch_np/numpy_tests/linalg/test_linalg.py::TestQR::test_stacked_inputs_size3_outer_size0_dt2, test/torch_np/numpy_tests/linalg/test_linalg.py::TestQR::test_stacked_inputs_size3_outer_size0_dt3, test/torch_np/numpy_tests/linalg/test_linalg.py::TestQR::test_stacked_inputs_size3_outer_size1_dt0, test/torch_np/numpy_tests/linalg/test_linalg.py::TestQR::test_stacked_inputs_size3_outer_size1_dt1, test/torch_np/numpy_tests/linalg/test_linalg.py::TestQR::test_stacked_inputs_size3_outer_size1_dt2, test/torch_np/numpy_tests/linalg/test_linalg.py::TestQR::test_stacked_inputs_size3_outer_size1_dt3, test/torch_np/numpy_tests/linalg/test_linalg.py::TestQR::test_stacked_inputs_size3_outer_size2_dt0, test/torch_np/numpy_tests/linalg/test_linalg.py::TestQR::test_stacked_inputs_size3_outer_size2_dt1, test/torch_np/numpy_tests/linalg/test_linalg.py::TestQR::test_stacked_inputs_size3_outer_size2_dt2, test/torch_np/numpy_tests/linalg/test_linalg.py::TestQR::test_stacked_inputs_size3_outer_size2_dt3, test/torch_np/numpy_tests/linalg/test_linalg.py::TestQR::test_stacked_inputs_size4_outer_size0_dt0, test/torch_np/numpy_tests/linalg/test_linalg.py::TestQR::test_stacked_inputs_size4_outer_size0_dt1, test/torch_np/numpy_tests/linalg/test_linalg.py::TestQR::test_stacked_inputs_size4_outer_size0_dt2, test/torch_np/numpy_tests/linalg/test_linalg.py::TestQR::test_stacked_inputs_size4_outer_size0_dt3, test/torch_np/numpy_tests/linalg/test_linalg.py::TestQR::test_stacked_inputs_size4_outer_size1_dt0, test/torch_np/numpy_tests/linalg/test_linalg.py::TestQR::test_stacked_inputs_size4_outer_size1_dt1, test/torch_np/numpy_tests/linalg/test_linalg.py::TestQR::test_stacked_inputs_size4_outer_size1_dt2, test/torch_np/numpy_tests/linalg/test_linalg.py::TestQR::test_stacked_inputs_size4_outer_size1_dt3, test/torch_np/numpy_tests/linalg/test_linalg.py::TestQR::test_stacked_inputs_size4_outer_size2_dt0, test/torch_np/numpy_tests/linalg/test_linalg.py::TestQR::test_stacked_inputs_size4_outer_size2_dt1, test/torch_np/numpy_tests/linalg/test_linalg.py::TestQR::test_stacked_inputs_size4_outer_size2_dt2, test/torch_np/numpy_tests/linalg/test_linalg.py::TestQR::test_stacked_inputs_size4_outer_size2_dt3, test/torch_np/numpy_tests/linalg/test_linalg.py::TestCholesky::test_0_size, test/torch_np/numpy_tests/linalg/test_linalg.py::TestCholesky::test_basic_property_shape0_dtype0, test/torch_np/numpy_tests/linalg/test_linalg.py::TestCholesky::test_basic_property_shape0_dtype1, test/torch_np/numpy_tests/linalg/test_linalg.py::TestCholesky::test_basic_property_shape0_dtype2, test/torch_np/numpy_tests/linalg/test_linalg.py::TestCholesky::test_basic_property_shape0_dtype3, test/torch_np/numpy_tests/linalg/test_linalg.py::TestCholesky::test_basic_property_shape1_dtype0, test/torch_np/numpy_tests/linalg/test_linalg.py::TestCholesky::test_basic_property_shape1_dtype1, test/torch_np/numpy_tests/linalg/test_linalg.py::TestCholesky::test_basic_property_shape1_dtype2, test/torch_np/numpy_tests/linalg/test_linalg.py::TestCholesky::test_basic_property_shape1_dtype3, test/torch_np/numpy_tests/linalg/test_linalg.py::TestCholesky::test_basic_property_shape2_dtype0, test/torch_np/numpy_tests/linalg/test_linalg.py::TestCholesky::test_basic_property_shape2_dtype1, test/torch_np/numpy_tests/linalg/test_linalg.py::TestCholesky::test_basic_property_shape2_dtype2, test/torch_np/numpy_tests/linalg/test_linalg.py::TestCholesky::test_basic_property_shape2_dtype3, test/torch_np/numpy_tests/linalg/test_linalg.py::TestCholesky::test_basic_property_shape3_dtype0, test/torch_np/numpy_tests/linalg/test_linalg.py::TestCholesky::test_basic_property_shape3_dtype1, test/torch_np/numpy_tests/linalg/test_linalg.py::TestCholesky::test_basic_property_shape3_dtype2, test/torch_np/numpy_tests/linalg/test_linalg.py::TestCholesky::test_basic_property_shape3_dtype3, test/torch_np/numpy_tests/linalg/test_linalg.py::TestCholesky::test_basic_property_shape4_dtype0, test/torch_np/numpy_tests/linalg/test_linalg.py::TestCholesky::test_basic_property_shape4_dtype1, test/torch_np/numpy_tests/linalg/test_linalg.py::TestCholesky::test_basic_property_shape4_dtype2, test/torch_np/numpy_tests/linalg/test_linalg.py::TestCholesky::test_basic_property_shape4_dtype3, test/torch_np/numpy_tests/linalg/test_linalg.py::TestMisc::test_byteorder_check, test/torch_np/numpy_tests/linalg/test_linalg.py::TestMisc::test_generalized_raise_multiloop, test/torch_np/numpy_tests/linalg/test_linalg.py::TestMisc::test_sdot_bug_8577, test/torch_np/numpy_tests/linalg/test_linalg.py::TestMisc::test_xerbla_override, test/torch_np/numpy_tests/linalg/test_linalg.py::TestMultiDot::test_basic_function_with_dynamic_programming_optimization, test/torch_np/numpy_tests/linalg/test_linalg.py::TestMultiDot::test_basic_function_with_three_arguments, test/torch_np/numpy_tests/linalg/test_linalg.py::TestMultiDot::test_basic_function_with_two_arguments, test/torch_np/numpy_tests/linalg/test_linalg.py::TestMultiDot::test_dynamic_programming_logic, test/torch_np/numpy_tests/linalg/test_linalg.py::TestMultiDot::test_dynamic_programming_optimization_and_out, test/torch_np/numpy_tests/linalg/test_linalg.py::TestMultiDot::test_three_arguments_and_out, test/torch_np/numpy_tests/linalg/test_linalg.py::TestMultiDot::test_too_few_input_arrays, test/torch_np/numpy_tests/linalg/test_linalg.py::TestMultiDot::test_two_arguments_and_out, test/torch_np/numpy_tests/linalg/test_linalg.py::TestMultiDot::test_vector_as_first_and_last_argument, test/torch_np/numpy_tests/linalg/test_linalg.py::TestMultiDot::test_vector_as_first_argument, test/torch_np/numpy_tests/linalg/test_linalg.py::TestMultiDot::test_vector_as_last_argument, test/torch_np/numpy_tests/linalg/test_linalg.py::TestTensorinv::test_non_square_handling_arr0_ind_2, test/torch_np/numpy_tests/linalg/test_linalg.py::TestTensorinv::test_non_square_handling_arr1_ind_1, test/torch_np/numpy_tests/linalg/test_linalg.py::TestTensorinv::test_tensorinv_ind_limit_ind_-2, test/torch_np/numpy_tests/linalg/test_linalg.py::TestTensorinv::test_tensorinv_ind_limit_ind_0, test/torch_np/numpy_tests/linalg/test_linalg.py::TestTensorinv::test_tensorinv_result, test/torch_np/numpy_tests/linalg/test_linalg.py::TestTensorinv::test_tensorinv_shape_shape0_ind_2, test/torch_np/numpy_tests/linalg/test_linalg.py::TestTensorinv::test_tensorinv_shape_shape1_ind_1, test/torch_np/numpy_tests/linalg/test_linalg.py::TestTensorsolve::test_non_square_handling_a0_axes0, test/torch_np/numpy_tests/linalg/test_linalg.py::TestTensorsolve::test_non_square_handling_a1_axes1, test/torch_np/numpy_tests/linalg/test_linalg.py::TestTensorsolve::test_tensorsolve_result_shape0, test/torch_np/numpy_tests/linalg/test_linalg.py::TestTensorsolve::test_tensorsolve_result_shape1, test/torch_np/numpy_tests/linalg/test_linalg.py::TestTensorsolve::test_tensorsolve_result_shape2, test/torch_np/numpy_tests/linalg/test_linalg.py::TestMisc2::test_blas64_dot, test/torch_np/numpy_tests/linalg/test_linalg.py::TestMisc2::test_blas64_geqrf_lwork_smoketest, test/torch_np/numpy_tests/linalg/test_linalg.py::TestMisc2::test_unsupported_commontype 2025-12-04T13:52:28.9100837Z 2025-12-04T13:52:28.9101195Z Finished torch_np/numpy_tests/linalg/test_linalg 1/1 ... [2025-12-04 13:52:28.897180][15196.839396517], took 0.18min 2025-12-04T13:52:28.9268263Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/torch_np.numpy_tests.linalg.test_linalg/torch_np.numpy_tests.linalg.test_linalg-5c045926bb031e0f.xml 2025-12-04T13:52:29.0051090Z Running torch_np/numpy_tests/core/test_dtype 1/1 ... [2025-12-04 13:52:29.004846][15196.947064228] 2025-12-04T13:52:29.0051584Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2025-12-04T13:52:29.0054461Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'torch_np/numpy_tests/core/test_dtype.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-12-04 13:52:29.005157] 2025-12-04T13:52:32.6258886Z 2025-12-04T13:52:32.6260008Z torch_np/numpy_tests/core/test_dtype 1/1 was successful, full logs can be found in artifacts with path test/test-reports/torch_np.numpy_tests.core.test_dtype_1.1_d6bdce85d660528e_.log 2025-12-04T13:52:32.6290620Z Running 102 items in this shard: test/torch_np/numpy_tests/core/test_dtype.py::TestBuiltin::test_equivalent_dtype_hashing, test/torch_np/numpy_tests/core/test_dtype.py::TestBuiltin::test_invalid_types, test/torch_np/numpy_tests/core/test_dtype.py::TestBuiltin::test_numeric_style_types_are_invalid_dtype_Bool, test/torch_np/numpy_tests/core/test_dtype.py::TestBuiltin::test_numeric_style_types_are_invalid_dtype_Bytes0, test/torch_np/numpy_tests/core/test_dtype.py::TestBuiltin::test_numeric_style_types_are_invalid_dtype_Complex128, test/torch_np/numpy_tests/core/test_dtype.py::TestBuiltin::test_numeric_style_types_are_invalid_dtype_Complex32, test/torch_np/numpy_tests/core/test_dtype.py::TestBuiltin::test_numeric_style_types_are_invalid_dtype_Complex64, test/torch_np/numpy_tests/core/test_dtype.py::TestBuiltin::test_numeric_style_types_are_invalid_dtype_Datetime64, test/torch_np/numpy_tests/core/test_dtype.py::TestBuiltin::test_numeric_style_types_are_invalid_dtype_Float128, test/torch_np/numpy_tests/core/test_dtype.py::TestBuiltin::test_numeric_style_types_are_invalid_dtype_Float16, test/torch_np/numpy_tests/core/test_dtype.py::TestBuiltin::test_numeric_style_types_are_invalid_dtype_Float32, test/torch_np/numpy_tests/core/test_dtype.py::TestBuiltin::test_numeric_style_types_are_invalid_dtype_Float64, test/torch_np/numpy_tests/core/test_dtype.py::TestBuiltin::test_numeric_style_types_are_invalid_dtype_Int16, test/torch_np/numpy_tests/core/test_dtype.py::TestBuiltin::test_numeric_style_types_are_invalid_dtype_Int32, test/torch_np/numpy_tests/core/test_dtype.py::TestBuiltin::test_numeric_style_types_are_invalid_dtype_Int64, test/torch_np/numpy_tests/core/test_dtype.py::TestBuiltin::test_numeric_style_types_are_invalid_dtype_Int8, test/torch_np/numpy_tests/core/test_dtype.py::TestBuiltin::test_numeric_style_types_are_invalid_dtype_Object0, test/torch_np/numpy_tests/core/test_dtype.py::TestBuiltin::test_numeric_style_types_are_invalid_dtype_Str0, test/torch_np/numpy_tests/core/test_dtype.py::TestBuiltin::test_numeric_style_types_are_invalid_dtype_Timedelta64, test/torch_np/numpy_tests/core/test_dtype.py::TestBuiltin::test_numeric_style_types_are_invalid_dtype_UInt16, test/torch_np/numpy_tests/core/test_dtype.py::TestBuiltin::test_numeric_style_types_are_invalid_dtype_UInt32, test/torch_np/numpy_tests/core/test_dtype.py::TestBuiltin::test_numeric_style_types_are_invalid_dtype_UInt64, test/torch_np/numpy_tests/core/test_dtype.py::TestBuiltin::test_numeric_style_types_are_invalid_dtype_UInt8, test/torch_np/numpy_tests/core/test_dtype.py::TestBuiltin::test_numeric_style_types_are_invalid_dtype_Uint32, test/torch_np/numpy_tests/core/test_dtype.py::TestBuiltin::test_numeric_style_types_are_invalid_dtype_Uint64, test/torch_np/numpy_tests/core/test_dtype.py::TestBuiltin::test_numeric_style_types_are_invalid_dtype_Void0, test/torch_np/numpy_tests/core/test_dtype.py::TestBuiltin::test_richcompare_invalid_dtype_comparison_operation0, test/torch_np/numpy_tests/core/test_dtype.py::TestBuiltin::test_richcompare_invalid_dtype_comparison_operation1, test/torch_np/numpy_tests/core/test_dtype.py::TestBuiltin::test_richcompare_invalid_dtype_comparison_operation2, test/torch_np/numpy_tests/core/test_dtype.py::TestBuiltin::test_richcompare_invalid_dtype_comparison_operation3, test/torch_np/numpy_tests/core/test_dtype.py::TestBuiltin::test_richcompare_invalid_dtype_equality, test/torch_np/numpy_tests/core/test_dtype.py::TestBuiltin::test_run_t0, test/torch_np/numpy_tests/core/test_dtype.py::TestBuiltin::test_run_t1, test/torch_np/numpy_tests/core/test_dtype.py::TestBuiltin::test_run_t2, test/torch_np/numpy_tests/core/test_dtype.py::TestBuiltin::test_run_t3, test/torch_np/numpy_tests/core/test_dtype.py::TestDtypeAttributeDeletion::test_dtype_non_writable_attributes_deletion, test/torch_np/numpy_tests/core/test_dtype.py::TestDtypeAttributeDeletion::test_dtype_writable_attributes_deletion, test/torch_np/numpy_tests/core/test_dtype.py::TestPickling::test_builtin_t0, test/torch_np/numpy_tests/core/test_dtype.py::TestPickling::test_builtin_t1, test/torch_np/numpy_tests/core/test_dtype.py::TestPickling::test_builtin_t2, test/torch_np/numpy_tests/core/test_dtype.py::TestPickling::test_builtin_t3, test/torch_np/numpy_tests/core/test_dtype.py::TestPickling::test_builtin_t4, test/torch_np/numpy_tests/core/test_dtype.py::TestPickling::test_pickle_types_DType11, test/torch_np/numpy_tests/core/test_dtype.py::TestPickling::test_pickle_types_bool__10, test/torch_np/numpy_tests/core/test_dtype.py::TestPickling::test_pickle_types_complex128_4, test/torch_np/numpy_tests/core/test_dtype.py::TestPickling::test_pickle_types_complex64_3, test/torch_np/numpy_tests/core/test_dtype.py::TestPickling::test_pickle_types_float16_0, test/torch_np/numpy_tests/core/test_dtype.py::TestPickling::test_pickle_types_float32_1, test/torch_np/numpy_tests/core/test_dtype.py::TestPickling::test_pickle_types_float64_2, test/torch_np/numpy_tests/core/test_dtype.py::TestPickling::test_pickle_types_int16_7, test/torch_np/numpy_tests/core/test_dtype.py::TestPickling::test_pickle_types_int32_8, test/torch_np/numpy_tests/core/test_dtype.py::TestPickling::test_pickle_types_int64_9, test/torch_np/numpy_tests/core/test_dtype.py::TestPickling::test_pickle_types_int8_6, test/torch_np/numpy_tests/core/test_dtype.py::TestPickling::test_pickle_types_uint8_5, test/torch_np/numpy_tests/core/test_dtype.py::TestPromotion::test_complex_other_value_based_complex64_complex64_None, test/torch_np/numpy_tests/core/test_dtype.py::TestPromotion::test_complex_other_value_based_float16_complex64_None, test/torch_np/numpy_tests/core/test_dtype.py::TestPromotion::test_complex_other_value_based_float32_complex64_None, test/torch_np/numpy_tests/core/test_dtype.py::TestPromotion::test_complex_other_value_based_other_4294967295_expected1_expected_weak1, test/torch_np/numpy_tests/core/test_dtype.py::TestPromotion::test_complex_other_value_based_other_65535_expected0_expected_weak0, test/torch_np/numpy_tests/core/test_dtype.py::TestPromotion::test_complex_scalar_value_based_other0_expected0, test/torch_np/numpy_tests/core/test_dtype.py::TestPromotion::test_complex_scalar_value_based_other1_expected1, test/torch_np/numpy_tests/core/test_dtype.py::TestPromotion::test_complex_scalar_value_based_other2_expected2, test/torch_np/numpy_tests/core/test_dtype.py::TestPromotion::test_complex_scalar_value_based_other3_expected3, test/torch_np/numpy_tests/core/test_dtype.py::TestPromotion::test_complex_scalar_value_based_other4_expected4, test/torch_np/numpy_tests/core/test_dtype.py::TestPromotion::test_complex_scalar_value_based_other5_expected5, test/torch_np/numpy_tests/core/test_dtype.py::TestPromotion::test_complex_scalar_value_based_other6_expected6, test/torch_np/numpy_tests/core/test_dtype.py::TestPromotion::test_permutations_do_not_influence_result_dtypes0_expected0, test/torch_np/numpy_tests/core/test_dtype.py::TestPromotion::test_permutations_do_not_influence_result_dtypes1_expected1, test/torch_np/numpy_tests/core/test_dtype.py::TestPromotion::test_permutations_do_not_influence_result_dtypes2_expected2, test/torch_np/numpy_tests/core/test_dtype.py::TestPromotion::test_permutations_do_not_influence_result_dtypes3_expected3, test/torch_np/numpy_tests/core/test_dtype.py::TestPromotion::test_permutations_do_not_influence_result_dtypes4_expected4, test/torch_np/numpy_tests/core/test_dtype.py::TestPromotion::test_permutations_do_not_influence_result_dtypes5_expected5, test/torch_np/numpy_tests/core/test_dtype.py::TestPromotion::test_permutations_do_not_influence_result_dtypes6_expected6, test/torch_np/numpy_tests/core/test_dtype.py::TestPromotion::test_permutations_do_not_influence_result_dtypes7_expected7, test/torch_np/numpy_tests/core/test_dtype.py::TestPromotion::test_permutations_do_not_influence_result_dtypes8_expected8, test/torch_np/numpy_tests/core/test_dtype.py::TestPromotion::test_permutations_do_not_influence_result_dtypes9_expected9, test/torch_np/numpy_tests/core/test_dtype.py::TestPromotion::test_python_integer_promotion_val_18446744073709551616, test/torch_np/numpy_tests/core/test_dtype.py::TestPromotion::test_python_integer_promotion_val_2, test/torch_np/numpy_tests/core/test_dtype.py::TestPromotion::test_python_integer_promotion_val_200, test/torch_np/numpy_tests/core/test_dtype.py::TestPromotion::test_python_integer_promotion_val_4294967296, test/torch_np/numpy_tests/core/test_dtype.py::TestPromotion::test_python_integer_promotion_val_9223372036854775808, test/torch_np/numpy_tests/core/test_dtype.py::TestMisc::test_dtypes_are_true, test/torch_np/numpy_tests/core/test_dtype.py::TestMisc::test_keyword_argument, test/torch_np/numpy_tests/core/test_dtype.py::TestFromDTypeAttribute::test_recursion, test/torch_np/numpy_tests/core/test_dtype.py::TestFromDTypeAttribute::test_simple, test/torch_np/numpy_tests/core/test_dtype.py::TestClassGetItem::test_dtype, test/torch_np/numpy_tests/core/test_dtype.py::TestClassGetItem::test_dtype_subclass_code_?, test/torch_np/numpy_tests/core/test_dtype.py::TestClassGetItem::test_dtype_subclass_code_B, test/torch_np/numpy_tests/core/test_dtype.py::TestClassGetItem::test_dtype_subclass_code_D, test/torch_np/numpy_tests/core/test_dtype.py::TestClassGetItem::test_dtype_subclass_code_F, test/torch_np/numpy_tests/core/test_dtype.py::TestClassGetItem::test_dtype_subclass_code_b, test/torch_np/numpy_tests/core/test_dtype.py::TestClassGetItem::test_dtype_subclass_code_d, test/torch_np/numpy_tests/core/test_dtype.py::TestClassGetItem::test_dtype_subclass_code_e, test/torch_np/numpy_tests/core/test_dtype.py::TestClassGetItem::test_dtype_subclass_code_f, test/torch_np/numpy_tests/core/test_dtype.py::TestClassGetItem::test_dtype_subclass_code_h, test/torch_np/numpy_tests/core/test_dtype.py::TestClassGetItem::test_dtype_subclass_code_i, test/torch_np/numpy_tests/core/test_dtype.py::TestClassGetItem::test_dtype_subclass_code_l, test/torch_np/numpy_tests/core/test_dtype.py::TestClassGetItem::test_subscript_scalar, test/torch_np/numpy_tests/core/test_dtype.py::TestClassGetItem::test_subscript_tuple_arg_len_0, test/torch_np/numpy_tests/core/test_dtype.py::TestClassGetItem::test_subscript_tuple_arg_len_1, test/torch_np/numpy_tests/core/test_dtype.py::TestClassGetItem::test_subscript_tuple_arg_len_2, test/torch_np/numpy_tests/core/test_dtype.py::TestClassGetItem::test_subscript_tuple_arg_len_3 2025-12-04T13:52:32.6318762Z 2025-12-04T13:52:32.6319020Z Finished torch_np/numpy_tests/core/test_dtype 1/1 ... [2025-12-04 13:52:32.625852][15200.56806869], took 0.06min 2025-12-04T13:52:32.6558120Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/torch_np.numpy_tests.core.test_dtype/torch_np.numpy_tests.core.test_dtype-14b751af817e6a8b.xml 2025-12-04T13:52:32.6879876Z Running lazy/test_debug_util 1/1 ... [2025-12-04 13:52:32.687730][15200.62994776] 2025-12-04T13:52:32.6880303Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2025-12-04T13:52:32.6883195Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'lazy/test_debug_util.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-12-04 13:52:32.688053] 2025-12-04T13:52:36.0080918Z 2025-12-04T13:52:36.0081748Z lazy/test_debug_util 1/1 was successful, full logs can be found in artifacts with path test/test-reports/lazy.test_debug_util_1.1_22a1fe9c50887d0a_.log 2025-12-04T13:52:36.0082666Z Running 1 items in this shard: test/lazy/test_debug_util.py::DebugUtilTest::test_get_python_frames 2025-12-04T13:52:36.0083059Z 2025-12-04T13:52:36.0083320Z Finished lazy/test_debug_util 1/1 ... [2025-12-04 13:52:36.007860][15203.950075969], took 0.06min 2025-12-04T13:52:36.0374177Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/lazy.test_debug_util/lazy.test_debug_util-84565a365481d73c.xml 2025-12-04T13:52:36.0899093Z Running nn/test_load_state_dict 1/1 ... [2025-12-04 13:52:36.089675][15204.031893614] 2025-12-04T13:52:36.0899553Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2025-12-04T13:52:36.0902249Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'nn/test_load_state_dict.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-12-04 13:52:36.089968] 2025-12-04T13:52:39.9113501Z 2025-12-04T13:52:39.9114830Z nn/test_load_state_dict 1/1 was successful, full logs can be found in artifacts with path test/test-reports/nn.test_load_state_dict_1.1_7f9d0868685e0061_.log 2025-12-04T13:52:39.9124978Z Running 29 items in this shard: test/nn/test_load_state_dict.py::TestLoadStateDict::test_load_state_dict_BC_swap_False, test/nn/test_load_state_dict.py::TestLoadStateDict::test_load_state_dict_BC_swap_True, test/nn/test_load_state_dict.py::TestLoadStateDict::test_load_state_dict_assign_meta_swap_False_keep_vars_False, test/nn/test_load_state_dict.py::TestLoadStateDict::test_load_state_dict_assign_meta_swap_False_keep_vars_True, test/nn/test_load_state_dict.py::TestLoadStateDict::test_load_state_dict_assign_meta_swap_True_keep_vars_False, test/nn/test_load_state_dict.py::TestLoadStateDict::test_load_state_dict_assign_meta_swap_True_keep_vars_True, test/nn/test_load_state_dict.py::TestLoadStateDict::test_load_state_dict_assign_shape_stride_swap_False, test/nn/test_load_state_dict.py::TestLoadStateDict::test_load_state_dict_assign_shape_stride_swap_True, test/nn/test_load_state_dict.py::TestLoadStateDict::test_load_state_dict_assign_with_optimizer_swap_False, test/nn/test_load_state_dict.py::TestLoadStateDict::test_load_state_dict_assign_with_optimizer_swap_True, test/nn/test_load_state_dict.py::TestLoadStateDict::test_load_state_dict_child_swap_False, test/nn/test_load_state_dict.py::TestLoadStateDict::test_load_state_dict_child_swap_True, test/nn/test_load_state_dict.py::TestLoadStateDict::test_load_state_dict_custom_swap_False, test/nn/test_load_state_dict.py::TestLoadStateDict::test_load_state_dict_custom_swap_True, test/nn/test_load_state_dict.py::TestLoadStateDict::test_load_state_dict_invalid_swap_False, test/nn/test_load_state_dict.py::TestLoadStateDict::test_load_state_dict_invalid_swap_True, test/nn/test_load_state_dict.py::TestLoadStateDict::test_load_state_dict_ref_cycle_swap_False, test/nn/test_load_state_dict.py::TestLoadStateDict::test_load_state_dict_swap_False, test/nn/test_load_state_dict.py::TestLoadStateDict::test_load_state_dict_swap_True, test/nn/test_load_state_dict.py::TestLoadStateDict::test_load_state_dict_type_swap_False, test/nn/test_load_state_dict.py::TestLoadStateDict::test_load_state_dict_type_swap_True, test/nn/test_load_state_dict.py::TestLoadStateDict::test_load_state_dict_warn_assign_swap_False, test/nn/test_load_state_dict.py::TestLoadStateDict::test_load_state_dict_warn_assign_swap_True, test/nn/test_load_state_dict.py::TestLoadStateDict::test_load_state_dict_with_unexpected_key_swap_False, test/nn/test_load_state_dict.py::TestLoadStateDict::test_load_state_dict_with_unexpected_key_swap_True, test/nn/test_load_state_dict.py::TestLoadStateDict::test_scalar_param_1d_tensor_raises_swap_False, test/nn/test_load_state_dict.py::TestLoadStateDict::test_scalar_param_1d_tensor_raises_swap_True, test/nn/test_load_state_dict.py::TestLoadStateDictSwap::test_swap_subclass_swap_True_assign_False, test/nn/test_load_state_dict.py::TestLoadStateDictSwap::test_swap_subclass_swap_True_assign_True 2025-12-04T13:52:39.9132740Z 2025-12-04T13:52:39.9132958Z Finished nn/test_load_state_dict 1/1 ... [2025-12-04 13:52:39.910980][15207.853191793], took 0.06min 2025-12-04T13:52:39.9426578Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/nn.test_load_state_dict/nn.test_load_state_dict-243a96ba7564bbce.xml 2025-12-04T13:52:39.9903796Z Running test_shape_ops 1/1 ... [2025-12-04 13:52:39.990139][15207.93235789] 2025-12-04T13:52:39.9904219Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2025-12-04T13:52:39.9907272Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'test_shape_ops.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-12-04 13:52:39.990466] 2025-12-04T13:52:44.8130600Z 2025-12-04T13:52:44.8131365Z test_shape_ops 1/1 was successful, full logs can be found in artifacts with path test/test-reports/test_shape_ops_1.1_e2c81c501f7ea980_.log 2025-12-04T13:52:44.8151351Z Running 99 items in this shard: test/test_shape_ops.py::TestShapeOpsCUDA::test_clamp_cuda_float32, test/test_shape_ops.py::TestShapeOpsCUDA::test_clamp_cuda_int64, test/test_shape_ops.py::TestShapeOpsCUDA::test_clamp_propagates_nans_cuda, test/test_shape_ops.py::TestShapeOpsCUDA::test_clamp_raises_arg_errors_cuda, test/test_shape_ops.py::TestShapeOpsCUDA::test_complex_rot90_cuda_complex128, test/test_shape_ops.py::TestShapeOpsCUDA::test_complex_rot90_cuda_complex64, test/test_shape_ops.py::TestShapeOpsCUDA::test_diag_cuda_bool, test/test_shape_ops.py::TestShapeOpsCUDA::test_diag_cuda_float32, test/test_shape_ops.py::TestShapeOpsCUDA::test_diagonal_cuda, test/test_shape_ops.py::TestShapeOpsCUDA::test_diagonal_multidim_cuda_float32, test/test_shape_ops.py::TestShapeOpsCUDA::test_flip_cuda_bfloat16, test/test_shape_ops.py::TestShapeOpsCUDA::test_flip_cuda_bool, test/test_shape_ops.py::TestShapeOpsCUDA::test_flip_cuda_complex128, test/test_shape_ops.py::TestShapeOpsCUDA::test_flip_cuda_complex64, test/test_shape_ops.py::TestShapeOpsCUDA::test_flip_cuda_float16, test/test_shape_ops.py::TestShapeOpsCUDA::test_flip_cuda_float32, test/test_shape_ops.py::TestShapeOpsCUDA::test_flip_cuda_float64, test/test_shape_ops.py::TestShapeOpsCUDA::test_flip_cuda_int16, test/test_shape_ops.py::TestShapeOpsCUDA::test_flip_cuda_int32, test/test_shape_ops.py::TestShapeOpsCUDA::test_flip_cuda_int64, test/test_shape_ops.py::TestShapeOpsCUDA::test_flip_cuda_int8, test/test_shape_ops.py::TestShapeOpsCUDA::test_flip_cuda_uint8, test/test_shape_ops.py::TestShapeOpsCUDA::test_flip_errors_cuda_bfloat16, test/test_shape_ops.py::TestShapeOpsCUDA::test_flip_errors_cuda_bool, test/test_shape_ops.py::TestShapeOpsCUDA::test_flip_errors_cuda_complex128, test/test_shape_ops.py::TestShapeOpsCUDA::test_flip_errors_cuda_complex64, test/test_shape_ops.py::TestShapeOpsCUDA::test_flip_errors_cuda_float16, test/test_shape_ops.py::TestShapeOpsCUDA::test_flip_errors_cuda_float32, test/test_shape_ops.py::TestShapeOpsCUDA::test_flip_errors_cuda_float64, test/test_shape_ops.py::TestShapeOpsCUDA::test_flip_errors_cuda_int16, test/test_shape_ops.py::TestShapeOpsCUDA::test_flip_errors_cuda_int32, test/test_shape_ops.py::TestShapeOpsCUDA::test_flip_errors_cuda_int64, test/test_shape_ops.py::TestShapeOpsCUDA::test_flip_errors_cuda_int8, test/test_shape_ops.py::TestShapeOpsCUDA::test_flip_errors_cuda_uint8, test/test_shape_ops.py::TestShapeOpsCUDA::test_flip_large_tensor_cuda, test/test_shape_ops.py::TestShapeOpsCUDA::test_flip_numpy_cuda_bfloat16, test/test_shape_ops.py::TestShapeOpsCUDA::test_flip_numpy_cuda_bool, test/test_shape_ops.py::TestShapeOpsCUDA::test_flip_numpy_cuda_complex128, test/test_shape_ops.py::TestShapeOpsCUDA::test_flip_numpy_cuda_complex64, test/test_shape_ops.py::TestShapeOpsCUDA::test_flip_numpy_cuda_float16, test/test_shape_ops.py::TestShapeOpsCUDA::test_flip_numpy_cuda_float32, test/test_shape_ops.py::TestShapeOpsCUDA::test_flip_numpy_cuda_float64, test/test_shape_ops.py::TestShapeOpsCUDA::test_flip_numpy_cuda_int16, test/test_shape_ops.py::TestShapeOpsCUDA::test_flip_numpy_cuda_int32, test/test_shape_ops.py::TestShapeOpsCUDA::test_flip_numpy_cuda_int64, test/test_shape_ops.py::TestShapeOpsCUDA::test_flip_numpy_cuda_int8, test/test_shape_ops.py::TestShapeOpsCUDA::test_flip_numpy_cuda_uint8, test/test_shape_ops.py::TestShapeOpsCUDA::test_flip_unsupported_dtype_cuda_quint2x4, test/test_shape_ops.py::TestShapeOpsCUDA::test_flip_unsupported_dtype_cuda_quint4x2, test/test_shape_ops.py::TestShapeOpsCUDA::test_fliplr_cuda_complex128, test/test_shape_ops.py::TestShapeOpsCUDA::test_fliplr_cuda_float64, test/test_shape_ops.py::TestShapeOpsCUDA::test_fliplr_cuda_int64, test/test_shape_ops.py::TestShapeOpsCUDA::test_fliplr_invalid_cuda_complex128, test/test_shape_ops.py::TestShapeOpsCUDA::test_fliplr_invalid_cuda_float64, test/test_shape_ops.py::TestShapeOpsCUDA::test_fliplr_invalid_cuda_int64, test/test_shape_ops.py::TestShapeOpsCUDA::test_flipud_cuda_complex128, test/test_shape_ops.py::TestShapeOpsCUDA::test_flipud_cuda_float64, test/test_shape_ops.py::TestShapeOpsCUDA::test_flipud_cuda_int64, test/test_shape_ops.py::TestShapeOpsCUDA::test_flipud_invalid_cuda_complex128, test/test_shape_ops.py::TestShapeOpsCUDA::test_flipud_invalid_cuda_float64, test/test_shape_ops.py::TestShapeOpsCUDA::test_flipud_invalid_cuda_int64, test/test_shape_ops.py::TestShapeOpsCUDA::test_movedim_cuda_complex128, test/test_shape_ops.py::TestShapeOpsCUDA::test_movedim_cuda_float32, test/test_shape_ops.py::TestShapeOpsCUDA::test_movedim_cuda_int64, test/test_shape_ops.py::TestShapeOpsCUDA::test_movedim_invalid_cuda_complex128, test/test_shape_ops.py::TestShapeOpsCUDA::test_movedim_invalid_cuda_float32, test/test_shape_ops.py::TestShapeOpsCUDA::test_movedim_invalid_cuda_int64, test/test_shape_ops.py::TestShapeOpsCUDA::test_nonzero_astuple_out_cuda, test/test_shape_ops.py::TestShapeOpsCUDA::test_nonzero_cuda_bfloat16, test/test_shape_ops.py::TestShapeOpsCUDA::test_nonzero_cuda_bool, test/test_shape_ops.py::TestShapeOpsCUDA::test_nonzero_cuda_float16, test/test_shape_ops.py::TestShapeOpsCUDA::test_nonzero_cuda_float32, test/test_shape_ops.py::TestShapeOpsCUDA::test_nonzero_cuda_float64, test/test_shape_ops.py::TestShapeOpsCUDA::test_nonzero_cuda_int16, test/test_shape_ops.py::TestShapeOpsCUDA::test_nonzero_cuda_int32, test/test_shape_ops.py::TestShapeOpsCUDA::test_nonzero_cuda_int64, test/test_shape_ops.py::TestShapeOpsCUDA::test_nonzero_cuda_int8, test/test_shape_ops.py::TestShapeOpsCUDA::test_nonzero_cuda_uint8, test/test_shape_ops.py::TestShapeOpsCUDA::test_nonzero_discontiguous_cuda, test/test_shape_ops.py::TestShapeOpsCUDA::test_nonzero_no_warning_cuda, test/test_shape_ops.py::TestShapeOpsCUDA::test_nonzero_non_diff_cuda, test/test_shape_ops.py::TestShapeOpsCUDA::test_rot90_cuda, test/test_shape_ops.py::TestShapeOpsCUDA::test_sparse_dense_dim_cuda_complex128, test/test_shape_ops.py::TestShapeOpsCUDA::test_sparse_dense_dim_cuda_float32, test/test_shape_ops.py::TestShapeOpsCUDA::test_sparse_dense_dim_cuda_int64, test/test_shape_ops.py::TestShapeOpsCUDA::test_tolist_cuda, test/test_shape_ops.py::TestShapeOpsCUDA::test_trace_cuda_float16, test/test_shape_ops.py::TestShapeOpsCUDA::test_trace_cuda_float32, test/test_shape_ops.py::TestShapeOpsCUDA::test_trace_cuda_float64, test/test_shape_ops.py::TestShapeOpsCUDA::test_trace_cuda_int16, test/test_shape_ops.py::TestShapeOpsCUDA::test_trace_cuda_int32, test/test_shape_ops.py::TestShapeOpsCUDA::test_trace_cuda_int64, test/test_shape_ops.py::TestShapeOpsCUDA::test_trace_cuda_int8, test/test_shape_ops.py::TestShapeOpsCUDA::test_trace_cuda_uint8, test/test_shape_ops.py::TestShapeOpsCUDA::test_unbind_cuda, test/test_shape_ops.py::TestShapeOpsCUDA::test_unfold_all_devices_and_dtypes_cuda, test/test_shape_ops.py::TestShapeOpsCUDA::test_unfold_backward_errors_cuda, test/test_shape_ops.py::TestShapeOpsCUDA::test_unfold_errors_cuda, test/test_shape_ops.py::TestShapeOpsCUDA::test_unfold_scalars_cuda 2025-12-04T13:52:44.8169849Z 2025-12-04T13:52:44.8170036Z Finished test_shape_ops 1/1 ... [2025-12-04 13:52:44.812852][15212.755068286], took 0.08min 2025-12-04T13:52:44.8440269Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/test_shape_ops/test_shape_ops-3d3d95d2af19f7b9.xml 2025-12-04T13:52:44.8904896Z Running profiler/test_memory_profiler 1/1 ... [2025-12-04 13:52:44.890251][15212.832470003] 2025-12-04T13:52:44.8905411Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2025-12-04T13:52:44.8908144Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'profiler/test_memory_profiler.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-12-04 13:52:44.890561] 2025-12-04T13:52:51.2656492Z 2025-12-04T13:52:51.2658096Z profiler/test_memory_profiler 1/1 was successful, full logs can be found in artifacts with path test/test-reports/profiler.test_memory_profiler_1.1_bd4cd8ef1d0be682_.log 2025-12-04T13:52:51.2668496Z Running 33 items in this shard: test/profiler/test_memory_profiler.py::TestMemoryProfiler::test_config_check, test/profiler/test_memory_profiler.py::TestIdentifyGradients::test_extract_gradients_from_module, test/profiler/test_memory_profiler.py::TestIdentifyGradients::test_extract_gradients_from_module_and_optimizer, test/profiler/test_memory_profiler.py::TestIdentifyGradients::test_extract_gradients_from_optimizer, test/profiler/test_memory_profiler.py::TestIdentifyGradients::test_extract_gradients_from_optimizer_set_to_none, test/profiler/test_memory_profiler.py::TestIdentifyGradients::test_extract_gradients_low_level, test/profiler/test_memory_profiler.py::TestDataFlow::test_data_flow_graph_complicated, test/profiler/test_memory_profiler.py::TestDataFlow::test_data_flow_graph_non_op_allocations, test/profiler/test_memory_profiler.py::TestDataFlow::test_data_flow_graph_simple, test/profiler/test_memory_profiler.py::TestDataFlow::test_data_flow_graph_simple_backward, test/profiler/test_memory_profiler.py::TestDataFlow::test_data_flow_graph_simple_inplace, test/profiler/test_memory_profiler.py::TestDataFlow::test_data_flow_graph_stacked, test/profiler/test_memory_profiler.py::TestDataFlow::test_data_flow_graph_with_annotations, test/profiler/test_memory_profiler.py::TestDataFlow::test_match_schemas, test/profiler/test_memory_profiler.py::TestDataFlow::test_match_schemas_backward, test/profiler/test_memory_profiler.py::TestDataFlow::test_match_schemas_tensorlist, test/profiler/test_memory_profiler.py::TestMemoryProfilerE2E::test_categories_e2e_sequential_fwd, test/profiler/test_memory_profiler.py::TestMemoryProfilerE2E::test_categories_e2e_sequential_fwd_bwd, test/profiler/test_memory_profiler.py::TestMemoryProfilerE2E::test_categories_e2e_simple_fwd, test/profiler/test_memory_profiler.py::TestMemoryProfilerE2E::test_categories_e2e_simple_fwd_bwd, test/profiler/test_memory_profiler.py::TestMemoryProfilerE2E::test_categories_e2e_simple_fwd_bwd_step, test/profiler/test_memory_profiler.py::TestMemoryProfilerE2E::test_categories_e2e_simple_module_fwd, test/profiler/test_memory_profiler.py::TestMemoryProfilerE2E::test_categories_e2e_simple_module_fwd_bwd, test/profiler/test_memory_profiler.py::TestMemoryProfilerE2E::test_categories_e2e_simple_module_fwd_bwd_step, test/profiler/test_memory_profiler.py::TestMemoryProfilerE2E::test_inputs_fwd, test/profiler/test_memory_profiler.py::TestMemoryProfilerE2E::test_inputs_fwd_bwd, test/profiler/test_memory_profiler.py::TestMemoryProfilerE2E::test_inputs_fwd_lazy, test/profiler/test_memory_profiler.py::TestMemoryProfilerE2E::test_lazily_initialized, test/profiler/test_memory_profiler.py::TestMemoryProfilerE2E::test_manual_optimizer_step, test/profiler/test_memory_profiler.py::TestMemoryProfilerE2E::test_memory_timeline, test/profiler/test_memory_profiler.py::TestMemoryProfilerE2E::test_parameters_and_gradients, test/profiler/test_memory_profiler.py::TestMemoryProfilerE2E::test_parameters_and_gradients_set_to_none, test/profiler/test_memory_profiler.py::TestMemoryProfilerTimelineCUDA::test_memory_timeline_no_id_cuda 2025-12-04T13:52:51.2678364Z 2025-12-04T13:52:51.2678622Z Finished profiler/test_memory_profiler 1/1 ... [2025-12-04 13:52:51.265272][15219.207485212], took 0.11min 2025-12-04T13:52:51.2970133Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/profiler.test_memory_profiler/profiler.test_memory_profiler-c5c6304c0eac96b6.xml 2025-12-04T13:52:51.3715128Z Running test_indexing 1/1 ... [2025-12-04 13:52:51.371270][15219.313488987] 2025-12-04T13:52:51.3715564Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2025-12-04T13:52:51.3718393Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'test_indexing.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-12-04 13:52:51.371571] 2025-12-04T13:53:11.1666283Z 2025-12-04T13:53:11.1667152Z test_indexing 1/1 was successful, full logs can be found in artifacts with path test/test-reports/test_indexing_1.1_ea804c0f86ef5646_.log 2025-12-04T13:53:11.1709145Z Running 186 items in this shard: test/test_indexing.py::TestIndexingCUDA::test_advancedindex_big_cuda, test/test_indexing.py::TestIndexingCUDA::test_advancedindex_cuda_float16, test/test_indexing.py::TestIndexingCUDA::test_advancedindex_cuda_float64, test/test_indexing.py::TestIndexingCUDA::test_basic_advanced_combined_cuda, test/test_indexing.py::TestIndexingCUDA::test_bool_indices_accumulate_cuda, test/test_indexing.py::TestIndexingCUDA::test_bool_indices_cuda, test/test_indexing.py::TestIndexingCUDA::test_bool_mask_assignment_cuda, test/test_indexing.py::TestIndexingCUDA::test_byte_mask2d_cuda, test/test_indexing.py::TestIndexingCUDA::test_byte_mask_accumulate_cuda, test/test_indexing.py::TestIndexingCUDA::test_byte_mask_cuda, test/test_indexing.py::TestIndexingCUDA::test_byte_tensor_assignment_cuda, test/test_indexing.py::TestIndexingCUDA::test_cpu_indices_cuda, test/test_indexing.py::TestIndexingCUDA::test_cuda_broadcast_index_use_deterministic_algorithms_cuda, test/test_indexing.py::TestIndexingCUDA::test_ellipsis_tensor_cuda, test/test_indexing.py::TestIndexingCUDA::test_empty_index_cuda, test/test_indexing.py::TestIndexingCUDA::test_empty_ndim_index_bool_cuda, test/test_indexing.py::TestIndexingCUDA::test_empty_ndim_index_cuda, test/test_indexing.py::TestIndexingCUDA::test_empty_slice_cuda, test/test_indexing.py::TestIndexingCUDA::test_errors_index_copy_cuda, test/test_indexing.py::TestIndexingCUDA::test_gather_take_along_dim_cross_device_cuda_float32, test/test_indexing.py::TestIndexingCUDA::test_getitem_scalars_cuda, test/test_indexing.py::TestIndexingCUDA::test_index_add_deterministic_cuda, test/test_indexing.py::TestIndexingCUDA::test_index_copy_cuda_bfloat16, test/test_indexing.py::TestIndexingCUDA::test_index_copy_cuda_bool, test/test_indexing.py::TestIndexingCUDA::test_index_copy_cuda_complex128, test/test_indexing.py::TestIndexingCUDA::test_index_copy_cuda_complex64, test/test_indexing.py::TestIndexingCUDA::test_index_copy_cuda_float16, test/test_indexing.py::TestIndexingCUDA::test_index_copy_cuda_float32, test/test_indexing.py::TestIndexingCUDA::test_index_copy_cuda_float64, test/test_indexing.py::TestIndexingCUDA::test_index_copy_cuda_int16, test/test_indexing.py::TestIndexingCUDA::test_index_copy_cuda_int32, test/test_indexing.py::TestIndexingCUDA::test_index_copy_cuda_int64, test/test_indexing.py::TestIndexingCUDA::test_index_copy_cuda_int8, test/test_indexing.py::TestIndexingCUDA::test_index_copy_cuda_uint8, test/test_indexing.py::TestIndexingCUDA::test_index_copy_deterministic_cuda, test/test_indexing.py::TestIndexingCUDA::test_index_copy_scalars_cuda_bfloat16, test/test_indexing.py::TestIndexingCUDA::test_index_copy_scalars_cuda_bool, test/test_indexing.py::TestIndexingCUDA::test_index_copy_scalars_cuda_complex128, test/test_indexing.py::TestIndexingCUDA::test_index_copy_scalars_cuda_complex64, test/test_indexing.py::TestIndexingCUDA::test_index_copy_scalars_cuda_float16, test/test_indexing.py::TestIndexingCUDA::test_index_copy_scalars_cuda_float32, test/test_indexing.py::TestIndexingCUDA::test_index_copy_scalars_cuda_float64, test/test_indexing.py::TestIndexingCUDA::test_index_copy_scalars_cuda_int16, test/test_indexing.py::TestIndexingCUDA::test_index_copy_scalars_cuda_int32, test/test_indexing.py::TestIndexingCUDA::test_index_copy_scalars_cuda_int64, test/test_indexing.py::TestIndexingCUDA::test_index_copy_scalars_cuda_int8, test/test_indexing.py::TestIndexingCUDA::test_index_copy_scalars_cuda_uint8, test/test_indexing.py::TestIndexingCUDA::test_index_cuda, test/test_indexing.py::TestIndexingCUDA::test_index_fill_cuda_bfloat16, test/test_indexing.py::TestIndexingCUDA::test_index_fill_cuda_bool, test/test_indexing.py::TestIndexingCUDA::test_index_fill_cuda_complex128, test/test_indexing.py::TestIndexingCUDA::test_index_fill_cuda_complex64, test/test_indexing.py::TestIndexingCUDA::test_index_fill_cuda_float16, test/test_indexing.py::TestIndexingCUDA::test_index_fill_cuda_float32, test/test_indexing.py::TestIndexingCUDA::test_index_fill_cuda_float64, test/test_indexing.py::TestIndexingCUDA::test_index_fill_cuda_int16, test/test_indexing.py::TestIndexingCUDA::test_index_fill_cuda_int32, test/test_indexing.py::TestIndexingCUDA::test_index_fill_cuda_int64, test/test_indexing.py::TestIndexingCUDA::test_index_fill_cuda_int8, test/test_indexing.py::TestIndexingCUDA::test_index_fill_cuda_uint8, test/test_indexing.py::TestIndexingCUDA::test_index_getitem_copy_bools_slices_cuda, test/test_indexing.py::TestIndexingCUDA::test_index_ind_dtype_cuda, test/test_indexing.py::TestIndexingCUDA::test_index_limits_cuda, test/test_indexing.py::TestIndexingCUDA::test_index_put_accumulate_duplicate_indices_cuda, test/test_indexing.py::TestIndexingCUDA::test_index_put_accumulate_empty_cuda, test/test_indexing.py::TestIndexingCUDA::test_index_put_accumulate_expanded_values_cuda, test/test_indexing.py::TestIndexingCUDA::test_index_put_accumulate_large_tensor_cuda, test/test_indexing.py::TestIndexingCUDA::test_index_put_accumulate_non_contiguous_cuda, test/test_indexing.py::TestIndexingCUDA::test_index_put_deterministic_with_optional_tensors_cuda, test/test_indexing.py::TestIndexingCUDA::test_index_put_large_indices_cuda, test/test_indexing.py::TestIndexingCUDA::test_index_put_non_accumulate_deterministic_cuda, test/test_indexing.py::TestIndexingCUDA::test_index_put_src_datatype_cuda_bfloat16, test/test_indexing.py::TestIndexingCUDA::test_index_put_src_datatype_cuda_bool, test/test_indexing.py::TestIndexingCUDA::test_index_put_src_datatype_cuda_complex128, test/test_indexing.py::TestIndexingCUDA::test_index_put_src_datatype_cuda_complex64, test/test_indexing.py::TestIndexingCUDA::test_index_put_src_datatype_cuda_float16, test/test_indexing.py::TestIndexingCUDA::test_index_put_src_datatype_cuda_float8_e4m3fn, test/test_indexing.py::TestIndexingCUDA::test_index_put_src_datatype_cuda_float8_e5m2, test/test_indexing.py::TestIndexingCUDA::test_index_put_src_datatype_cuda_int64, test/test_indexing.py::TestIndexingCUDA::test_index_reduce_reduce_amax_cuda_bfloat16, test/test_indexing.py::TestIndexingCUDA::test_index_reduce_reduce_amax_cuda_float16, test/test_indexing.py::TestIndexingCUDA::test_index_reduce_reduce_amax_cuda_float32, test/test_indexing.py::TestIndexingCUDA::test_index_reduce_reduce_amax_cuda_float64, test/test_indexing.py::TestIndexingCUDA::test_index_reduce_reduce_amax_cuda_int16, test/test_indexing.py::TestIndexingCUDA::test_index_reduce_reduce_amax_cuda_int32, test/test_indexing.py::TestIndexingCUDA::test_index_reduce_reduce_amax_cuda_int64, test/test_indexing.py::TestIndexingCUDA::test_index_reduce_reduce_amax_cuda_int8, test/test_indexing.py::TestIndexingCUDA::test_index_reduce_reduce_amax_cuda_uint8, test/test_indexing.py::TestIndexingCUDA::test_index_reduce_reduce_amin_cuda_bfloat16, test/test_indexing.py::TestIndexingCUDA::test_index_reduce_reduce_amin_cuda_float16, test/test_indexing.py::TestIndexingCUDA::test_index_reduce_reduce_amin_cuda_float32, test/test_indexing.py::TestIndexingCUDA::test_index_reduce_reduce_amin_cuda_float64, test/test_indexing.py::TestIndexingCUDA::test_index_reduce_reduce_amin_cuda_int16, test/test_indexing.py::TestIndexingCUDA::test_index_reduce_reduce_amin_cuda_int32, test/test_indexing.py::TestIndexingCUDA::test_index_reduce_reduce_amin_cuda_int64, test/test_indexing.py::TestIndexingCUDA::test_index_reduce_reduce_amin_cuda_int8, test/test_indexing.py::TestIndexingCUDA::test_index_reduce_reduce_amin_cuda_uint8, test/test_indexing.py::TestIndexingCUDA::test_index_reduce_reduce_mean_cuda_bfloat16, test/test_indexing.py::TestIndexingCUDA::test_index_reduce_reduce_mean_cuda_float16, test/test_indexing.py::TestIndexingCUDA::test_index_reduce_reduce_mean_cuda_float32, test/test_indexing.py::TestIndexingCUDA::test_index_reduce_reduce_mean_cuda_float64, test/test_indexing.py::TestIndexingCUDA::test_index_reduce_reduce_mean_cuda_int16, test/test_indexing.py::TestIndexingCUDA::test_index_reduce_reduce_mean_cuda_int32, test/test_indexing.py::TestIndexingCUDA::test_index_reduce_reduce_mean_cuda_int64, test/test_indexing.py::TestIndexingCUDA::test_index_reduce_reduce_mean_cuda_int8, test/test_indexing.py::TestIndexingCUDA::test_index_reduce_reduce_mean_cuda_uint8, test/test_indexing.py::TestIndexingCUDA::test_index_reduce_reduce_prod_cuda_bfloat16, test/test_indexing.py::TestIndexingCUDA::test_index_reduce_reduce_prod_cuda_float16, test/test_indexing.py::TestIndexingCUDA::test_index_reduce_reduce_prod_cuda_float32, test/test_indexing.py::TestIndexingCUDA::test_index_reduce_reduce_prod_cuda_float64, test/test_indexing.py::TestIndexingCUDA::test_index_reduce_reduce_prod_cuda_int16, test/test_indexing.py::TestIndexingCUDA::test_index_reduce_reduce_prod_cuda_int32, test/test_indexing.py::TestIndexingCUDA::test_index_reduce_reduce_prod_cuda_int64, test/test_indexing.py::TestIndexingCUDA::test_index_reduce_reduce_prod_cuda_int8, test/test_indexing.py::TestIndexingCUDA::test_index_reduce_reduce_prod_cuda_uint8, test/test_indexing.py::TestIndexingCUDA::test_index_scalar_with_bool_mask_cuda, test/test_indexing.py::TestIndexingCUDA::test_index_select_cuda_bfloat16, test/test_indexing.py::TestIndexingCUDA::test_index_select_cuda_bool, test/test_indexing.py::TestIndexingCUDA::test_index_select_cuda_complex128, test/test_indexing.py::TestIndexingCUDA::test_index_select_cuda_complex64, test/test_indexing.py::TestIndexingCUDA::test_index_select_cuda_float16, test/test_indexing.py::TestIndexingCUDA::test_index_select_cuda_float32, test/test_indexing.py::TestIndexingCUDA::test_index_select_cuda_float64, test/test_indexing.py::TestIndexingCUDA::test_index_select_cuda_float8_e4m3fn, test/test_indexing.py::TestIndexingCUDA::test_index_select_cuda_float8_e4m3fnuz, test/test_indexing.py::TestIndexingCUDA::test_index_select_cuda_float8_e5m2, test/test_indexing.py::TestIndexingCUDA::test_index_select_cuda_float8_e5m2fnuz, test/test_indexing.py::TestIndexingCUDA::test_index_select_cuda_int16, test/test_indexing.py::TestIndexingCUDA::test_index_select_cuda_int32, test/test_indexing.py::TestIndexingCUDA::test_index_select_cuda_int64, test/test_indexing.py::TestIndexingCUDA::test_index_select_cuda_int8, test/test_indexing.py::TestIndexingCUDA::test_index_select_cuda_uint8, test/test_indexing.py::TestIndexingCUDA::test_index_setitem_bools_slices_cuda, test/test_indexing.py::TestIndexingCUDA::test_index_src_datatype_cuda_bfloat16, test/test_indexing.py::TestIndexingCUDA::test_index_src_datatype_cuda_bool, test/test_indexing.py::TestIndexingCUDA::test_index_src_datatype_cuda_float16, test/test_indexing.py::TestIndexingCUDA::test_index_src_datatype_cuda_int64, test/test_indexing.py::TestIndexingCUDA::test_int_assignment_cuda, test/test_indexing.py::TestIndexingCUDA::test_int_indices2d_cuda, test/test_indexing.py::TestIndexingCUDA::test_int_indices_broadcast_cuda, test/test_indexing.py::TestIndexingCUDA::test_int_indices_cuda, test/test_indexing.py::TestIndexingCUDA::test_invalid_device_cuda, test/test_indexing.py::TestIndexingCUDA::test_invalid_index_cuda, test/test_indexing.py::TestIndexingCUDA::test_jit_indexing_cuda, test/test_indexing.py::TestIndexingCUDA::test_list_indices_cuda, test/test_indexing.py::TestIndexingCUDA::test_multi_dimensional_bool_mask_assignment_cuda, test/test_indexing.py::TestIndexingCUDA::test_multi_dimensional_bool_mask_cuda, test/test_indexing.py::TestIndexingCUDA::test_multiple_bool_indices_cuda, test/test_indexing.py::TestIndexingCUDA::test_multiple_byte_mask_cuda, test/test_indexing.py::TestIndexingCUDA::test_multiple_int_cuda, test/test_indexing.py::TestIndexingCUDA::test_none_cuda, test/test_indexing.py::TestIndexingCUDA::test_out_of_bound_index_cuda, test/test_indexing.py::TestIndexingCUDA::test_set_item_to_scalar_tensor_cuda, test/test_indexing.py::TestIndexingCUDA::test_setitem_expansion_error_cuda, test/test_indexing.py::TestIndexingCUDA::test_setitem_scalars_cuda, test/test_indexing.py::TestIndexingCUDA::test_single_int_cuda, test/test_indexing.py::TestIndexingCUDA::test_step_assignment_cuda, test/test_indexing.py::TestIndexingCUDA::test_step_cuda, test/test_indexing.py::TestIndexingCUDA::test_take_along_dim_cuda_float32, test/test_indexing.py::TestIndexingCUDA::test_take_along_dim_cuda_int64, test/test_indexing.py::TestIndexingCUDA::test_take_along_dim_invalid_cuda_float32, test/test_indexing.py::TestIndexingCUDA::test_take_along_dim_invalid_cuda_int64, test/test_indexing.py::TestIndexingCUDA::test_unravel_index_errors_cuda, test/test_indexing.py::TestIndexingCUDA::test_variable_slicing_cuda, test/test_indexing.py::TestIndexingCUDA::test_zero_dim_index_cuda, test/test_indexing.py::NumpyTestsCUDA::test_boolean_assignment_value_mismatch_cuda, test/test_indexing.py::NumpyTestsCUDA::test_boolean_indexing_alldims_cuda, test/test_indexing.py::NumpyTestsCUDA::test_boolean_indexing_onedim_cuda, test/test_indexing.py::NumpyTestsCUDA::test_boolean_indexing_twodim_cuda, test/test_indexing.py::NumpyTestsCUDA::test_boolean_indexing_weirdness_cuda, test/test_indexing.py::NumpyTestsCUDA::test_boolean_indexing_weirdness_tensors_cuda, test/test_indexing.py::NumpyTestsCUDA::test_boolean_list_indexing_cuda, test/test_indexing.py::NumpyTestsCUDA::test_boolean_shape_mismatch_cuda, test/test_indexing.py::NumpyTestsCUDA::test_broadcast_subspace_cuda, test/test_indexing.py::NumpyTestsCUDA::test_broaderrors_indexing_cuda, test/test_indexing.py::NumpyTestsCUDA::test_ellipsis_index_cuda, test/test_indexing.py::NumpyTestsCUDA::test_empty_fancy_index_cuda, test/test_indexing.py::NumpyTestsCUDA::test_empty_tuple_index_cuda, test/test_indexing.py::NumpyTestsCUDA::test_everything_returns_views_cuda, test/test_indexing.py::NumpyTestsCUDA::test_index_is_larger_cuda, test/test_indexing.py::NumpyTestsCUDA::test_index_no_floats_cuda, test/test_indexing.py::NumpyTestsCUDA::test_none_index_cuda, test/test_indexing.py::NumpyTestsCUDA::test_single_bool_index_cuda, test/test_indexing.py::NumpyTestsCUDA::test_single_int_index_cuda, test/test_indexing.py::NumpyTestsCUDA::test_trivial_fancy_out_of_bounds_cuda, test/test_indexing.py::NumpyTestsCUDA::test_truncate_leading_1s_cuda 2025-12-04T13:53:11.1748815Z 2025-12-04T13:53:11.1749003Z Finished test_indexing 1/1 ... [2025-12-04 13:53:11.166714][15239.108930952], took 0.33min 2025-12-04T13:53:11.1989467Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/test_indexing/test_indexing-05fc579dc970f1f4.xml 2025-12-04T13:53:11.2711300Z Running torch_np/numpy_tests/core/test_getlimits 1/1 ... [2025-12-04 13:53:11.270887][15239.213105696] 2025-12-04T13:53:11.2711824Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2025-12-04T13:53:11.2714754Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'torch_np/numpy_tests/core/test_getlimits.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-12-04 13:53:11.271230] 2025-12-04T13:53:14.6920783Z 2025-12-04T13:53:14.6921830Z torch_np/numpy_tests/core/test_getlimits 1/1 was successful, full logs can be found in artifacts with path test/test-reports/torch_np.numpy_tests.core.test_getlimits_1.1_13078b14d88df97c_.log 2025-12-04T13:53:14.6927044Z Running 17 items in this shard: test/torch_np/numpy_tests/core/test_getlimits.py::TestPythonFloat::test_singleton, test/torch_np/numpy_tests/core/test_getlimits.py::TestHalf::test_singleton, test/torch_np/numpy_tests/core/test_getlimits.py::TestSingle::test_singleton, test/torch_np/numpy_tests/core/test_getlimits.py::TestDouble::test_singleton, test/torch_np/numpy_tests/core/test_getlimits.py::TestFinfo::test_basic, test/torch_np/numpy_tests/core/test_getlimits.py::TestFinfo::test_basic_missing, test/torch_np/numpy_tests/core/test_getlimits.py::TestIinfo::test_basic, test/torch_np/numpy_tests/core/test_getlimits.py::TestIinfo::test_unsigned_max_T0, test/torch_np/numpy_tests/core/test_getlimits.py::TestIinfo::test_unsigned_max_T1, test/torch_np/numpy_tests/core/test_getlimits.py::TestIinfo::test_unsigned_max_T2, test/torch_np/numpy_tests/core/test_getlimits.py::TestIinfo::test_unsigned_max_T3, test/torch_np/numpy_tests/core/test_getlimits.py::TestRepr::test_finfo_repr, test/torch_np/numpy_tests/core/test_getlimits.py::TestRepr::test_iinfo_repr, test/torch_np/numpy_tests/core/test_getlimits.py::TestMisc::test_instances, test/torch_np/numpy_tests/core/test_getlimits.py::TestMisc::test_known_types, test/torch_np/numpy_tests/core/test_getlimits.py::TestMisc::test_plausible_finfo, test/torch_np/numpy_tests/core/test_getlimits.py::TestMisc::test_subnormal_warning 2025-12-04T13:53:14.6931050Z 2025-12-04T13:53:14.6931326Z Finished torch_np/numpy_tests/core/test_getlimits 1/1 ... [2025-12-04 13:53:14.691753][15242.633960926], took 0.06min 2025-12-04T13:53:14.7229803Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/torch_np.numpy_tests.core.test_getlimits/torch_np.numpy_tests.core.test_getlimits-9dc4d2d2b9b5672c.xml 2025-12-04T13:53:14.7511805Z Running torch_np/test_ndarray_methods 1/1 ... [2025-12-04 13:53:14.750944][15242.693162978] 2025-12-04T13:53:14.7512286Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2025-12-04T13:53:14.7515358Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'torch_np/test_ndarray_methods.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-12-04 13:53:14.751267] 2025-12-04T13:53:21.3270206Z 2025-12-04T13:53:21.3271333Z torch_np/test_ndarray_methods 1/1 was successful, full logs can be found in artifacts with path test/test-reports/torch_np.test_ndarray_methods_1.1_570f98179517d221_.log 2025-12-04T13:53:21.3362539Z Running 342 items in this shard: test/torch_np/test_ndarray_methods.py::TestIndexing::test_indexing_simple, test/torch_np/test_ndarray_methods.py::TestIndexing::test_setitem, test/torch_np/test_ndarray_methods.py::TestReshape::test_reshape_function, test/torch_np/test_ndarray_methods.py::TestReshape::test_reshape_method, test/torch_np/test_ndarray_methods.py::TestTranspose::test_transpose_function, test/torch_np/test_ndarray_methods.py::TestTranspose::test_transpose_method, test/torch_np/test_ndarray_methods.py::TestRavel::test_ravel_function, test/torch_np/test_ndarray_methods.py::TestRavel::test_ravel_method, test/torch_np/test_ndarray_methods.py::TestNonzero::test_array_method, test/torch_np/test_ndarray_methods.py::TestNonzero::test_nonzero_onedim, test/torch_np/test_ndarray_methods.py::TestNonzero::test_nonzero_trivial, test/torch_np/test_ndarray_methods.py::TestNonzero::test_nonzero_twodim, test/torch_np/test_ndarray_methods.py::TestNonzero::test_sparse, test/torch_np/test_ndarray_methods.py::TestArgmaxArgminCommon::test_all_method_max, test/torch_np/test_ndarray_methods.py::TestArgmaxArgminCommon::test_all_method_min, test/torch_np/test_ndarray_methods.py::TestArgmaxArgminCommon::test_np_argmin_argmax_keepdims_size0_axis0_method0, test/torch_np/test_ndarray_methods.py::TestArgmaxArgminCommon::test_np_argmin_argmax_keepdims_size0_axis0_method1, test/torch_np/test_ndarray_methods.py::TestArgmaxArgminCommon::test_np_argmin_argmax_keepdims_size10_axis_-1_method0, test/torch_np/test_ndarray_methods.py::TestArgmaxArgminCommon::test_np_argmin_argmax_keepdims_size10_axis_-1_method1, test/torch_np/test_ndarray_methods.py::TestArgmaxArgminCommon::test_np_argmin_argmax_keepdims_size11_axis_0_method0, test/torch_np/test_ndarray_methods.py::TestArgmaxArgminCommon::test_np_argmin_argmax_keepdims_size11_axis_0_method1, test/torch_np/test_ndarray_methods.py::TestArgmaxArgminCommon::test_np_argmin_argmax_keepdims_size12_axis_1_method0, test/torch_np/test_ndarray_methods.py::TestArgmaxArgminCommon::test_np_argmin_argmax_keepdims_size12_axis_1_method1, test/torch_np/test_ndarray_methods.py::TestArgmaxArgminCommon::test_np_argmin_argmax_keepdims_size13_axis13_method0, test/torch_np/test_ndarray_methods.py::TestArgmaxArgminCommon::test_np_argmin_argmax_keepdims_size13_axis13_method1, test/torch_np/test_ndarray_methods.py::TestArgmaxArgminCommon::test_np_argmin_argmax_keepdims_size14_axis_-2_method0, test/torch_np/test_ndarray_methods.py::TestArgmaxArgminCommon::test_np_argmin_argmax_keepdims_size14_axis_-2_method1, test/torch_np/test_ndarray_methods.py::TestArgmaxArgminCommon::test_np_argmin_argmax_keepdims_size15_axis_-1_method0, test/torch_np/test_ndarray_methods.py::TestArgmaxArgminCommon::test_np_argmin_argmax_keepdims_size15_axis_-1_method1, test/torch_np/test_ndarray_methods.py::TestArgmaxArgminCommon::test_np_argmin_argmax_keepdims_size16_axis_0_method0, test/torch_np/test_ndarray_methods.py::TestArgmaxArgminCommon::test_np_argmin_argmax_keepdims_size16_axis_0_method1, test/torch_np/test_ndarray_methods.py::TestArgmaxArgminCommon::test_np_argmin_argmax_keepdims_size17_axis_1_method0, test/torch_np/test_ndarray_methods.py::TestArgmaxArgminCommon::test_np_argmin_argmax_keepdims_size17_axis_1_method1, test/torch_np/test_ndarray_methods.py::TestArgmaxArgminCommon::test_np_argmin_argmax_keepdims_size18_axis18_method0, test/torch_np/test_ndarray_methods.py::TestArgmaxArgminCommon::test_np_argmin_argmax_keepdims_size18_axis18_method1, test/torch_np/test_ndarray_methods.py::TestArgmaxArgminCommon::test_np_argmin_argmax_keepdims_size19_axis_-3_method0, test/torch_np/test_ndarray_methods.py::TestArgmaxArgminCommon::test_np_argmin_argmax_keepdims_size19_axis_-3_method1, test/torch_np/test_ndarray_methods.py::TestArgmaxArgminCommon::test_np_argmin_argmax_keepdims_size1_axis_-1_method0, test/torch_np/test_ndarray_methods.py::TestArgmaxArgminCommon::test_np_argmin_argmax_keepdims_size1_axis_-1_method1, test/torch_np/test_ndarray_methods.py::TestArgmaxArgminCommon::test_np_argmin_argmax_keepdims_size20_axis_-2_method0, test/torch_np/test_ndarray_methods.py::TestArgmaxArgminCommon::test_np_argmin_argmax_keepdims_size20_axis_-2_method1, test/torch_np/test_ndarray_methods.py::TestArgmaxArgminCommon::test_np_argmin_argmax_keepdims_size21_axis_-1_method0, test/torch_np/test_ndarray_methods.py::TestArgmaxArgminCommon::test_np_argmin_argmax_keepdims_size21_axis_-1_method1, test/torch_np/test_ndarray_methods.py::TestArgmaxArgminCommon::test_np_argmin_argmax_keepdims_size22_axis_0_method0, test/torch_np/test_ndarray_methods.py::TestArgmaxArgminCommon::test_np_argmin_argmax_keepdims_size22_axis_0_method1, test/torch_np/test_ndarray_methods.py::TestArgmaxArgminCommon::test_np_argmin_argmax_keepdims_size23_axis_1_method0, test/torch_np/test_ndarray_methods.py::TestArgmaxArgminCommon::test_np_argmin_argmax_keepdims_size23_axis_1_method1, test/torch_np/test_ndarray_methods.py::TestArgmaxArgminCommon::test_np_argmin_argmax_keepdims_size24_axis_2_method0, test/torch_np/test_ndarray_methods.py::TestArgmaxArgminCommon::test_np_argmin_argmax_keepdims_size24_axis_2_method1, test/torch_np/test_ndarray_methods.py::TestArgmaxArgminCommon::test_np_argmin_argmax_keepdims_size25_axis25_method0, test/torch_np/test_ndarray_methods.py::TestArgmaxArgminCommon::test_np_argmin_argmax_keepdims_size25_axis25_method1, test/torch_np/test_ndarray_methods.py::TestArgmaxArgminCommon::test_np_argmin_argmax_keepdims_size26_axis_-3_method0, test/torch_np/test_ndarray_methods.py::TestArgmaxArgminCommon::test_np_argmin_argmax_keepdims_size26_axis_-3_method1, test/torch_np/test_ndarray_methods.py::TestArgmaxArgminCommon::test_np_argmin_argmax_keepdims_size27_axis_-2_method0, test/torch_np/test_ndarray_methods.py::TestArgmaxArgminCommon::test_np_argmin_argmax_keepdims_size27_axis_-2_method1, test/torch_np/test_ndarray_methods.py::TestArgmaxArgminCommon::test_np_argmin_argmax_keepdims_size28_axis_-1_method0, test/torch_np/test_ndarray_methods.py::TestArgmaxArgminCommon::test_np_argmin_argmax_keepdims_size28_axis_-1_method1, test/torch_np/test_ndarray_methods.py::TestArgmaxArgminCommon::test_np_argmin_argmax_keepdims_size29_axis_0_method0, test/torch_np/test_ndarray_methods.py::TestArgmaxArgminCommon::test_np_argmin_argmax_keepdims_size29_axis_0_method1, test/torch_np/test_ndarray_methods.py::TestArgmaxArgminCommon::test_np_argmin_argmax_keepdims_size2_axis_0_method0, test/torch_np/test_ndarray_methods.py::TestArgmaxArgminCommon::test_np_argmin_argmax_keepdims_size2_axis_0_method1, test/torch_np/test_ndarray_methods.py::TestArgmaxArgminCommon::test_np_argmin_argmax_keepdims_size30_axis_1_method0, test/torch_np/test_ndarray_methods.py::TestArgmaxArgminCommon::test_np_argmin_argmax_keepdims_size30_axis_1_method1, test/torch_np/test_ndarray_methods.py::TestArgmaxArgminCommon::test_np_argmin_argmax_keepdims_size31_axis_2_method0, test/torch_np/test_ndarray_methods.py::TestArgmaxArgminCommon::test_np_argmin_argmax_keepdims_size31_axis_2_method1, test/torch_np/test_ndarray_methods.py::TestArgmaxArgminCommon::test_np_argmin_argmax_keepdims_size32_axis32_method0, test/torch_np/test_ndarray_methods.py::TestArgmaxArgminCommon::test_np_argmin_argmax_keepdims_size32_axis32_method1, test/torch_np/test_ndarray_methods.py::TestArgmaxArgminCommon::test_np_argmin_argmax_keepdims_size33_axis_-4_method0, test/torch_np/test_ndarray_methods.py::TestArgmaxArgminCommon::test_np_argmin_argmax_keepdims_size33_axis_-4_method1, test/torch_np/test_ndarray_methods.py::TestArgmaxArgminCommon::test_np_argmin_argmax_keepdims_size34_axis_-3_method0, test/torch_np/test_ndarray_methods.py::TestArgmaxArgminCommon::test_np_argmin_argmax_keepdims_size34_axis_-3_method1, test/torch_np/test_ndarray_methods.py::TestArgmaxArgminCommon::test_np_argmin_argmax_keepdims_size35_axis_-2_method0, test/torch_np/test_ndarray_methods.py::TestArgmaxArgminCommon::test_np_argmin_argmax_keepdims_size35_axis_-2_method1, test/torch_np/test_ndarray_methods.py::TestArgmaxArgminCommon::test_np_argmin_argmax_keepdims_size36_axis_-1_method0, test/torch_np/test_ndarray_methods.py::TestArgmaxArgminCommon::test_np_argmin_argmax_keepdims_size36_axis_-1_method1, test/torch_np/test_ndarray_methods.py::TestArgmaxArgminCommon::test_np_argmin_argmax_keepdims_size37_axis_0_method0, test/torch_np/test_ndarray_methods.py::TestArgmaxArgminCommon::test_np_argmin_argmax_keepdims_size37_axis_0_method1, test/torch_np/test_ndarray_methods.py::TestArgmaxArgminCommon::test_np_argmin_argmax_keepdims_size38_axis_1_method0, test/torch_np/test_ndarray_methods.py::TestArgmaxArgminCommon::test_np_argmin_argmax_keepdims_size38_axis_1_method1, test/torch_np/test_ndarray_methods.py::TestArgmaxArgminCommon::test_np_argmin_argmax_keepdims_size39_axis_2_method0, test/torch_np/test_ndarray_methods.py::TestArgmaxArgminCommon::test_np_argmin_argmax_keepdims_size39_axis_2_method1, test/torch_np/test_ndarray_methods.py::TestArgmaxArgminCommon::test_np_argmin_argmax_keepdims_size3_axis3_method0, test/torch_np/test_ndarray_methods.py::TestArgmaxArgminCommon::test_np_argmin_argmax_keepdims_size3_axis3_method1, test/torch_np/test_ndarray_methods.py::TestArgmaxArgminCommon::test_np_argmin_argmax_keepdims_size40_axis_3_method0, test/torch_np/test_ndarray_methods.py::TestArgmaxArgminCommon::test_np_argmin_argmax_keepdims_size40_axis_3_method1, test/torch_np/test_ndarray_methods.py::TestArgmaxArgminCommon::test_np_argmin_argmax_keepdims_size41_axis41_method0, test/torch_np/test_ndarray_methods.py::TestArgmaxArgminCommon::test_np_argmin_argmax_keepdims_size41_axis41_method1, test/torch_np/test_ndarray_methods.py::TestArgmaxArgminCommon::test_np_argmin_argmax_keepdims_size42_axis_-4_method0, test/torch_np/test_ndarray_methods.py::TestArgmaxArgminCommon::test_np_argmin_argmax_keepdims_size42_axis_-4_method1, test/torch_np/test_ndarray_methods.py::TestArgmaxArgminCommon::test_np_argmin_argmax_keepdims_size43_axis_-3_method0, test/torch_np/test_ndarray_methods.py::TestArgmaxArgminCommon::test_np_argmin_argmax_keepdims_size43_axis_-3_method1, test/torch_np/test_ndarray_methods.py::TestArgmaxArgminCommon::test_np_argmin_argmax_keepdims_size44_axis_-2_method0, test/torch_np/test_ndarray_methods.py::TestArgmaxArgminCommon::test_np_argmin_argmax_keepdims_size44_axis_-2_method1, test/torch_np/test_ndarray_methods.py::TestArgmaxArgminCommon::test_np_argmin_argmax_keepdims_size45_axis_-1_method0, test/torch_np/test_ndarray_methods.py::TestArgmaxArgminCommon::test_np_argmin_argmax_keepdims_size45_axis_-1_method1, test/torch_np/test_ndarray_methods.py::TestArgmaxArgminCommon::test_np_argmin_argmax_keepdims_size46_axis_0_method0, test/torch_np/test_ndarray_methods.py::TestArgmaxArgminCommon::test_np_argmin_argmax_keepdims_size46_axis_0_method1, test/torch_np/test_ndarray_methods.py::TestArgmaxArgminCommon::test_np_argmin_argmax_keepdims_size47_axis_1_method0, test/torch_np/test_ndarray_methods.py::TestArgmaxArgminCommon::test_np_argmin_argmax_keepdims_size47_axis_1_method1, test/torch_np/test_ndarray_methods.py::TestArgmaxArgminCommon::test_np_argmin_argmax_keepdims_size48_axis_2_method0, test/torch_np/test_ndarray_methods.py::TestArgmaxArgminCommon::test_np_argmin_argmax_keepdims_size48_axis_2_method1, test/torch_np/test_ndarray_methods.py::TestArgmaxArgminCommon::test_np_argmin_argmax_keepdims_size49_axis_3_method0, test/torch_np/test_ndarray_methods.py::TestArgmaxArgminCommon::test_np_argmin_argmax_keepdims_size49_axis_3_method1, test/torch_np/test_ndarray_methods.py::TestArgmaxArgminCommon::test_np_argmin_argmax_keepdims_size4_axis_-2_method0, test/torch_np/test_ndarray_methods.py::TestArgmaxArgminCommon::test_np_argmin_argmax_keepdims_size4_axis_-2_method1, test/torch_np/test_ndarray_methods.py::TestArgmaxArgminCommon::test_np_argmin_argmax_keepdims_size50_axis50_method0, test/torch_np/test_ndarray_methods.py::TestArgmaxArgminCommon::test_np_argmin_argmax_keepdims_size50_axis50_method1, test/torch_np/test_ndarray_methods.py::TestArgmaxArgminCommon::test_np_argmin_argmax_keepdims_size51_axis_-4_method0, test/torch_np/test_ndarray_methods.py::TestArgmaxArgminCommon::test_np_argmin_argmax_keepdims_size51_axis_-4_method1, test/torch_np/test_ndarray_methods.py::TestArgmaxArgminCommon::test_np_argmin_argmax_keepdims_size52_axis_-3_method0, test/torch_np/test_ndarray_methods.py::TestArgmaxArgminCommon::test_np_argmin_argmax_keepdims_size52_axis_-3_method1, test/torch_np/test_ndarray_methods.py::TestArgmaxArgminCommon::test_np_argmin_argmax_keepdims_size53_axis_-2_method0, test/torch_np/test_ndarray_methods.py::TestArgmaxArgminCommon::test_np_argmin_argmax_keepdims_size53_axis_-2_method1, test/torch_np/test_ndarray_methods.py::TestArgmaxArgminCommon::test_np_argmin_argmax_keepdims_size54_axis_-1_method0, test/torch_np/test_ndarray_methods.py::TestArgmaxArgminCommon::test_np_argmin_argmax_keepdims_size54_axis_-1_method1, test/torch_np/test_ndarray_methods.py::TestArgmaxArgminCommon::test_np_argmin_argmax_keepdims_size55_axis_0_method0, test/torch_np/test_ndarray_methods.py::TestArgmaxArgminCommon::test_np_argmin_argmax_keepdims_size55_axis_0_method1, test/torch_np/test_ndarray_methods.py::TestArgmaxArgminCommon::test_np_argmin_argmax_keepdims_size56_axis_1_method0, test/torch_np/test_ndarray_methods.py::TestArgmaxArgminCommon::test_np_argmin_argmax_keepdims_size56_axis_1_method1, test/torch_np/test_ndarray_methods.py::TestArgmaxArgminCommon::test_np_argmin_argmax_keepdims_size57_axis_2_method0, test/torch_np/test_ndarray_methods.py::TestArgmaxArgminCommon::test_np_argmin_argmax_keepdims_size57_axis_2_method1, test/torch_np/test_ndarray_methods.py::TestArgmaxArgminCommon::test_np_argmin_argmax_keepdims_size58_axis_3_method0, test/torch_np/test_ndarray_methods.py::TestArgmaxArgminCommon::test_np_argmin_argmax_keepdims_size58_axis_3_method1, test/torch_np/test_ndarray_methods.py::TestArgmaxArgminCommon::test_np_argmin_argmax_keepdims_size59_axis59_method0, test/torch_np/test_ndarray_methods.py::TestArgmaxArgminCommon::test_np_argmin_argmax_keepdims_size59_axis59_method1, test/torch_np/test_ndarray_methods.py::TestArgmaxArgminCommon::test_np_argmin_argmax_keepdims_size5_axis_-1_method0, test/torch_np/test_ndarray_methods.py::TestArgmaxArgminCommon::test_np_argmin_argmax_keepdims_size5_axis_-1_method1, test/torch_np/test_ndarray_methods.py::TestArgmaxArgminCommon::test_np_argmin_argmax_keepdims_size60_axis_-4_method0, test/torch_np/test_ndarray_methods.py::TestArgmaxArgminCommon::test_np_argmin_argmax_keepdims_size60_axis_-4_method1, test/torch_np/test_ndarray_methods.py::TestArgmaxArgminCommon::test_np_argmin_argmax_keepdims_size61_axis_-3_method0, test/torch_np/test_ndarray_methods.py::TestArgmaxArgminCommon::test_np_argmin_argmax_keepdims_size61_axis_-3_method1, test/torch_np/test_ndarray_methods.py::TestArgmaxArgminCommon::test_np_argmin_argmax_keepdims_size62_axis_-2_method0, test/torch_np/test_ndarray_methods.py::TestArgmaxArgminCommon::test_np_argmin_argmax_keepdims_size62_axis_-2_method1, test/torch_np/test_ndarray_methods.py::TestArgmaxArgminCommon::test_np_argmin_argmax_keepdims_size63_axis_-1_method0, test/torch_np/test_ndarray_methods.py::TestArgmaxArgminCommon::test_np_argmin_argmax_keepdims_size63_axis_-1_method1, test/torch_np/test_ndarray_methods.py::TestArgmaxArgminCommon::test_np_argmin_argmax_keepdims_size64_axis_0_method0, test/torch_np/test_ndarray_methods.py::TestArgmaxArgminCommon::test_np_argmin_argmax_keepdims_size64_axis_0_method1, test/torch_np/test_ndarray_methods.py::TestArgmaxArgminCommon::test_np_argmin_argmax_keepdims_size65_axis_1_method0, test/torch_np/test_ndarray_methods.py::TestArgmaxArgminCommon::test_np_argmin_argmax_keepdims_size65_axis_1_method1, test/torch_np/test_ndarray_methods.py::TestArgmaxArgminCommon::test_np_argmin_argmax_keepdims_size66_axis_2_method0, test/torch_np/test_ndarray_methods.py::TestArgmaxArgminCommon::test_np_argmin_argmax_keepdims_size66_axis_2_method1, test/torch_np/test_ndarray_methods.py::TestArgmaxArgminCommon::test_np_argmin_argmax_keepdims_size67_axis_3_method0, test/torch_np/test_ndarray_methods.py::TestArgmaxArgminCommon::test_np_argmin_argmax_keepdims_size67_axis_3_method1, test/torch_np/test_ndarray_methods.py::TestArgmaxArgminCommon::test_np_argmin_argmax_keepdims_size68_axis68_method0, test/torch_np/test_ndarray_methods.py::TestArgmaxArgminCommon::test_np_argmin_argmax_keepdims_size68_axis68_method1, test/torch_np/test_ndarray_methods.py::TestArgmaxArgminCommon::test_np_argmin_argmax_keepdims_size69_axis_-1_method0, test/torch_np/test_ndarray_methods.py::TestArgmaxArgminCommon::test_np_argmin_argmax_keepdims_size69_axis_-1_method1, test/torch_np/test_ndarray_methods.py::TestArgmaxArgminCommon::test_np_argmin_argmax_keepdims_size6_axis_0_method0, test/torch_np/test_ndarray_methods.py::TestArgmaxArgminCommon::test_np_argmin_argmax_keepdims_size6_axis_0_method1, test/torch_np/test_ndarray_methods.py::TestArgmaxArgminCommon::test_np_argmin_argmax_keepdims_size70_axis_0_method0, test/torch_np/test_ndarray_methods.py::TestArgmaxArgminCommon::test_np_argmin_argmax_keepdims_size70_axis_0_method1, test/torch_np/test_ndarray_methods.py::TestArgmaxArgminCommon::test_np_argmin_argmax_keepdims_size71_axis71_method0, test/torch_np/test_ndarray_methods.py::TestArgmaxArgminCommon::test_np_argmin_argmax_keepdims_size71_axis71_method1, test/torch_np/test_ndarray_methods.py::TestArgmaxArgminCommon::test_np_argmin_argmax_keepdims_size72_axis_-1_method0, test/torch_np/test_ndarray_methods.py::TestArgmaxArgminCommon::test_np_argmin_argmax_keepdims_size72_axis_-1_method1, test/torch_np/test_ndarray_methods.py::TestArgmaxArgminCommon::test_np_argmin_argmax_keepdims_size73_axis_0_method0, test/torch_np/test_ndarray_methods.py::TestArgmaxArgminCommon::test_np_argmin_argmax_keepdims_size73_axis_0_method1, test/torch_np/test_ndarray_methods.py::TestArgmaxArgminCommon::test_np_argmin_argmax_keepdims_size74_axis74_method0, test/torch_np/test_ndarray_methods.py::TestArgmaxArgminCommon::test_np_argmin_argmax_keepdims_size74_axis74_method1, test/torch_np/test_ndarray_methods.py::TestArgmaxArgminCommon::test_np_argmin_argmax_keepdims_size75_axis_-1_method0, test/torch_np/test_ndarray_methods.py::TestArgmaxArgminCommon::test_np_argmin_argmax_keepdims_size75_axis_-1_method1, test/torch_np/test_ndarray_methods.py::TestArgmaxArgminCommon::test_np_argmin_argmax_keepdims_size76_axis_0_method0, test/torch_np/test_ndarray_methods.py::TestArgmaxArgminCommon::test_np_argmin_argmax_keepdims_size76_axis_0_method1, test/torch_np/test_ndarray_methods.py::TestArgmaxArgminCommon::test_np_argmin_argmax_keepdims_size77_axis77_method0, test/torch_np/test_ndarray_methods.py::TestArgmaxArgminCommon::test_np_argmin_argmax_keepdims_size77_axis77_method1, test/torch_np/test_ndarray_methods.py::TestArgmaxArgminCommon::test_np_argmin_argmax_keepdims_size7_axis_1_method0, test/torch_np/test_ndarray_methods.py::TestArgmaxArgminCommon::test_np_argmin_argmax_keepdims_size7_axis_1_method1, test/torch_np/test_ndarray_methods.py::TestArgmaxArgminCommon::test_np_argmin_argmax_keepdims_size8_axis8_method0, test/torch_np/test_ndarray_methods.py::TestArgmaxArgminCommon::test_np_argmin_argmax_keepdims_size8_axis8_method1, test/torch_np/test_ndarray_methods.py::TestArgmaxArgminCommon::test_np_argmin_argmax_keepdims_size9_axis_-2_method0, test/torch_np/test_ndarray_methods.py::TestArgmaxArgminCommon::test_np_argmin_argmax_keepdims_size9_axis_-2_method1, test/torch_np/test_ndarray_methods.py::TestArgmaxArgminCommon::test_np_vs_ndarray_arr_method_argmax_np_method0, test/torch_np/test_ndarray_methods.py::TestArgmaxArgminCommon::test_np_vs_ndarray_arr_method_argmin_np_method1, test/torch_np/test_ndarray_methods.py::TestArgmaxArgminCommon::test_np_vs_ndarray_positional_arr_method_argmax_np_method0, test/torch_np/test_ndarray_methods.py::TestArgmaxArgminCommon::test_np_vs_ndarray_positional_arr_method_argmin_np_method1, test/torch_np/test_ndarray_methods.py::TestArgmaxArgminCommon::test_output_shape_method_argmax, test/torch_np/test_ndarray_methods.py::TestArgmaxArgminCommon::test_output_shape_method_argmin, test/torch_np/test_ndarray_methods.py::TestArgmaxArgminCommon::test_ret_is_out_ndim_0_method_argmax, test/torch_np/test_ndarray_methods.py::TestArgmaxArgminCommon::test_ret_is_out_ndim_0_method_argmin, test/torch_np/test_ndarray_methods.py::TestArgmaxArgminCommon::test_ret_is_out_ndim_1_method_argmax, test/torch_np/test_ndarray_methods.py::TestArgmaxArgminCommon::test_ret_is_out_ndim_1_method_argmin, test/torch_np/test_ndarray_methods.py::TestArgmax::test_combinations_data0, test/torch_np/test_ndarray_methods.py::TestArgmax::test_combinations_data1, test/torch_np/test_ndarray_methods.py::TestArgmax::test_combinations_data10, test/torch_np/test_ndarray_methods.py::TestArgmax::test_combinations_data11, test/torch_np/test_ndarray_methods.py::TestArgmax::test_combinations_data12, test/torch_np/test_ndarray_methods.py::TestArgmax::test_combinations_data13, test/torch_np/test_ndarray_methods.py::TestArgmax::test_combinations_data14, test/torch_np/test_ndarray_methods.py::TestArgmax::test_combinations_data15, test/torch_np/test_ndarray_methods.py::TestArgmax::test_combinations_data16, test/torch_np/test_ndarray_methods.py::TestArgmax::test_combinations_data17, test/torch_np/test_ndarray_methods.py::TestArgmax::test_combinations_data18, test/torch_np/test_ndarray_methods.py::TestArgmax::test_combinations_data19, test/torch_np/test_ndarray_methods.py::TestArgmax::test_combinations_data2, test/torch_np/test_ndarray_methods.py::TestArgmax::test_combinations_data20, test/torch_np/test_ndarray_methods.py::TestArgmax::test_combinations_data21, test/torch_np/test_ndarray_methods.py::TestArgmax::test_combinations_data22, test/torch_np/test_ndarray_methods.py::TestArgmax::test_combinations_data23, test/torch_np/test_ndarray_methods.py::TestArgmax::test_combinations_data24, test/torch_np/test_ndarray_methods.py::TestArgmax::test_combinations_data25, test/torch_np/test_ndarray_methods.py::TestArgmax::test_combinations_data26, test/torch_np/test_ndarray_methods.py::TestArgmax::test_combinations_data27, test/torch_np/test_ndarray_methods.py::TestArgmax::test_combinations_data28, test/torch_np/test_ndarray_methods.py::TestArgmax::test_combinations_data29, test/torch_np/test_ndarray_methods.py::TestArgmax::test_combinations_data3, test/torch_np/test_ndarray_methods.py::TestArgmax::test_combinations_data30, test/torch_np/test_ndarray_methods.py::TestArgmax::test_combinations_data31, test/torch_np/test_ndarray_methods.py::TestArgmax::test_combinations_data32, test/torch_np/test_ndarray_methods.py::TestArgmax::test_combinations_data33, test/torch_np/test_ndarray_methods.py::TestArgmax::test_combinations_data34, test/torch_np/test_ndarray_methods.py::TestArgmax::test_combinations_data35, test/torch_np/test_ndarray_methods.py::TestArgmax::test_combinations_data36, test/torch_np/test_ndarray_methods.py::TestArgmax::test_combinations_data37, test/torch_np/test_ndarray_methods.py::TestArgmax::test_combinations_data38, test/torch_np/test_ndarray_methods.py::TestArgmax::test_combinations_data39, test/torch_np/test_ndarray_methods.py::TestArgmax::test_combinations_data4, test/torch_np/test_ndarray_methods.py::TestArgmax::test_combinations_data40, test/torch_np/test_ndarray_methods.py::TestArgmax::test_combinations_data41, test/torch_np/test_ndarray_methods.py::TestArgmax::test_combinations_data42, test/torch_np/test_ndarray_methods.py::TestArgmax::test_combinations_data43, test/torch_np/test_ndarray_methods.py::TestArgmax::test_combinations_data44, test/torch_np/test_ndarray_methods.py::TestArgmax::test_combinations_data45, test/torch_np/test_ndarray_methods.py::TestArgmax::test_combinations_data46, test/torch_np/test_ndarray_methods.py::TestArgmax::test_combinations_data47, test/torch_np/test_ndarray_methods.py::TestArgmax::test_combinations_data48, test/torch_np/test_ndarray_methods.py::TestArgmax::test_combinations_data49, test/torch_np/test_ndarray_methods.py::TestArgmax::test_combinations_data5, test/torch_np/test_ndarray_methods.py::TestArgmax::test_combinations_data50, test/torch_np/test_ndarray_methods.py::TestArgmax::test_combinations_data51, test/torch_np/test_ndarray_methods.py::TestArgmax::test_combinations_data52, test/torch_np/test_ndarray_methods.py::TestArgmax::test_combinations_data53, test/torch_np/test_ndarray_methods.py::TestArgmax::test_combinations_data54, test/torch_np/test_ndarray_methods.py::TestArgmax::test_combinations_data55, test/torch_np/test_ndarray_methods.py::TestArgmax::test_combinations_data56, test/torch_np/test_ndarray_methods.py::TestArgmax::test_combinations_data57, test/torch_np/test_ndarray_methods.py::TestArgmax::test_combinations_data58, test/torch_np/test_ndarray_methods.py::TestArgmax::test_combinations_data59, test/torch_np/test_ndarray_methods.py::TestArgmax::test_combinations_data6, test/torch_np/test_ndarray_methods.py::TestArgmax::test_combinations_data60, test/torch_np/test_ndarray_methods.py::TestArgmax::test_combinations_data61, test/torch_np/test_ndarray_methods.py::TestArgmax::test_combinations_data62, test/torch_np/test_ndarray_methods.py::TestArgmax::test_combinations_data63, test/torch_np/test_ndarray_methods.py::TestArgmax::test_combinations_data64, test/torch_np/test_ndarray_methods.py::TestArgmax::test_combinations_data65, test/torch_np/test_ndarray_methods.py::TestArgmax::test_combinations_data66, test/torch_np/test_ndarray_methods.py::TestArgmax::test_combinations_data67, test/torch_np/test_ndarray_methods.py::TestArgmax::test_combinations_data68, test/torch_np/test_ndarray_methods.py::TestArgmax::test_combinations_data69, test/torch_np/test_ndarray_methods.py::TestArgmax::test_combinations_data7, test/torch_np/test_ndarray_methods.py::TestArgmax::test_combinations_data70, test/torch_np/test_ndarray_methods.py::TestArgmax::test_combinations_data71, test/torch_np/test_ndarray_methods.py::TestArgmax::test_combinations_data72, test/torch_np/test_ndarray_methods.py::TestArgmax::test_combinations_data73, test/torch_np/test_ndarray_methods.py::TestArgmax::test_combinations_data8, test/torch_np/test_ndarray_methods.py::TestArgmax::test_combinations_data9, test/torch_np/test_ndarray_methods.py::TestArgmax::test_maximum_signed_integers, test/torch_np/test_ndarray_methods.py::TestArgmin::test_combinations_data0, test/torch_np/test_ndarray_methods.py::TestArgmin::test_combinations_data1, test/torch_np/test_ndarray_methods.py::TestArgmin::test_combinations_data10, test/torch_np/test_ndarray_methods.py::TestArgmin::test_combinations_data11, test/torch_np/test_ndarray_methods.py::TestArgmin::test_combinations_data12, test/torch_np/test_ndarray_methods.py::TestArgmin::test_combinations_data13, test/torch_np/test_ndarray_methods.py::TestArgmin::test_combinations_data14, test/torch_np/test_ndarray_methods.py::TestArgmin::test_combinations_data15, test/torch_np/test_ndarray_methods.py::TestArgmin::test_combinations_data16, test/torch_np/test_ndarray_methods.py::TestArgmin::test_combinations_data17, test/torch_np/test_ndarray_methods.py::TestArgmin::test_combinations_data18, test/torch_np/test_ndarray_methods.py::TestArgmin::test_combinations_data19, test/torch_np/test_ndarray_methods.py::TestArgmin::test_combinations_data2, test/torch_np/test_ndarray_methods.py::TestArgmin::test_combinations_data20, test/torch_np/test_ndarray_methods.py::TestArgmin::test_combinations_data21, test/torch_np/test_ndarray_methods.py::TestArgmin::test_combinations_data22, test/torch_np/test_ndarray_methods.py::TestArgmin::test_combinations_data23, test/torch_np/test_ndarray_methods.py::TestArgmin::test_combinations_data24, test/torch_np/test_ndarray_methods.py::TestArgmin::test_combinations_data25, test/torch_np/test_ndarray_methods.py::TestArgmin::test_combinations_data26, test/torch_np/test_ndarray_methods.py::TestArgmin::test_combinations_data27, test/torch_np/test_ndarray_methods.py::TestArgmin::test_combinations_data28, test/torch_np/test_ndarray_methods.py::TestArgmin::test_combinations_data29, test/torch_np/test_ndarray_methods.py::TestArgmin::test_combinations_data3, test/torch_np/test_ndarray_methods.py::TestArgmin::test_combinations_data30, test/torch_np/test_ndarray_methods.py::TestArgmin::test_combinations_data31, test/torch_np/test_ndarray_methods.py::TestArgmin::test_combinations_data32, test/torch_np/test_ndarray_methods.py::TestArgmin::test_combinations_data33, test/torch_np/test_ndarray_methods.py::TestArgmin::test_combinations_data34, test/torch_np/test_ndarray_methods.py::TestArgmin::test_combinations_data35, test/torch_np/test_ndarray_methods.py::TestArgmin::test_combinations_data36, test/torch_np/test_ndarray_methods.py::TestArgmin::test_combinations_data37, test/torch_np/test_ndarray_methods.py::TestArgmin::test_combinations_data38, test/torch_np/test_ndarray_methods.py::TestArgmin::test_combinations_data39, test/torch_np/test_ndarray_methods.py::TestArgmin::test_combinations_data4, test/torch_np/test_ndarray_methods.py::TestArgmin::test_combinations_data40, test/torch_np/test_ndarray_methods.py::TestArgmin::test_combinations_data41, test/torch_np/test_ndarray_methods.py::TestArgmin::test_combinations_data42, test/torch_np/test_ndarray_methods.py::TestArgmin::test_combinations_data43, test/torch_np/test_ndarray_methods.py::TestArgmin::test_combinations_data44, test/torch_np/test_ndarray_methods.py::TestArgmin::test_combinations_data45, test/torch_np/test_ndarray_methods.py::TestArgmin::test_combinations_data46, test/torch_np/test_ndarray_methods.py::TestArgmin::test_combinations_data47, test/torch_np/test_ndarray_methods.py::TestArgmin::test_combinations_data48, test/torch_np/test_ndarray_methods.py::TestArgmin::test_combinations_data49, test/torch_np/test_ndarray_methods.py::TestArgmin::test_combinations_data5, test/torch_np/test_ndarray_methods.py::TestArgmin::test_combinations_data50, test/torch_np/test_ndarray_methods.py::TestArgmin::test_combinations_data51, test/torch_np/test_ndarray_methods.py::TestArgmin::test_combinations_data52, test/torch_np/test_ndarray_methods.py::TestArgmin::test_combinations_data53, test/torch_np/test_ndarray_methods.py::TestArgmin::test_combinations_data54, test/torch_np/test_ndarray_methods.py::TestArgmin::test_combinations_data55, test/torch_np/test_ndarray_methods.py::TestArgmin::test_combinations_data56, test/torch_np/test_ndarray_methods.py::TestArgmin::test_combinations_data57, test/torch_np/test_ndarray_methods.py::TestArgmin::test_combinations_data58, test/torch_np/test_ndarray_methods.py::TestArgmin::test_combinations_data59, test/torch_np/test_ndarray_methods.py::TestArgmin::test_combinations_data6, test/torch_np/test_ndarray_methods.py::TestArgmin::test_combinations_data60, test/torch_np/test_ndarray_methods.py::TestArgmin::test_combinations_data61, test/torch_np/test_ndarray_methods.py::TestArgmin::test_combinations_data62, test/torch_np/test_ndarray_methods.py::TestArgmin::test_combinations_data63, test/torch_np/test_ndarray_methods.py::TestArgmin::test_combinations_data64, test/torch_np/test_ndarray_methods.py::TestArgmin::test_combinations_data65, test/torch_np/test_ndarray_methods.py::TestArgmin::test_combinations_data66, test/torch_np/test_ndarray_methods.py::TestArgmin::test_combinations_data67, test/torch_np/test_ndarray_methods.py::TestArgmin::test_combinations_data68, test/torch_np/test_ndarray_methods.py::TestArgmin::test_combinations_data69, test/torch_np/test_ndarray_methods.py::TestArgmin::test_combinations_data7, test/torch_np/test_ndarray_methods.py::TestArgmin::test_combinations_data70, test/torch_np/test_ndarray_methods.py::TestArgmin::test_combinations_data71, test/torch_np/test_ndarray_methods.py::TestArgmin::test_combinations_data72, test/torch_np/test_ndarray_methods.py::TestArgmin::test_combinations_data73, test/torch_np/test_ndarray_methods.py::TestArgmin::test_combinations_data8, test/torch_np/test_ndarray_methods.py::TestArgmin::test_combinations_data9, test/torch_np/test_ndarray_methods.py::TestArgmin::test_minimum_signed_integers, test/torch_np/test_ndarray_methods.py::TestAmax::test_basic, test/torch_np/test_ndarray_methods.py::TestAmin::test_basic, test/torch_np/test_ndarray_methods.py::TestContains::test_contains, test/torch_np/test_ndarray_methods.py::TestNoExtraMethods::test_extra_methods_name_fn, test/torch_np/test_ndarray_methods.py::TestNoExtraMethods::test_extra_methods_name_ivar, test/torch_np/test_ndarray_methods.py::TestNoExtraMethods::test_extra_methods_name_method, test/torch_np/test_ndarray_methods.py::TestNoExtraMethods::test_extra_methods_name_name, test/torch_np/test_ndarray_methods.py::TestNoExtraMethods::test_extra_methods_name_plain, test/torch_np/test_ndarray_methods.py::TestNoExtraMethods::test_extra_methods_name_rvar, test/torch_np/test_ndarray_methods.py::TestIter::test_iter_1d, test/torch_np/test_ndarray_methods.py::TestIter::test_iter_2d 2025-12-04T13:53:21.3451464Z 2025-12-04T13:53:21.3451706Z Finished torch_np/test_ndarray_methods 1/1 ... [2025-12-04 13:53:21.327271][15249.269488425], took 0.11min 2025-12-04T13:53:21.3593889Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/torch_np.test_ndarray_methods/torch_np.test_ndarray_methods-30a1cc814f90c4ef.xml 2025-12-04T13:53:21.4405330Z Running test_view_ops 1/1 ... [2025-12-04 13:53:21.440296][15249.382514127] 2025-12-04T13:53:21.4405731Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2025-12-04T13:53:21.4408674Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'test_view_ops.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-12-04 13:53:21.440613] 2025-12-04T13:53:38.1814849Z 2025-12-04T13:53:38.1819725Z test_view_ops 1/1 was successful, full logs can be found in artifacts with path test/test-reports/test_view_ops_1.1_843de7c7b78fe573_.log 2025-12-04T13:53:38.1876178Z Running 279 items in this shard: test/test_view_ops.py::TestViewOpsCUDA::test_T_view_cuda, test/test_view_ops.py::TestViewOpsCUDA::test_advanced_indexing_assignment_cuda, test/test_view_ops.py::TestViewOpsCUDA::test_advanced_indexing_nonview_cuda, test/test_view_ops.py::TestViewOpsCUDA::test_as_strided_gradients_cuda, test/test_view_ops.py::TestViewOpsCUDA::test_as_strided_inplace_view_cuda, test/test_view_ops.py::TestViewOpsCUDA::test_as_strided_view_cuda, test/test_view_ops.py::TestViewOpsCUDA::test_basic_indexing_ellipses_view_cuda, test/test_view_ops.py::TestViewOpsCUDA::test_basic_indexing_newaxis_view_cuda, test/test_view_ops.py::TestViewOpsCUDA::test_basic_indexing_slice_view_cuda, test/test_view_ops.py::TestViewOpsCUDA::test_chunk_view_cuda, test/test_view_ops.py::TestViewOpsCUDA::test_conj_imag_view_cuda_complex128, test/test_view_ops.py::TestViewOpsCUDA::test_conj_imag_view_cuda_complex64, test/test_view_ops.py::TestViewOpsCUDA::test_conj_self_cuda_bfloat16, test/test_view_ops.py::TestViewOpsCUDA::test_conj_self_cuda_float16, test/test_view_ops.py::TestViewOpsCUDA::test_conj_self_cuda_float32, test/test_view_ops.py::TestViewOpsCUDA::test_conj_self_cuda_float64, test/test_view_ops.py::TestViewOpsCUDA::test_conj_self_cuda_int16, test/test_view_ops.py::TestViewOpsCUDA::test_conj_self_cuda_int32, test/test_view_ops.py::TestViewOpsCUDA::test_conj_self_cuda_int64, test/test_view_ops.py::TestViewOpsCUDA::test_conj_self_cuda_int8, test/test_view_ops.py::TestViewOpsCUDA::test_conj_self_cuda_uint8, test/test_view_ops.py::TestViewOpsCUDA::test_conj_view_with_shared_memory_cuda, test/test_view_ops.py::TestViewOpsCUDA::test_contiguous_nonview_cuda, test/test_view_ops.py::TestViewOpsCUDA::test_contiguous_self_cuda, test/test_view_ops.py::TestViewOpsCUDA::test_diagonal_view_cuda, test/test_view_ops.py::TestViewOpsCUDA::test_expand_as_view_cuda, test/test_view_ops.py::TestViewOpsCUDA::test_expand_view_cuda, test/test_view_ops.py::TestViewOpsCUDA::test_flatten_nonview_cuda, test/test_view_ops.py::TestViewOpsCUDA::test_flatten_view_cuda, test/test_view_ops.py::TestViewOpsCUDA::test_imag_noncomplex_cuda_bfloat16, test/test_view_ops.py::TestViewOpsCUDA::test_imag_noncomplex_cuda_float16, test/test_view_ops.py::TestViewOpsCUDA::test_imag_noncomplex_cuda_float32, test/test_view_ops.py::TestViewOpsCUDA::test_imag_noncomplex_cuda_float64, test/test_view_ops.py::TestViewOpsCUDA::test_imag_noncomplex_cuda_int16, test/test_view_ops.py::TestViewOpsCUDA::test_imag_noncomplex_cuda_int32, test/test_view_ops.py::TestViewOpsCUDA::test_imag_noncomplex_cuda_int64, test/test_view_ops.py::TestViewOpsCUDA::test_imag_noncomplex_cuda_int8, test/test_view_ops.py::TestViewOpsCUDA::test_imag_noncomplex_cuda_uint8, test/test_view_ops.py::TestViewOpsCUDA::test_movedim_view_cuda, test/test_view_ops.py::TestViewOpsCUDA::test_narrow_view_cuda, test/test_view_ops.py::TestViewOpsCUDA::test_permute_view_cuda, test/test_view_ops.py::TestViewOpsCUDA::test_real_imag_view_cuda_complex128, test/test_view_ops.py::TestViewOpsCUDA::test_real_imag_view_cuda_complex64, test/test_view_ops.py::TestViewOpsCUDA::test_reshape_as_view_cuda, test/test_view_ops.py::TestViewOpsCUDA::test_reshape_nonview_cuda, test/test_view_ops.py::TestViewOpsCUDA::test_reshape_view_cuda, test/test_view_ops.py::TestViewOpsCUDA::test_select_view_cuda, test/test_view_ops.py::TestViewOpsCUDA::test_set_real_imag_cuda_complex128_bfloat16, test/test_view_ops.py::TestViewOpsCUDA::test_set_real_imag_cuda_complex128_bool, test/test_view_ops.py::TestViewOpsCUDA::test_set_real_imag_cuda_complex128_complex128, test/test_view_ops.py::TestViewOpsCUDA::test_set_real_imag_cuda_complex128_complex64, test/test_view_ops.py::TestViewOpsCUDA::test_set_real_imag_cuda_complex128_float16, test/test_view_ops.py::TestViewOpsCUDA::test_set_real_imag_cuda_complex128_float32, test/test_view_ops.py::TestViewOpsCUDA::test_set_real_imag_cuda_complex128_float64, test/test_view_ops.py::TestViewOpsCUDA::test_set_real_imag_cuda_complex128_int16, test/test_view_ops.py::TestViewOpsCUDA::test_set_real_imag_cuda_complex128_int32, test/test_view_ops.py::TestViewOpsCUDA::test_set_real_imag_cuda_complex128_int64, test/test_view_ops.py::TestViewOpsCUDA::test_set_real_imag_cuda_complex128_int8, test/test_view_ops.py::TestViewOpsCUDA::test_set_real_imag_cuda_complex128_uint8, test/test_view_ops.py::TestViewOpsCUDA::test_set_real_imag_cuda_complex64_bfloat16, test/test_view_ops.py::TestViewOpsCUDA::test_set_real_imag_cuda_complex64_bool, test/test_view_ops.py::TestViewOpsCUDA::test_set_real_imag_cuda_complex64_complex128, test/test_view_ops.py::TestViewOpsCUDA::test_set_real_imag_cuda_complex64_complex64, test/test_view_ops.py::TestViewOpsCUDA::test_set_real_imag_cuda_complex64_float16, test/test_view_ops.py::TestViewOpsCUDA::test_set_real_imag_cuda_complex64_float32, test/test_view_ops.py::TestViewOpsCUDA::test_set_real_imag_cuda_complex64_float64, test/test_view_ops.py::TestViewOpsCUDA::test_set_real_imag_cuda_complex64_int16, test/test_view_ops.py::TestViewOpsCUDA::test_set_real_imag_cuda_complex64_int32, test/test_view_ops.py::TestViewOpsCUDA::test_set_real_imag_cuda_complex64_int64, test/test_view_ops.py::TestViewOpsCUDA::test_set_real_imag_cuda_complex64_int8, test/test_view_ops.py::TestViewOpsCUDA::test_set_real_imag_cuda_complex64_uint8, test/test_view_ops.py::TestViewOpsCUDA::test_split_view_cuda, test/test_view_ops.py::TestViewOpsCUDA::test_squeeze_inplace_view_cuda, test/test_view_ops.py::TestViewOpsCUDA::test_squeeze_view_cuda, test/test_view_ops.py::TestViewOpsCUDA::test_t_inplace_view_cuda, test/test_view_ops.py::TestViewOpsCUDA::test_t_view_cuda, test/test_view_ops.py::TestViewOpsCUDA::test_transpose_inplace_view_cuda, test/test_view_ops.py::TestViewOpsCUDA::test_transpose_view_cuda, test/test_view_ops.py::TestViewOpsCUDA::test_unbind_cuda, test/test_view_ops.py::TestViewOpsCUDA::test_unbind_view_cuda, test/test_view_ops.py::TestViewOpsCUDA::test_unfold_view_cuda, test/test_view_ops.py::TestViewOpsCUDA::test_unsqueeze_inplace_view_cuda, test/test_view_ops.py::TestViewOpsCUDA::test_unsqueeze_view_cuda, test/test_view_ops.py::TestViewOpsCUDA::test_view_as_complex_cuda, test/test_view_ops.py::TestViewOpsCUDA::test_view_as_real_cuda_complex128, test/test_view_ops.py::TestViewOpsCUDA::test_view_as_real_cuda_complex32, test/test_view_ops.py::TestViewOpsCUDA::test_view_as_real_cuda_complex64, test/test_view_ops.py::TestViewOpsCUDA::test_view_as_view_cuda, test/test_view_ops.py::TestViewOpsCUDA::test_view_copy_cuda, test/test_view_ops.py::TestViewOpsCUDA::test_view_copy_out_cuda, test/test_view_ops.py::TestViewOpsCUDA::test_view_copy_output_contiguous_cuda, test/test_view_ops.py::TestViewOpsCUDA::test_view_dtype_new_cuda_bool, test/test_view_ops.py::TestViewOpsCUDA::test_view_dtype_new_cuda_complex128, test/test_view_ops.py::TestViewOpsCUDA::test_view_dtype_new_cuda_complex64, test/test_view_ops.py::TestViewOpsCUDA::test_view_dtype_new_cuda_float16, test/test_view_ops.py::TestViewOpsCUDA::test_view_dtype_new_cuda_float32, test/test_view_ops.py::TestViewOpsCUDA::test_view_dtype_new_cuda_float64, test/test_view_ops.py::TestViewOpsCUDA::test_view_dtype_new_cuda_int16, test/test_view_ops.py::TestViewOpsCUDA::test_view_dtype_new_cuda_int32, test/test_view_ops.py::TestViewOpsCUDA::test_view_dtype_new_cuda_int64, test/test_view_ops.py::TestViewOpsCUDA::test_view_dtype_new_cuda_int8, test/test_view_ops.py::TestViewOpsCUDA::test_view_dtype_new_cuda_uint8, test/test_view_ops.py::TestViewOpsCUDA::test_view_dtype_upsize_errors_cuda_bfloat16, test/test_view_ops.py::TestViewOpsCUDA::test_view_dtype_upsize_errors_cuda_bool, test/test_view_ops.py::TestViewOpsCUDA::test_view_dtype_upsize_errors_cuda_complex128, test/test_view_ops.py::TestViewOpsCUDA::test_view_dtype_upsize_errors_cuda_complex64, test/test_view_ops.py::TestViewOpsCUDA::test_view_dtype_upsize_errors_cuda_float16, test/test_view_ops.py::TestViewOpsCUDA::test_view_dtype_upsize_errors_cuda_float32, test/test_view_ops.py::TestViewOpsCUDA::test_view_dtype_upsize_errors_cuda_float64, test/test_view_ops.py::TestViewOpsCUDA::test_view_dtype_upsize_errors_cuda_int16, test/test_view_ops.py::TestViewOpsCUDA::test_view_dtype_upsize_errors_cuda_int32, test/test_view_ops.py::TestViewOpsCUDA::test_view_dtype_upsize_errors_cuda_int64, test/test_view_ops.py::TestViewOpsCUDA::test_view_dtype_upsize_errors_cuda_int8, test/test_view_ops.py::TestViewOpsCUDA::test_view_dtype_upsize_errors_cuda_uint8, test/test_view_ops.py::TestViewOpsCUDA::test_view_tensor_dsplit_cuda_bfloat16, test/test_view_ops.py::TestViewOpsCUDA::test_view_tensor_dsplit_cuda_bool, test/test_view_ops.py::TestViewOpsCUDA::test_view_tensor_dsplit_cuda_complex128, test/test_view_ops.py::TestViewOpsCUDA::test_view_tensor_dsplit_cuda_complex64, test/test_view_ops.py::TestViewOpsCUDA::test_view_tensor_dsplit_cuda_float16, test/test_view_ops.py::TestViewOpsCUDA::test_view_tensor_dsplit_cuda_float32, test/test_view_ops.py::TestViewOpsCUDA::test_view_tensor_dsplit_cuda_float64, test/test_view_ops.py::TestViewOpsCUDA::test_view_tensor_dsplit_cuda_int16, test/test_view_ops.py::TestViewOpsCUDA::test_view_tensor_dsplit_cuda_int32, test/test_view_ops.py::TestViewOpsCUDA::test_view_tensor_dsplit_cuda_int64, test/test_view_ops.py::TestViewOpsCUDA::test_view_tensor_dsplit_cuda_int8, test/test_view_ops.py::TestViewOpsCUDA::test_view_tensor_dsplit_cuda_uint8, test/test_view_ops.py::TestViewOpsCUDA::test_view_tensor_hsplit_cuda_bfloat16, test/test_view_ops.py::TestViewOpsCUDA::test_view_tensor_hsplit_cuda_bool, test/test_view_ops.py::TestViewOpsCUDA::test_view_tensor_hsplit_cuda_complex128, test/test_view_ops.py::TestViewOpsCUDA::test_view_tensor_hsplit_cuda_complex64, test/test_view_ops.py::TestViewOpsCUDA::test_view_tensor_hsplit_cuda_float16, test/test_view_ops.py::TestViewOpsCUDA::test_view_tensor_hsplit_cuda_float32, test/test_view_ops.py::TestViewOpsCUDA::test_view_tensor_hsplit_cuda_float64, test/test_view_ops.py::TestViewOpsCUDA::test_view_tensor_hsplit_cuda_int16, test/test_view_ops.py::TestViewOpsCUDA::test_view_tensor_hsplit_cuda_int32, test/test_view_ops.py::TestViewOpsCUDA::test_view_tensor_hsplit_cuda_int64, test/test_view_ops.py::TestViewOpsCUDA::test_view_tensor_hsplit_cuda_int8, test/test_view_ops.py::TestViewOpsCUDA::test_view_tensor_hsplit_cuda_uint8, test/test_view_ops.py::TestViewOpsCUDA::test_view_tensor_split_cuda_bfloat16, test/test_view_ops.py::TestViewOpsCUDA::test_view_tensor_split_cuda_bool, test/test_view_ops.py::TestViewOpsCUDA::test_view_tensor_split_cuda_complex128, test/test_view_ops.py::TestViewOpsCUDA::test_view_tensor_split_cuda_complex64, test/test_view_ops.py::TestViewOpsCUDA::test_view_tensor_split_cuda_float16, test/test_view_ops.py::TestViewOpsCUDA::test_view_tensor_split_cuda_float32, test/test_view_ops.py::TestViewOpsCUDA::test_view_tensor_split_cuda_float64, test/test_view_ops.py::TestViewOpsCUDA::test_view_tensor_split_cuda_int16, test/test_view_ops.py::TestViewOpsCUDA::test_view_tensor_split_cuda_int32, test/test_view_ops.py::TestViewOpsCUDA::test_view_tensor_split_cuda_int64, test/test_view_ops.py::TestViewOpsCUDA::test_view_tensor_split_cuda_int8, test/test_view_ops.py::TestViewOpsCUDA::test_view_tensor_split_cuda_uint8, test/test_view_ops.py::TestViewOpsCUDA::test_view_tensor_vsplit_cuda_bfloat16, test/test_view_ops.py::TestViewOpsCUDA::test_view_tensor_vsplit_cuda_bool, test/test_view_ops.py::TestViewOpsCUDA::test_view_tensor_vsplit_cuda_complex128, test/test_view_ops.py::TestViewOpsCUDA::test_view_tensor_vsplit_cuda_complex64, test/test_view_ops.py::TestViewOpsCUDA::test_view_tensor_vsplit_cuda_float16, test/test_view_ops.py::TestViewOpsCUDA::test_view_tensor_vsplit_cuda_float32, test/test_view_ops.py::TestViewOpsCUDA::test_view_tensor_vsplit_cuda_float64, test/test_view_ops.py::TestViewOpsCUDA::test_view_tensor_vsplit_cuda_int16, test/test_view_ops.py::TestViewOpsCUDA::test_view_tensor_vsplit_cuda_int32, test/test_view_ops.py::TestViewOpsCUDA::test_view_tensor_vsplit_cuda_int64, test/test_view_ops.py::TestViewOpsCUDA::test_view_tensor_vsplit_cuda_int8, test/test_view_ops.py::TestViewOpsCUDA::test_view_tensor_vsplit_cuda_uint8, test/test_view_ops.py::TestViewOpsCUDA::test_view_view_cuda, test/test_view_ops.py::TestOldViewOpsCUDA::test_T_cuda, test/test_view_ops.py::TestOldViewOpsCUDA::test_as_strided_overflow_storage_offset_cuda, test/test_view_ops.py::TestOldViewOpsCUDA::test_atleast_cuda_complex128, test/test_view_ops.py::TestOldViewOpsCUDA::test_atleast_cuda_complex64, test/test_view_ops.py::TestOldViewOpsCUDA::test_atleast_cuda_float16, test/test_view_ops.py::TestOldViewOpsCUDA::test_atleast_cuda_float32, test/test_view_ops.py::TestOldViewOpsCUDA::test_atleast_cuda_float64, test/test_view_ops.py::TestOldViewOpsCUDA::test_atleast_cuda_int16, test/test_view_ops.py::TestOldViewOpsCUDA::test_atleast_cuda_int32, test/test_view_ops.py::TestOldViewOpsCUDA::test_atleast_cuda_int64, test/test_view_ops.py::TestOldViewOpsCUDA::test_atleast_cuda_int8, test/test_view_ops.py::TestOldViewOpsCUDA::test_atleast_cuda_uint8, test/test_view_ops.py::TestOldViewOpsCUDA::test_atleast_gradient_cuda, test/test_view_ops.py::TestOldViewOpsCUDA::test_big_transpose_cuda, test/test_view_ops.py::TestOldViewOpsCUDA::test_broadcast_shapes_cuda, test/test_view_ops.py::TestOldViewOpsCUDA::test_broadcast_tensors_cuda_float32, test/test_view_ops.py::TestOldViewOpsCUDA::test_broadcast_to_cuda_bool, test/test_view_ops.py::TestOldViewOpsCUDA::test_broadcast_to_cuda_complex128, test/test_view_ops.py::TestOldViewOpsCUDA::test_broadcast_to_cuda_complex64, test/test_view_ops.py::TestOldViewOpsCUDA::test_broadcast_to_cuda_float16, test/test_view_ops.py::TestOldViewOpsCUDA::test_broadcast_to_cuda_float32, test/test_view_ops.py::TestOldViewOpsCUDA::test_broadcast_to_cuda_float64, test/test_view_ops.py::TestOldViewOpsCUDA::test_broadcast_to_cuda_int16, test/test_view_ops.py::TestOldViewOpsCUDA::test_broadcast_to_cuda_int32, test/test_view_ops.py::TestOldViewOpsCUDA::test_broadcast_to_cuda_int64, test/test_view_ops.py::TestOldViewOpsCUDA::test_broadcast_to_cuda_int8, test/test_view_ops.py::TestOldViewOpsCUDA::test_broadcast_to_cuda_uint8, test/test_view_ops.py::TestOldViewOpsCUDA::test_chunk_cuda, test/test_view_ops.py::TestOldViewOpsCUDA::test_conj_neg_view_numpy_error_cuda, test/test_view_ops.py::TestOldViewOpsCUDA::test_contiguous_cuda, test/test_view_ops.py::TestOldViewOpsCUDA::test_crow_col_indices_cuda, test/test_view_ops.py::TestOldViewOpsCUDA::test_empty_reshape_cuda, test/test_view_ops.py::TestOldViewOpsCUDA::test_expand_cuda, test/test_view_ops.py::TestOldViewOpsCUDA::test_flatten_cuda, test/test_view_ops.py::TestOldViewOpsCUDA::test_memory_format_resize__cuda, test/test_view_ops.py::TestOldViewOpsCUDA::test_memory_format_resize_as_cuda, test/test_view_ops.py::TestOldViewOpsCUDA::test_narrow_cuda, test/test_view_ops.py::TestOldViewOpsCUDA::test_narrow_tensor_cuda, test/test_view_ops.py::TestOldViewOpsCUDA::test_python_types_cuda, test/test_view_ops.py::TestOldViewOpsCUDA::test_ravel_cuda, test/test_view_ops.py::TestOldViewOpsCUDA::test_reshape_cuda, test/test_view_ops.py::TestOldViewOpsCUDA::test_reshape_view_semantics_cuda_bfloat16, test/test_view_ops.py::TestOldViewOpsCUDA::test_reshape_view_semantics_cuda_bool, test/test_view_ops.py::TestOldViewOpsCUDA::test_reshape_view_semantics_cuda_complex128, test/test_view_ops.py::TestOldViewOpsCUDA::test_reshape_view_semantics_cuda_complex64, test/test_view_ops.py::TestOldViewOpsCUDA::test_reshape_view_semantics_cuda_float16, test/test_view_ops.py::TestOldViewOpsCUDA::test_reshape_view_semantics_cuda_float32, test/test_view_ops.py::TestOldViewOpsCUDA::test_reshape_view_semantics_cuda_float64, test/test_view_ops.py::TestOldViewOpsCUDA::test_reshape_view_semantics_cuda_int16, test/test_view_ops.py::TestOldViewOpsCUDA::test_reshape_view_semantics_cuda_int32, test/test_view_ops.py::TestOldViewOpsCUDA::test_reshape_view_semantics_cuda_int64, test/test_view_ops.py::TestOldViewOpsCUDA::test_reshape_view_semantics_cuda_int8, test/test_view_ops.py::TestOldViewOpsCUDA::test_reshape_view_semantics_cuda_uint8, test/test_view_ops.py::TestOldViewOpsCUDA::test_resize_all_dtypes_and_devices_cuda, test/test_view_ops.py::TestOldViewOpsCUDA::test_resize_as_all_dtypes_and_devices_cuda, test/test_view_ops.py::TestOldViewOpsCUDA::test_resize_as_preserves_strides_cuda, test/test_view_ops.py::TestOldViewOpsCUDA::test_resize_overflow_cuda, test/test_view_ops.py::TestOldViewOpsCUDA::test_split_cuda, test/test_view_ops.py::TestOldViewOpsCUDA::test_t_cuda, test/test_view_ops.py::TestOldViewOpsCUDA::test_tensor_split_errors_cuda, test/test_view_ops.py::TestOldViewOpsCUDA::test_tensor_split_indices_cuda_bool, test/test_view_ops.py::TestOldViewOpsCUDA::test_tensor_split_indices_cuda_complex128, test/test_view_ops.py::TestOldViewOpsCUDA::test_tensor_split_indices_cuda_complex64, test/test_view_ops.py::TestOldViewOpsCUDA::test_tensor_split_indices_cuda_float16, test/test_view_ops.py::TestOldViewOpsCUDA::test_tensor_split_indices_cuda_float32, test/test_view_ops.py::TestOldViewOpsCUDA::test_tensor_split_indices_cuda_float64, test/test_view_ops.py::TestOldViewOpsCUDA::test_tensor_split_indices_cuda_int16, test/test_view_ops.py::TestOldViewOpsCUDA::test_tensor_split_indices_cuda_int32, test/test_view_ops.py::TestOldViewOpsCUDA::test_tensor_split_indices_cuda_int64, test/test_view_ops.py::TestOldViewOpsCUDA::test_tensor_split_indices_cuda_int8, test/test_view_ops.py::TestOldViewOpsCUDA::test_tensor_split_indices_cuda_uint8, test/test_view_ops.py::TestOldViewOpsCUDA::test_tensor_split_sections_cuda_bool, test/test_view_ops.py::TestOldViewOpsCUDA::test_tensor_split_sections_cuda_complex128, test/test_view_ops.py::TestOldViewOpsCUDA::test_tensor_split_sections_cuda_complex64, test/test_view_ops.py::TestOldViewOpsCUDA::test_tensor_split_sections_cuda_float16, test/test_view_ops.py::TestOldViewOpsCUDA::test_tensor_split_sections_cuda_float32, test/test_view_ops.py::TestOldViewOpsCUDA::test_tensor_split_sections_cuda_float64, test/test_view_ops.py::TestOldViewOpsCUDA::test_tensor_split_sections_cuda_int16, test/test_view_ops.py::TestOldViewOpsCUDA::test_tensor_split_sections_cuda_int32, test/test_view_ops.py::TestOldViewOpsCUDA::test_tensor_split_sections_cuda_int64, test/test_view_ops.py::TestOldViewOpsCUDA::test_tensor_split_sections_cuda_int8, test/test_view_ops.py::TestOldViewOpsCUDA::test_tensor_split_sections_cuda_uint8, test/test_view_ops.py::TestOldViewOpsCUDA::test_transpose_invalid_cuda_complex128, test/test_view_ops.py::TestOldViewOpsCUDA::test_transpose_invalid_cuda_float32, test/test_view_ops.py::TestOldViewOpsCUDA::test_transpose_invalid_cuda_int64, test/test_view_ops.py::TestOldViewOpsCUDA::test_transpose_vs_numpy_cuda_complex128, test/test_view_ops.py::TestOldViewOpsCUDA::test_transpose_vs_numpy_cuda_float32, test/test_view_ops.py::TestOldViewOpsCUDA::test_transpose_vs_numpy_cuda_int64, test/test_view_ops.py::TestOldViewOpsCUDA::test_transposes_cuda_bfloat16, test/test_view_ops.py::TestOldViewOpsCUDA::test_transposes_cuda_bool, test/test_view_ops.py::TestOldViewOpsCUDA::test_transposes_cuda_complex128, test/test_view_ops.py::TestOldViewOpsCUDA::test_transposes_cuda_complex64, test/test_view_ops.py::TestOldViewOpsCUDA::test_transposes_cuda_float16, test/test_view_ops.py::TestOldViewOpsCUDA::test_transposes_cuda_float32, test/test_view_ops.py::TestOldViewOpsCUDA::test_transposes_cuda_float64, test/test_view_ops.py::TestOldViewOpsCUDA::test_transposes_cuda_int16, test/test_view_ops.py::TestOldViewOpsCUDA::test_transposes_cuda_int32, test/test_view_ops.py::TestOldViewOpsCUDA::test_transposes_cuda_int64, test/test_view_ops.py::TestOldViewOpsCUDA::test_transposes_cuda_int8, test/test_view_ops.py::TestOldViewOpsCUDA::test_transposes_cuda_uint8, test/test_view_ops.py::TestOldViewOpsCUDA::test_transposes_errors_cuda_bfloat16, test/test_view_ops.py::TestOldViewOpsCUDA::test_transposes_errors_cuda_bool, test/test_view_ops.py::TestOldViewOpsCUDA::test_transposes_errors_cuda_complex128, test/test_view_ops.py::TestOldViewOpsCUDA::test_transposes_errors_cuda_complex64, test/test_view_ops.py::TestOldViewOpsCUDA::test_transposes_errors_cuda_float16, test/test_view_ops.py::TestOldViewOpsCUDA::test_transposes_errors_cuda_float32, test/test_view_ops.py::TestOldViewOpsCUDA::test_transposes_errors_cuda_float64, test/test_view_ops.py::TestOldViewOpsCUDA::test_transposes_errors_cuda_int16, test/test_view_ops.py::TestOldViewOpsCUDA::test_transposes_errors_cuda_int32, test/test_view_ops.py::TestOldViewOpsCUDA::test_transposes_errors_cuda_int64, test/test_view_ops.py::TestOldViewOpsCUDA::test_transposes_errors_cuda_int8, test/test_view_ops.py::TestOldViewOpsCUDA::test_transposes_errors_cuda_uint8, test/test_view_ops.py::TestOldViewOpsCUDA::test_unsqueeze_cuda, test/test_view_ops.py::TestOldViewOpsCUDA::test_view_all_dtypes_and_devices_cuda, test/test_view_ops.py::TestOldViewOpsCUDA::test_view_cuda, test/test_view_ops.py::TestOldViewOpsCUDA::test_view_empty_cuda 2025-12-04T13:53:38.1930868Z 2025-12-04T13:53:38.1931059Z Finished test_view_ops 1/1 ... [2025-12-04 13:53:38.181896][15266.124111563], took 0.28min 2025-12-04T13:53:38.2144837Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/test_view_ops/test_view_ops-6b7df7431405f809.xml 2025-12-04T13:53:38.2843909Z Running test_type_info 1/1 ... [2025-12-04 13:53:38.284152][15266.226370092] 2025-12-04T13:53:38.2844325Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2025-12-04T13:53:38.2847245Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'test_type_info.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-12-04 13:53:38.284472] 2025-12-04T13:53:41.6047065Z 2025-12-04T13:53:41.6047826Z test_type_info 1/1 was successful, full logs can be found in artifacts with path test/test-reports/test_type_info_1.1_6aa92fc777d561da_.log 2025-12-04T13:53:41.6049356Z Running 5 items in this shard: test/test_type_info.py::TestDTypeInfo::test_finfo, test/test_type_info.py::TestDTypeInfo::test_iinfo, test/test_type_info.py::TestDTypeInfo::test_invalid_input, test/test_type_info.py::TestDTypeInfo::test_to_complex, test/test_type_info.py::TestDTypeInfo::test_to_real 2025-12-04T13:53:41.6050397Z 2025-12-04T13:53:41.6050891Z Finished test_type_info 1/1 ... [2025-12-04 13:53:41.604485][15269.546699215], took 0.06min 2025-12-04T13:53:41.6360763Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/test_type_info/test_type_info-d4f338100cffb528.xml 2025-12-04T13:53:41.6663907Z Running functorch/test_aotdispatch 1/1 ... [2025-12-04 13:53:41.666161][15269.6083791] 2025-12-04T13:53:41.6664512Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2025-12-04T13:53:41.6667333Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'functorch/test_aotdispatch.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-12-04 13:53:41.666474] 2025-12-04T13:55:29.9536413Z 2025-12-04T13:55:29.9537688Z functorch/test_aotdispatch 1/1 was successful, full logs can be found in artifacts with path test/test-reports/functorch.test_aotdispatch_1.1_a56217decbf84b1c_.log 2025-12-04T13:55:29.9699442Z Running 537 items in this shard: test/functorch/test_aotdispatch.py::TestAOTAutograd::test_alias_of_intermediate_detach_backend_aot_eager_view_replay_for_aliased_outputs_False_dynamic_shapes_False, test/functorch/test_aotdispatch.py::TestAOTAutograd::test_alias_of_intermediate_detach_backend_aot_eager_view_replay_for_aliased_outputs_False_dynamic_shapes_True, test/functorch/test_aotdispatch.py::TestAOTAutograd::test_alias_of_intermediate_detach_backend_aot_eager_view_replay_for_aliased_outputs_True_dynamic_shapes_False, test/functorch/test_aotdispatch.py::TestAOTAutograd::test_alias_of_intermediate_detach_backend_aot_eager_view_replay_for_aliased_outputs_True_dynamic_shapes_True, test/functorch/test_aotdispatch.py::TestAOTAutograd::test_alias_of_intermediate_detach_backend_inductor_view_replay_for_aliased_outputs_False_dynamic_shapes_False, test/functorch/test_aotdispatch.py::TestAOTAutograd::test_alias_of_intermediate_detach_backend_inductor_view_replay_for_aliased_outputs_False_dynamic_shapes_True, test/functorch/test_aotdispatch.py::TestAOTAutograd::test_alias_of_intermediate_detach_backend_inductor_view_replay_for_aliased_outputs_True_dynamic_shapes_False, test/functorch/test_aotdispatch.py::TestAOTAutograd::test_alias_of_intermediate_detach_backend_inductor_view_replay_for_aliased_outputs_True_dynamic_shapes_True, test/functorch/test_aotdispatch.py::TestAOTAutograd::test_autocast_disable_guard, test/functorch/test_aotdispatch.py::TestAOTAutograd::test_backward_mutation_data, test/functorch/test_aotdispatch.py::TestAOTAutograd::test_backward_mutation_forward_inputs, test/functorch/test_aotdispatch.py::TestAOTAutograd::test_backward_mutation_forward_inputs_create_graph, test/functorch/test_aotdispatch.py::TestAOTAutograd::test_backward_mutation_metadata, test/functorch/test_aotdispatch.py::TestAOTAutograd::test_backward_mutation_on_grad_out, test/functorch/test_aotdispatch.py::TestAOTAutograd::test_backward_pass_autocast_custom, test/functorch/test_aotdispatch.py::TestAOTAutograd::test_backward_pass_autocast_off, test/functorch/test_aotdispatch.py::TestAOTAutograd::test_backward_pass_autocast_on, test/functorch/test_aotdispatch.py::TestAOTAutograd::test_batch_norm_amp, test/functorch/test_aotdispatch.py::TestAOTAutograd::test_batchnorm, test/functorch/test_aotdispatch.py::TestAOTAutograd::test_batchnorm_inference, test/functorch/test_aotdispatch.py::TestAOTAutograd::test_buffer_batch_norm, test/functorch/test_aotdispatch.py::TestAOTAutograd::test_buffer_copied_in_graph, test/functorch/test_aotdispatch.py::TestAOTAutograd::test_buffer_copied_in_graph_with_different_shapes, test/functorch/test_aotdispatch.py::TestAOTAutograd::test_compilation_context, test/functorch/test_aotdispatch.py::TestAOTAutograd::test_complex_linear, test/functorch/test_aotdispatch.py::TestAOTAutograd::test_composite_impl_compile, test/functorch/test_aotdispatch.py::TestAOTAutograd::test_custom_autograd, test/functorch/test_aotdispatch.py::TestAOTAutograd::test_custom_tensor_metadata, test/functorch/test_aotdispatch.py::TestAOTAutograd::test_default_partitioner_saves_symints_not_tensors_for_bw, test/functorch/test_aotdispatch.py::TestAOTAutograd::test_dupe_arg, test/functorch/test_aotdispatch.py::TestAOTAutograd::test_dupe_arg_returned_as_output, test/functorch/test_aotdispatch.py::TestAOTAutograd::test_dupe_arg_torture, test/functorch/test_aotdispatch.py::TestAOTAutograd::test_duplicated_arguments_on_tensor_overlap, test/functorch/test_aotdispatch.py::TestAOTAutograd::test_dynamic_output_aliases_input_view_meta_replay, test/functorch/test_aotdispatch.py::TestAOTAutograd::test_dynamic_shape_output_not_in_bw_graph, test/functorch/test_aotdispatch.py::TestAOTAutograd::test_embedding_bag_view_dynamic, test/functorch/test_aotdispatch.py::TestAOTAutograd::test_fw_bw_mutation_no_functionalization1, test/functorch/test_aotdispatch.py::TestAOTAutograd::test_fw_bw_mutation_no_functionalization2, test/functorch/test_aotdispatch.py::TestAOTAutograd::test_grad_context, test/functorch/test_aotdispatch.py::TestAOTAutograd::test_inference_mode, test/functorch/test_aotdispatch.py::TestAOTAutograd::test_inner_grad, test/functorch/test_aotdispatch.py::TestAOTAutograd::test_input_aliased_with_mutation_output_alias, test/functorch/test_aotdispatch.py::TestAOTAutograd::test_input_data_and_metadata_mutation, test/functorch/test_aotdispatch.py::TestAOTAutograd::test_input_data_and_metadata_mutation_aliases_other_input, test/functorch/test_aotdispatch.py::TestAOTAutograd::test_input_inplace_requires_grad_true, test/functorch/test_aotdispatch.py::TestAOTAutograd::test_input_metadata_mutation_aliases, test/functorch/test_aotdispatch.py::TestAOTAutograd::test_input_mutation_alias_everything, test/functorch/test_aotdispatch.py::TestAOTAutograd::test_input_mutation_aliases_and_none_require_gradients, test/functorch/test_aotdispatch.py::TestAOTAutograd::test_input_mutation_aliases_and_output_alias, test/functorch/test_aotdispatch.py::TestAOTAutograd::test_input_mutation_aliases_bases_out_of_order, test/functorch/test_aotdispatch.py::TestAOTAutograd::test_input_mutation_aliases_other_input, test/functorch/test_aotdispatch.py::TestAOTAutograd::test_input_mutation_aliases_other_input2, test/functorch/test_aotdispatch.py::TestAOTAutograd::test_input_mutation_and_output_view, test/functorch/test_aotdispatch.py::TestAOTAutograd::test_input_mutation_batchnorm, test/functorch/test_aotdispatch.py::TestAOTAutograd::test_input_mutation_false_aliasing, test/functorch/test_aotdispatch.py::TestAOTAutograd::test_input_mutation_hidden_from_autograd_aliasing, test/functorch/test_aotdispatch.py::TestAOTAutograd::test_input_mutation_is_output, test/functorch/test_aotdispatch.py::TestAOTAutograd::test_input_mutation_metadata, test/functorch/test_aotdispatch.py::TestAOTAutograd::test_input_mutation_metadata2, test/functorch/test_aotdispatch.py::TestAOTAutograd::test_input_mutation_modifies_autograd_meta_of_aliases, test/functorch/test_aotdispatch.py::TestAOTAutograd::test_input_mutation_multiple, test/functorch/test_aotdispatch.py::TestAOTAutograd::test_input_mutation_noncontiguous, test/functorch/test_aotdispatch.py::TestAOTAutograd::test_input_mutation_output_view_multiple, test/functorch/test_aotdispatch.py::TestAOTAutograd::test_input_mutation_requires_grad_detach, test/functorch/test_aotdispatch.py::TestAOTAutograd::test_input_mutation_requires_grad_no_grad, test/functorch/test_aotdispatch.py::TestAOTAutograd::test_input_mutation_requires_grad_no_grad_detach_mixed, test/functorch/test_aotdispatch.py::TestAOTAutograd::test_input_mutation_requires_grad_no_grad_inference_graph, test/functorch/test_aotdispatch.py::TestAOTAutograd::test_input_mutation_return, test/functorch/test_aotdispatch.py::TestAOTAutograd::test_input_mutation_set__input_mutation, test/functorch/test_aotdispatch.py::TestAOTAutograd::test_input_mutation_set__nop, test/functorch/test_aotdispatch.py::TestAOTAutograd::test_input_mutation_simple, test/functorch/test_aotdispatch.py::TestAOTAutograd::test_input_mutation_simple_with_none_and_nontensor, test/functorch/test_aotdispatch.py::TestAOTAutograd::test_input_mutation_storage_resize_before_set_, test/functorch/test_aotdispatch.py::TestAOTAutograd::test_input_mutation_storage_resize_down, test/functorch/test_aotdispatch.py::TestAOTAutograd::test_input_mutation_storage_resize_down_and_set_, test/functorch/test_aotdispatch.py::TestAOTAutograd::test_input_mutation_storage_resize_up, test/functorch/test_aotdispatch.py::TestAOTAutograd::test_input_output_aliase_custom_autograd_function, test/functorch/test_aotdispatch.py::TestAOTAutograd::test_input_output_view_metadata_mutate_multiple, test/functorch/test_aotdispatch.py::TestAOTAutograd::test_input_output_view_mutate_multiple, test/functorch/test_aotdispatch.py::TestAOTAutograd::test_input_output_view_simple, test/functorch/test_aotdispatch.py::TestAOTAutograd::test_invalid_dupe, test/functorch/test_aotdispatch.py::TestAOTAutograd::test_invalid_dupe_fake, test/functorch/test_aotdispatch.py::TestAOTAutograd::test_invalid_dupe_left_bias, test/functorch/test_aotdispatch.py::TestAOTAutograd::test_invalid_requires_grad, test/functorch/test_aotdispatch.py::TestAOTAutograd::test_invalid_requires_grad_fake, test/functorch/test_aotdispatch.py::TestAOTAutograd::test_list_codegen, test/functorch/test_aotdispatch.py::TestAOTAutograd::test_mark_activations_dynamic, test/functorch/test_aotdispatch.py::TestAOTAutograd::test_mark_activations_dynamic_with_nested, test/functorch/test_aotdispatch.py::TestAOTAutograd::test_mark_outputs_dynamic_use_autograd_False, test/functorch/test_aotdispatch.py::TestAOTAutograd::test_mark_outputs_dynamic_use_autograd_True, test/functorch/test_aotdispatch.py::TestAOTAutograd::test_mem_leak_from_save_for_bw, test/functorch/test_aotdispatch.py::TestAOTAutograd::test_module, test/functorch/test_aotdispatch.py::TestAOTAutograd::test_multi_output, test/functorch/test_aotdispatch.py::TestAOTAutograd::test_multi_output_list, test/functorch/test_aotdispatch.py::TestAOTAutograd::test_mutates_input_noncontiguous, test/functorch/test_aotdispatch.py::TestAOTAutograd::test_nested_subclasses, test/functorch/test_aotdispatch.py::TestAOTAutograd::test_nested_subclasses_complicated_inps, test/functorch/test_aotdispatch.py::TestAOTAutograd::test_nested_subclasses_complicated_inps_mixed, test/functorch/test_aotdispatch.py::TestAOTAutograd::test_nested_subclasses_non_homogenous, test/functorch/test_aotdispatch.py::TestAOTAutograd::test_nested_subclasses_non_nested_grad, test/functorch/test_aotdispatch.py::TestAOTAutograd::test_new_inp_requires_grad_now, test/functorch/test_aotdispatch.py::TestAOTAutograd::test_no_grad_input_output, test/functorch/test_aotdispatch.py::TestAOTAutograd::test_non_tensor_and_none_inputs, test/functorch/test_aotdispatch.py::TestAOTAutograd::test_nonidempotent_amp, test/functorch/test_aotdispatch.py::TestAOTAutograd::test_output_aliases_input_multi_output_view, test/functorch/test_aotdispatch.py::TestAOTAutograd::test_output_aliases_input_multi_output_view_should_raise_autograd_error, test/functorch/test_aotdispatch.py::TestAOTAutograd::test_output_aliases_input_view_meta_replay, test/functorch/test_aotdispatch.py::TestAOTAutograd::test_output_aliases_intermediate_and_returned, test/functorch/test_aotdispatch.py::TestAOTAutograd::test_output_aliases_intermediate_and_returned_different_grad, test/functorch/test_aotdispatch.py::TestAOTAutograd::test_output_aliases_intermediate_and_returned_flipped, test/functorch/test_aotdispatch.py::TestAOTAutograd::test_output_aliases_intermediate_inplace_view, test/functorch/test_aotdispatch.py::TestAOTAutograd::test_output_aliases_intermediate_inplace_view_and_view, test/functorch/test_aotdispatch.py::TestAOTAutograd::test_output_aliases_intermediate_inplace_view_with_detach, test/functorch/test_aotdispatch.py::TestAOTAutograd::test_output_aliases_intermediate_multi_output_view, test/functorch/test_aotdispatch.py::TestAOTAutograd::test_output_aliases_intermediate_multiple, test/functorch/test_aotdispatch.py::TestAOTAutograd::test_output_aliases_intermediate_multiple_mixed, test/functorch/test_aotdispatch.py::TestAOTAutograd::test_output_aliases_intermediate_mutation_linear, test/functorch/test_aotdispatch.py::TestAOTAutograd::test_output_aliases_intermediate_no_grad, test/functorch/test_aotdispatch.py::TestAOTAutograd::test_output_aliases_intermediate_returned_multiple_times, test/functorch/test_aotdispatch.py::TestAOTAutograd::test_output_aliases_intermediate_single, test/functorch/test_aotdispatch.py::TestAOTAutograd::test_output_aliases_intermediate_view_meta_replay, test/functorch/test_aotdispatch.py::TestAOTAutograd::test_output_aliases_multiple_inputs_get_correct_one, test/functorch/test_aotdispatch.py::TestAOTAutograd::test_output_aliases_output_view_meta_replay, test/functorch/test_aotdispatch.py::TestAOTAutograd::test_output_all_alias_types, test/functorch/test_aotdispatch.py::TestAOTAutograd::test_output_dict, test/functorch/test_aotdispatch.py::TestAOTAutograd::test_output_op_depending_on_symint, test/functorch/test_aotdispatch.py::TestAOTAutograd::test_outputs_are_aliased, test/functorch/test_aotdispatch.py::TestAOTAutograd::test_real_weights_in_symbolic_mode, test/functorch/test_aotdispatch.py::TestAOTAutograd::test_real_weights_in_symbolic_mode_with_inplace_ops, test/functorch/test_aotdispatch.py::TestAOTAutograd::test_saved_tensors_hooks_mutations_raise, test/functorch/test_aotdispatch.py::TestAOTAutograd::test_set__and_data_mutation_bad, test/functorch/test_aotdispatch.py::TestAOTAutograd::test_set__and_data_mutation_good, test/functorch/test_aotdispatch.py::TestAOTAutograd::test_set__not_allowed, test/functorch/test_aotdispatch.py::TestAOTAutograd::test_set__steals_view_chain, test/functorch/test_aotdispatch.py::TestAOTAutograd::test_single_output, test/functorch/test_aotdispatch.py::TestAOTAutograd::test_some_output_requires_grad_input_doesnt, test/functorch/test_aotdispatch.py::TestAOTAutograd::test_some_outputs_dont_require_grad_non_view, test/functorch/test_aotdispatch.py::TestAOTAutograd::test_some_outputs_dont_require_grad_view, test/functorch/test_aotdispatch.py::TestAOTAutograd::test_squeeze_mutation, test/functorch/test_aotdispatch.py::TestAOTAutograd::test_subclass_metadata_mutation_req_grad_False, test/functorch/test_aotdispatch.py::TestAOTAutograd::test_subclass_metadata_mutation_req_grad_True, test/functorch/test_aotdispatch.py::TestAOTAutograd::test_subclasses_mixed, test/functorch/test_aotdispatch.py::TestAOTAutograd::test_subclasses_mixed_mode, test/functorch/test_aotdispatch.py::TestAOTAutograd::test_synthetic_base_base_attribute_is_none, test/functorch/test_aotdispatch.py::TestAOTAutograd::test_view_and_inplace_view, test/functorch/test_aotdispatch.py::TestAOTAutograd::test_view_detach, test/functorch/test_aotdispatch.py::TestAOTExport::test_aot_export_ban_dropout_mut_pre_dispatch, test/functorch/test_aotdispatch.py::TestAOTExport::test_aot_export_forward_mutation_multiple_mut, test/functorch/test_aotdispatch.py::TestAOTExport::test_aot_export_forward_mutation_no_buffer_mut, test/functorch/test_aotdispatch.py::TestAOTExport::test_aot_export_functionalized_rng_banned, test/functorch/test_aotdispatch.py::TestAOTExport::test_aot_export_input_dupes_banned, test/functorch/test_aotdispatch.py::TestAOTExport::test_aot_export_input_mutation, test/functorch/test_aotdispatch.py::TestAOTExport::test_aot_export_input_mutation_on_input_requiring_grad_banned, test/functorch/test_aotdispatch.py::TestAOTExport::test_aot_export_input_mutation_on_parameter_banned, test/functorch/test_aotdispatch.py::TestAOTExport::test_aot_export_metadata_mutation_banned, test/functorch/test_aotdispatch.py::TestAOTExport::test_aot_export_module_joint, test/functorch/test_aotdispatch.py::TestAOTExport::test_aot_export_multiple_outputs_require_grad_banned, test/functorch/test_aotdispatch.py::TestAOTExport::test_aot_export_predispatch_buffer_mutation_metadata, test/functorch/test_aotdispatch.py::TestAOTExport::test_aot_export_predispatch_composite_implicit_inplace, test/functorch/test_aotdispatch.py::TestAOTExport::test_aot_export_predispatch_composite_implicit_linear, test/functorch/test_aotdispatch.py::TestAOTExport::test_aot_export_predispatch_contiguous, test/functorch/test_aotdispatch.py::TestAOTExport::test_aot_export_predispatch_conv_and_bn, test/functorch/test_aotdispatch.py::TestAOTExport::test_aot_export_predispatch_func_composite_implicit, test/functorch/test_aotdispatch.py::TestAOTExport::test_aot_export_predispatch_func_simple, test/functorch/test_aotdispatch.py::TestAOTExport::test_aot_export_predispatch_func_view, test/functorch/test_aotdispatch.py::TestAOTExport::test_aot_export_predispatch_map_1, test/functorch/test_aotdispatch.py::TestAOTExport::test_aot_export_predispatch_map_2, test/functorch/test_aotdispatch.py::TestAOTExport::test_aot_export_predispatch_outdtype, test/functorch/test_aotdispatch.py::TestAOTExport::test_aot_export_predispatch_reshape, test/functorch/test_aotdispatch.py::TestAOTExport::test_aot_export_predispatch_with_autograd_op, test/functorch/test_aotdispatch.py::TestAOTExport::test_aot_export_predispatch_with_cond, test/functorch/test_aotdispatch.py::TestAOTExport::test_aot_export_predispatch_with_cond_nested, test/functorch/test_aotdispatch.py::TestAOTExport::test_aot_export_simplified_basic, test/functorch/test_aotdispatch.py::TestAOTExport::test_aot_export_simplified_pytrees_banned, test/functorch/test_aotdispatch.py::TestAOTExport::test_aot_export_synthetic_bases_banned, test/functorch/test_aotdispatch.py::TestAOTExport::test_aot_export_unbacked_arg, test/functorch/test_aotdispatch.py::TestAOTExport::test_aot_export_with_torch_cond, test/functorch/test_aotdispatch.py::TestPartitioning::test_autocast, test/functorch/test_aotdispatch.py::TestPartitioning::test_contiguous, test/functorch/test_aotdispatch.py::TestPartitioning::test_custom_partitioner_fn, test/functorch/test_aotdispatch.py::TestPartitioning::test_default_partitioner_getitem, test/functorch/test_aotdispatch.py::TestPartitioning::test_default_partitioner_output_tensor_shape_tensor, test/functorch/test_aotdispatch.py::TestPartitioning::test_generate_gives_inference_graph, test/functorch/test_aotdispatch.py::TestPartitioning::test_meta_tensor_inplace_op, test/functorch/test_aotdispatch.py::TestPartitioning::test_min_cut_partitioner, test/functorch/test_aotdispatch.py::TestPartitioning::test_min_cut_partitioner_output_tensor_shape_tensor, test/functorch/test_aotdispatch.py::TestPartitioning::test_min_cut_partitioner_raise_getitems, test/functorch/test_aotdispatch.py::TestPartitioning::test_min_cut_partitioner_save_shape, test/functorch/test_aotdispatch.py::TestPartitioning::test_preserve_random, test/functorch/test_aotdispatch.py::TestPartitioning::test_quantize_activation_duplicate_nodes, test/functorch/test_aotdispatch.py::TestPartitioning::test_recompute_partitioning, test/functorch/test_aotdispatch.py::TestAOTDispatch::test_aot_dispatch_incorrect_backward, test/functorch/test_aotdispatch.py::TestAOTDispatch::test_aot_dispatch_inference, test/functorch/test_aotdispatch.py::TestAOTDispatch::test_aot_dispatch_input_data_and_metadata_mutation, test/functorch/test_aotdispatch.py::TestAOTDispatch::test_aot_dispatch_input_metadata_mutation, test/functorch/test_aotdispatch.py::TestAOTDispatch::test_aot_dispatch_input_mutation, test/functorch/test_aotdispatch.py::TestAOTDispatch::test_aot_dispatch_input_mutation_and_output_alias, test/functorch/test_aotdispatch.py::TestAOTDispatch::test_aot_dispatch_output_alias, test/functorch/test_aotdispatch.py::TestAOTDispatch::test_aot_dispatch_output_requires_grad_in_no_grad, test/functorch/test_aotdispatch.py::TestAOTDispatch::test_aot_dispatch_output_requires_grad_in_no_grad_views, test/functorch/test_aotdispatch.py::TestAOTDispatch::test_aot_dispatch_simple, test/functorch/test_aotdispatch.py::TestAOTModuleSimplified::test_aot_module_simplified, test/functorch/test_aotdispatch.py::TestAOTModuleSimplified::test_aot_module_simplified_dynamic, test/functorch/test_aotdispatch.py::TestAOTModuleSimplified::test_aot_module_simplified_fake_tensor_gm_raises, test/functorch/test_aotdispatch.py::TestAOTModuleSimplified::test_aot_module_simplified_preserves_stack_trace, test/functorch/test_aotdispatch.py::TestAOTModuleSimplified::test_aot_module_simplified_preserves_stack_trace_from_mutation, test/functorch/test_aotdispatch.py::TestAOTModuleSimplified::test_aot_test_subclasses_with_tensor_factories, test/functorch/test_aotdispatch.py::TestAOTModuleSimplified::test_flex_attn_noncontiguous_tangents, test/functorch/test_aotdispatch.py::TestAOTModuleSimplified::test_grads_no_force_contiguous_dense, test/functorch/test_aotdispatch.py::TestAOTModuleSimplified::test_grads_no_force_contiguous_nested_subclass, test/functorch/test_aotdispatch.py::TestAOTModuleSimplified::test_grads_no_force_contiguous_nested_tensor_tangent, test/functorch/test_aotdispatch.py::TestAOTModuleSimplified::test_grads_no_force_contiguous_subclass, test/functorch/test_aotdispatch.py::TestAOTModuleSimplified::test_inductor_freezing_with_subclasses, test/functorch/test_aotdispatch.py::TestAOTModuleSimplified::test_inference_python_dispatcher, test/functorch/test_aotdispatch.py::TestAOTModuleSimplified::test_layer_norm, test/functorch/test_aotdispatch.py::TestAOTModuleSimplified::test_lift_fresh_copy_in_graph, test/functorch/test_aotdispatch.py::TestAOTModuleSimplified::test_noncontig_nonmemformat_tangents_dynamic_shapes_False_test_subclasses_False_device_cpu, test/functorch/test_aotdispatch.py::TestAOTModuleSimplified::test_noncontig_nonmemformat_tangents_dynamic_shapes_False_test_subclasses_False_device_cuda, test/functorch/test_aotdispatch.py::TestAOTModuleSimplified::test_noncontig_nonmemformat_tangents_dynamic_shapes_False_test_subclasses_True_device_cpu, test/functorch/test_aotdispatch.py::TestAOTModuleSimplified::test_noncontig_nonmemformat_tangents_dynamic_shapes_False_test_subclasses_True_device_cuda, test/functorch/test_aotdispatch.py::TestAOTModuleSimplified::test_noncontig_nonmemformat_tangents_dynamic_shapes_True_test_subclasses_False_device_cpu, test/functorch/test_aotdispatch.py::TestAOTModuleSimplified::test_noncontig_nonmemformat_tangents_dynamic_shapes_True_test_subclasses_False_device_cuda, test/functorch/test_aotdispatch.py::TestAOTModuleSimplified::test_noncontig_nonmemformat_tangents_dynamic_shapes_True_test_subclasses_True_device_cpu, test/functorch/test_aotdispatch.py::TestAOTModuleSimplified::test_noncontig_nonmemformat_tangents_dynamic_shapes_True_test_subclasses_True_device_cuda, test/functorch/test_aotdispatch.py::TestAOTModuleSimplified::test_rms_norm, test/functorch/test_aotdispatch.py::TestAOTModuleSimplified::test_rrelu, test/functorch/test_aotdispatch.py::TestAOTModuleSimplified::test_rrelu_with_noise_mutation, test/functorch/test_aotdispatch.py::TestAOTModuleSimplified::test_saved_tensors_hooks_base_saved_tensors_hooks_filtering_mode_all, test/functorch/test_aotdispatch.py::TestAOTModuleSimplified::test_saved_tensors_hooks_base_saved_tensors_hooks_filtering_mode_donated, test/functorch/test_aotdispatch.py::TestAOTModuleSimplified::test_saved_tensors_hooks_base_saved_tensors_hooks_filtering_mode_no_static, test/functorch/test_aotdispatch.py::TestAOTModuleSimplified::test_saved_tensors_hooks_donated_buffers, test/functorch/test_aotdispatch.py::TestAOTModuleSimplified::test_saved_tensors_hooks_params, test/functorch/test_aotdispatch.py::TestAOTModuleSimplified::test_saved_tensors_hooks_recompile, test/functorch/test_aotdispatch.py::TestAOTModuleSimplified::test_subclass_parameters, test/functorch/test_aotdispatch.py::TestAOTModuleSimplified::test_subclass_parameters_torture_case, test/functorch/test_aotdispatch.py::TestAOTModuleSimplified::test_tangent_type_coercion, test/functorch/test_aotdispatch.py::TestAOTModuleSimplified::test_wrong_guess_tangent_type, test/functorch/test_aotdispatch.py::TestAOTAutogradWithDynamo::test_alias_of_intermediate_detach_backend_aot_eager_view_replay_for_aliased_outputs_False_dynamic_shapes_False, test/functorch/test_aotdispatch.py::TestAOTAutogradWithDynamo::test_alias_of_intermediate_detach_backend_aot_eager_view_replay_for_aliased_outputs_False_dynamic_shapes_True, test/functorch/test_aotdispatch.py::TestAOTAutogradWithDynamo::test_alias_of_intermediate_detach_backend_aot_eager_view_replay_for_aliased_outputs_True_dynamic_shapes_False, test/functorch/test_aotdispatch.py::TestAOTAutogradWithDynamo::test_alias_of_intermediate_detach_backend_aot_eager_view_replay_for_aliased_outputs_True_dynamic_shapes_True, test/functorch/test_aotdispatch.py::TestAOTAutogradWithDynamo::test_alias_of_intermediate_detach_backend_inductor_view_replay_for_aliased_outputs_False_dynamic_shapes_False, test/functorch/test_aotdispatch.py::TestAOTAutogradWithDynamo::test_alias_of_intermediate_detach_backend_inductor_view_replay_for_aliased_outputs_False_dynamic_shapes_True, test/functorch/test_aotdispatch.py::TestAOTAutogradWithDynamo::test_alias_of_intermediate_detach_backend_inductor_view_replay_for_aliased_outputs_True_dynamic_shapes_False, test/functorch/test_aotdispatch.py::TestAOTAutogradWithDynamo::test_alias_of_intermediate_detach_backend_inductor_view_replay_for_aliased_outputs_True_dynamic_shapes_True, test/functorch/test_aotdispatch.py::TestAOTAutogradWithDynamo::test_autocast_disable_guard, test/functorch/test_aotdispatch.py::TestAOTAutogradWithDynamo::test_backward_mutation_data, test/functorch/test_aotdispatch.py::TestAOTAutogradWithDynamo::test_backward_mutation_forward_inputs, test/functorch/test_aotdispatch.py::TestAOTAutogradWithDynamo::test_backward_mutation_forward_inputs_create_graph, test/functorch/test_aotdispatch.py::TestAOTAutogradWithDynamo::test_backward_mutation_metadata, test/functorch/test_aotdispatch.py::TestAOTAutogradWithDynamo::test_backward_mutation_on_grad_out, test/functorch/test_aotdispatch.py::TestAOTAutogradWithDynamo::test_backward_pass_autocast_custom, test/functorch/test_aotdispatch.py::TestAOTAutogradWithDynamo::test_backward_pass_autocast_off, test/functorch/test_aotdispatch.py::TestAOTAutogradWithDynamo::test_backward_pass_autocast_on, test/functorch/test_aotdispatch.py::TestAOTAutogradWithDynamo::test_batch_norm_amp, test/functorch/test_aotdispatch.py::TestAOTAutogradWithDynamo::test_batchnorm, test/functorch/test_aotdispatch.py::TestAOTAutogradWithDynamo::test_batchnorm_inference, test/functorch/test_aotdispatch.py::TestAOTAutogradWithDynamo::test_buffer_batch_norm, test/functorch/test_aotdispatch.py::TestAOTAutogradWithDynamo::test_buffer_copied_in_graph, test/functorch/test_aotdispatch.py::TestAOTAutogradWithDynamo::test_buffer_copied_in_graph_with_different_shapes, test/functorch/test_aotdispatch.py::TestAOTAutogradWithDynamo::test_compilation_context, test/functorch/test_aotdispatch.py::TestAOTAutogradWithDynamo::test_complex_linear, test/functorch/test_aotdispatch.py::TestAOTAutogradWithDynamo::test_composite_impl_compile, test/functorch/test_aotdispatch.py::TestAOTAutogradWithDynamo::test_custom_autograd, test/functorch/test_aotdispatch.py::TestAOTAutogradWithDynamo::test_custom_tensor_metadata, test/functorch/test_aotdispatch.py::TestAOTAutogradWithDynamo::test_default_partitioner_saves_symints_not_tensors_for_bw, test/functorch/test_aotdispatch.py::TestAOTAutogradWithDynamo::test_dupe_arg, test/functorch/test_aotdispatch.py::TestAOTAutogradWithDynamo::test_dupe_arg_returned_as_output, test/functorch/test_aotdispatch.py::TestAOTAutogradWithDynamo::test_dupe_arg_torture, test/functorch/test_aotdispatch.py::TestAOTAutogradWithDynamo::test_duplicated_arguments_on_tensor_overlap, test/functorch/test_aotdispatch.py::TestAOTAutogradWithDynamo::test_dynamic_output_aliases_input_view_meta_replay, test/functorch/test_aotdispatch.py::TestAOTAutogradWithDynamo::test_dynamic_shape_output_not_in_bw_graph, test/functorch/test_aotdispatch.py::TestAOTAutogradWithDynamo::test_embedding_bag_view_dynamic, test/functorch/test_aotdispatch.py::TestAOTAutogradWithDynamo::test_fw_bw_mutation_no_functionalization1, test/functorch/test_aotdispatch.py::TestAOTAutogradWithDynamo::test_fw_bw_mutation_no_functionalization2, test/functorch/test_aotdispatch.py::TestAOTAutogradWithDynamo::test_grad_context, test/functorch/test_aotdispatch.py::TestAOTAutogradWithDynamo::test_inference_mode, test/functorch/test_aotdispatch.py::TestAOTAutogradWithDynamo::test_inner_grad, test/functorch/test_aotdispatch.py::TestAOTAutogradWithDynamo::test_input_aliased_with_mutation_output_alias, test/functorch/test_aotdispatch.py::TestAOTAutogradWithDynamo::test_input_data_and_metadata_mutation, test/functorch/test_aotdispatch.py::TestAOTAutogradWithDynamo::test_input_data_and_metadata_mutation_aliases_other_input, test/functorch/test_aotdispatch.py::TestAOTAutogradWithDynamo::test_input_inplace_requires_grad_true, test/functorch/test_aotdispatch.py::TestAOTAutogradWithDynamo::test_input_metadata_mutation_aliases, test/functorch/test_aotdispatch.py::TestAOTAutogradWithDynamo::test_input_mutation_alias_everything, test/functorch/test_aotdispatch.py::TestAOTAutogradWithDynamo::test_input_mutation_aliases_and_none_require_gradients, test/functorch/test_aotdispatch.py::TestAOTAutogradWithDynamo::test_input_mutation_aliases_and_output_alias, test/functorch/test_aotdispatch.py::TestAOTAutogradWithDynamo::test_input_mutation_aliases_bases_out_of_order, test/functorch/test_aotdispatch.py::TestAOTAutogradWithDynamo::test_input_mutation_aliases_other_input, test/functorch/test_aotdispatch.py::TestAOTAutogradWithDynamo::test_input_mutation_aliases_other_input2, test/functorch/test_aotdispatch.py::TestAOTAutogradWithDynamo::test_input_mutation_and_output_view, test/functorch/test_aotdispatch.py::TestAOTAutogradWithDynamo::test_input_mutation_batchnorm, test/functorch/test_aotdispatch.py::TestAOTAutogradWithDynamo::test_input_mutation_false_aliasing, test/functorch/test_aotdispatch.py::TestAOTAutogradWithDynamo::test_input_mutation_hidden_from_autograd_aliasing, test/functorch/test_aotdispatch.py::TestAOTAutogradWithDynamo::test_input_mutation_is_output, test/functorch/test_aotdispatch.py::TestAOTAutogradWithDynamo::test_input_mutation_metadata, test/functorch/test_aotdispatch.py::TestAOTAutogradWithDynamo::test_input_mutation_metadata2, test/functorch/test_aotdispatch.py::TestAOTAutogradWithDynamo::test_input_mutation_modifies_autograd_meta_of_aliases, test/functorch/test_aotdispatch.py::TestAOTAutogradWithDynamo::test_input_mutation_multiple, test/functorch/test_aotdispatch.py::TestAOTAutogradWithDynamo::test_input_mutation_noncontiguous, test/functorch/test_aotdispatch.py::TestAOTAutogradWithDynamo::test_input_mutation_output_view_multiple, test/functorch/test_aotdispatch.py::TestAOTAutogradWithDynamo::test_input_mutation_requires_grad_detach, test/functorch/test_aotdispatch.py::TestAOTAutogradWithDynamo::test_input_mutation_requires_grad_no_grad, test/functorch/test_aotdispatch.py::TestAOTAutogradWithDynamo::test_input_mutation_requires_grad_no_grad_detach_mixed, test/functorch/test_aotdispatch.py::TestAOTAutogradWithDynamo::test_input_mutation_requires_grad_no_grad_inference_graph, test/functorch/test_aotdispatch.py::TestAOTAutogradWithDynamo::test_input_mutation_return, test/functorch/test_aotdispatch.py::TestAOTAutogradWithDynamo::test_input_mutation_set__input_mutation, test/functorch/test_aotdispatch.py::TestAOTAutogradWithDynamo::test_input_mutation_set__nop, test/functorch/test_aotdispatch.py::TestAOTAutogradWithDynamo::test_input_mutation_simple, test/functorch/test_aotdispatch.py::TestAOTAutogradWithDynamo::test_input_mutation_simple_with_none_and_nontensor, test/functorch/test_aotdispatch.py::TestAOTAutogradWithDynamo::test_input_mutation_storage_resize_before_set_, test/functorch/test_aotdispatch.py::TestAOTAutogradWithDynamo::test_input_mutation_storage_resize_down, test/functorch/test_aotdispatch.py::TestAOTAutogradWithDynamo::test_input_mutation_storage_resize_down_and_set_, test/functorch/test_aotdispatch.py::TestAOTAutogradWithDynamo::test_input_mutation_storage_resize_up, test/functorch/test_aotdispatch.py::TestAOTAutogradWithDynamo::test_input_output_aliase_custom_autograd_function, test/functorch/test_aotdispatch.py::TestAOTAutogradWithDynamo::test_input_output_view_metadata_mutate_multiple, test/functorch/test_aotdispatch.py::TestAOTAutogradWithDynamo::test_input_output_view_mutate_multiple, test/functorch/test_aotdispatch.py::TestAOTAutogradWithDynamo::test_input_output_view_simple, test/functorch/test_aotdispatch.py::TestAOTAutogradWithDynamo::test_inputs_overlapping_unsqueeze_with_mutation, test/functorch/test_aotdispatch.py::TestAOTAutogradWithDynamo::test_inputs_overlapping_with_mutation_guard_base, test/functorch/test_aotdispatch.py::TestAOTAutogradWithDynamo::test_invalid_dupe, test/functorch/test_aotdispatch.py::TestAOTAutogradWithDynamo::test_invalid_dupe_fake, test/functorch/test_aotdispatch.py::TestAOTAutogradWithDynamo::test_invalid_dupe_left_bias, test/functorch/test_aotdispatch.py::TestAOTAutogradWithDynamo::test_invalid_requires_grad, test/functorch/test_aotdispatch.py::TestAOTAutogradWithDynamo::test_invalid_requires_grad_fake, test/functorch/test_aotdispatch.py::TestAOTAutogradWithDynamo::test_list_codegen, test/functorch/test_aotdispatch.py::TestAOTAutogradWithDynamo::test_mark_activations_dynamic, test/functorch/test_aotdispatch.py::TestAOTAutogradWithDynamo::test_mark_activations_dynamic_with_nested, test/functorch/test_aotdispatch.py::TestAOTAutogradWithDynamo::test_mark_outputs_dynamic_use_autograd_False, test/functorch/test_aotdispatch.py::TestAOTAutogradWithDynamo::test_mark_outputs_dynamic_use_autograd_True, test/functorch/test_aotdispatch.py::TestAOTAutogradWithDynamo::test_mem_leak_from_save_for_bw, test/functorch/test_aotdispatch.py::TestAOTAutogradWithDynamo::test_module, test/functorch/test_aotdispatch.py::TestAOTAutogradWithDynamo::test_multi_output, test/functorch/test_aotdispatch.py::TestAOTAutogradWithDynamo::test_multi_output_list, test/functorch/test_aotdispatch.py::TestAOTAutogradWithDynamo::test_mutates_input_noncontiguous, test/functorch/test_aotdispatch.py::TestAOTAutogradWithDynamo::test_mutation_of_input_in_fw_and_bw, test/functorch/test_aotdispatch.py::TestAOTAutogradWithDynamo::test_mutations_in_bw_detached_from_tangent, test/functorch/test_aotdispatch.py::TestAOTAutogradWithDynamo::test_nested_subclasses, test/functorch/test_aotdispatch.py::TestAOTAutogradWithDynamo::test_nested_subclasses_complicated_inps, test/functorch/test_aotdispatch.py::TestAOTAutogradWithDynamo::test_nested_subclasses_complicated_inps_mixed, test/functorch/test_aotdispatch.py::TestAOTAutogradWithDynamo::test_nested_subclasses_non_homogenous, test/functorch/test_aotdispatch.py::TestAOTAutogradWithDynamo::test_nested_subclasses_non_nested_grad, test/functorch/test_aotdispatch.py::TestAOTAutogradWithDynamo::test_new_inp_requires_grad_now, test/functorch/test_aotdispatch.py::TestAOTAutogradWithDynamo::test_no_grad_input_output, test/functorch/test_aotdispatch.py::TestAOTAutogradWithDynamo::test_non_tensor_and_none_inputs, test/functorch/test_aotdispatch.py::TestAOTAutogradWithDynamo::test_nonidempotent_amp, test/functorch/test_aotdispatch.py::TestAOTAutogradWithDynamo::test_output_aliases_input_multi_output_view, test/functorch/test_aotdispatch.py::TestAOTAutogradWithDynamo::test_output_aliases_input_multi_output_view_should_raise_autograd_error, test/functorch/test_aotdispatch.py::TestAOTAutogradWithDynamo::test_output_aliases_input_view_meta_replay, test/functorch/test_aotdispatch.py::TestAOTAutogradWithDynamo::test_output_aliases_intermediate_and_returned, test/functorch/test_aotdispatch.py::TestAOTAutogradWithDynamo::test_output_aliases_intermediate_and_returned_different_grad, test/functorch/test_aotdispatch.py::TestAOTAutogradWithDynamo::test_output_aliases_intermediate_and_returned_flipped, test/functorch/test_aotdispatch.py::TestAOTAutogradWithDynamo::test_output_aliases_intermediate_inplace_view, test/functorch/test_aotdispatch.py::TestAOTAutogradWithDynamo::test_output_aliases_intermediate_inplace_view_and_view, test/functorch/test_aotdispatch.py::TestAOTAutogradWithDynamo::test_output_aliases_intermediate_inplace_view_with_detach, test/functorch/test_aotdispatch.py::TestAOTAutogradWithDynamo::test_output_aliases_intermediate_multi_output_view, test/functorch/test_aotdispatch.py::TestAOTAutogradWithDynamo::test_output_aliases_intermediate_multiple, test/functorch/test_aotdispatch.py::TestAOTAutogradWithDynamo::test_output_aliases_intermediate_multiple_mixed, test/functorch/test_aotdispatch.py::TestAOTAutogradWithDynamo::test_output_aliases_intermediate_mutation_linear, test/functorch/test_aotdispatch.py::TestAOTAutogradWithDynamo::test_output_aliases_intermediate_no_grad, test/functorch/test_aotdispatch.py::TestAOTAutogradWithDynamo::test_output_aliases_intermediate_returned_multiple_times, test/functorch/test_aotdispatch.py::TestAOTAutogradWithDynamo::test_output_aliases_intermediate_single, test/functorch/test_aotdispatch.py::TestAOTAutogradWithDynamo::test_output_aliases_intermediate_view_meta_replay, test/functorch/test_aotdispatch.py::TestAOTAutogradWithDynamo::test_output_aliases_multiple_inputs_get_correct_one, test/functorch/test_aotdispatch.py::TestAOTAutogradWithDynamo::test_output_aliases_output_view_meta_replay, test/functorch/test_aotdispatch.py::TestAOTAutogradWithDynamo::test_output_all_alias_types, test/functorch/test_aotdispatch.py::TestAOTAutogradWithDynamo::test_output_dict, test/functorch/test_aotdispatch.py::TestAOTAutogradWithDynamo::test_output_op_depending_on_symint, test/functorch/test_aotdispatch.py::TestAOTAutogradWithDynamo::test_outputs_are_aliased, test/functorch/test_aotdispatch.py::TestAOTAutogradWithDynamo::test_real_weights_in_symbolic_mode, test/functorch/test_aotdispatch.py::TestAOTAutogradWithDynamo::test_real_weights_in_symbolic_mode_with_inplace_ops, test/functorch/test_aotdispatch.py::TestAOTAutogradWithDynamo::test_saved_tensors_hooks_mutations_raise, test/functorch/test_aotdispatch.py::TestAOTAutogradWithDynamo::test_set__and_data_mutation_bad, test/functorch/test_aotdispatch.py::TestAOTAutogradWithDynamo::test_set__and_data_mutation_good, test/functorch/test_aotdispatch.py::TestAOTAutogradWithDynamo::test_set__not_allowed, test/functorch/test_aotdispatch.py::TestAOTAutogradWithDynamo::test_set__steals_view_chain, test/functorch/test_aotdispatch.py::TestAOTAutogradWithDynamo::test_single_output, test/functorch/test_aotdispatch.py::TestAOTAutogradWithDynamo::test_some_output_requires_grad_input_doesnt, test/functorch/test_aotdispatch.py::TestAOTAutogradWithDynamo::test_some_outputs_dont_require_grad_non_view, test/functorch/test_aotdispatch.py::TestAOTAutogradWithDynamo::test_some_outputs_dont_require_grad_view, test/functorch/test_aotdispatch.py::TestAOTAutogradWithDynamo::test_squeeze_mutation, test/functorch/test_aotdispatch.py::TestAOTAutogradWithDynamo::test_subclass_metadata_mutation_req_grad_False, test/functorch/test_aotdispatch.py::TestAOTAutogradWithDynamo::test_subclass_metadata_mutation_req_grad_True, test/functorch/test_aotdispatch.py::TestAOTAutogradWithDynamo::test_subclasses_mixed, test/functorch/test_aotdispatch.py::TestAOTAutogradWithDynamo::test_subclasses_mixed_mode, test/functorch/test_aotdispatch.py::TestAOTAutogradWithDynamo::test_synthetic_base_base_attribute_is_none, test/functorch/test_aotdispatch.py::TestAOTAutogradWithDynamo::test_view_and_inplace_view, test/functorch/test_aotdispatch.py::TestAOTAutogradWithDynamo::test_view_detach, test/functorch/test_aotdispatch.py::TestAOTAutogradWithCache::test_alias_of_intermediate_detach_backend_aot_eager_view_replay_for_aliased_outputs_False_dynamic_shapes_False, test/functorch/test_aotdispatch.py::TestAOTAutogradWithCache::test_alias_of_intermediate_detach_backend_aot_eager_view_replay_for_aliased_outputs_False_dynamic_shapes_True, test/functorch/test_aotdispatch.py::TestAOTAutogradWithCache::test_alias_of_intermediate_detach_backend_aot_eager_view_replay_for_aliased_outputs_True_dynamic_shapes_False, test/functorch/test_aotdispatch.py::TestAOTAutogradWithCache::test_alias_of_intermediate_detach_backend_aot_eager_view_replay_for_aliased_outputs_True_dynamic_shapes_True, test/functorch/test_aotdispatch.py::TestAOTAutogradWithCache::test_alias_of_intermediate_detach_backend_inductor_view_replay_for_aliased_outputs_False_dynamic_shapes_False, test/functorch/test_aotdispatch.py::TestAOTAutogradWithCache::test_alias_of_intermediate_detach_backend_inductor_view_replay_for_aliased_outputs_False_dynamic_shapes_True, test/functorch/test_aotdispatch.py::TestAOTAutogradWithCache::test_alias_of_intermediate_detach_backend_inductor_view_replay_for_aliased_outputs_True_dynamic_shapes_False, test/functorch/test_aotdispatch.py::TestAOTAutogradWithCache::test_alias_of_intermediate_detach_backend_inductor_view_replay_for_aliased_outputs_True_dynamic_shapes_True, test/functorch/test_aotdispatch.py::TestAOTAutogradWithCache::test_autocast_disable_guard, test/functorch/test_aotdispatch.py::TestAOTAutogradWithCache::test_backward_mutation_data, test/functorch/test_aotdispatch.py::TestAOTAutogradWithCache::test_backward_mutation_forward_inputs, test/functorch/test_aotdispatch.py::TestAOTAutogradWithCache::test_backward_mutation_forward_inputs_create_graph, test/functorch/test_aotdispatch.py::TestAOTAutogradWithCache::test_backward_mutation_metadata, test/functorch/test_aotdispatch.py::TestAOTAutogradWithCache::test_backward_mutation_on_grad_out, test/functorch/test_aotdispatch.py::TestAOTAutogradWithCache::test_backward_pass_autocast_custom, test/functorch/test_aotdispatch.py::TestAOTAutogradWithCache::test_backward_pass_autocast_off, test/functorch/test_aotdispatch.py::TestAOTAutogradWithCache::test_backward_pass_autocast_on, test/functorch/test_aotdispatch.py::TestAOTAutogradWithCache::test_batch_norm_amp, test/functorch/test_aotdispatch.py::TestAOTAutogradWithCache::test_batchnorm, test/functorch/test_aotdispatch.py::TestAOTAutogradWithCache::test_batchnorm_inference, test/functorch/test_aotdispatch.py::TestAOTAutogradWithCache::test_buffer_batch_norm, test/functorch/test_aotdispatch.py::TestAOTAutogradWithCache::test_buffer_copied_in_graph, test/functorch/test_aotdispatch.py::TestAOTAutogradWithCache::test_buffer_copied_in_graph_with_different_shapes, test/functorch/test_aotdispatch.py::TestAOTAutogradWithCache::test_compilation_context, test/functorch/test_aotdispatch.py::TestAOTAutogradWithCache::test_complex_linear, test/functorch/test_aotdispatch.py::TestAOTAutogradWithCache::test_composite_impl_compile, test/functorch/test_aotdispatch.py::TestAOTAutogradWithCache::test_custom_autograd, test/functorch/test_aotdispatch.py::TestAOTAutogradWithCache::test_custom_tensor_metadata, test/functorch/test_aotdispatch.py::TestAOTAutogradWithCache::test_default_partitioner_saves_symints_not_tensors_for_bw, test/functorch/test_aotdispatch.py::TestAOTAutogradWithCache::test_dupe_arg, test/functorch/test_aotdispatch.py::TestAOTAutogradWithCache::test_dupe_arg_returned_as_output, test/functorch/test_aotdispatch.py::TestAOTAutogradWithCache::test_dupe_arg_torture, test/functorch/test_aotdispatch.py::TestAOTAutogradWithCache::test_duplicated_arguments_on_tensor_overlap, test/functorch/test_aotdispatch.py::TestAOTAutogradWithCache::test_dynamic_output_aliases_input_view_meta_replay, test/functorch/test_aotdispatch.py::TestAOTAutogradWithCache::test_dynamic_shape_output_not_in_bw_graph, test/functorch/test_aotdispatch.py::TestAOTAutogradWithCache::test_embedding_bag_view_dynamic, test/functorch/test_aotdispatch.py::TestAOTAutogradWithCache::test_fw_bw_mutation_no_functionalization1, test/functorch/test_aotdispatch.py::TestAOTAutogradWithCache::test_fw_bw_mutation_no_functionalization2, test/functorch/test_aotdispatch.py::TestAOTAutogradWithCache::test_grad_context, test/functorch/test_aotdispatch.py::TestAOTAutogradWithCache::test_inference_mode, test/functorch/test_aotdispatch.py::TestAOTAutogradWithCache::test_inner_grad, test/functorch/test_aotdispatch.py::TestAOTAutogradWithCache::test_input_aliased_with_mutation_output_alias, test/functorch/test_aotdispatch.py::TestAOTAutogradWithCache::test_input_data_and_metadata_mutation, test/functorch/test_aotdispatch.py::TestAOTAutogradWithCache::test_input_data_and_metadata_mutation_aliases_other_input, test/functorch/test_aotdispatch.py::TestAOTAutogradWithCache::test_input_inplace_requires_grad_true, test/functorch/test_aotdispatch.py::TestAOTAutogradWithCache::test_input_metadata_mutation_aliases, test/functorch/test_aotdispatch.py::TestAOTAutogradWithCache::test_input_mutation_alias_everything, test/functorch/test_aotdispatch.py::TestAOTAutogradWithCache::test_input_mutation_aliases_and_none_require_gradients, test/functorch/test_aotdispatch.py::TestAOTAutogradWithCache::test_input_mutation_aliases_and_output_alias, test/functorch/test_aotdispatch.py::TestAOTAutogradWithCache::test_input_mutation_aliases_bases_out_of_order, test/functorch/test_aotdispatch.py::TestAOTAutogradWithCache::test_input_mutation_aliases_other_input, test/functorch/test_aotdispatch.py::TestAOTAutogradWithCache::test_input_mutation_aliases_other_input2, test/functorch/test_aotdispatch.py::TestAOTAutogradWithCache::test_input_mutation_and_output_view, test/functorch/test_aotdispatch.py::TestAOTAutogradWithCache::test_input_mutation_batchnorm, test/functorch/test_aotdispatch.py::TestAOTAutogradWithCache::test_input_mutation_false_aliasing, test/functorch/test_aotdispatch.py::TestAOTAutogradWithCache::test_input_mutation_hidden_from_autograd_aliasing, test/functorch/test_aotdispatch.py::TestAOTAutogradWithCache::test_input_mutation_is_output, test/functorch/test_aotdispatch.py::TestAOTAutogradWithCache::test_input_mutation_metadata, test/functorch/test_aotdispatch.py::TestAOTAutogradWithCache::test_input_mutation_metadata2, test/functorch/test_aotdispatch.py::TestAOTAutogradWithCache::test_input_mutation_modifies_autograd_meta_of_aliases, test/functorch/test_aotdispatch.py::TestAOTAutogradWithCache::test_input_mutation_multiple, test/functorch/test_aotdispatch.py::TestAOTAutogradWithCache::test_input_mutation_noncontiguous, test/functorch/test_aotdispatch.py::TestAOTAutogradWithCache::test_input_mutation_output_view_multiple, test/functorch/test_aotdispatch.py::TestAOTAutogradWithCache::test_input_mutation_requires_grad_detach, test/functorch/test_aotdispatch.py::TestAOTAutogradWithCache::test_input_mutation_requires_grad_no_grad, test/functorch/test_aotdispatch.py::TestAOTAutogradWithCache::test_input_mutation_requires_grad_no_grad_detach_mixed, test/functorch/test_aotdispatch.py::TestAOTAutogradWithCache::test_input_mutation_requires_grad_no_grad_inference_graph, test/functorch/test_aotdispatch.py::TestAOTAutogradWithCache::test_input_mutation_return, test/functorch/test_aotdispatch.py::TestAOTAutogradWithCache::test_input_mutation_set__input_mutation, test/functorch/test_aotdispatch.py::TestAOTAutogradWithCache::test_input_mutation_set__nop, test/functorch/test_aotdispatch.py::TestAOTAutogradWithCache::test_input_mutation_simple, test/functorch/test_aotdispatch.py::TestAOTAutogradWithCache::test_input_mutation_simple_with_none_and_nontensor, test/functorch/test_aotdispatch.py::TestAOTAutogradWithCache::test_input_mutation_storage_resize_before_set_, test/functorch/test_aotdispatch.py::TestAOTAutogradWithCache::test_input_mutation_storage_resize_down, test/functorch/test_aotdispatch.py::TestAOTAutogradWithCache::test_input_mutation_storage_resize_down_and_set_, test/functorch/test_aotdispatch.py::TestAOTAutogradWithCache::test_input_mutation_storage_resize_up, test/functorch/test_aotdispatch.py::TestAOTAutogradWithCache::test_input_output_aliase_custom_autograd_function, test/functorch/test_aotdispatch.py::TestAOTAutogradWithCache::test_input_output_view_metadata_mutate_multiple, test/functorch/test_aotdispatch.py::TestAOTAutogradWithCache::test_input_output_view_mutate_multiple, test/functorch/test_aotdispatch.py::TestAOTAutogradWithCache::test_input_output_view_simple, test/functorch/test_aotdispatch.py::TestAOTAutogradWithCache::test_inputs_overlapping_unsqueeze_with_mutation, test/functorch/test_aotdispatch.py::TestAOTAutogradWithCache::test_inputs_overlapping_with_mutation_guard_base, test/functorch/test_aotdispatch.py::TestAOTAutogradWithCache::test_invalid_dupe, test/functorch/test_aotdispatch.py::TestAOTAutogradWithCache::test_invalid_dupe_fake, test/functorch/test_aotdispatch.py::TestAOTAutogradWithCache::test_invalid_dupe_left_bias, test/functorch/test_aotdispatch.py::TestAOTAutogradWithCache::test_invalid_requires_grad, test/functorch/test_aotdispatch.py::TestAOTAutogradWithCache::test_invalid_requires_grad_fake, test/functorch/test_aotdispatch.py::TestAOTAutogradWithCache::test_list_codegen, test/functorch/test_aotdispatch.py::TestAOTAutogradWithCache::test_mark_activations_dynamic, test/functorch/test_aotdispatch.py::TestAOTAutogradWithCache::test_mark_activations_dynamic_with_nested, test/functorch/test_aotdispatch.py::TestAOTAutogradWithCache::test_mark_outputs_dynamic_use_autograd_False, test/functorch/test_aotdispatch.py::TestAOTAutogradWithCache::test_mark_outputs_dynamic_use_autograd_True, test/functorch/test_aotdispatch.py::TestAOTAutogradWithCache::test_mem_leak_from_save_for_bw, test/functorch/test_aotdispatch.py::TestAOTAutogradWithCache::test_module, test/functorch/test_aotdispatch.py::TestAOTAutogradWithCache::test_multi_output, test/functorch/test_aotdispatch.py::TestAOTAutogradWithCache::test_multi_output_list, test/functorch/test_aotdispatch.py::TestAOTAutogradWithCache::test_mutates_input_noncontiguous, test/functorch/test_aotdispatch.py::TestAOTAutogradWithCache::test_mutation_of_input_in_fw_and_bw, test/functorch/test_aotdispatch.py::TestAOTAutogradWithCache::test_mutations_in_bw_detached_from_tangent, test/functorch/test_aotdispatch.py::TestAOTAutogradWithCache::test_nested_subclasses, test/functorch/test_aotdispatch.py::TestAOTAutogradWithCache::test_nested_subclasses_complicated_inps, test/functorch/test_aotdispatch.py::TestAOTAutogradWithCache::test_nested_subclasses_complicated_inps_mixed, test/functorch/test_aotdispatch.py::TestAOTAutogradWithCache::test_nested_subclasses_non_homogenous, test/functorch/test_aotdispatch.py::TestAOTAutogradWithCache::test_nested_subclasses_non_nested_grad, test/functorch/test_aotdispatch.py::TestAOTAutogradWithCache::test_new_inp_requires_grad_now, test/functorch/test_aotdispatch.py::TestAOTAutogradWithCache::test_no_grad_input_output, test/functorch/test_aotdispatch.py::TestAOTAutogradWithCache::test_non_tensor_and_none_inputs, test/functorch/test_aotdispatch.py::TestAOTAutogradWithCache::test_nonidempotent_amp, test/functorch/test_aotdispatch.py::TestAOTAutogradWithCache::test_output_aliases_input_multi_output_view, test/functorch/test_aotdispatch.py::TestAOTAutogradWithCache::test_output_aliases_input_multi_output_view_should_raise_autograd_error, test/functorch/test_aotdispatch.py::TestAOTAutogradWithCache::test_output_aliases_input_view_meta_replay, test/functorch/test_aotdispatch.py::TestAOTAutogradWithCache::test_output_aliases_intermediate_and_returned, test/functorch/test_aotdispatch.py::TestAOTAutogradWithCache::test_output_aliases_intermediate_and_returned_different_grad, test/functorch/test_aotdispatch.py::TestAOTAutogradWithCache::test_output_aliases_intermediate_and_returned_flipped, test/functorch/test_aotdispatch.py::TestAOTAutogradWithCache::test_output_aliases_intermediate_inplace_view, test/functorch/test_aotdispatch.py::TestAOTAutogradWithCache::test_output_aliases_intermediate_inplace_view_and_view, test/functorch/test_aotdispatch.py::TestAOTAutogradWithCache::test_output_aliases_intermediate_inplace_view_with_detach, test/functorch/test_aotdispatch.py::TestAOTAutogradWithCache::test_output_aliases_intermediate_multi_output_view, test/functorch/test_aotdispatch.py::TestAOTAutogradWithCache::test_output_aliases_intermediate_multiple, test/functorch/test_aotdispatch.py::TestAOTAutogradWithCache::test_output_aliases_intermediate_multiple_mixed, test/functorch/test_aotdispatch.py::TestAOTAutogradWithCache::test_output_aliases_intermediate_mutation_linear, test/functorch/test_aotdispatch.py::TestAOTAutogradWithCache::test_output_aliases_intermediate_no_grad, test/functorch/test_aotdispatch.py::TestAOTAutogradWithCache::test_output_aliases_intermediate_returned_multiple_times, test/functorch/test_aotdispatch.py::TestAOTAutogradWithCache::test_output_aliases_intermediate_single, test/functorch/test_aotdispatch.py::TestAOTAutogradWithCache::test_output_aliases_intermediate_view_meta_replay, test/functorch/test_aotdispatch.py::TestAOTAutogradWithCache::test_output_aliases_multiple_inputs_get_correct_one, test/functorch/test_aotdispatch.py::TestAOTAutogradWithCache::test_output_aliases_output_view_meta_replay, test/functorch/test_aotdispatch.py::TestAOTAutogradWithCache::test_output_all_alias_types, test/functorch/test_aotdispatch.py::TestAOTAutogradWithCache::test_output_dict, test/functorch/test_aotdispatch.py::TestAOTAutogradWithCache::test_output_op_depending_on_symint, test/functorch/test_aotdispatch.py::TestAOTAutogradWithCache::test_outputs_are_aliased, test/functorch/test_aotdispatch.py::TestAOTAutogradWithCache::test_real_weights_in_symbolic_mode, test/functorch/test_aotdispatch.py::TestAOTAutogradWithCache::test_real_weights_in_symbolic_mode_with_inplace_ops, test/functorch/test_aotdispatch.py::TestAOTAutogradWithCache::test_saved_tensors_hooks_mutations_raise, test/functorch/test_aotdispatch.py::TestAOTAutogradWithCache::test_set__and_data_mutation_bad, test/functorch/test_aotdispatch.py::TestAOTAutogradWithCache::test_set__and_data_mutation_good, test/functorch/test_aotdispatch.py::TestAOTAutogradWithCache::test_set__not_allowed, test/functorch/test_aotdispatch.py::TestAOTAutogradWithCache::test_set__steals_view_chain, test/functorch/test_aotdispatch.py::TestAOTAutogradWithCache::test_single_output, test/functorch/test_aotdispatch.py::TestAOTAutogradWithCache::test_some_output_requires_grad_input_doesnt, test/functorch/test_aotdispatch.py::TestAOTAutogradWithCache::test_some_outputs_dont_require_grad_non_view, test/functorch/test_aotdispatch.py::TestAOTAutogradWithCache::test_some_outputs_dont_require_grad_view, test/functorch/test_aotdispatch.py::TestAOTAutogradWithCache::test_squeeze_mutation, test/functorch/test_aotdispatch.py::TestAOTAutogradWithCache::test_subclass_metadata_mutation_req_grad_False, test/functorch/test_aotdispatch.py::TestAOTAutogradWithCache::test_subclass_metadata_mutation_req_grad_True, test/functorch/test_aotdispatch.py::TestAOTAutogradWithCache::test_subclasses_mixed, test/functorch/test_aotdispatch.py::TestAOTAutogradWithCache::test_subclasses_mixed_mode, test/functorch/test_aotdispatch.py::TestAOTAutogradWithCache::test_synthetic_base_base_attribute_is_none, test/functorch/test_aotdispatch.py::TestAOTAutogradWithCache::test_view_and_inplace_view, test/functorch/test_aotdispatch.py::TestAOTAutogradWithCache::test_view_detach 2025-12-04T13:55:29.9856374Z 2025-12-04T13:55:29.9856632Z Finished functorch/test_aotdispatch 1/1 ... [2025-12-04 13:55:29.954258][15377.896473633], took 1.80min 2025-12-04T13:55:29.9868363Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/functorch.test_aotdispatch/functorch.test_aotdispatch-c6f1fb2e499ba29b.xml 2025-12-04T13:55:30.0670871Z Running test_native_mha 1/1 ... [2025-12-04 13:55:30.066821][15378.009039575] 2025-12-04T13:55:30.0671300Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2025-12-04T13:55:30.0673999Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'test_native_mha.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-12-04 13:55:30.067140] 2025-12-04T13:55:34.4392170Z 2025-12-04T13:55:34.4392929Z test_native_mha 1/1 was successful, full logs can be found in artifacts with path test/test-reports/test_native_mha_1.1_19664d87c5964474_.log 2025-12-04T13:55:34.4422069Z Running 54 items in this shard: test/test_native_mha.py::TestMHADeviceTypeCUDA::test_native_multihead_attention_cuda_float16, test/test_native_mha.py::TestMHADeviceTypeCUDA::test_native_multihead_attention_cuda_float32, test/test_native_mha.py::TestMHADeviceTypeCUDA::test_native_multihead_encoder_decoder_attention_cuda_float16, test/test_native_mha.py::TestMHADeviceTypeCUDA::test_native_multihead_encoder_decoder_attention_cuda_float32, test/test_native_mha.py::TestMHADeviceTypeCUDA::test_native_multihead_self_attention_use_nt_False_use_padding_False_pad_all_False_need_weights_False_average_attn_weights_False_fused_False_cuda_float16, test/test_native_mha.py::TestMHADeviceTypeCUDA::test_native_multihead_self_attention_use_nt_False_use_padding_False_pad_all_False_need_weights_False_average_attn_weights_False_fused_False_cuda_float32, test/test_native_mha.py::TestMHADeviceTypeCUDA::test_native_multihead_self_attention_use_nt_False_use_padding_False_pad_all_False_need_weights_False_average_attn_weights_False_fused_True_cuda_float16, test/test_native_mha.py::TestMHADeviceTypeCUDA::test_native_multihead_self_attention_use_nt_False_use_padding_False_pad_all_False_need_weights_False_average_attn_weights_False_fused_True_cuda_float32, test/test_native_mha.py::TestMHADeviceTypeCUDA::test_native_multihead_self_attention_use_nt_False_use_padding_False_pad_all_False_need_weights_False_average_attn_weights_True_fused_False_cuda_float16, test/test_native_mha.py::TestMHADeviceTypeCUDA::test_native_multihead_self_attention_use_nt_False_use_padding_False_pad_all_False_need_weights_False_average_attn_weights_True_fused_False_cuda_float32, test/test_native_mha.py::TestMHADeviceTypeCUDA::test_native_multihead_self_attention_use_nt_False_use_padding_False_pad_all_False_need_weights_False_average_attn_weights_True_fused_True_cuda_float16, test/test_native_mha.py::TestMHADeviceTypeCUDA::test_native_multihead_self_attention_use_nt_False_use_padding_False_pad_all_False_need_weights_False_average_attn_weights_True_fused_True_cuda_float32, test/test_native_mha.py::TestMHADeviceTypeCUDA::test_native_multihead_self_attention_use_nt_False_use_padding_True_pad_all_False_need_weights_False_average_attn_weights_False_fused_False_cuda_float16, test/test_native_mha.py::TestMHADeviceTypeCUDA::test_native_multihead_self_attention_use_nt_False_use_padding_True_pad_all_False_need_weights_False_average_attn_weights_False_fused_False_cuda_float32, test/test_native_mha.py::TestMHADeviceTypeCUDA::test_native_multihead_self_attention_use_nt_False_use_padding_True_pad_all_False_need_weights_False_average_attn_weights_False_fused_True_cuda_float16, test/test_native_mha.py::TestMHADeviceTypeCUDA::test_native_multihead_self_attention_use_nt_False_use_padding_True_pad_all_False_need_weights_False_average_attn_weights_False_fused_True_cuda_float32, test/test_native_mha.py::TestMHADeviceTypeCUDA::test_native_multihead_self_attention_use_nt_False_use_padding_True_pad_all_False_need_weights_False_average_attn_weights_True_fused_False_cuda_float16, test/test_native_mha.py::TestMHADeviceTypeCUDA::test_native_multihead_self_attention_use_nt_False_use_padding_True_pad_all_False_need_weights_False_average_attn_weights_True_fused_False_cuda_float32, test/test_native_mha.py::TestMHADeviceTypeCUDA::test_native_multihead_self_attention_use_nt_False_use_padding_True_pad_all_False_need_weights_False_average_attn_weights_True_fused_True_cuda_float16, test/test_native_mha.py::TestMHADeviceTypeCUDA::test_native_multihead_self_attention_use_nt_False_use_padding_True_pad_all_False_need_weights_False_average_attn_weights_True_fused_True_cuda_float32, test/test_native_mha.py::TestMHADeviceTypeCUDA::test_native_multihead_self_attention_use_nt_False_use_padding_True_pad_all_True_need_weights_False_average_attn_weights_False_fused_False_cuda_float16, test/test_native_mha.py::TestMHADeviceTypeCUDA::test_native_multihead_self_attention_use_nt_False_use_padding_True_pad_all_True_need_weights_False_average_attn_weights_False_fused_False_cuda_float32, test/test_native_mha.py::TestMHADeviceTypeCUDA::test_native_multihead_self_attention_use_nt_False_use_padding_True_pad_all_True_need_weights_False_average_attn_weights_False_fused_True_cuda_float16, test/test_native_mha.py::TestMHADeviceTypeCUDA::test_native_multihead_self_attention_use_nt_False_use_padding_True_pad_all_True_need_weights_False_average_attn_weights_False_fused_True_cuda_float32, test/test_native_mha.py::TestMHADeviceTypeCUDA::test_native_multihead_self_attention_use_nt_False_use_padding_True_pad_all_True_need_weights_False_average_attn_weights_True_fused_False_cuda_float16, test/test_native_mha.py::TestMHADeviceTypeCUDA::test_native_multihead_self_attention_use_nt_False_use_padding_True_pad_all_True_need_weights_False_average_attn_weights_True_fused_False_cuda_float32, test/test_native_mha.py::TestMHADeviceTypeCUDA::test_native_multihead_self_attention_use_nt_False_use_padding_True_pad_all_True_need_weights_False_average_attn_weights_True_fused_True_cuda_float16, test/test_native_mha.py::TestMHADeviceTypeCUDA::test_native_multihead_self_attention_use_nt_False_use_padding_True_pad_all_True_need_weights_False_average_attn_weights_True_fused_True_cuda_float32, test/test_native_mha.py::TestMHADeviceTypeCUDA::test_native_multihead_self_attention_use_nt_True_use_padding_False_pad_all_False_need_weights_False_average_attn_weights_False_fused_False_cuda_float16, test/test_native_mha.py::TestMHADeviceTypeCUDA::test_native_multihead_self_attention_use_nt_True_use_padding_False_pad_all_False_need_weights_False_average_attn_weights_False_fused_False_cuda_float32, test/test_native_mha.py::TestMHADeviceTypeCUDA::test_native_multihead_self_attention_use_nt_True_use_padding_False_pad_all_False_need_weights_False_average_attn_weights_False_fused_True_cuda_float16, test/test_native_mha.py::TestMHADeviceTypeCUDA::test_native_multihead_self_attention_use_nt_True_use_padding_False_pad_all_False_need_weights_False_average_attn_weights_False_fused_True_cuda_float32, test/test_native_mha.py::TestMHADeviceTypeCUDA::test_native_multihead_self_attention_use_nt_True_use_padding_False_pad_all_False_need_weights_False_average_attn_weights_True_fused_False_cuda_float16, test/test_native_mha.py::TestMHADeviceTypeCUDA::test_native_multihead_self_attention_use_nt_True_use_padding_False_pad_all_False_need_weights_False_average_attn_weights_True_fused_False_cuda_float32, test/test_native_mha.py::TestMHADeviceTypeCUDA::test_native_multihead_self_attention_use_nt_True_use_padding_False_pad_all_False_need_weights_False_average_attn_weights_True_fused_True_cuda_float16, test/test_native_mha.py::TestMHADeviceTypeCUDA::test_native_multihead_self_attention_use_nt_True_use_padding_False_pad_all_False_need_weights_False_average_attn_weights_True_fused_True_cuda_float32, test/test_native_mha.py::TestMHADeviceTypeCUDA::test_native_multihead_self_attention_use_nt_True_use_padding_True_pad_all_False_need_weights_False_average_attn_weights_False_fused_False_cuda_float16, test/test_native_mha.py::TestMHADeviceTypeCUDA::test_native_multihead_self_attention_use_nt_True_use_padding_True_pad_all_False_need_weights_False_average_attn_weights_False_fused_False_cuda_float32, test/test_native_mha.py::TestMHADeviceTypeCUDA::test_native_multihead_self_attention_use_nt_True_use_padding_True_pad_all_False_need_weights_False_average_attn_weights_False_fused_True_cuda_float16, test/test_native_mha.py::TestMHADeviceTypeCUDA::test_native_multihead_self_attention_use_nt_True_use_padding_True_pad_all_False_need_weights_False_average_attn_weights_False_fused_True_cuda_float32, test/test_native_mha.py::TestMHADeviceTypeCUDA::test_native_multihead_self_attention_use_nt_True_use_padding_True_pad_all_False_need_weights_False_average_attn_weights_True_fused_False_cuda_float16, test/test_native_mha.py::TestMHADeviceTypeCUDA::test_native_multihead_self_attention_use_nt_True_use_padding_True_pad_all_False_need_weights_False_average_attn_weights_True_fused_False_cuda_float32, test/test_native_mha.py::TestMHADeviceTypeCUDA::test_native_multihead_self_attention_use_nt_True_use_padding_True_pad_all_False_need_weights_False_average_attn_weights_True_fused_True_cuda_float16, test/test_native_mha.py::TestMHADeviceTypeCUDA::test_native_multihead_self_attention_use_nt_True_use_padding_True_pad_all_False_need_weights_False_average_attn_weights_True_fused_True_cuda_float32, test/test_native_mha.py::TestMHADeviceTypeCUDA::test_native_multihead_self_attention_use_nt_True_use_padding_True_pad_all_True_need_weights_False_average_attn_weights_False_fused_False_cuda_float16, test/test_native_mha.py::TestMHADeviceTypeCUDA::test_native_multihead_self_attention_use_nt_True_use_padding_True_pad_all_True_need_weights_False_average_attn_weights_False_fused_False_cuda_float32, test/test_native_mha.py::TestMHADeviceTypeCUDA::test_native_multihead_self_attention_use_nt_True_use_padding_True_pad_all_True_need_weights_False_average_attn_weights_False_fused_True_cuda_float16, test/test_native_mha.py::TestMHADeviceTypeCUDA::test_native_multihead_self_attention_use_nt_True_use_padding_True_pad_all_True_need_weights_False_average_attn_weights_False_fused_True_cuda_float32, test/test_native_mha.py::TestMHADeviceTypeCUDA::test_native_multihead_self_attention_use_nt_True_use_padding_True_pad_all_True_need_weights_False_average_attn_weights_True_fused_False_cuda_float16, test/test_native_mha.py::TestMHADeviceTypeCUDA::test_native_multihead_self_attention_use_nt_True_use_padding_True_pad_all_True_need_weights_False_average_attn_weights_True_fused_False_cuda_float32, test/test_native_mha.py::TestMHADeviceTypeCUDA::test_native_multihead_self_attention_use_nt_True_use_padding_True_pad_all_True_need_weights_False_average_attn_weights_True_fused_True_cuda_float16, test/test_native_mha.py::TestMHADeviceTypeCUDA::test_native_multihead_self_attention_use_nt_True_use_padding_True_pad_all_True_need_weights_False_average_attn_weights_True_fused_True_cuda_float32, test/test_native_mha.py::TestMHADeviceTypeCUDA::test_transform_bias_rescale_qkv_cuda_float32, test/test_native_mha.py::TestMHADeviceTypeCUDA::test_transform_bias_rescale_qkv_nested_cuda_float32 2025-12-04T13:55:34.4449168Z 2025-12-04T13:55:34.4449363Z Finished test_native_mha 1/1 ... [2025-12-04 13:55:34.439015][15382.381225917], took 0.07min 2025-12-04T13:55:34.4713266Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/test_native_mha/test_native_mha-948cc387c4d4054b.xml 2025-12-04T13:55:34.5025287Z Running torch_np/numpy_tests/core/test_numerictypes 1/1 ... [2025-12-04 13:55:34.502269][15382.444487478] 2025-12-04T13:55:34.5025802Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2025-12-04T13:55:34.5028590Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'torch_np/numpy_tests/core/test_numerictypes.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-12-04 13:55:34.502585] 2025-12-04T13:55:37.9233303Z 2025-12-04T13:55:37.9234271Z torch_np/numpy_tests/core/test_numerictypes 1/1 was successful, full logs can be found in artifacts with path test/test-reports/torch_np.numpy_tests.core.test_numerictypes_1.1_cb1f018ca535f77c_.log 2025-12-04T13:55:37.9246213Z Running 34 items in this shard: test/torch_np/numpy_tests/core/test_numerictypes.py::TestCommonType::test_scalar_loses1, test/torch_np/numpy_tests/core/test_numerictypes.py::TestCommonType::test_scalar_loses2, test/torch_np/numpy_tests/core/test_numerictypes.py::TestCommonType::test_scalar_wins, test/torch_np/numpy_tests/core/test_numerictypes.py::TestCommonType::test_scalar_wins2, test/torch_np/numpy_tests/core/test_numerictypes.py::TestCommonType::test_scalar_wins3, test/torch_np/numpy_tests/core/test_numerictypes.py::TestIsSubDType::test_both_abstract, test/torch_np/numpy_tests/core/test_numerictypes.py::TestIsSubDType::test_nondtype_nonscalartype, test/torch_np/numpy_tests/core/test_numerictypes.py::TestIsSubDType::test_same, test/torch_np/numpy_tests/core/test_numerictypes.py::TestIsSubDType::test_sibling_class, test/torch_np/numpy_tests/core/test_numerictypes.py::TestIsSubDType::test_subclass, test/torch_np/numpy_tests/core/test_numerictypes.py::TestIsSubDType::test_subclass_backwards, test/torch_np/numpy_tests/core/test_numerictypes.py::TestBitName::test_abstract, test/torch_np/numpy_tests/core/test_numerictypes.py::TestDocStrings::test_platform_dependent_aliases, test/torch_np/numpy_tests/core/test_numerictypes.py::TestScalarTypeNames::test_names_are_undersood_by_dtype_t0, test/torch_np/numpy_tests/core/test_numerictypes.py::TestScalarTypeNames::test_names_are_undersood_by_dtype_t1, test/torch_np/numpy_tests/core/test_numerictypes.py::TestScalarTypeNames::test_names_are_undersood_by_dtype_t2, test/torch_np/numpy_tests/core/test_numerictypes.py::TestScalarTypeNames::test_names_are_undersood_by_dtype_t3, test/torch_np/numpy_tests/core/test_numerictypes.py::TestScalarTypeNames::test_names_are_undersood_by_dtype_t4, test/torch_np/numpy_tests/core/test_numerictypes.py::TestScalarTypeNames::test_names_are_undersood_by_dtype_t5, test/torch_np/numpy_tests/core/test_numerictypes.py::TestScalarTypeNames::test_names_are_undersood_by_dtype_t6, test/torch_np/numpy_tests/core/test_numerictypes.py::TestScalarTypeNames::test_names_are_undersood_by_dtype_t7, test/torch_np/numpy_tests/core/test_numerictypes.py::TestScalarTypeNames::test_names_are_undersood_by_dtype_t8, test/torch_np/numpy_tests/core/test_numerictypes.py::TestScalarTypeNames::test_names_are_undersood_by_dtype_t9, test/torch_np/numpy_tests/core/test_numerictypes.py::TestScalarTypeNames::test_names_are_unique, test/torch_np/numpy_tests/core/test_numerictypes.py::TestScalarTypeNames::test_names_reflect_attributes_t0, test/torch_np/numpy_tests/core/test_numerictypes.py::TestScalarTypeNames::test_names_reflect_attributes_t1, test/torch_np/numpy_tests/core/test_numerictypes.py::TestScalarTypeNames::test_names_reflect_attributes_t2, test/torch_np/numpy_tests/core/test_numerictypes.py::TestScalarTypeNames::test_names_reflect_attributes_t3, test/torch_np/numpy_tests/core/test_numerictypes.py::TestScalarTypeNames::test_names_reflect_attributes_t4, test/torch_np/numpy_tests/core/test_numerictypes.py::TestScalarTypeNames::test_names_reflect_attributes_t5, test/torch_np/numpy_tests/core/test_numerictypes.py::TestScalarTypeNames::test_names_reflect_attributes_t6, test/torch_np/numpy_tests/core/test_numerictypes.py::TestScalarTypeNames::test_names_reflect_attributes_t7, test/torch_np/numpy_tests/core/test_numerictypes.py::TestScalarTypeNames::test_names_reflect_attributes_t8, test/torch_np/numpy_tests/core/test_numerictypes.py::TestScalarTypeNames::test_names_reflect_attributes_t9 2025-12-04T13:55:37.9256184Z 2025-12-04T13:55:37.9256469Z Finished torch_np/numpy_tests/core/test_numerictypes 1/1 ... [2025-12-04 13:55:37.923007][15385.865219379], took 0.06min 2025-12-04T13:55:37.9554752Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/torch_np.numpy_tests.core.test_numerictypes/torch_np.numpy_tests.core.test_numerictypes-39c198af6c0e3f47.xml 2025-12-04T13:55:38.0161455Z Running test_function_schema 1/1 ... [2025-12-04 13:55:38.015899][15385.958117912] 2025-12-04T13:55:38.0161899Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2025-12-04T13:55:38.0164543Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'test_function_schema.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-12-04 13:55:38.016188] 2025-12-04T13:55:41.5366819Z 2025-12-04T13:55:41.5368482Z test_function_schema 1/1 was successful, full logs can be found in artifacts with path test/test-reports/test_function_schema_1.1_9409a5d6f9c95b37_.log 2025-12-04T13:55:41.5373804Z Running 15 items in this shard: test/test_function_schema.py::TestFunctionSchema::test_backward_compatible_arguments, test/test_function_schema.py::TestFunctionSchema::test_backward_compatible_outputs, test/test_function_schema.py::TestFunctionSchema::test_backward_compatible_structure, test/test_function_schema.py::TestFunctionSchema::test_backward_compatible_with_smart_serialization, test/test_function_schema.py::TestFunctionSchema::test_forward_compatible_arguments_real_use_case, test/test_function_schema.py::TestFunctionSchema::test_forward_compatible_arguments_with_out, test/test_function_schema.py::TestFunctionSchema::test_forward_compatible_arguments_without_out, test/test_function_schema.py::TestFunctionSchema::test_hash_schema, test/test_function_schema.py::TestFunctionSchema::test_out_schema, test/test_function_schema.py::TestFunctionSchema::test_schema_error, test/test_function_schema.py::TestFunctionSchema::test_serialize_and_deserialize, test/test_function_schema.py::TestFunctionSchema::test_string_optional_parameter_default_value, test/test_function_schema.py::TestFunctionSchema::test_sym_int_argument_properly_parsed, test/test_function_schema.py::TestFunctionSchema::test_tensor_list_alias_annotation_properly_parsed, test/test_function_schema.py::TestFunctionSchema::test_tensor_option_arguments_properly_parsed 2025-12-04T13:55:41.5377742Z 2025-12-04T13:55:41.5378750Z Finished test_function_schema 1/1 ... [2025-12-04 13:55:41.536333][15389.478540487], took 0.06min 2025-12-04T13:55:41.5699372Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/test_function_schema/test_function_schema-aaa43be3c6542b34.xml 2025-12-04T13:55:41.6009659Z Running nn/test_init 1/1 ... [2025-12-04 13:55:41.600753][15389.542971765] 2025-12-04T13:55:41.6010242Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2025-12-04T13:55:41.6013219Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'nn/test_init.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-12-04 13:55:41.601071] 2025-12-04T13:55:48.0766808Z 2025-12-04T13:55:48.0767647Z nn/test_init 1/1 was successful, full logs can be found in artifacts with path test/test-reports/nn.test_init_1.1_7a042e75797ecf7c_.log 2025-12-04T13:55:48.0774566Z Running 30 items in this shard: test/nn/test_init.py::TestNNInit::test_calculate_gain_leaky_relu, test/nn/test_init.py::TestNNInit::test_calculate_gain_leaky_relu_only_accepts_numbers, test/nn/test_init.py::TestNNInit::test_calculate_gain_linear, test/nn/test_init.py::TestNNInit::test_calculate_gain_nonlinear, test/nn/test_init.py::TestNNInit::test_calculate_gain_only_accepts_valid_nonlinearities, test/nn/test_init.py::TestNNInit::test_constant, test/nn/test_init.py::TestNNInit::test_deprecation, test/nn/test_init.py::TestNNInit::test_dirac_identity, test/nn/test_init.py::TestNNInit::test_dirac_only_works_on_3_4_5d_inputs, test/nn/test_init.py::TestNNInit::test_dirac_properties, test/nn/test_init.py::TestNNInit::test_eye, test/nn/test_init.py::TestNNInit::test_eye_only_works_on_2d_inputs, test/nn/test_init.py::TestNNInit::test_kaiming_normal, test/nn/test_init.py::TestNNInit::test_kaiming_normal_errors_on_inputs_smaller_than_2d, test/nn/test_init.py::TestNNInit::test_kaiming_normal_warning_on_0element_tensor, test/nn/test_init.py::TestNNInit::test_kaiming_uniform, test/nn/test_init.py::TestNNInit::test_kaiming_uniform_errors_on_inputs_smaller_than_2d, test/nn/test_init.py::TestNNInit::test_kaiming_uniform_warning_on_0element_tensor, test/nn/test_init.py::TestNNInit::test_normal, test/nn/test_init.py::TestNNInit::test_ones_and_zeros, test/nn/test_init.py::TestNNInit::test_orthogonal, test/nn/test_init.py::TestNNInit::test_sparse_default_std, test/nn/test_init.py::TestNNInit::test_sparse_only_works_on_2d_inputs, test/nn/test_init.py::TestNNInit::test_trunc_normal, test/nn/test_init.py::TestNNInit::test_trunc_normal_generator, test/nn/test_init.py::TestNNInit::test_uniform, test/nn/test_init.py::TestNNInit::test_xavier_normal, test/nn/test_init.py::TestNNInit::test_xavier_normal_errors_on_inputs_smaller_than_2d, test/nn/test_init.py::TestNNInit::test_xavier_uniform, test/nn/test_init.py::TestNNInit::test_xavier_uniform_errors_on_inputs_smaller_than_2d 2025-12-04T13:55:48.0780345Z 2025-12-04T13:55:48.0780534Z Finished nn/test_init 1/1 ... [2025-12-04 13:55:48.076327][15396.018539442], took 0.11min 2025-12-04T13:55:48.1092214Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/nn.test_init/nn.test_init-f88b1f9c9665697b.xml 2025-12-04T13:55:48.1915853Z Running torch_np/test_scalars_0D_arrays 1/1 ... [2025-12-04 13:55:48.191329][15396.133548188] 2025-12-04T13:55:48.1916347Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2025-12-04T13:55:48.1919020Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'torch_np/test_scalars_0D_arrays.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-12-04 13:55:48.191640] 2025-12-04T13:55:51.5620547Z 2025-12-04T13:55:51.5621433Z torch_np/test_scalars_0D_arrays 1/1 was successful, full logs can be found in artifacts with path test/test-reports/torch_np.test_scalars_0D_arrays_1.1_a5e82800420456e1_.log 2025-12-04T13:55:51.5630547Z Running 33 items in this shard: test/torch_np/test_scalars_0D_arrays.py::TestArrayScalars::test_array_scalar_basic_array, test/torch_np/test_scalars_0D_arrays.py::TestArrayScalars::test_array_scalar_basic_asarray, test/torch_np/test_scalars_0D_arrays.py::TestArrayScalars::test_array_scalar_basic_asarray_int, test/torch_np/test_scalars_0D_arrays.py::TestArrayScalars::test_array_scalar_basic_int64, test/torch_np/test_scalars_0D_arrays.py::TestArrayScalars::test_conversion_to_int_array, test/torch_np/test_scalars_0D_arrays.py::TestArrayScalars::test_conversion_to_int_asarray, test/torch_np/test_scalars_0D_arrays.py::TestArrayScalars::test_conversion_to_int_asarray_int, test/torch_np/test_scalars_0D_arrays.py::TestArrayScalars::test_conversion_to_int_int64, test/torch_np/test_scalars_0D_arrays.py::TestArrayScalars::test_decay_to_py_scalar_array, test/torch_np/test_scalars_0D_arrays.py::TestArrayScalars::test_decay_to_py_scalar_asarray, test/torch_np/test_scalars_0D_arrays.py::TestArrayScalars::test_decay_to_py_scalar_asarray_int, test/torch_np/test_scalars_0D_arrays.py::TestArrayScalars::test_decay_to_py_scalar_int64, test/torch_np/test_scalars_0D_arrays.py::TestArrayScalars::test_scalar_comparisons, test/torch_np/test_scalars_0D_arrays.py::TestIsScalar::test_is_not_scalar_value0, test/torch_np/test_scalars_0D_arrays.py::TestIsScalar::test_is_not_scalar_value1, test/torch_np/test_scalars_0D_arrays.py::TestIsScalar::test_is_not_scalar_value10, test/torch_np/test_scalars_0D_arrays.py::TestIsScalar::test_is_not_scalar_value11, test/torch_np/test_scalars_0D_arrays.py::TestIsScalar::test_is_not_scalar_value4, test/torch_np/test_scalars_0D_arrays.py::TestIsScalar::test_is_not_scalar_value5, test/torch_np/test_scalars_0D_arrays.py::TestIsScalar::test_is_not_scalar_value6, test/torch_np/test_scalars_0D_arrays.py::TestIsScalar::test_is_not_scalar_value7, test/torch_np/test_scalars_0D_arrays.py::TestIsScalar::test_is_not_scalar_value8, test/torch_np/test_scalars_0D_arrays.py::TestIsScalar::test_is_not_scalar_value9, test/torch_np/test_scalars_0D_arrays.py::TestIsScalar::test_is_not_scalar_value_s, test/torch_np/test_scalars_0D_arrays.py::TestIsScalar::test_is_not_scalar_value_string, test/torch_np/test_scalars_0D_arrays.py::TestIsScalar::test_is_scalar_array_0D, test/torch_np/test_scalars_0D_arrays.py::TestIsScalar::test_is_scalar_array_1D, test/torch_np/test_scalars_0D_arrays.py::TestIsScalar::test_is_scalar_array_2D, test/torch_np/test_scalars_0D_arrays.py::TestIsScalar::test_is_scalar_float32, test/torch_np/test_scalars_0D_arrays.py::TestIsScalar::test_is_scalar_int, test/torch_np/test_scalars_0D_arrays.py::TestIsScalar::test_is_scalar_list, test/torch_np/test_scalars_0D_arrays.py::TestIsScalar::test_is_scalar_list-list, test/torch_np/test_scalars_0D_arrays.py::TestIsScalar::test_is_scalar_literal 2025-12-04T13:55:51.5638198Z 2025-12-04T13:55:51.5638439Z Finished torch_np/test_scalars_0D_arrays 1/1 ... [2025-12-04 13:55:51.561827][15399.504044097], took 0.06min 2025-12-04T13:55:51.5945563Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/torch_np.test_scalars_0D_arrays/torch_np.test_scalars_0D_arrays-94a518ca987c13a8.xml 2025-12-04T13:55:51.6243308Z Running test_cuda_primary_ctx 1/1 ... [2025-12-04 13:55:51.624100][15399.566318406] 2025-12-04T13:55:51.6243753Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2025-12-04T13:55:51.6246687Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'test_cuda_primary_ctx.py', '--shard-id=1', '--num-shards=1', '-v', '--subprocess', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-12-04 13:55:51.624418] 2025-12-04T13:56:08.0642592Z 2025-12-04T13:56:08.0643415Z test_cuda_primary_ctx 1/1 was successful, full logs can be found in artifacts with path test/test-reports/test_cuda_primary_ctx_1.1_989b62dfd14bfc8d_.log 2025-12-04T13:56:08.0645305Z Running 4 items in this shard: test/test_cuda_primary_ctx.py::TestCudaPrimaryCtx::test_copy, test/test_cuda_primary_ctx.py::TestCudaPrimaryCtx::test_pin_memory, test/test_cuda_primary_ctx.py::TestCudaPrimaryCtx::test_set_device_0, test/test_cuda_primary_ctx.py::TestCudaPrimaryCtx::test_str_repr 2025-12-04T13:56:08.0646805Z Running 1 items in this shard: test/test_cuda_primary_ctx.py::TestCudaPrimaryCtx::test_copy 2025-12-04T13:56:08.0647458Z Running 1 items in this shard: test/test_cuda_primary_ctx.py::TestCudaPrimaryCtx::test_pin_memory 2025-12-04T13:56:08.0648277Z Running 1 items in this shard: test/test_cuda_primary_ctx.py::TestCudaPrimaryCtx::test_set_device_0 2025-12-04T13:56:08.0649067Z Running 1 items in this shard: test/test_cuda_primary_ctx.py::TestCudaPrimaryCtx::test_str_repr 2025-12-04T13:56:08.0649450Z 2025-12-04T13:56:08.0649704Z Finished test_cuda_primary_ctx 1/1 ... [2025-12-04 13:56:08.064126][15416.00634204], took 0.27min 2025-12-04T13:56:08.0974963Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/test_cuda_primary_ctx/test_cuda_primary_ctx-4703821b78d2c346.xml 2025-12-04T13:56:08.1637799Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/test_cuda_primary_ctx/test_cuda_primary_ctx-b8c58bb378a7356d.xml 2025-12-04T13:56:08.2238541Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/test_cuda_primary_ctx/test_cuda_primary_ctx-772a569d1e3ca8af.xml 2025-12-04T13:56:08.2560947Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/test_cuda_primary_ctx/test_cuda_primary_ctx-6abad4fe89929138.xml 2025-12-04T13:56:08.3341126Z Running profiler/test_profiler_tree 1/1 ... [2025-12-04 13:56:08.333888][15416.276105981] 2025-12-04T13:56:08.3341597Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2025-12-04T13:56:08.3344337Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'profiler/test_profiler_tree.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-12-04 13:56:08.334181] 2025-12-04T13:56:11.8548500Z 2025-12-04T13:56:11.8549493Z profiler/test_profiler_tree 1/1 was successful, full logs can be found in artifacts with path test/test-reports/profiler.test_profiler_tree_1.1_afa882d88da46829_.log 2025-12-04T13:56:11.8553771Z Running 10 items in this shard: test/profiler/test_profiler_tree.py::TestProfilerTree::test_profiler_experimental_tree, test/profiler/test_profiler_tree.py::TestProfilerTree::test_profiler_experimental_tree_cuda, test/profiler/test_profiler_tree.py::TestProfilerTree::test_profiler_experimental_tree_cuda_detailed, test/profiler/test_profiler_tree.py::TestProfilerTree::test_profiler_experimental_tree_cuda_with_stream, test/profiler/test_profiler_tree.py::TestProfilerTree::test_profiler_experimental_tree_with_memory, test/profiler/test_profiler_tree.py::TestProfilerTree::test_profiler_experimental_tree_with_memory_and_stack, test/profiler/test_profiler_tree.py::TestProfilerTree::test_profiler_experimental_tree_with_record_function, test/profiler/test_profiler_tree.py::TestProfilerTree::test_profiler_experimental_tree_with_stack_and_modules, test/profiler/test_profiler_tree.py::TestProfilerTree::test_profiler_experimental_tree_with_stack_and_torch_dispatch, test/profiler/test_profiler_tree.py::TestProfilerTree::test_profiler_experimental_tree_with_stack_and_torch_function 2025-12-04T13:56:11.8557379Z 2025-12-04T13:56:11.8557616Z Finished profiler/test_profiler_tree 1/1 ... [2025-12-04 13:56:11.854574][15419.796783409], took 0.06min 2025-12-04T13:56:11.8889660Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/profiler.test_profiler_tree/profiler.test_profiler_tree-96e743ca3bbe44d8.xml 2025-12-04T13:56:11.9220641Z Running test_dlpack 1/1 ... [2025-12-04 13:56:11.921829][15419.86404738] 2025-12-04T13:56:11.9221040Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2025-12-04T13:56:11.9224097Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'test_dlpack.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-12-04 13:56:11.922140] 2025-12-04T13:56:15.8936826Z 2025-12-04T13:56:15.8937887Z test_dlpack 1/1 was successful, full logs can be found in artifacts with path test/test-reports/test_dlpack_1.1_2aa44fcbfbf54c48_.log 2025-12-04T13:56:15.8976999Z Running 154 items in this shard: test/test_dlpack.py::TestTorchDlPackCUDA::test_automatically_select_in_creation_cuda, test/test_dlpack.py::TestTorchDlPackCUDA::test_copy_cuda, test/test_dlpack.py::TestTorchDlPackCUDA::test_dlpack_capsule_conversion_cuda_bfloat16, test/test_dlpack.py::TestTorchDlPackCUDA::test_dlpack_capsule_conversion_cuda_bool, test/test_dlpack.py::TestTorchDlPackCUDA::test_dlpack_capsule_conversion_cuda_complex128, test/test_dlpack.py::TestTorchDlPackCUDA::test_dlpack_capsule_conversion_cuda_complex64, test/test_dlpack.py::TestTorchDlPackCUDA::test_dlpack_capsule_conversion_cuda_float16, test/test_dlpack.py::TestTorchDlPackCUDA::test_dlpack_capsule_conversion_cuda_float32, test/test_dlpack.py::TestTorchDlPackCUDA::test_dlpack_capsule_conversion_cuda_float64, test/test_dlpack.py::TestTorchDlPackCUDA::test_dlpack_capsule_conversion_cuda_int16, test/test_dlpack.py::TestTorchDlPackCUDA::test_dlpack_capsule_conversion_cuda_int32, test/test_dlpack.py::TestTorchDlPackCUDA::test_dlpack_capsule_conversion_cuda_int64, test/test_dlpack.py::TestTorchDlPackCUDA::test_dlpack_capsule_conversion_cuda_int8, test/test_dlpack.py::TestTorchDlPackCUDA::test_dlpack_capsule_conversion_cuda_uint16, test/test_dlpack.py::TestTorchDlPackCUDA::test_dlpack_capsule_conversion_cuda_uint32, test/test_dlpack.py::TestTorchDlPackCUDA::test_dlpack_capsule_conversion_cuda_uint64, test/test_dlpack.py::TestTorchDlPackCUDA::test_dlpack_capsule_conversion_cuda_uint8, test/test_dlpack.py::TestTorchDlPackCUDA::test_dlpack_conversion_with_diff_streams_cuda_bfloat16, test/test_dlpack.py::TestTorchDlPackCUDA::test_dlpack_conversion_with_diff_streams_cuda_bool, test/test_dlpack.py::TestTorchDlPackCUDA::test_dlpack_conversion_with_diff_streams_cuda_complex128, test/test_dlpack.py::TestTorchDlPackCUDA::test_dlpack_conversion_with_diff_streams_cuda_complex64, test/test_dlpack.py::TestTorchDlPackCUDA::test_dlpack_conversion_with_diff_streams_cuda_float16, test/test_dlpack.py::TestTorchDlPackCUDA::test_dlpack_conversion_with_diff_streams_cuda_float32, test/test_dlpack.py::TestTorchDlPackCUDA::test_dlpack_conversion_with_diff_streams_cuda_float64, test/test_dlpack.py::TestTorchDlPackCUDA::test_dlpack_conversion_with_diff_streams_cuda_int16, test/test_dlpack.py::TestTorchDlPackCUDA::test_dlpack_conversion_with_diff_streams_cuda_int32, test/test_dlpack.py::TestTorchDlPackCUDA::test_dlpack_conversion_with_diff_streams_cuda_int64, test/test_dlpack.py::TestTorchDlPackCUDA::test_dlpack_conversion_with_diff_streams_cuda_int8, test/test_dlpack.py::TestTorchDlPackCUDA::test_dlpack_conversion_with_diff_streams_cuda_uint8, test/test_dlpack.py::TestTorchDlPackCUDA::test_dlpack_conversion_with_diff_streams_narrow_precision_cuda_float4_e2m1fn_x2, test/test_dlpack.py::TestTorchDlPackCUDA::test_dlpack_conversion_with_diff_streams_narrow_precision_cuda_float8_e4m3fn, test/test_dlpack.py::TestTorchDlPackCUDA::test_dlpack_conversion_with_diff_streams_narrow_precision_cuda_float8_e4m3fnuz, test/test_dlpack.py::TestTorchDlPackCUDA::test_dlpack_conversion_with_diff_streams_narrow_precision_cuda_float8_e5m2, test/test_dlpack.py::TestTorchDlPackCUDA::test_dlpack_conversion_with_diff_streams_narrow_precision_cuda_float8_e5m2fnuz, test/test_dlpack.py::TestTorchDlPackCUDA::test_dlpack_conversion_with_diff_streams_narrow_precision_cuda_float8_e8m0fnu, test/test_dlpack.py::TestTorchDlPackCUDA::test_dlpack_conversion_with_streams_cuda_bfloat16, test/test_dlpack.py::TestTorchDlPackCUDA::test_dlpack_conversion_with_streams_cuda_bool, test/test_dlpack.py::TestTorchDlPackCUDA::test_dlpack_conversion_with_streams_cuda_complex128, test/test_dlpack.py::TestTorchDlPackCUDA::test_dlpack_conversion_with_streams_cuda_complex64, test/test_dlpack.py::TestTorchDlPackCUDA::test_dlpack_conversion_with_streams_cuda_float16, test/test_dlpack.py::TestTorchDlPackCUDA::test_dlpack_conversion_with_streams_cuda_float32, test/test_dlpack.py::TestTorchDlPackCUDA::test_dlpack_conversion_with_streams_cuda_float64, test/test_dlpack.py::TestTorchDlPackCUDA::test_dlpack_conversion_with_streams_cuda_int16, test/test_dlpack.py::TestTorchDlPackCUDA::test_dlpack_conversion_with_streams_cuda_int32, test/test_dlpack.py::TestTorchDlPackCUDA::test_dlpack_conversion_with_streams_cuda_int64, test/test_dlpack.py::TestTorchDlPackCUDA::test_dlpack_conversion_with_streams_cuda_int8, test/test_dlpack.py::TestTorchDlPackCUDA::test_dlpack_conversion_with_streams_cuda_uint8, test/test_dlpack.py::TestTorchDlPackCUDA::test_dlpack_conversion_with_streams_narrow_precision_cuda_float4_e2m1fn_x2, test/test_dlpack.py::TestTorchDlPackCUDA::test_dlpack_conversion_with_streams_narrow_precision_cuda_float8_e4m3fn, test/test_dlpack.py::TestTorchDlPackCUDA::test_dlpack_conversion_with_streams_narrow_precision_cuda_float8_e4m3fnuz, test/test_dlpack.py::TestTorchDlPackCUDA::test_dlpack_conversion_with_streams_narrow_precision_cuda_float8_e5m2, test/test_dlpack.py::TestTorchDlPackCUDA::test_dlpack_conversion_with_streams_narrow_precision_cuda_float8_e5m2fnuz, test/test_dlpack.py::TestTorchDlPackCUDA::test_dlpack_conversion_with_streams_narrow_precision_cuda_float8_e8m0fnu, test/test_dlpack.py::TestTorchDlPackCUDA::test_dlpack_convert_default_stream_cuda, test/test_dlpack.py::TestTorchDlPackCUDA::test_dlpack_cuda_per_thread_stream_cuda, test/test_dlpack.py::TestTorchDlPackCUDA::test_dlpack_default_stream_cuda, test/test_dlpack.py::TestTorchDlPackCUDA::test_dlpack_export_is_conj_cuda, test/test_dlpack.py::TestTorchDlPackCUDA::test_dlpack_export_non_strided_cuda, test/test_dlpack.py::TestTorchDlPackCUDA::test_dlpack_export_requires_grad_cuda, test/test_dlpack.py::TestTorchDlPackCUDA::test_dlpack_invalid_cpu_stream_cuda, test/test_dlpack.py::TestTorchDlPackCUDA::test_dlpack_invalid_cuda_streams_cuda, test/test_dlpack.py::TestTorchDlPackCUDA::test_dlpack_invalid_rocm_streams_cuda, test/test_dlpack.py::TestTorchDlPackCUDA::test_dlpack_normalize_strides_cuda, test/test_dlpack.py::TestTorchDlPackCUDA::test_dlpack_protocol_conversion_cuda_bfloat16, test/test_dlpack.py::TestTorchDlPackCUDA::test_dlpack_protocol_conversion_cuda_bool, test/test_dlpack.py::TestTorchDlPackCUDA::test_dlpack_protocol_conversion_cuda_complex128, test/test_dlpack.py::TestTorchDlPackCUDA::test_dlpack_protocol_conversion_cuda_complex64, test/test_dlpack.py::TestTorchDlPackCUDA::test_dlpack_protocol_conversion_cuda_float16, test/test_dlpack.py::TestTorchDlPackCUDA::test_dlpack_protocol_conversion_cuda_float32, test/test_dlpack.py::TestTorchDlPackCUDA::test_dlpack_protocol_conversion_cuda_float64, test/test_dlpack.py::TestTorchDlPackCUDA::test_dlpack_protocol_conversion_cuda_int16, test/test_dlpack.py::TestTorchDlPackCUDA::test_dlpack_protocol_conversion_cuda_int32, test/test_dlpack.py::TestTorchDlPackCUDA::test_dlpack_protocol_conversion_cuda_int64, test/test_dlpack.py::TestTorchDlPackCUDA::test_dlpack_protocol_conversion_cuda_int8, test/test_dlpack.py::TestTorchDlPackCUDA::test_dlpack_protocol_conversion_cuda_uint16, test/test_dlpack.py::TestTorchDlPackCUDA::test_dlpack_protocol_conversion_cuda_uint32, test/test_dlpack.py::TestTorchDlPackCUDA::test_dlpack_protocol_conversion_cuda_uint64, test/test_dlpack.py::TestTorchDlPackCUDA::test_dlpack_protocol_conversion_cuda_uint8, test/test_dlpack.py::TestTorchDlPackCUDA::test_dlpack_shared_storage_cuda, test/test_dlpack.py::TestTorchDlPackCUDA::test_dlpack_tensor_invalid_stream_cuda_bfloat16, test/test_dlpack.py::TestTorchDlPackCUDA::test_dlpack_tensor_invalid_stream_cuda_complex128, test/test_dlpack.py::TestTorchDlPackCUDA::test_dlpack_tensor_invalid_stream_cuda_complex64, test/test_dlpack.py::TestTorchDlPackCUDA::test_dlpack_tensor_invalid_stream_cuda_float16, test/test_dlpack.py::TestTorchDlPackCUDA::test_dlpack_tensor_invalid_stream_cuda_float32, test/test_dlpack.py::TestTorchDlPackCUDA::test_dlpack_tensor_invalid_stream_cuda_float64, test/test_dlpack.py::TestTorchDlPackCUDA::test_dlpack_tensor_invalid_stream_cuda_int16, test/test_dlpack.py::TestTorchDlPackCUDA::test_dlpack_tensor_invalid_stream_cuda_int32, test/test_dlpack.py::TestTorchDlPackCUDA::test_dlpack_tensor_invalid_stream_cuda_int64, test/test_dlpack.py::TestTorchDlPackCUDA::test_dlpack_tensor_invalid_stream_cuda_int8, test/test_dlpack.py::TestTorchDlPackCUDA::test_dlpack_tensor_invalid_stream_cuda_uint8, test/test_dlpack.py::TestTorchDlPackCUDA::test_dlpack_tensor_on_different_device_cuda, test/test_dlpack.py::TestTorchDlPackCUDA::test_dlpack_unsupported_dtype_error_cuda, test/test_dlpack.py::TestTorchDlPackCUDA::test_from_dlpack_cuda_bfloat16, test/test_dlpack.py::TestTorchDlPackCUDA::test_from_dlpack_cuda_bool, test/test_dlpack.py::TestTorchDlPackCUDA::test_from_dlpack_cuda_complex128, test/test_dlpack.py::TestTorchDlPackCUDA::test_from_dlpack_cuda_complex64, test/test_dlpack.py::TestTorchDlPackCUDA::test_from_dlpack_cuda_float16, test/test_dlpack.py::TestTorchDlPackCUDA::test_from_dlpack_cuda_float32, test/test_dlpack.py::TestTorchDlPackCUDA::test_from_dlpack_cuda_float64, test/test_dlpack.py::TestTorchDlPackCUDA::test_from_dlpack_cuda_int16, test/test_dlpack.py::TestTorchDlPackCUDA::test_from_dlpack_cuda_int32, test/test_dlpack.py::TestTorchDlPackCUDA::test_from_dlpack_cuda_int64, test/test_dlpack.py::TestTorchDlPackCUDA::test_from_dlpack_cuda_int8, test/test_dlpack.py::TestTorchDlPackCUDA::test_from_dlpack_cuda_uint16, test/test_dlpack.py::TestTorchDlPackCUDA::test_from_dlpack_cuda_uint32, test/test_dlpack.py::TestTorchDlPackCUDA::test_from_dlpack_cuda_uint64, test/test_dlpack.py::TestTorchDlPackCUDA::test_from_dlpack_cuda_uint8, test/test_dlpack.py::TestTorchDlPackCUDA::test_from_dlpack_dtype_cuda_bfloat16, test/test_dlpack.py::TestTorchDlPackCUDA::test_from_dlpack_dtype_cuda_bool, test/test_dlpack.py::TestTorchDlPackCUDA::test_from_dlpack_dtype_cuda_complex128, test/test_dlpack.py::TestTorchDlPackCUDA::test_from_dlpack_dtype_cuda_complex64, test/test_dlpack.py::TestTorchDlPackCUDA::test_from_dlpack_dtype_cuda_float16, test/test_dlpack.py::TestTorchDlPackCUDA::test_from_dlpack_dtype_cuda_float32, test/test_dlpack.py::TestTorchDlPackCUDA::test_from_dlpack_dtype_cuda_float64, test/test_dlpack.py::TestTorchDlPackCUDA::test_from_dlpack_dtype_cuda_int16, test/test_dlpack.py::TestTorchDlPackCUDA::test_from_dlpack_dtype_cuda_int32, test/test_dlpack.py::TestTorchDlPackCUDA::test_from_dlpack_dtype_cuda_int64, test/test_dlpack.py::TestTorchDlPackCUDA::test_from_dlpack_dtype_cuda_int8, test/test_dlpack.py::TestTorchDlPackCUDA::test_from_dlpack_dtype_cuda_uint16, test/test_dlpack.py::TestTorchDlPackCUDA::test_from_dlpack_dtype_cuda_uint32, test/test_dlpack.py::TestTorchDlPackCUDA::test_from_dlpack_dtype_cuda_uint64, test/test_dlpack.py::TestTorchDlPackCUDA::test_from_dlpack_dtype_cuda_uint8, test/test_dlpack.py::TestTorchDlPackCUDA::test_from_dlpack_noncontinguous_cuda_bfloat16, test/test_dlpack.py::TestTorchDlPackCUDA::test_from_dlpack_noncontinguous_cuda_bool, test/test_dlpack.py::TestTorchDlPackCUDA::test_from_dlpack_noncontinguous_cuda_complex128, test/test_dlpack.py::TestTorchDlPackCUDA::test_from_dlpack_noncontinguous_cuda_complex64, test/test_dlpack.py::TestTorchDlPackCUDA::test_from_dlpack_noncontinguous_cuda_float16, test/test_dlpack.py::TestTorchDlPackCUDA::test_from_dlpack_noncontinguous_cuda_float32, test/test_dlpack.py::TestTorchDlPackCUDA::test_from_dlpack_noncontinguous_cuda_float64, test/test_dlpack.py::TestTorchDlPackCUDA::test_from_dlpack_noncontinguous_cuda_int16, test/test_dlpack.py::TestTorchDlPackCUDA::test_from_dlpack_noncontinguous_cuda_int32, test/test_dlpack.py::TestTorchDlPackCUDA::test_from_dlpack_noncontinguous_cuda_int64, test/test_dlpack.py::TestTorchDlPackCUDA::test_from_dlpack_noncontinguous_cuda_int8, test/test_dlpack.py::TestTorchDlPackCUDA::test_from_dlpack_noncontinguous_cuda_uint16, test/test_dlpack.py::TestTorchDlPackCUDA::test_from_dlpack_noncontinguous_cuda_uint32, test/test_dlpack.py::TestTorchDlPackCUDA::test_from_dlpack_noncontinguous_cuda_uint64, test/test_dlpack.py::TestTorchDlPackCUDA::test_from_dlpack_noncontinguous_cuda_uint8, test/test_dlpack.py::TestTorchDlPackCUDA::test_max_version_cuda, test/test_dlpack.py::TestTorchDlPackCUDA::test_needs_copy_error_cuda, test/test_dlpack.py::TestTorchDlPackCUDA::test_no_copy_cuda, test/test_dlpack.py::TestTorchDlPackCUDA::test_numpy_dlpack_protocol_conversion_cuda_complex128, test/test_dlpack.py::TestTorchDlPackCUDA::test_numpy_dlpack_protocol_conversion_cuda_complex64, test/test_dlpack.py::TestTorchDlPackCUDA::test_numpy_dlpack_protocol_conversion_cuda_float16, test/test_dlpack.py::TestTorchDlPackCUDA::test_numpy_dlpack_protocol_conversion_cuda_float32, test/test_dlpack.py::TestTorchDlPackCUDA::test_numpy_dlpack_protocol_conversion_cuda_float64, test/test_dlpack.py::TestTorchDlPackCUDA::test_numpy_dlpack_protocol_conversion_cuda_int16, test/test_dlpack.py::TestTorchDlPackCUDA::test_numpy_dlpack_protocol_conversion_cuda_int32, test/test_dlpack.py::TestTorchDlPackCUDA::test_numpy_dlpack_protocol_conversion_cuda_int64, test/test_dlpack.py::TestTorchDlPackCUDA::test_numpy_dlpack_protocol_conversion_cuda_int8, test/test_dlpack.py::TestTorchDlPackCUDA::test_numpy_dlpack_protocol_conversion_cuda_uint16, test/test_dlpack.py::TestTorchDlPackCUDA::test_numpy_dlpack_protocol_conversion_cuda_uint32, test/test_dlpack.py::TestTorchDlPackCUDA::test_numpy_dlpack_protocol_conversion_cuda_uint64, test/test_dlpack.py::TestTorchDlPackCUDA::test_numpy_dlpack_protocol_conversion_cuda_uint8, test/test_dlpack.py::TestTorchDlPackCUDA::test_unsupported_device_error_cuda 2025-12-04T13:56:15.9014711Z 2025-12-04T13:56:15.9014894Z Finished test_dlpack 1/1 ... [2025-12-04 13:56:15.893692][15423.835905774], took 0.07min 2025-12-04T13:56:15.9281697Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/test_dlpack/test_dlpack-f4bd83620146cc39.xml 2025-12-04T13:56:15.9611007Z Running lazy/test_reuse_ir 1/1 ... [2025-12-04 13:56:15.960837][15423.903054433] 2025-12-04T13:56:15.9611685Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2025-12-04T13:56:15.9614515Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'lazy/test_reuse_ir.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-12-04 13:56:15.961166] 2025-12-04T13:56:19.4821876Z 2025-12-04T13:56:19.4822690Z lazy/test_reuse_ir 1/1 was successful, full logs can be found in artifacts with path test/test-reports/lazy.test_reuse_ir_1.1_6a37ac220f458a3d_.log 2025-12-04T13:56:19.4824569Z Running 4 items in this shard: test/lazy/test_reuse_ir.py::TestLazyReuseIr::testAdd, test/lazy/test_reuse_ir.py::TestLazyReuseIr::testAddSub, test/lazy/test_reuse_ir.py::TestLazyReuseIr::testAddSubFallback, test/lazy/test_reuse_ir.py::TestLazyReuseIr::testBatchNorm 2025-12-04T13:56:19.4825562Z 2025-12-04T13:56:19.4825824Z Finished lazy/test_reuse_ir 1/1 ... [2025-12-04 13:56:19.481882][15427.424089483], took 0.06min 2025-12-04T13:56:19.5164798Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/lazy.test_reuse_ir/lazy.test_reuse_ir-494000dbe63af35c.xml 2025-12-04T13:56:19.5486293Z Running test_functional_autograd_benchmark 1/1 ... [2025-12-04 13:56:19.548386][15427.490605159] 2025-12-04T13:56:19.5486780Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2025-12-04T13:56:19.5489483Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'test_functional_autograd_benchmark.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-12-04 13:56:19.548682] 2025-12-04T13:56:42.2491600Z 2025-12-04T13:56:42.2492946Z test_functional_autograd_benchmark 1/1 was successful, full logs can be found in artifacts with path test/test-reports/test_functional_autograd_benchmark_1.1_62fb1a6c4d9b68b7_.log 2025-12-04T13:56:42.2495431Z Running 2 items in this shard: test/test_functional_autograd_benchmark.py::TestFunctionalAutogradBenchmark::test_fast_tasks, test/test_functional_autograd_benchmark.py::TestFunctionalAutogradBenchmark::test_slow_tasks 2025-12-04T13:56:42.2496694Z 2025-12-04T13:56:42.2497041Z Finished test_functional_autograd_benchmark 1/1 ... [2025-12-04 13:56:42.248962][15450.191179443], took 0.38min 2025-12-04T13:56:42.2827267Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/test_functional_autograd_benchmark/test_functional_autograd_benchmark-632b3fce6e67c954.xml 2025-12-04T13:56:42.3590436Z Running distributions/test_distributions 1/1 ... [2025-12-04 13:56:42.358815][15450.301033484] 2025-12-04T13:56:42.3590930Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2025-12-04T13:56:42.3593466Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributions/test_distributions.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-12-04 13:56:42.359104] 2025-12-04T13:57:40.3154368Z 2025-12-04T13:57:40.3156636Z distributions/test_distributions 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributions.test_distributions_1.1_c4dba1c406e36ca0_.log 2025-12-04T13:57:40.3222650Z Running 230 items in this shard: test/distributions/test_distributions.py::TestDistributions::test_argmax_relaxed_categorical, test/distributions/test_distributions.py::TestDistributions::test_bernoulli, test/distributions/test_distributions.py::TestDistributions::test_bernoulli_3d, test/distributions/test_distributions.py::TestDistributions::test_bernoulli_enumerate_support, test/distributions/test_distributions.py::TestDistributions::test_beta_log_prob, test/distributions/test_distributions.py::TestDistributions::test_beta_sample, test/distributions/test_distributions.py::TestDistributions::test_beta_shape, test/distributions/test_distributions.py::TestDistributions::test_beta_underflow, test/distributions/test_distributions.py::TestDistributions::test_beta_underflow_gpu, test/distributions/test_distributions.py::TestDistributions::test_binomial, test/distributions/test_distributions.py::TestDistributions::test_binomial_bfloat16, test/distributions/test_distributions.py::TestDistributions::test_binomial_enumerate_support, test/distributions/test_distributions.py::TestDistributions::test_binomial_extreme_vals, test/distributions/test_distributions.py::TestDistributions::test_binomial_half, test/distributions/test_distributions.py::TestDistributions::test_binomial_log_prob_and_entropy, test/distributions/test_distributions.py::TestDistributions::test_binomial_log_prob_vectorized_count, test/distributions/test_distributions.py::TestDistributions::test_binomial_sample, test/distributions/test_distributions.py::TestDistributions::test_binomial_stable, test/distributions/test_distributions.py::TestDistributions::test_binomial_vectorized_count, test/distributions/test_distributions.py::TestDistributions::test_categorical_1d, test/distributions/test_distributions.py::TestDistributions::test_categorical_2d, test/distributions/test_distributions.py::TestDistributions::test_categorical_enumerate_support, test/distributions/test_distributions.py::TestDistributions::test_cauchy, test/distributions/test_distributions.py::TestDistributions::test_cdf_icdf_inverse, test/distributions/test_distributions.py::TestDistributions::test_cdf_log_prob, test/distributions/test_distributions.py::TestDistributions::test_chi2_sample, test/distributions/test_distributions.py::TestDistributions::test_chi2_shape, test/distributions/test_distributions.py::TestDistributions::test_continuous_bernoulli, test/distributions/test_distributions.py::TestDistributions::test_continuous_bernoulli_3d, test/distributions/test_distributions.py::TestDistributions::test_dirichlet_log_prob, test/distributions/test_distributions.py::TestDistributions::test_dirichlet_log_prob_zero, test/distributions/test_distributions.py::TestDistributions::test_dirichlet_mode, test/distributions/test_distributions.py::TestDistributions::test_dirichlet_sample, test/distributions/test_distributions.py::TestDistributions::test_dirichlet_shape, test/distributions/test_distributions.py::TestDistributions::test_distribution_expand, test/distributions/test_distributions.py::TestDistributions::test_distribution_subclass_expand, test/distributions/test_distributions.py::TestDistributions::test_enumerate_support_type, test/distributions/test_distributions.py::TestDistributions::test_exponential, test/distributions/test_distributions.py::TestDistributions::test_exponential_sample, test/distributions/test_distributions.py::TestDistributions::test_fishersnedecor, test/distributions/test_distributions.py::TestDistributions::test_fishersnedecor_sample, test/distributions/test_distributions.py::TestDistributions::test_gamma_gpu_sample, test/distributions/test_distributions.py::TestDistributions::test_gamma_gpu_shape, test/distributions/test_distributions.py::TestDistributions::test_gamma_log_prob_at_boundary, test/distributions/test_distributions.py::TestDistributions::test_gamma_sample, test/distributions/test_distributions.py::TestDistributions::test_gamma_shape, test/distributions/test_distributions.py::TestDistributions::test_generalized_pareto, test/distributions/test_distributions.py::TestDistributions::test_generalized_pareto_sample, test/distributions/test_distributions.py::TestDistributions::test_geometric, test/distributions/test_distributions.py::TestDistributions::test_geometric_log_prob_and_entropy, test/distributions/test_distributions.py::TestDistributions::test_geometric_sample, test/distributions/test_distributions.py::TestDistributions::test_gumbel, test/distributions/test_distributions.py::TestDistributions::test_gumbel_sample, test/distributions/test_distributions.py::TestDistributions::test_halfcauchy, test/distributions/test_distributions.py::TestDistributions::test_halfnormal, test/distributions/test_distributions.py::TestDistributions::test_halfnormal_logprob, test/distributions/test_distributions.py::TestDistributions::test_halfnormal_sample, test/distributions/test_distributions.py::TestDistributions::test_has_examples, test/distributions/test_distributions.py::TestDistributions::test_independent_expand, test/distributions/test_distributions.py::TestDistributions::test_independent_shape, test/distributions/test_distributions.py::TestDistributions::test_invalid_parameter_broadcasting, test/distributions/test_distributions.py::TestDistributions::test_inversegamma, test/distributions/test_distributions.py::TestDistributions::test_inversegamma_sample, test/distributions/test_distributions.py::TestDistributions::test_kumaraswamy_mean_variance, test/distributions/test_distributions.py::TestDistributions::test_kumaraswamy_shape, test/distributions/test_distributions.py::TestDistributions::test_laplace, test/distributions/test_distributions.py::TestDistributions::test_laplace_sample, test/distributions/test_distributions.py::TestDistributions::test_lazy_property_grad, test/distributions/test_distributions.py::TestDistributions::test_lkj_cholesky_log_prob, test/distributions/test_distributions.py::TestDistributions::test_logisticnormal, test/distributions/test_distributions.py::TestDistributions::test_logisticnormal_logprob, test/distributions/test_distributions.py::TestDistributions::test_logisticnormal_sample, test/distributions/test_distributions.py::TestDistributions::test_lognormal, test/distributions/test_distributions.py::TestDistributions::test_lognormal_logprob, test/distributions/test_distributions.py::TestDistributions::test_lognormal_sample, test/distributions/test_distributions.py::TestDistributions::test_lowrank_multivariate_normal_log_prob, test/distributions/test_distributions.py::TestDistributions::test_lowrank_multivariate_normal_moments, test/distributions/test_distributions.py::TestDistributions::test_lowrank_multivariate_normal_properties, test/distributions/test_distributions.py::TestDistributions::test_lowrank_multivariate_normal_sample, test/distributions/test_distributions.py::TestDistributions::test_lowrank_multivariate_normal_shape, test/distributions/test_distributions.py::TestDistributions::test_mixture_same_family_binomial_log_prob, test/distributions/test_distributions.py::TestDistributions::test_mixture_same_family_normal_log_prob, test/distributions/test_distributions.py::TestDistributions::test_mixture_same_family_sample, test/distributions/test_distributions.py::TestDistributions::test_mixture_same_family_shape, test/distributions/test_distributions.py::TestDistributions::test_mode, test/distributions/test_distributions.py::TestDistributions::test_multinomial_1d, test/distributions/test_distributions.py::TestDistributions::test_multinomial_1d_log_prob_and_entropy, test/distributions/test_distributions.py::TestDistributions::test_multinomial_2d, test/distributions/test_distributions.py::TestDistributions::test_multinomial_sequential_draw, test/distributions/test_distributions.py::TestDistributions::test_multivariate_normal_log_prob, test/distributions/test_distributions.py::TestDistributions::test_multivariate_normal_moments, test/distributions/test_distributions.py::TestDistributions::test_multivariate_normal_properties, test/distributions/test_distributions.py::TestDistributions::test_multivariate_normal_sample, test/distributions/test_distributions.py::TestDistributions::test_multivariate_normal_shape, test/distributions/test_distributions.py::TestDistributions::test_multivariate_normal_stable_with_precision_matrix, test/distributions/test_distributions.py::TestDistributions::test_negative_binomial, test/distributions/test_distributions.py::TestDistributions::test_negative_binomial_log_prob, test/distributions/test_distributions.py::TestDistributions::test_negative_binomial_log_prob_vectorized_count, test/distributions/test_distributions.py::TestDistributions::test_normal, test/distributions/test_distributions.py::TestDistributions::test_normal_sample, test/distributions/test_distributions.py::TestDistributions::test_one_hot_categorical_1d, test/distributions/test_distributions.py::TestDistributions::test_one_hot_categorical_2d, test/distributions/test_distributions.py::TestDistributions::test_one_hot_categorical_enumerate_support, test/distributions/test_distributions.py::TestDistributions::test_pareto, test/distributions/test_distributions.py::TestDistributions::test_pareto_sample, test/distributions/test_distributions.py::TestDistributions::test_poisson_forward_ad, test/distributions/test_distributions.py::TestDistributions::test_poisson_gpu_sample, test/distributions/test_distributions.py::TestDistributions::test_poisson_log_prob, test/distributions/test_distributions.py::TestDistributions::test_poisson_sample, test/distributions/test_distributions.py::TestDistributions::test_poisson_shape, test/distributions/test_distributions.py::TestDistributions::test_relaxed_bernoulli, test/distributions/test_distributions.py::TestDistributions::test_relaxed_one_hot_categorical_1d, test/distributions/test_distributions.py::TestDistributions::test_relaxed_one_hot_categorical_2d, test/distributions/test_distributions.py::TestDistributions::test_repr, test/distributions/test_distributions.py::TestDistributions::test_rounded_relaxed_bernoulli, test/distributions/test_distributions.py::TestDistributions::test_rsample_requires_grad, test/distributions/test_distributions.py::TestDistributions::test_sample_detached, test/distributions/test_distributions.py::TestDistributions::test_studentT, test/distributions/test_distributions.py::TestDistributions::test_studentT_log_prob, test/distributions/test_distributions.py::TestDistributions::test_studentT_sample, test/distributions/test_distributions.py::TestDistributions::test_support_attributes, test/distributions/test_distributions.py::TestDistributions::test_torch_binomial_dtype_errors, test/distributions/test_distributions.py::TestDistributions::test_uniform, test/distributions/test_distributions.py::TestDistributions::test_valid_parameter_broadcasting, test/distributions/test_distributions.py::TestDistributions::test_vonmises_logprob, test/distributions/test_distributions.py::TestDistributions::test_vonmises_sample, test/distributions/test_distributions.py::TestDistributions::test_wishart_log_prob, test/distributions/test_distributions.py::TestDistributions::test_wishart_moments, test/distributions/test_distributions.py::TestDistributions::test_wishart_properties, test/distributions/test_distributions.py::TestDistributions::test_wishart_sample, test/distributions/test_distributions.py::TestDistributions::test_wishart_shape, test/distributions/test_distributions.py::TestDistributions::test_wishart_stable_with_precision_matrix, test/distributions/test_distributions.py::TestDistributions::test_zero_excluded_binomial, test/distributions/test_distributions.py::TestRsample::test_beta_wrt_alpha, test/distributions/test_distributions.py::TestRsample::test_beta_wrt_beta, test/distributions/test_distributions.py::TestRsample::test_chi2, test/distributions/test_distributions.py::TestRsample::test_dirichlet_multivariate, test/distributions/test_distributions.py::TestRsample::test_dirichlet_on_diagonal, test/distributions/test_distributions.py::TestRsample::test_dirichlet_tangent_field, test/distributions/test_distributions.py::TestRsample::test_gamma, test/distributions/test_distributions.py::TestDistributionShapes::test_bernoulli_shape_scalar_params, test/distributions/test_distributions.py::TestDistributionShapes::test_bernoulli_shape_tensor_params, test/distributions/test_distributions.py::TestDistributionShapes::test_beta_shape_scalar_params, test/distributions/test_distributions.py::TestDistributionShapes::test_beta_shape_tensor_params, test/distributions/test_distributions.py::TestDistributionShapes::test_binomial_shape, test/distributions/test_distributions.py::TestDistributionShapes::test_binomial_shape_vectorized_n, test/distributions/test_distributions.py::TestDistributionShapes::test_categorical_shape, test/distributions/test_distributions.py::TestDistributionShapes::test_cauchy_shape_scalar_params, test/distributions/test_distributions.py::TestDistributionShapes::test_cauchy_shape_tensor_params, test/distributions/test_distributions.py::TestDistributionShapes::test_chi2_shape_scalar_params, test/distributions/test_distributions.py::TestDistributionShapes::test_chi2_shape_tensor_params, test/distributions/test_distributions.py::TestDistributionShapes::test_continuous_bernoulli_shape_scalar_params, test/distributions/test_distributions.py::TestDistributionShapes::test_continuous_bernoulli_shape_tensor_params, test/distributions/test_distributions.py::TestDistributionShapes::test_dirichlet_shape, test/distributions/test_distributions.py::TestDistributionShapes::test_entropy_shape, test/distributions/test_distributions.py::TestDistributionShapes::test_exponential_shape_scalar_param, test/distributions/test_distributions.py::TestDistributionShapes::test_exponential_shape_tensor_param, test/distributions/test_distributions.py::TestDistributionShapes::test_gamma_shape_scalar_params, test/distributions/test_distributions.py::TestDistributionShapes::test_gamma_shape_tensor_params, test/distributions/test_distributions.py::TestDistributionShapes::test_geometric_shape_scalar_params, test/distributions/test_distributions.py::TestDistributionShapes::test_geometric_shape_tensor_params, test/distributions/test_distributions.py::TestDistributionShapes::test_gumbel_shape_scalar_params, test/distributions/test_distributions.py::TestDistributionShapes::test_halfcauchy_shape_scalar_params, test/distributions/test_distributions.py::TestDistributionShapes::test_halfcauchy_shape_tensor_params, test/distributions/test_distributions.py::TestDistributionShapes::test_kumaraswamy_shape_scalar_params, test/distributions/test_distributions.py::TestDistributionShapes::test_laplace_shape_scalar_params, test/distributions/test_distributions.py::TestDistributionShapes::test_laplace_shape_tensor_params, test/distributions/test_distributions.py::TestDistributionShapes::test_mixture_same_family_mean_shape, test/distributions/test_distributions.py::TestDistributionShapes::test_mixture_same_family_shape, test/distributions/test_distributions.py::TestDistributionShapes::test_multinomial_shape, test/distributions/test_distributions.py::TestDistributionShapes::test_normal_shape_scalar_params, test/distributions/test_distributions.py::TestDistributionShapes::test_normal_shape_tensor_params, test/distributions/test_distributions.py::TestDistributionShapes::test_one_hot_categorical_shape, test/distributions/test_distributions.py::TestDistributionShapes::test_pareto_shape_scalar_params, test/distributions/test_distributions.py::TestDistributionShapes::test_studentT_shape_scalar_params, test/distributions/test_distributions.py::TestDistributionShapes::test_studentT_shape_tensor_params, test/distributions/test_distributions.py::TestDistributionShapes::test_uniform_shape_scalar_params, test/distributions/test_distributions.py::TestDistributionShapes::test_uniform_shape_tensor_params, test/distributions/test_distributions.py::TestDistributionShapes::test_vonmises_shape_scalar_params, test/distributions/test_distributions.py::TestDistributionShapes::test_vonmises_shape_tensor_params, test/distributions/test_distributions.py::TestDistributionShapes::test_weibull_scale_scalar_params, test/distributions/test_distributions.py::TestDistributionShapes::test_wishart_shape_scalar_params, test/distributions/test_distributions.py::TestDistributionShapes::test_wishart_shape_tensor_params, test/distributions/test_distributions.py::TestKL::test_entropy_exponential_family, test/distributions/test_distributions.py::TestKL::test_entropy_monte_carlo, test/distributions/test_distributions.py::TestKL::test_kl_edgecases, test/distributions/test_distributions.py::TestKL::test_kl_exponential_family, test/distributions/test_distributions.py::TestKL::test_kl_infinite, test/distributions/test_distributions.py::TestKL::test_kl_lowrank_multivariate_normal, test/distributions/test_distributions.py::TestKL::test_kl_lowrank_multivariate_normal_batched, test/distributions/test_distributions.py::TestKL::test_kl_monte_carlo, test/distributions/test_distributions.py::TestKL::test_kl_multivariate_normal, test/distributions/test_distributions.py::TestKL::test_kl_multivariate_normal_batched, test/distributions/test_distributions.py::TestKL::test_kl_multivariate_normal_batched_broadcasted, test/distributions/test_distributions.py::TestKL::test_kl_shape, test/distributions/test_distributions.py::TestKL::test_kl_transformed, test/distributions/test_distributions.py::TestConstraints::test_params_constraints, test/distributions/test_distributions.py::TestConstraints::test_support_constraints, test/distributions/test_distributions.py::TestNumericalStability::test_bernoulli_gradient, test/distributions/test_distributions.py::TestNumericalStability::test_bernoulli_with_logits_overflow, test/distributions/test_distributions.py::TestNumericalStability::test_bernoulli_with_logits_underflow, test/distributions/test_distributions.py::TestNumericalStability::test_categorical_log_prob, test/distributions/test_distributions.py::TestNumericalStability::test_categorical_log_prob_with_logits, test/distributions/test_distributions.py::TestNumericalStability::test_continuous_bernoulli_gradient, test/distributions/test_distributions.py::TestNumericalStability::test_continuous_bernoulli_with_logits_overflow, test/distributions/test_distributions.py::TestNumericalStability::test_continuous_bernoulli_with_logits_underflow, test/distributions/test_distributions.py::TestNumericalStability::test_multinomial_log_prob, test/distributions/test_distributions.py::TestNumericalStability::test_multinomial_log_prob_with_logits, test/distributions/test_distributions.py::TestLazyLogitsInitialization::test_lazy_logits_initialization, test/distributions/test_distributions.py::TestLazyLogitsInitialization::test_lazy_probs_initialization, test/distributions/test_distributions.py::TestAgainstScipy::test_cdf, test/distributions/test_distributions.py::TestAgainstScipy::test_icdf, test/distributions/test_distributions.py::TestAgainstScipy::test_mean, test/distributions/test_distributions.py::TestAgainstScipy::test_variance_stddev, test/distributions/test_distributions.py::TestFunctors::test_cat_event_dim, test/distributions/test_distributions.py::TestFunctors::test_cat_transform, test/distributions/test_distributions.py::TestFunctors::test_cat_transform_non_uniform, test/distributions/test_distributions.py::TestFunctors::test_stack_transform, test/distributions/test_distributions.py::TestValidation::test_invalid, test/distributions/test_distributions.py::TestValidation::test_invalid_log_probs_arg, test/distributions/test_distributions.py::TestValidation::test_valid, test/distributions/test_distributions.py::TestValidation::test_warning_unimplemented_constraints, test/distributions/test_distributions.py::TestJit::test_cdf, test/distributions/test_distributions.py::TestJit::test_entropy, test/distributions/test_distributions.py::TestJit::test_enumerate_support, test/distributions/test_distributions.py::TestJit::test_log_prob, test/distributions/test_distributions.py::TestJit::test_mean, test/distributions/test_distributions.py::TestJit::test_rsample, test/distributions/test_distributions.py::TestJit::test_sample, test/distributions/test_distributions.py::TestJit::test_variance 2025-12-04T13:57:40.3284967Z 2025-12-04T13:57:40.3285245Z Finished distributions/test_distributions 1/1 ... [2025-12-04 13:57:40.315788][15508.258003434], took 0.97min 2025-12-04T13:57:40.3502624Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/distributions.test_distributions/distributions.test_distributions-0cc6093cedae9699.xml 2025-12-04T13:57:42.2660134Z Uploading artifacts took 1.84 seconds 2025-12-04T13:57:45.6726827Z Running test batch 'tests to run' cost 14628.88 seconds 2025-12-04T13:57:45.6739051Z Emitting td_test_failure_stats_v2 2025-12-04T13:57:45.6742568Z Writing 1 documents to S3 ossci-raw-job-status/ossci_uploaded_metrics/td_test_failure_stats_v2_1764856665_35dfebe0d11911f085f30242ac110002 2025-12-04T13:57:45.7946849Z Done! Finish writing document to S3 ossci-raw-job-status/ossci_uploaded_metrics/td_test_failure_stats_v2_1764856665_35dfebe0d11911f085f30242ac110002 2025-12-04T13:57:45.7948003Z inductor/test_cuda_select_algorithm 1/1 failed! 2025-12-04T13:57:46.4007618Z 2025-12-04T13:57:46.4008142Z real 243m54.151s 2025-12-04T13:57:46.4008407Z user 226m46.715s 2025-12-04T13:57:46.4008633Z sys 36m28.668s 2025-12-04T13:57:46.4008836Z + sccache_epilogue 2025-12-04T13:57:46.4009107Z + echo '::group::Sccache Compilation Log' 2025-12-04T13:57:46.4009713Z ##[group]Sccache Compilation Log 2025-12-04T13:57:46.4010371Z + echo '=================== sccache compilation log ===================' 2025-12-04T13:57:46.4010745Z =================== sccache compilation log =================== 2025-12-04T13:57:46.4011299Z + python /var/lib/jenkins/workspace/.ci/pytorch/print_sccache_log.py /var/lib/jenkins/sccache_error.log 2025-12-04T13:57:46.4141478Z + echo '=========== If your build fails, please take a look at the log above for possible reasons ===========' 2025-12-04T13:57:46.4142142Z =========== If your build fails, please take a look at the log above for possible reasons =========== 2025-12-04T13:57:46.4142597Z + sccache --show-stats 2025-12-04T13:57:46.4181139Z Compile requests 3826 2025-12-04T13:57:46.4182068Z Compile requests executed 217 2025-12-04T13:57:46.4182942Z Cache hits 122 2025-12-04T13:57:46.4183493Z Cache hits (C/C++) 122 2025-12-04T13:57:46.4184229Z Cache misses 75 2025-12-04T13:57:46.4184621Z Cache misses (C/C++) 75 2025-12-04T13:57:46.4185004Z Cache hits rate 61.93 % 2025-12-04T13:57:46.4185287Z Cache hits rate (C/C++) 61.93 % 2025-12-04T13:57:46.4185526Z Cache timeouts 0 2025-12-04T13:57:46.4185902Z Cache read errors 0 2025-12-04T13:57:46.4186255Z Forced recaches 0 2025-12-04T13:57:46.4186617Z Cache write errors 0 2025-12-04T13:57:46.4186974Z Cache errors 0 2025-12-04T13:57:46.4187208Z Compilations 75 2025-12-04T13:57:46.4187433Z Compilation failures 20 2025-12-04T13:57:46.4188621Z Non-cacheable compilations 0 2025-12-04T13:57:46.4188881Z Non-cacheable calls 101 2025-12-04T13:57:46.4189105Z Non-compilation calls 3508 2025-12-04T13:57:46.4189331Z Unsupported compiler calls 0 2025-12-04T13:57:46.4189676Z Average cache write 0.053 s 2025-12-04T13:57:46.4189913Z Average compiler 9.578 s 2025-12-04T13:57:46.4190213Z Average cache read hit 0.026 s 2025-12-04T13:57:46.4190464Z Failed distributed compilations 0 2025-12-04T13:57:46.4190624Z 2025-12-04T13:57:46.4190703Z Non-cacheable reasons: 2025-12-04T13:57:46.4190888Z unknown source language 59 2025-12-04T13:57:46.4191101Z -E 42 2025-12-04T13:57:46.4191247Z 2025-12-04T13:57:46.4191429Z Cache location s3, name: ossci-compiler-cache-circleci-v2, prefix: / 2025-12-04T13:57:46.4191762Z Version (client) 0.10.0 2025-12-04T13:57:46.4191986Z + sccache --stop-server 2025-12-04T13:57:46.4206208Z Stopping sccache server... 2025-12-04T13:57:46.4208824Z Compile requests 3826 2025-12-04T13:57:46.4209267Z Compile requests executed 217 2025-12-04T13:57:46.4209545Z Cache hits 122 2025-12-04T13:57:46.4209835Z Cache hits (C/C++) 122 2025-12-04T13:57:46.4210101Z Cache misses 75 2025-12-04T13:57:46.4210361Z Cache misses (C/C++) 75 2025-12-04T13:57:46.4210634Z Cache hits rate 61.93 % 2025-12-04T13:57:46.4210911Z Cache hits rate (C/C++) 61.93 % 2025-12-04T13:57:46.4211185Z Cache timeouts 0 2025-12-04T13:57:46.4211445Z Cache read errors 0 2025-12-04T13:57:46.4211700Z Forced recaches 0 2025-12-04T13:57:46.4211955Z Cache write errors 0 2025-12-04T13:57:46.4212207Z Cache errors 0 2025-12-04T13:57:46.4212466Z Compilations 75 2025-12-04T13:57:46.4212726Z Compilation failures 20 2025-12-04T13:57:46.4213009Z Non-cacheable compilations 0 2025-12-04T13:57:46.4213230Z Non-cacheable calls 101 2025-12-04T13:57:46.4213511Z Non-compilation calls 3508 2025-12-04T13:57:46.4213895Z Unsupported compiler calls 0 2025-12-04T13:57:46.4214415Z Average cache write 0.053 s 2025-12-04T13:57:46.4214789Z Average compiler 9.578 s 2025-12-04T13:57:46.4215071Z Average cache read hit 0.026 s 2025-12-04T13:57:46.4215302Z Failed distributed compilations 0 2025-12-04T13:57:46.4215451Z 2025-12-04T13:57:46.4215537Z Non-cacheable reasons: 2025-12-04T13:57:46.4215723Z unknown source language 59 2025-12-04T13:57:46.4215934Z -E 42 2025-12-04T13:57:46.4216078Z 2025-12-04T13:57:46.4216253Z Cache location s3, name: ossci-compiler-cache-circleci-v2, prefix: / 2025-12-04T13:57:46.4216579Z Version (client) 0.10.0 2025-12-04T13:57:46.4216809Z + echo ::endgroup:: 2025-12-04T13:57:46.4217177Z ##[endgroup] 2025-12-04T13:57:46.4217335Z + cleanup_workspace 2025-12-04T13:57:46.4217698Z + echo 'sudo may print the following warning message that can be ignored. The chown command will still run.' 2025-12-04T13:57:46.4218259Z sudo may print the following warning message that can be ignored. The chown command will still run. 2025-12-04T13:57:46.4218720Z + echo ' sudo: setrlimit(RLIMIT_STACK): Operation not permitted' 2025-12-04T13:57:46.4219056Z sudo: setrlimit(RLIMIT_STACK): Operation not permitted 2025-12-04T13:57:46.4219441Z + echo 'For more details refer to https://github.com/sudo-project/sudo/issues/42' 2025-12-04T13:57:46.4219866Z For more details refer to https://github.com/sudo-project/sudo/issues/42 2025-12-04T13:57:46.4220197Z + sudo chown -R 1000 /var/lib/jenkins/workspace 2025-12-04T13:57:47.4557663Z ##[error]Process completed with exit code 1. 2025-12-04T13:57:47.4614340Z Prepare all required actions 2025-12-04T13:57:47.4614698Z Getting action download info 2025-12-04T13:57:47.6434068Z ##[group]Run ./.github/actions/pytest-cache-upload 2025-12-04T13:57:47.6434326Z with: 2025-12-04T13:57:47.6434487Z cache_dir: .pytest_cache 2025-12-04T13:57:47.6434792Z shard: 6 2025-12-04T13:57:47.6434979Z sha: ffd9b0fb4355e97af82fc42cf185c3ffa0fc0a32 2025-12-04T13:57:47.6435231Z test_config: default 2025-12-04T13:57:47.6435497Z job_identifier: periodic_linux-jammy-cuda12.8-py3.10-gcc11-debug 2025-12-04T13:57:47.6435797Z env: 2025-12-04T13:57:47.6435950Z GIT_DEFAULT_BRANCH: main 2025-12-04T13:57:47.6436145Z HAS_NVIDIA_GPU: true 2025-12-04T13:57:47.6436376Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2025-12-04T13:57:47.6436764Z DOCKER_CONTAINER_ID: 657e31a8be774492437c4338a4b954650ffa0203d69f0088e2d19ede1e69df31 2025-12-04T13:57:47.6437117Z ##[endgroup] 2025-12-04T13:57:47.6464216Z ##[group]Run nick-fields/retry@v3.0.0 2025-12-04T13:57:47.6464465Z with: 2025-12-04T13:57:47.6464610Z shell: bash 2025-12-04T13:57:47.6464777Z timeout_minutes: 5 2025-12-04T13:57:47.6464955Z max_attempts: 5 2025-12-04T13:57:47.6465135Z retry_wait_seconds: 30 2025-12-04T13:57:47.6465388Z command: set -eu python3 -m pip install boto3==1.35.42 2025-12-04T13:57:47.6465690Z polling_interval_seconds: 1 2025-12-04T13:57:47.6465905Z warning_on_retry: true 2025-12-04T13:57:47.6466093Z continue_on_error: false 2025-12-04T13:57:47.6466282Z env: 2025-12-04T13:57:47.6466438Z GIT_DEFAULT_BRANCH: main 2025-12-04T13:57:47.6466623Z HAS_NVIDIA_GPU: true 2025-12-04T13:57:47.6466853Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2025-12-04T13:57:47.6467270Z DOCKER_CONTAINER_ID: 657e31a8be774492437c4338a4b954650ffa0203d69f0088e2d19ede1e69df31 2025-12-04T13:57:47.6467639Z ##[endgroup] 2025-12-04T13:57:47.9798479Z Defaulting to user installation because normal site-packages is not writeable 2025-12-04T13:57:49.0378048Z Collecting boto3==1.35.42 2025-12-04T13:57:49.0655385Z Downloading boto3-1.35.42-py3-none-any.whl (139 kB) 2025-12-04T13:57:49.1352702Z Collecting s3transfer<0.11.0,>=0.10.0 2025-12-04T13:57:49.1407030Z Downloading s3transfer-0.10.4-py3-none-any.whl (83 kB) 2025-12-04T13:57:49.1465220Z Requirement already satisfied: jmespath<2.0.0,>=0.7.1 in /usr/lib/python3.9/site-packages (from boto3==1.35.42) (0.10.0) 2025-12-04T13:57:50.2447775Z Collecting botocore<1.36.0,>=1.35.42 2025-12-04T13:57:50.2485355Z Downloading botocore-1.35.99-py3-none-any.whl (13.3 MB) 2025-12-04T13:57:50.3830956Z Requirement already satisfied: python-dateutil<3.0.0,>=2.1 in /usr/lib/python3.9/site-packages (from botocore<1.36.0,>=1.35.42->boto3==1.35.42) (2.8.1) 2025-12-04T13:57:50.3838535Z Requirement already satisfied: urllib3<1.27,>=1.25.4 in /usr/lib/python3.9/site-packages (from botocore<1.36.0,>=1.35.42->boto3==1.35.42) (1.25.10) 2025-12-04T13:57:50.5390812Z Requirement already satisfied: six>=1.5 in /usr/lib/python3.9/site-packages (from python-dateutil<3.0.0,>=2.1->botocore<1.36.0,>=1.35.42->boto3==1.35.42) (1.15.0) 2025-12-04T13:57:50.6163244Z Installing collected packages: botocore, s3transfer, boto3 2025-12-04T13:57:51.1977007Z Successfully installed boto3-1.35.42 botocore-1.35.99 s3transfer-0.10.4 2025-12-04T13:57:51.7181896Z Command completed after 1 attempt(s). 2025-12-04T13:57:51.7245343Z ##[group]Run python3 .github/scripts/pytest_cache.py \ 2025-12-04T13:57:51.7245688Z python3 .github/scripts/pytest_cache.py \ 2025-12-04T13:57:51.7245947Z  --upload \ 2025-12-04T13:57:51.7246180Z  --cache_dir "$GITHUB_WORKSPACE/$CACHE_DIR" \ 2025-12-04T13:57:51.7246457Z  --pr_identifier "$GITHUB_REF" \ 2025-12-04T13:57:51.7246715Z  --job_identifier "$JOB_IDENTIFIER" \ 2025-12-04T13:57:51.7246955Z  --sha "$SHA" \ 2025-12-04T13:57:51.7247156Z  --test_config "$TEST_CONFIG" \ 2025-12-04T13:57:51.7247725Z  --shard "$SHARD" \ 2025-12-04T13:57:51.7247950Z  --repo "$REPO" \ 2025-12-04T13:57:51.7248360Z  --temp_dir "$RUNNER_TEMP" \ 2025-12-04T13:57:51.7261176Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-12-04T13:57:51.7261456Z env: 2025-12-04T13:57:51.7261621Z GIT_DEFAULT_BRANCH: main 2025-12-04T13:57:51.7261811Z HAS_NVIDIA_GPU: true 2025-12-04T13:57:51.7262145Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2025-12-04T13:57:51.7262537Z DOCKER_CONTAINER_ID: 657e31a8be774492437c4338a4b954650ffa0203d69f0088e2d19ede1e69df31 2025-12-04T13:57:51.7262899Z CACHE_DIR: .pytest_cache 2025-12-04T13:57:51.7263185Z JOB_IDENTIFIER: periodic_linux-jammy-cuda12.8-py3.10-gcc11-debug 2025-12-04T13:57:51.7263508Z SHA: ffd9b0fb4355e97af82fc42cf185c3ffa0fc0a32 2025-12-04T13:57:51.7263858Z TEST_CONFIG: default 2025-12-04T13:57:51.7264032Z SHARD: 6 2025-12-04T13:57:51.7264190Z REPO: pytorch/pytorch 2025-12-04T13:57:51.7264369Z ##[endgroup] 2025-12-04T13:57:52.1104405Z PR identifier for `refs/heads/main` is `96e092540d6b3c4076e3d2bc6f1f9013` 2025-12-04T13:57:52.1106267Z Uploading cache with args Namespace(upload=True, download=False, cache_dir='/home/ec2-user/actions-runner/_work/pytorch/pytorch/.pytest_cache', pr_identifier='refs/heads/main', job_identifier='periodic_linux-jammy-cuda12.8-py3.10-gcc11-debug', sha='ffd9b0fb4355e97af82fc42cf185c3ffa0fc0a32', test_config='default', shard='6', repo='pytorch/pytorch', temp_dir='/home/ec2-user/actions-runner/_work/_temp', bucket=None) 2025-12-04T13:57:52.1108056Z Zipping /home/ec2-user/actions-runner/_work/pytorch/pytorch/.pytest_cache 2025-12-04T13:57:52.1109175Z to /home/ec2-user/actions-runner/_work/_temp/zip-upload/pytest_cache/pytorch/pytorch/96e092540d6b3c4076e3d2bc6f1f9013/periodic_linux-jammy-cuda12_8-py3_10-gcc11-debug/ffd9b0fb4355e97af82fc42cf185c3ffa0fc0a32/default/6 2025-12-04T13:57:52.1110906Z Uploading /home/ec2-user/actions-runner/_work/_temp/zip-upload/pytest_cache/pytorch/pytorch/96e092540d6b3c4076e3d2bc6f1f9013/periodic_linux-jammy-cuda12_8-py3_10-gcc11-debug/ffd9b0fb4355e97af82fc42cf185c3ffa0fc0a32/default/6.zip 2025-12-04T13:57:52.1112530Z to s3://gha-artifacts/pytest_cache/pytorch/pytorch/96e092540d6b3c4076e3d2bc6f1f9013/periodic_linux-jammy-cuda12_8-py3_10-gcc11-debug/ffd9b0fb4355e97af82fc42cf185c3ffa0fc0a32/default/6.zip 2025-12-04T13:57:52.1487118Z ##[group]Run cat test/**/*_toprint.log || true 2025-12-04T13:57:52.1487604Z cat test/**/*_toprint.log || true 2025-12-04T13:57:52.1496264Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-12-04T13:57:52.1496558Z env: 2025-12-04T13:57:52.1496737Z GIT_DEFAULT_BRANCH: main 2025-12-04T13:57:52.1496931Z HAS_NVIDIA_GPU: true 2025-12-04T13:57:52.1497168Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2025-12-04T13:57:52.1497561Z DOCKER_CONTAINER_ID: 657e31a8be774492437c4338a4b954650ffa0203d69f0088e2d19ede1e69df31 2025-12-04T13:57:52.1497905Z ##[endgroup] 2025-12-04T13:57:52.1595028Z cat: 'test/**/*_toprint.log': No such file or directory 2025-12-04T13:57:52.1620730Z ##[group]Run kill "$MONITOR_SCRIPT_PID" 2025-12-04T13:57:52.1621031Z kill "$MONITOR_SCRIPT_PID" 2025-12-04T13:57:52.1628612Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-12-04T13:57:52.1628908Z env: 2025-12-04T13:57:52.1629072Z GIT_DEFAULT_BRANCH: main 2025-12-04T13:57:52.1629270Z HAS_NVIDIA_GPU: true 2025-12-04T13:57:52.1629499Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2025-12-04T13:57:52.1629889Z DOCKER_CONTAINER_ID: 657e31a8be774492437c4338a4b954650ffa0203d69f0088e2d19ede1e69df31 2025-12-04T13:57:52.1630255Z MONITOR_SCRIPT_PID: 60603 2025-12-04T13:57:52.1630451Z ##[endgroup] 2025-12-04T13:57:52.1657976Z /home/ec2-user/actions-runner/_work/_temp/efdb5319-5457-481d-94be-a043e48fd655.sh: line 1: kill: (60603) - No such process 2025-12-04T13:57:52.1660706Z ##[error]Process completed with exit code 1. 2025-12-04T13:57:52.1752600Z Prepare all required actions 2025-12-04T13:57:52.1752962Z Getting action download info 2025-12-04T13:57:52.4264576Z Download action repository 'seemethere/upload-artifact-s3@v5' (SHA:baba72d0712b404f646cebe0730933554ebce96a) 2025-12-04T13:57:52.6707862Z Download action repository 'actions/upload-artifact@v4' (SHA:ea165f8d65b6e75b540449e92b4886f43607fa02) 2025-12-04T13:57:53.1671861Z ##[group]Run ./.github/actions/upload-test-artifacts 2025-12-04T13:57:53.1672242Z with: 2025-12-04T13:57:53.1672545Z file-suffix: test-default-6-7-linux.g6.4xlarge.experimental.nvidia.gpu_57120265687 2025-12-04T13:57:53.1672916Z s3-bucket: gha-artifacts 2025-12-04T13:57:53.1673101Z env: 2025-12-04T13:57:53.1673248Z GIT_DEFAULT_BRANCH: main 2025-12-04T13:57:53.1673431Z HAS_NVIDIA_GPU: true 2025-12-04T13:57:53.1673662Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2025-12-04T13:57:53.1674041Z DOCKER_CONTAINER_ID: 657e31a8be774492437c4338a4b954650ffa0203d69f0088e2d19ede1e69df31 2025-12-04T13:57:53.1674412Z ##[endgroup] 2025-12-04T13:57:53.1694654Z ##[group]Run # Remove any previous test jsons if they exist 2025-12-04T13:57:53.1694985Z # Remove any previous test jsons if they exist 2025-12-04T13:57:53.1695254Z rm -f test-jsons-*.zip 2025-12-04T13:57:53.1695567Z zip -r "test-jsons-${FILE_SUFFIX}.zip" test/test-reports -i '*.json' 2025-12-04T13:57:53.1703683Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-12-04T13:57:53.1703986Z env: 2025-12-04T13:57:53.1704152Z GIT_DEFAULT_BRANCH: main 2025-12-04T13:57:53.1704355Z HAS_NVIDIA_GPU: true 2025-12-04T13:57:53.1704579Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2025-12-04T13:57:53.1704968Z DOCKER_CONTAINER_ID: 657e31a8be774492437c4338a4b954650ffa0203d69f0088e2d19ede1e69df31 2025-12-04T13:57:53.1705459Z FILE_SUFFIX: test-default-6-7-linux.g6.4xlarge.experimental.nvidia.gpu_57120265687 2025-12-04T13:57:53.1705798Z ##[endgroup] 2025-12-04T13:57:53.1928239Z adding: test/test-reports/td_exclusions-b52d94079bac4e8ce784.json (deflated 82%) 2025-12-04T13:57:53.1954674Z adding: test/test-reports/python-pytest/inductor.test_torchinductor/inductor.test_torchinductor-410bad42b3bb466c.json (deflated 94%) 2025-12-04T13:57:53.1982396Z adding: test/test-reports/python-pytest/inductor.test_torchinductor_dynamic_shapes/inductor.test_torchinductor_dynamic_shapes-1c753e0c38ee0fc9.json (deflated 94%) 2025-12-04T13:57:53.1983909Z adding: test/test-reports/python-pytest/inductor.test_kernel_benchmark/inductor.test_kernel_benchmark-a01600ecae601eef.json (deflated 86%) 2025-12-04T13:57:53.1989163Z adding: test/test-reports/python-pytest/inductor.test_torchinductor_opinfo/inductor.test_torchinductor_opinfo-d9c51466d8748c04.json (deflated 96%) 2025-12-04T13:57:53.1994461Z adding: test/test-reports/python-pytest/inductor.test_torchinductor_opinfo/inductor.test_torchinductor_opinfo-eb72b3d6271f1e7e.json (deflated 95%) 2025-12-04T13:57:53.2046253Z adding: test/test-reports/python-pytest/dynamo.test_repros/dynamo.test_repros-df80b1b40a6198cc.json (deflated 92%) 2025-12-04T13:57:53.2047397Z adding: test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-9c66153466707d74.json (deflated 87%) 2025-12-04T13:57:53.2048533Z adding: test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-11449d70dc195ff4.json (deflated 86%) 2025-12-04T13:57:53.2049624Z adding: test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-6c093515a384d9fc.json (deflated 86%) 2025-12-04T13:57:53.2050713Z adding: test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-7ea688002f0554bd.json (deflated 87%) 2025-12-04T13:57:53.2051805Z adding: test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-759714b1d86e3c4c.json (deflated 86%) 2025-12-04T13:57:53.2053118Z adding: test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-7a190e1b598a9023.json (deflated 86%) 2025-12-04T13:57:53.2054202Z adding: test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-c45e67249599e6c2.json (deflated 87%) 2025-12-04T13:57:53.2055242Z adding: test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-8112676a6b085940.json (deflated 86%) 2025-12-04T13:57:53.2056231Z adding: test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-2280a7982827e03d.json (deflated 86%) 2025-12-04T13:57:53.2057079Z adding: test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-ecd8523a161788bb.json (deflated 87%) 2025-12-04T13:57:53.2057916Z adding: test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-d90aab22855519e4.json (deflated 86%) 2025-12-04T13:57:53.2058763Z adding: test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-5cb074eac86f05e5.json (deflated 86%) 2025-12-04T13:57:53.2059599Z adding: test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-7681ec3fadfa4a43.json (deflated 87%) 2025-12-04T13:57:53.2060445Z adding: test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-49c0a334019b663e.json (deflated 86%) 2025-12-04T13:57:53.2061292Z adding: test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-a4c3d2e69c8841e3.json (deflated 86%) 2025-12-04T13:57:53.2062147Z adding: test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-0f60f7321324a86a.json (deflated 87%) 2025-12-04T13:57:53.2062991Z adding: test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-294e2abb077cceb2.json (deflated 86%) 2025-12-04T13:57:53.2063921Z adding: test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-b24025703d1a5b1f.json (deflated 86%) 2025-12-04T13:57:53.2064767Z adding: test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-531381cfa4745e03.json (deflated 87%) 2025-12-04T13:57:53.2065621Z adding: test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-bb14ed3958363693.json (deflated 86%) 2025-12-04T13:57:53.2066524Z adding: test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-ec0356b9a2627168.json (deflated 86%) 2025-12-04T13:57:53.2067364Z adding: test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-c81b585b68f6cf60.json (deflated 87%) 2025-12-04T13:57:53.2068208Z adding: test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-647efcde8452d0c7.json (deflated 86%) 2025-12-04T13:57:53.2069053Z adding: test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-abfd2b283c5f3df6.json (deflated 86%) 2025-12-04T13:57:53.2069978Z adding: test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-e0f3393878857c02.json (deflated 87%) 2025-12-04T13:57:53.2070893Z adding: test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-d5da2e41e9fcecb9.json (deflated 86%) 2025-12-04T13:57:53.2071762Z adding: test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-5fb75fde833c163c.json (deflated 86%) 2025-12-04T13:57:53.2072691Z adding: test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-2f9606f02b59f87b.json (deflated 87%) 2025-12-04T13:57:53.2074613Z adding: test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-94458b88fceefe4e.json (deflated 86%) 2025-12-04T13:57:53.2075495Z adding: test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-6787c22d74999180.json (deflated 86%) 2025-12-04T13:57:53.2076327Z adding: test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-cdf948359e80bb41.json (deflated 87%) 2025-12-04T13:57:53.2077244Z adding: test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-bd192fd6fb52c26e.json (deflated 86%) 2025-12-04T13:57:53.2078607Z adding: test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-7d48593db7bb6f60.json (deflated 86%) 2025-12-04T13:57:53.2079871Z adding: test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-91ed99958e7f9cc9.json (deflated 87%) 2025-12-04T13:57:53.2080955Z adding: test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-f4094aba19e671d4.json (deflated 86%) 2025-12-04T13:57:53.2082002Z adding: test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-204edfd0c56893dd.json (deflated 86%) 2025-12-04T13:57:53.2083056Z adding: test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-5d3545fec05b298e.json (deflated 87%) 2025-12-04T13:57:53.2084120Z adding: test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-c2388e46be16ba24.json (deflated 86%) 2025-12-04T13:57:53.2085187Z adding: test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-e9cf01ceaaab6f89.json (deflated 86%) 2025-12-04T13:57:53.2086049Z adding: test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-ff104bb89c063280.json (deflated 87%) 2025-12-04T13:57:53.2086887Z adding: test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-61993e92defaa624.json (deflated 86%) 2025-12-04T13:57:53.2087735Z adding: test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-16a80675dc6db2f9.json (deflated 86%) 2025-12-04T13:57:53.2088577Z adding: test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-601480a9a7ef82d7.json (deflated 87%) 2025-12-04T13:57:53.2089548Z adding: test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-fda38496367dc0a9.json (deflated 86%) 2025-12-04T13:57:53.2090425Z adding: test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-13c7a98252f889c4.json (deflated 86%) 2025-12-04T13:57:53.2091299Z adding: test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-d1b821ab0fc5c966.json (deflated 87%) 2025-12-04T13:57:53.2092296Z adding: test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-d70416164f95a17a.json (deflated 86%) 2025-12-04T13:57:53.2093190Z adding: test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-a1832326f6932688.json (deflated 86%) 2025-12-04T13:57:53.2094102Z adding: test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-1965a836c7535ef2.json (deflated 87%) 2025-12-04T13:57:53.2095088Z adding: test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-12f95c060627756e.json (deflated 86%) 2025-12-04T13:57:53.2096165Z adding: test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-c82b9fa27197518a.json (deflated 86%) 2025-12-04T13:57:53.2097016Z adding: test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-36ce77ac1c8c9763.json (deflated 87%) 2025-12-04T13:57:53.2097879Z adding: test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-e4f38ecbc39decbb.json (deflated 86%) 2025-12-04T13:57:53.2098833Z adding: test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-e20c92a1eaab2de9.json (deflated 86%) 2025-12-04T13:57:53.2099846Z adding: test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-2048f1d3bc205458.json (deflated 87%) 2025-12-04T13:57:53.2100800Z adding: test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-d670b011b4f3da72.json (deflated 86%) 2025-12-04T13:57:53.2101680Z adding: test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-56e0b2ecffd4aa2f.json (deflated 86%) 2025-12-04T13:57:53.2102523Z adding: test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-1e9198c3a728666b.json (deflated 87%) 2025-12-04T13:57:53.2103380Z adding: test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-c1c9525c22bede8d.json (deflated 86%) 2025-12-04T13:57:53.2104398Z adding: test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-cc4ec46989be7746.json (deflated 86%) 2025-12-04T13:57:53.2105305Z adding: test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-3f42debbb8d01db8.json (deflated 87%) 2025-12-04T13:57:53.2106236Z adding: test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-92851bf92613c0d7.json (deflated 86%) 2025-12-04T13:57:53.2107080Z adding: test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-3610b3094a676974.json (deflated 86%) 2025-12-04T13:57:53.2108055Z adding: test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-dba8bfd8733d5e09.json (deflated 87%) 2025-12-04T13:57:53.2109016Z adding: test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-6c64eb8af833b17a.json (deflated 86%) 2025-12-04T13:57:53.2109979Z adding: test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-f878b685b7c3b00a.json (deflated 86%) 2025-12-04T13:57:53.2110993Z adding: test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-232061e660a2ff26.json (deflated 87%) 2025-12-04T13:57:53.2111959Z adding: test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-fe4ea69fd2c99307.json (deflated 86%) 2025-12-04T13:57:53.2112820Z adding: test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-4bc55824cd7c5544.json (deflated 86%) 2025-12-04T13:57:53.2113654Z adding: test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-c2fc9a8f37b6f624.json (deflated 87%) 2025-12-04T13:57:53.2114505Z adding: test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-df0458d9683e0406.json (deflated 86%) 2025-12-04T13:57:53.2115429Z adding: test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-1f5454d7efc5fad5.json (deflated 86%) 2025-12-04T13:57:53.2116318Z adding: test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-0e640f0818a9f631.json (deflated 87%) 2025-12-04T13:57:53.2117428Z adding: test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-e912b3c1622dc5a6.json (deflated 86%) 2025-12-04T13:57:53.2118377Z adding: test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-9bf082709afd4107.json (deflated 86%) 2025-12-04T13:57:53.2119252Z adding: test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-417661a6a446d83a.json (deflated 87%) 2025-12-04T13:57:53.2120174Z adding: test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-2d84d0ab3f5cbf16.json (deflated 86%) 2025-12-04T13:57:53.2121099Z adding: test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-3c429c2af5dedece.json (deflated 86%) 2025-12-04T13:57:53.2121942Z adding: test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-b9ae48951e1b6488.json (deflated 87%) 2025-12-04T13:57:53.2122793Z adding: test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-a9948af371c98fbd.json (deflated 86%) 2025-12-04T13:57:53.2123732Z adding: test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-f058047ef4d14b46.json (deflated 86%) 2025-12-04T13:57:53.2124638Z adding: test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-5d247f52fbae6573.json (deflated 87%) 2025-12-04T13:57:53.2125597Z adding: test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-b685fc5b32b50b90.json (deflated 86%) 2025-12-04T13:57:53.2126549Z adding: test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-bc2410b75aa40646.json (deflated 86%) 2025-12-04T13:57:53.2127458Z adding: test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-71cfec7d40715d91.json (deflated 87%) 2025-12-04T13:57:53.2128412Z adding: test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-c9dbc1d234ce26dc.json (deflated 86%) 2025-12-04T13:57:53.2129346Z adding: test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-8a076bf12e47858e.json (deflated 86%) 2025-12-04T13:57:53.2130257Z adding: test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-bb0bac5f3b803919.json (deflated 87%) 2025-12-04T13:57:53.2131173Z adding: test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-0645969e8b49cd4b.json (deflated 86%) 2025-12-04T13:57:53.2132137Z adding: test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-20accf8a0f4e3024.json (deflated 86%) 2025-12-04T13:57:53.2133060Z adding: test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-55dc1e1f737da98a.json (deflated 87%) 2025-12-04T13:57:53.2134018Z adding: test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-458d9d61d3eb81db.json (deflated 86%) 2025-12-04T13:57:53.2134954Z adding: test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-0cf2b0bead25849b.json (deflated 86%) 2025-12-04T13:57:53.2135894Z adding: test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-e47d4d68d26f0ad1.json (deflated 87%) 2025-12-04T13:57:53.2136824Z adding: test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-044dfd62cda981dc.json (deflated 86%) 2025-12-04T13:57:53.2137763Z adding: test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-f4c669524343c1b1.json (deflated 86%) 2025-12-04T13:57:53.2138787Z adding: test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-9f9264fbba4173a1.json (deflated 87%) 2025-12-04T13:57:53.2139637Z adding: test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-c85f34f0c90d933d.json (deflated 86%) 2025-12-04T13:57:53.2140487Z adding: test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-4b2e7fd1ed886dfe.json (deflated 86%) 2025-12-04T13:57:53.2141473Z adding: test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-f54f533501201f35.json (deflated 87%) 2025-12-04T13:57:53.2142404Z adding: test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-cbbd83fa84eb990f.json (deflated 86%) 2025-12-04T13:57:53.2143330Z adding: test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-67e17348ec4a01d8.json (deflated 86%) 2025-12-04T13:57:53.2144356Z adding: test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-f72cf4cc8675ff8a.json (deflated 87%) 2025-12-04T13:57:53.2145279Z adding: test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-bd700a0c553d5cfe.json (deflated 86%) 2025-12-04T13:57:53.2146133Z adding: test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-28b04335f093e360.json (deflated 86%) 2025-12-04T13:57:53.2147053Z adding: test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-857ce5cc87a4a050.json (deflated 87%) 2025-12-04T13:57:53.2148029Z adding: test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-4e0b90b7cea75841.json (deflated 86%) 2025-12-04T13:57:53.2148947Z adding: test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-14983efcd69bf6ac.json (deflated 86%) 2025-12-04T13:57:53.2149786Z adding: test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-2104559418af7772.json (deflated 87%) 2025-12-04T13:57:53.2150791Z adding: test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-49eccf27aa184bc6.json (deflated 86%) 2025-12-04T13:57:53.2151701Z adding: test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-71736827b1650ca5.json (deflated 86%) 2025-12-04T13:57:53.2152607Z adding: test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-b5c5c102e82be0c3.json (deflated 87%) 2025-12-04T13:57:53.2153534Z adding: test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-6b2f788f9a7d50ec.json (deflated 86%) 2025-12-04T13:57:53.2154432Z adding: test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-61d7ccaaf8cfd370.json (deflated 86%) 2025-12-04T13:57:53.2155280Z adding: test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-cc96100da976ce4e.json (deflated 87%) 2025-12-04T13:57:53.2156220Z adding: test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-e2ca8845caeab461.json (deflated 86%) 2025-12-04T13:57:53.2157163Z adding: test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-ea9e1f7f9950a9ad.json (deflated 86%) 2025-12-04T13:57:53.2158027Z adding: test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-778cc31dcd227b51.json (deflated 87%) 2025-12-04T13:57:53.2158973Z adding: test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-d8517590777959a7.json (deflated 86%) 2025-12-04T13:57:53.2160001Z adding: test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-7c1369cc4e5815f4.json (deflated 86%) 2025-12-04T13:57:53.2160994Z adding: test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-f43c4b9f36164ff6.json (deflated 87%) 2025-12-04T13:57:53.2161940Z adding: test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-d56aa1824afdfad3.json (deflated 86%) 2025-12-04T13:57:53.2162876Z adding: test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-6a9d289c68c250dc.json (deflated 86%) 2025-12-04T13:57:53.2163716Z adding: test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-04e5068c64f117f5.json (deflated 87%) 2025-12-04T13:57:53.2164621Z adding: test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-85ab96cf3cf02016.json (deflated 86%) 2025-12-04T13:57:53.2165495Z adding: test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-888ba6ece96c2997.json (deflated 86%) 2025-12-04T13:57:53.2166342Z adding: test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-58cf8d0fed87176f.json (deflated 87%) 2025-12-04T13:57:53.2167287Z adding: test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-0022ebcfefe802db.json (deflated 86%) 2025-12-04T13:57:53.2168225Z adding: test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-dac6db8ed6c97e37.json (deflated 86%) 2025-12-04T13:57:53.2169130Z adding: test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-040b4092def69522.json (deflated 87%) 2025-12-04T13:57:53.2170088Z adding: test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-084397a772e6c7f7.json (deflated 86%) 2025-12-04T13:57:53.2171008Z adding: test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-2942f663738ee80d.json (deflated 86%) 2025-12-04T13:57:53.2171918Z adding: test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-ac0d8e76747a1a1b.json (deflated 87%) 2025-12-04T13:57:53.2172837Z adding: test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-7f452630b39063ce.json (deflated 86%) 2025-12-04T13:57:53.2173770Z adding: test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-bb668a52e682b2dd.json (deflated 86%) 2025-12-04T13:57:53.2174694Z adding: test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-60466f6816d813eb.json (deflated 87%) 2025-12-04T13:57:53.2175718Z adding: test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-3b52d4f5cebad92d.json (deflated 86%) 2025-12-04T13:57:53.2176621Z adding: test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-2ff4d68a3a0ad43c.json (deflated 86%) 2025-12-04T13:57:53.2177537Z adding: test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-7240a6bd78b3c1ba.json (deflated 87%) 2025-12-04T13:57:53.2178830Z adding: test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-7cb1dae7cd5b9c6d.json (deflated 86%) 2025-12-04T13:57:53.2179790Z adding: test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-1238d86169b8b87a.json (deflated 86%) 2025-12-04T13:57:53.2180991Z adding: test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-86e4cbaa0e57366b.json (deflated 87%) 2025-12-04T13:57:53.2181894Z adding: test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-1c1918cf02456db6.json (deflated 86%) 2025-12-04T13:57:53.2182852Z adding: test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-4d7a6e50e4cb57cd.json (deflated 86%) 2025-12-04T13:57:53.2183834Z adding: test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-cd0db30d38ee28ec.json (deflated 87%) 2025-12-04T13:57:53.2184846Z adding: test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-97331d68f5e395ac.json (deflated 86%) 2025-12-04T13:57:53.2185759Z adding: test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-266b0df1c83533e9.json (deflated 86%) 2025-12-04T13:57:53.2186671Z adding: test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-0728186d53d0914c.json (deflated 87%) 2025-12-04T13:57:53.2187628Z adding: test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-ace5a67d4c3c2a01.json (deflated 86%) 2025-12-04T13:57:53.2188589Z adding: test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-eeeef84645c1bb52.json (deflated 86%) 2025-12-04T13:57:53.2189444Z adding: test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-ffbbb699e0b6f991.json (deflated 87%) 2025-12-04T13:57:53.2190443Z adding: test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-90e4a91c9bbb810f.json (deflated 86%) 2025-12-04T13:57:53.2191347Z adding: test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-2ee8d3487d6282f0.json (deflated 86%) 2025-12-04T13:57:53.2192247Z adding: test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-2ab4541f9392b10c.json (deflated 87%) 2025-12-04T13:57:53.2193219Z adding: test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-ce43d3c4fee34ad0.json (deflated 86%) 2025-12-04T13:57:53.2194143Z adding: test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-d04b19868e6e9971.json (deflated 86%) 2025-12-04T13:57:53.2194999Z adding: test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-cf6ace4ea5db8dfb.json (deflated 87%) 2025-12-04T13:57:53.2195960Z adding: test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-8b65a3dabcd39e70.json (deflated 86%) 2025-12-04T13:57:53.2196916Z adding: test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-c85748e91a458eac.json (deflated 86%) 2025-12-04T13:57:53.2197783Z adding: test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-85bca917f8457708.json (deflated 87%) 2025-12-04T13:57:53.2198650Z adding: test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-6a70392d2e5080cc.json (deflated 86%) 2025-12-04T13:57:53.2199589Z adding: test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-1b506e26b1410ad4.json (deflated 86%) 2025-12-04T13:57:53.2200446Z adding: test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-f7f9c3afca4dce49.json (deflated 87%) 2025-12-04T13:57:53.2201354Z adding: test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-c41d68e69339995e.json (deflated 86%) 2025-12-04T13:57:53.2202340Z adding: test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-dfa47b8dd194f01b.json (deflated 86%) 2025-12-04T13:57:53.2203301Z adding: test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-6b849be23abc3e95.json (deflated 87%) 2025-12-04T13:57:53.2204189Z adding: test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-1eeada475e93262f.json (deflated 86%) 2025-12-04T13:57:53.2205043Z adding: test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-3a596bc2a015c0a5.json (deflated 86%) 2025-12-04T13:57:53.2205990Z adding: test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-ff629078718dd71f.json (deflated 87%) 2025-12-04T13:57:53.2206832Z adding: test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-2cc0c50f93d222cd.json (deflated 86%) 2025-12-04T13:57:53.2207702Z adding: test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-ca536e35d6aacc32.json (deflated 86%) 2025-12-04T13:57:53.2208577Z adding: test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-732261681906a561.json (deflated 87%) 2025-12-04T13:57:53.2209519Z adding: test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-dd0f5dd918ba20c2.json (deflated 86%) 2025-12-04T13:57:53.2210393Z adding: test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-2967af23244bf7ea.json (deflated 86%) 2025-12-04T13:57:53.2211234Z adding: test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-defe8bc47342e70f.json (stored 0%) 2025-12-04T13:57:53.2240849Z adding: test/test-reports/python-pytest/inductor.test_compile_subprocess/inductor.test_compile_subprocess-576cab637f4fea56.json (deflated 96%) 2025-12-04T13:57:53.2241717Z adding: test/test-reports/python-pytest/inductor.test_compile_subprocess/inductor.test_compile_subprocess-c5e45fed58b5f3dc.json (deflated 74%) 2025-12-04T13:57:53.2253557Z adding: test/test-reports/python-pytest/inductor.test_compile_subprocess/inductor.test_compile_subprocess-825047c883c2cfeb.json (deflated 96%) 2025-12-04T13:57:53.2254373Z adding: test/test-reports/python-pytest/inductor.test_deterministic/inductor.test_deterministic-3de4e0d14d6571c6.json (deflated 84%) 2025-12-04T13:57:53.2255511Z adding: test/test-reports/python-pytest/dynamo.test_fx_graph_runnable/dynamo.test_fx_graph_runnable-4db38b1d3487bc19.json (deflated 89%) 2025-12-04T13:57:53.2256636Z adding: test/test-reports/python-pytest/inductor.test_scatter_optimization/inductor.test_scatter_optimization-7db92b732e333421.json (deflated 84%) 2025-12-04T13:57:53.2260072Z adding: test/test-reports/python-pytest/inductor.test_padding/inductor.test_padding-2fcd8838081bfd5e.json (deflated 91%) 2025-12-04T13:57:53.2260893Z adding: test/test-reports/python-pytest/dynamo.test_callback/dynamo.test_callback-dbddf90c7da99eb5.json (deflated 61%) 2025-12-04T13:57:53.2270868Z adding: test/test-reports/python-pytest/inductor.test_cudagraph_trees_expandable_segments/inductor.test_cudagraph_trees_expandable_segments-e8ff2a368a15c33d.json (deflated 93%) 2025-12-04T13:57:53.2272163Z adding: test/test-reports/python-pytest/inductor.test_cudagraph_trees_expandable_segments/inductor.test_cudagraph_trees_expandable_segments-99f752af3a8e76fa.json (deflated 57%) 2025-12-04T13:57:53.2276441Z adding: test/test-reports/python-pytest/inductor.test_cudagraph_trees_expandable_segments/inductor.test_cudagraph_trees_expandable_segments-3b9cbca8ed6c6a9f.json (deflated 92%) 2025-12-04T13:57:53.2293447Z adding: test/test-reports/python-pytest/test_autograd/test_autograd-428700f7d6c2123c.json (deflated 93%) 2025-12-04T13:57:53.2296236Z adding: test/test-reports/python-pytest/test_dataloader/test_dataloader-67cc6525e23b4ee5.json (deflated 90%) 2025-12-04T13:57:53.2306050Z adding: test/test-reports/python-pytest/test_decomp/test_decomp-0baa94e9decc4066.json (deflated 95%) 2025-12-04T13:57:53.2315791Z adding: test/test-reports/python-pytest/test_decomp/test_decomp-febede57690ded63.json (deflated 95%) 2025-12-04T13:57:53.2351063Z adding: test/test-reports/python-pytest/test_ops_fwd_gradients/test_ops_fwd_gradients-308b20ea781f4444.json (deflated 97%) 2025-12-04T13:57:53.2364164Z adding: test/test-reports/python-pytest/test_ops_jit/test_ops_jit-8aecbb695376eca7.json (deflated 95%) 2025-12-04T13:57:53.2399719Z adding: test/test-reports/python-pytest/test_nestedtensor/test_nestedtensor-3292279b9435c183.json (deflated 98%) 2025-12-04T13:57:53.2480908Z adding: test/test-reports/python-pytest/test_ops/test_ops-383153bca27cf48b.json (deflated 96%) 2025-12-04T13:57:53.2510050Z adding: test/test-reports/python-pytest/functorch.test_ops/functorch.test_ops-78f655d60a12ab9a.json (deflated 95%) 2025-12-04T13:57:53.2566610Z adding: test/test-reports/python-pytest/functorch.test_vmap/functorch.test_vmap-e514136507c8340c.json (deflated 96%) 2025-12-04T13:57:53.2567484Z adding: test/test-reports/python-pytest/dynamo.test_after_aot/dynamo.test_after_aot-392fab221b48829f.json (deflated 59%) 2025-12-04T13:57:53.2570085Z adding: test/test-reports/python-pytest/inductor.test_snode_runtime/inductor.test_snode_runtime-a0450ad8c50e7ed1.json (deflated 92%) 2025-12-04T13:57:53.2727941Z adding: test/test-reports/python-pytest/inductor.test_minifier/inductor.test_minifier-02a4962c79ea03ff.json (deflated 97%) 2025-12-04T13:57:53.2762039Z adding: test/test-reports/python-pytest/inductor.test_compiled_autograd/inductor.test_compiled_autograd-4b5e4b19f368f980.json (deflated 93%) 2025-12-04T13:57:53.2774199Z adding: test/test-reports/python-pytest/inductor.test_torchinductor_strided_blocks/inductor.test_torchinductor_strided_blocks-2a8dc82a74796451.json (deflated 97%) 2025-12-04T13:57:53.2775044Z adding: test/test-reports/python-pytest/inductor.test_pad_mm/inductor.test_pad_mm-e95ecbf722b780e7.json (deflated 94%) 2025-12-04T13:57:53.2775904Z adding: test/test-reports/python-pytest/inductor.test_triton_syntax/inductor.test_triton_syntax-445bbe43cdf23550.json (deflated 62%) 2025-12-04T13:57:53.2776932Z adding: test/test-reports/python-pytest/inductor.test_triton_extension_backend/inductor.test_triton_extension_backend-a1753ebb46ebf85e.json (stored 0%) 2025-12-04T13:57:53.2782145Z adding: test/test-reports/python-pytest/test_sparse_semi_structured/test_sparse_semi_structured-8ce56cf98dd64834.json (deflated 96%) 2025-12-04T13:57:53.2789762Z adding: test/test-reports/python-pytest/inductor.test_gpu_cpp_wrapper/inductor.test_gpu_cpp_wrapper-5772573b88e52d5b.json (deflated 94%) 2025-12-04T13:57:53.2790793Z adding: test/test-reports/python-pytest/inductor.test_control_deps/inductor.test_control_deps-2309f470268f6ec0.json (deflated 47%) 2025-12-04T13:57:53.2791763Z adding: test/test-reports/python-pytest/inductor.test_benchmarking/inductor.test_benchmarking-2c8c5f4991e26591.json (deflated 91%) 2025-12-04T13:57:53.2792760Z adding: test/test-reports/python-pytest/inductor.test_best_config/inductor.test_best_config-0c0fda7542001e9d.json (deflated 52%) 2025-12-04T13:57:53.2793647Z adding: test/test-reports/python-pytest/export.test_tools/export.test_tools-2cb5c16e1bf3618d.json (deflated 56%) 2025-12-04T13:57:53.2806128Z adding: test/test-reports/python-pytest/inductor.test_compiled_optimizers/inductor.test_compiled_optimizers-c666cafe979acadb.json (deflated 96%) 2025-12-04T13:57:53.2821232Z adding: test/test-reports/python-pytest/inductor.test_control_flow/inductor.test_control_flow-7c7af97e129282ca.json (deflated 97%) 2025-12-04T13:57:53.2822149Z adding: test/test-reports/python-pytest/test_per_overload_api/test_per_overload_api-380dd5ceb2a22fbd.json (deflated 75%) 2025-12-04T13:57:53.2823033Z adding: test/test-reports/python-pytest/test_multiprocessing/test_multiprocessing-1ba78cf82b9f2c08.json (deflated 89%) 2025-12-04T13:57:53.2840194Z adding: test/test-reports/python-pytest/functorch.test_control_flow/functorch.test_control_flow-f3515033a541aea0.json (deflated 96%) 2025-12-04T13:57:53.2841160Z adding: test/test-reports/python-pytest/test_tensorboard/test_tensorboard-fca5410f597c8a55.json (deflated 90%) 2025-12-04T13:57:53.2843634Z adding: test/test-reports/python-pytest/torch_np.numpy_tests.core.test_indexing/torch_np.numpy_tests.core.test_indexing-55ec670873971da8.json (deflated 93%) 2025-12-04T13:57:53.2844691Z adding: test/test-reports/python-pytest/test_futures/test_futures-365879bca82fb89f.json (deflated 92%) 2025-12-04T13:57:53.2854176Z adding: test/test-reports/python-pytest/test_tensor_creation_ops/test_tensor_creation_ops-3fdf3b4be21ec825.json (deflated 96%) 2025-12-04T13:57:53.2859780Z adding: test/test-reports/python-pytest/torch_np.numpy_tests.linalg.test_linalg/torch_np.numpy_tests.linalg.test_linalg-5c045926bb031e0f.json (deflated 97%) 2025-12-04T13:57:53.2862690Z adding: test/test-reports/python-pytest/torch_np.numpy_tests.core.test_dtype/torch_np.numpy_tests.core.test_dtype-14b751af817e6a8b.json (deflated 97%) 2025-12-04T13:57:53.2863727Z adding: test/test-reports/python-pytest/lazy.test_debug_util/lazy.test_debug_util-84565a365481d73c.json (deflated 33%) 2025-12-04T13:57:53.2864570Z adding: test/test-reports/python-pytest/nn.test_load_state_dict/nn.test_load_state_dict-243a96ba7564bbce.json (deflated 94%) 2025-12-04T13:57:53.2865414Z adding: test/test-reports/python-pytest/test_shape_ops/test_shape_ops-3d3d95d2af19f7b9.json (deflated 96%) 2025-12-04T13:57:53.2866592Z adding: test/test-reports/python-pytest/profiler.test_memory_profiler/profiler.test_memory_profiler-c5c6304c0eac96b6.json (deflated 87%) 2025-12-04T13:57:53.2870097Z adding: test/test-reports/python-pytest/test_indexing/test_indexing-05fc579dc970f1f4.json (deflated 95%) 2025-12-04T13:57:53.2871151Z adding: test/test-reports/python-pytest/torch_np.numpy_tests.core.test_getlimits/torch_np.numpy_tests.core.test_getlimits-9dc4d2d2b9b5672c.json (deflated 91%) 2025-12-04T13:57:53.2877247Z adding: test/test-reports/python-pytest/torch_np.test_ndarray_methods/torch_np.test_ndarray_methods-30a1cc814f90c4ef.json (deflated 98%) 2025-12-04T13:57:53.2882506Z adding: test/test-reports/python-pytest/test_view_ops/test_view_ops-6b7df7431405f809.json (deflated 95%) 2025-12-04T13:57:53.2883316Z adding: test/test-reports/python-pytest/test_type_info/test_type_info-d4f338100cffb528.json (deflated 83%) 2025-12-04T13:57:53.2900321Z adding: test/test-reports/python-pytest/functorch.test_aotdispatch/functorch.test_aotdispatch-c6f1fb2e499ba29b.json (deflated 95%) 2025-12-04T13:57:53.2901670Z adding: test/test-reports/python-pytest/test_native_mha/test_native_mha-948cc387c4d4054b.json (deflated 95%) 2025-12-04T13:57:53.2902843Z adding: test/test-reports/python-pytest/torch_np.numpy_tests.core.test_numerictypes/torch_np.numpy_tests.core.test_numerictypes-39c198af6c0e3f47.json (deflated 94%) 2025-12-04T13:57:53.2903977Z adding: test/test-reports/python-pytest/test_function_schema/test_function_schema-aaa43be3c6542b34.json (deflated 91%) 2025-12-04T13:57:53.2904779Z adding: test/test-reports/python-pytest/nn.test_init/nn.test_init-f88b1f9c9665697b.json (deflated 91%) 2025-12-04T13:57:53.2905624Z adding: test/test-reports/python-pytest/torch_np.test_scalars_0D_arrays/torch_np.test_scalars_0D_arrays-94a518ca987c13a8.json (deflated 96%) 2025-12-04T13:57:53.2906378Z adding: test/test-reports/python-pytest/test_cuda_primary_ctx/test_cuda_primary_ctx-4703821b78d2c346.json (deflated 43%) 2025-12-04T13:57:53.2907075Z adding: test/test-reports/python-pytest/test_cuda_primary_ctx/test_cuda_primary_ctx-b8c58bb378a7356d.json (deflated 42%) 2025-12-04T13:57:53.2907917Z adding: test/test-reports/python-pytest/test_cuda_primary_ctx/test_cuda_primary_ctx-772a569d1e3ca8af.json (deflated 35%) 2025-12-04T13:57:53.2908585Z adding: test/test-reports/python-pytest/test_cuda_primary_ctx/test_cuda_primary_ctx-6abad4fe89929138.json (deflated 42%) 2025-12-04T13:57:53.2909456Z adding: test/test-reports/python-pytest/profiler.test_profiler_tree/profiler.test_profiler_tree-96e743ca3bbe44d8.json (deflated 87%) 2025-12-04T13:57:53.2910164Z adding: test/test-reports/python-pytest/test_dlpack/test_dlpack-f4bd83620146cc39.json (deflated 97%) 2025-12-04T13:57:53.2910774Z adding: test/test-reports/python-pytest/lazy.test_reuse_ir/lazy.test_reuse_ir-494000dbe63af35c.json (deflated 78%) 2025-12-04T13:57:53.2911516Z adding: test/test-reports/python-pytest/test_functional_autograd_benchmark/test_functional_autograd_benchmark-632b3fce6e67c954.json (deflated 63%) 2025-12-04T13:57:53.2916483Z adding: test/test-reports/python-pytest/distributions.test_distributions/distributions.test_distributions-0cc6093cedae9699.json (deflated 95%) 2025-12-04T13:57:53.2943268Z ##[group]Run # Remove any previous test reports if they exist 2025-12-04T13:57:53.2943744Z # Remove any previous test reports if they exist 2025-12-04T13:57:53.2944038Z rm -f test-reports-*.zip 2025-12-04T13:57:53.2944392Z zip -r "test-reports-${FILE_SUFFIX}.zip" test/test-reports -i '*.xml' -i '*.csv' 2025-12-04T13:57:53.2952056Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-12-04T13:57:53.2952340Z env: 2025-12-04T13:57:53.2952506Z GIT_DEFAULT_BRANCH: main 2025-12-04T13:57:53.2952701Z HAS_NVIDIA_GPU: true 2025-12-04T13:57:53.2952928Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2025-12-04T13:57:53.2953342Z DOCKER_CONTAINER_ID: 657e31a8be774492437c4338a4b954650ffa0203d69f0088e2d19ede1e69df31 2025-12-04T13:57:53.2953851Z FILE_SUFFIX: test-default-6-7-linux.g6.4xlarge.experimental.nvidia.gpu_57120265687 2025-12-04T13:57:53.2954204Z ##[endgroup] 2025-12-04T13:57:53.3100925Z adding: test/test-reports/python-pytest/inductor.test_torchinductor/inductor.test_torchinductor-410bad42b3bb466c.xml (deflated 93%) 2025-12-04T13:57:53.3125288Z adding: test/test-reports/python-pytest/inductor.test_torchinductor_dynamic_shapes/inductor.test_torchinductor_dynamic_shapes-1c753e0c38ee0fc9.xml (deflated 93%) 2025-12-04T13:57:53.3126774Z adding: test/test-reports/python-pytest/inductor.test_kernel_benchmark/inductor.test_kernel_benchmark-a01600ecae601eef.xml (deflated 84%) 2025-12-04T13:57:53.3130821Z adding: test/test-reports/python-pytest/inductor.test_torchinductor_opinfo/inductor.test_torchinductor_opinfo-d9c51466d8748c04.xml (deflated 92%) 2025-12-04T13:57:53.3134878Z adding: test/test-reports/python-pytest/inductor.test_torchinductor_opinfo/inductor.test_torchinductor_opinfo-eb72b3d6271f1e7e.xml (deflated 92%) 2025-12-04T13:57:53.3186006Z adding: test/test-reports/python-pytest/dynamo.test_repros/dynamo.test_repros-df80b1b40a6198cc.xml (deflated 92%) 2025-12-04T13:57:53.3187206Z adding: test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-9c66153466707d74.xml (deflated 87%) 2025-12-04T13:57:53.3188509Z adding: test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-11449d70dc195ff4.xml (deflated 85%) 2025-12-04T13:57:53.3189745Z adding: test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-6c093515a384d9fc.xml (deflated 85%) 2025-12-04T13:57:53.3190959Z adding: test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-7ea688002f0554bd.xml (deflated 87%) 2025-12-04T13:57:53.3192222Z adding: test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-759714b1d86e3c4c.xml (deflated 85%) 2025-12-04T13:57:53.3193421Z adding: test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-7a190e1b598a9023.xml (deflated 85%) 2025-12-04T13:57:53.3194567Z adding: test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-c45e67249599e6c2.xml (deflated 87%) 2025-12-04T13:57:53.3195946Z adding: test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-8112676a6b085940.xml (deflated 85%) 2025-12-04T13:57:53.3196922Z adding: test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-2280a7982827e03d.xml (deflated 85%) 2025-12-04T13:57:53.3197839Z adding: test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-ecd8523a161788bb.xml (deflated 87%) 2025-12-04T13:57:53.3198918Z adding: test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-d90aab22855519e4.xml (deflated 85%) 2025-12-04T13:57:53.3199953Z adding: test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-5cb074eac86f05e5.xml (deflated 85%) 2025-12-04T13:57:53.3215628Z adding: test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-7681ec3fadfa4a43.xml (deflated 86%) 2025-12-04T13:57:53.3216638Z adding: test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-49c0a334019b663e.xml (deflated 85%) 2025-12-04T13:57:53.3217553Z adding: test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-a4c3d2e69c8841e3.xml (deflated 85%) 2025-12-04T13:57:53.3218427Z adding: test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-0f60f7321324a86a.xml (deflated 86%) 2025-12-04T13:57:53.3219324Z adding: test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-294e2abb077cceb2.xml (deflated 85%) 2025-12-04T13:57:53.3220190Z adding: test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-b24025703d1a5b1f.xml (deflated 85%) 2025-12-04T13:57:53.3221050Z adding: test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-531381cfa4745e03.xml (deflated 86%) 2025-12-04T13:57:53.3221903Z adding: test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-bb14ed3958363693.xml (deflated 85%) 2025-12-04T13:57:53.3222764Z adding: test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-ec0356b9a2627168.xml (deflated 85%) 2025-12-04T13:57:53.3223735Z adding: test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-c81b585b68f6cf60.xml (deflated 86%) 2025-12-04T13:57:53.3224727Z adding: test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-647efcde8452d0c7.xml (deflated 85%) 2025-12-04T13:57:53.3225595Z adding: test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-abfd2b283c5f3df6.xml (deflated 85%) 2025-12-04T13:57:53.3226471Z adding: test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-e0f3393878857c02.xml (deflated 86%) 2025-12-04T13:57:53.3227341Z adding: test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-d5da2e41e9fcecb9.xml (deflated 85%) 2025-12-04T13:57:53.3228220Z adding: test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-5fb75fde833c163c.xml (deflated 85%) 2025-12-04T13:57:53.3229080Z adding: test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-2f9606f02b59f87b.xml (deflated 86%) 2025-12-04T13:57:53.3229941Z adding: test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-94458b88fceefe4e.xml (deflated 85%) 2025-12-04T13:57:53.3230795Z adding: test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-6787c22d74999180.xml (deflated 85%) 2025-12-04T13:57:53.3231734Z adding: test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-cdf948359e80bb41.xml (deflated 86%) 2025-12-04T13:57:53.3232598Z adding: test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-bd192fd6fb52c26e.xml (deflated 85%) 2025-12-04T13:57:53.3233452Z adding: test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-7d48593db7bb6f60.xml (deflated 85%) 2025-12-04T13:57:53.3234360Z adding: test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-91ed99958e7f9cc9.xml (deflated 86%) 2025-12-04T13:57:53.3235214Z adding: test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-f4094aba19e671d4.xml (deflated 85%) 2025-12-04T13:57:53.3236070Z adding: test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-204edfd0c56893dd.xml (deflated 85%) 2025-12-04T13:57:53.3236948Z adding: test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-5d3545fec05b298e.xml (deflated 86%) 2025-12-04T13:57:53.3237798Z adding: test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-c2388e46be16ba24.xml (deflated 85%) 2025-12-04T13:57:53.3238649Z adding: test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-e9cf01ceaaab6f89.xml (deflated 85%) 2025-12-04T13:57:53.3239517Z adding: test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-ff104bb89c063280.xml (deflated 86%) 2025-12-04T13:57:53.3240376Z adding: test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-61993e92defaa624.xml (deflated 85%) 2025-12-04T13:57:53.3241233Z adding: test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-16a80675dc6db2f9.xml (deflated 85%) 2025-12-04T13:57:53.3242080Z adding: test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-601480a9a7ef82d7.xml (deflated 86%) 2025-12-04T13:57:53.3242948Z adding: test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-fda38496367dc0a9.xml (deflated 85%) 2025-12-04T13:57:53.3243812Z adding: test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-13c7a98252f889c4.xml (deflated 85%) 2025-12-04T13:57:53.3244714Z adding: test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-d1b821ab0fc5c966.xml (deflated 86%) 2025-12-04T13:57:53.3245579Z adding: test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-d70416164f95a17a.xml (deflated 85%) 2025-12-04T13:57:53.3246433Z adding: test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-a1832326f6932688.xml (deflated 85%) 2025-12-04T13:57:53.3247275Z adding: test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-1965a836c7535ef2.xml (deflated 86%) 2025-12-04T13:57:53.3248133Z adding: test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-12f95c060627756e.xml (deflated 85%) 2025-12-04T13:57:53.3248978Z adding: test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-c82b9fa27197518a.xml (deflated 85%) 2025-12-04T13:57:53.3249837Z adding: test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-36ce77ac1c8c9763.xml (deflated 86%) 2025-12-04T13:57:53.3250704Z adding: test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-e4f38ecbc39decbb.xml (deflated 85%) 2025-12-04T13:57:53.3251890Z adding: test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-e20c92a1eaab2de9.xml (deflated 85%) 2025-12-04T13:57:53.3252764Z adding: test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-2048f1d3bc205458.xml (deflated 86%) 2025-12-04T13:57:53.3253618Z adding: test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-d670b011b4f3da72.xml (deflated 85%) 2025-12-04T13:57:53.3254509Z adding: test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-56e0b2ecffd4aa2f.xml (deflated 85%) 2025-12-04T13:57:53.3255366Z adding: test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-1e9198c3a728666b.xml (deflated 86%) 2025-12-04T13:57:53.3256218Z adding: test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-c1c9525c22bede8d.xml (deflated 85%) 2025-12-04T13:57:53.3257077Z adding: test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-cc4ec46989be7746.xml (deflated 85%) 2025-12-04T13:57:53.3257932Z adding: test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-3f42debbb8d01db8.xml (deflated 86%) 2025-12-04T13:57:53.3258788Z adding: test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-92851bf92613c0d7.xml (deflated 85%) 2025-12-04T13:57:53.3259645Z adding: test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-3610b3094a676974.xml (deflated 85%) 2025-12-04T13:57:53.3260489Z adding: test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-dba8bfd8733d5e09.xml (deflated 86%) 2025-12-04T13:57:53.3261338Z adding: test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-6c64eb8af833b17a.xml (deflated 85%) 2025-12-04T13:57:53.3262194Z adding: test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-f878b685b7c3b00a.xml (deflated 85%) 2025-12-04T13:57:53.3263037Z adding: test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-232061e660a2ff26.xml (deflated 86%) 2025-12-04T13:57:53.3263964Z adding: test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-fe4ea69fd2c99307.xml (deflated 85%) 2025-12-04T13:57:53.3264862Z adding: test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-4bc55824cd7c5544.xml (deflated 85%) 2025-12-04T13:57:53.3265726Z adding: test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-c2fc9a8f37b6f624.xml (deflated 86%) 2025-12-04T13:57:53.3266583Z adding: test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-df0458d9683e0406.xml (deflated 85%) 2025-12-04T13:57:53.3267432Z adding: test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-1f5454d7efc5fad5.xml (deflated 85%) 2025-12-04T13:57:53.3268281Z adding: test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-0e640f0818a9f631.xml (deflated 86%) 2025-12-04T13:57:53.3269130Z adding: test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-e912b3c1622dc5a6.xml (deflated 85%) 2025-12-04T13:57:53.3269977Z adding: test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-9bf082709afd4107.xml (deflated 85%) 2025-12-04T13:57:53.3270823Z adding: test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-417661a6a446d83a.xml (deflated 86%) 2025-12-04T13:57:53.3271756Z adding: test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-2d84d0ab3f5cbf16.xml (deflated 85%) 2025-12-04T13:57:53.3272612Z adding: test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-3c429c2af5dedece.xml (deflated 85%) 2025-12-04T13:57:53.3273458Z adding: test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-b9ae48951e1b6488.xml (deflated 87%) 2025-12-04T13:57:53.3274355Z adding: test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-a9948af371c98fbd.xml (deflated 85%) 2025-12-04T13:57:53.3275200Z adding: test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-f058047ef4d14b46.xml (deflated 85%) 2025-12-04T13:57:53.3276059Z adding: test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-5d247f52fbae6573.xml (deflated 86%) 2025-12-04T13:57:53.3276903Z adding: test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-b685fc5b32b50b90.xml (deflated 85%) 2025-12-04T13:57:53.3277751Z adding: test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-bc2410b75aa40646.xml (deflated 85%) 2025-12-04T13:57:53.3278875Z adding: test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-71cfec7d40715d91.xml (deflated 86%) 2025-12-04T13:57:53.3279733Z adding: test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-c9dbc1d234ce26dc.xml (deflated 85%) 2025-12-04T13:57:53.3280588Z adding: test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-8a076bf12e47858e.xml (deflated 85%) 2025-12-04T13:57:53.3281441Z adding: test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-bb0bac5f3b803919.xml (deflated 87%) 2025-12-04T13:57:53.3282293Z adding: test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-0645969e8b49cd4b.xml (deflated 85%) 2025-12-04T13:57:53.3283152Z adding: test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-20accf8a0f4e3024.xml (deflated 85%) 2025-12-04T13:57:53.3283999Z adding: test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-55dc1e1f737da98a.xml (deflated 86%) 2025-12-04T13:57:53.3284939Z adding: test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-458d9d61d3eb81db.xml (deflated 85%) 2025-12-04T13:57:53.3285788Z adding: test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-0cf2b0bead25849b.xml (deflated 85%) 2025-12-04T13:57:53.3286644Z adding: test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-e47d4d68d26f0ad1.xml (deflated 86%) 2025-12-04T13:57:53.3287487Z adding: test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-044dfd62cda981dc.xml (deflated 85%) 2025-12-04T13:57:53.3288336Z adding: test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-f4c669524343c1b1.xml (deflated 85%) 2025-12-04T13:57:53.3289196Z adding: test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-9f9264fbba4173a1.xml (deflated 86%) 2025-12-04T13:57:53.3290051Z adding: test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-c85f34f0c90d933d.xml (deflated 85%) 2025-12-04T13:57:53.3290911Z adding: test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-4b2e7fd1ed886dfe.xml (deflated 85%) 2025-12-04T13:57:53.3291891Z adding: test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-f54f533501201f35.xml (deflated 86%) 2025-12-04T13:57:53.3292742Z adding: test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-cbbd83fa84eb990f.xml (deflated 85%) 2025-12-04T13:57:53.3293597Z adding: test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-67e17348ec4a01d8.xml (deflated 85%) 2025-12-04T13:57:53.3294515Z adding: test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-f72cf4cc8675ff8a.xml (deflated 86%) 2025-12-04T13:57:53.3295375Z adding: test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-bd700a0c553d5cfe.xml (deflated 85%) 2025-12-04T13:57:53.3296223Z adding: test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-28b04335f093e360.xml (deflated 85%) 2025-12-04T13:57:53.3297087Z adding: test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-857ce5cc87a4a050.xml (deflated 87%) 2025-12-04T13:57:53.3297942Z adding: test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-4e0b90b7cea75841.xml (deflated 85%) 2025-12-04T13:57:53.3298795Z adding: test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-14983efcd69bf6ac.xml (deflated 85%) 2025-12-04T13:57:53.3299657Z adding: test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-2104559418af7772.xml (deflated 86%) 2025-12-04T13:57:53.3300506Z adding: test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-49eccf27aa184bc6.xml (deflated 85%) 2025-12-04T13:57:53.3301365Z adding: test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-71736827b1650ca5.xml (deflated 85%) 2025-12-04T13:57:53.3302219Z adding: test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-b5c5c102e82be0c3.xml (deflated 86%) 2025-12-04T13:57:53.3303069Z adding: test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-6b2f788f9a7d50ec.xml (deflated 85%) 2025-12-04T13:57:53.3303986Z adding: test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-61d7ccaaf8cfd370.xml (deflated 85%) 2025-12-04T13:57:53.3304899Z adding: test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-cc96100da976ce4e.xml (deflated 87%) 2025-12-04T13:57:53.3305755Z adding: test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-e2ca8845caeab461.xml (deflated 85%) 2025-12-04T13:57:53.3306614Z adding: test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-ea9e1f7f9950a9ad.xml (deflated 85%) 2025-12-04T13:57:53.3307477Z adding: test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-778cc31dcd227b51.xml (deflated 86%) 2025-12-04T13:57:53.3308323Z adding: test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-d8517590777959a7.xml (deflated 85%) 2025-12-04T13:57:53.3309176Z adding: test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-7c1369cc4e5815f4.xml (deflated 85%) 2025-12-04T13:57:53.3310031Z adding: test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-f43c4b9f36164ff6.xml (deflated 86%) 2025-12-04T13:57:53.3310877Z adding: test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-d56aa1824afdfad3.xml (deflated 85%) 2025-12-04T13:57:53.3311834Z adding: test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-6a9d289c68c250dc.xml (deflated 85%) 2025-12-04T13:57:53.3312693Z adding: test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-04e5068c64f117f5.xml (deflated 86%) 2025-12-04T13:57:53.3313543Z adding: test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-85ab96cf3cf02016.xml (deflated 85%) 2025-12-04T13:57:53.3314432Z adding: test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-888ba6ece96c2997.xml (deflated 85%) 2025-12-04T13:57:53.3315291Z adding: test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-58cf8d0fed87176f.xml (deflated 86%) 2025-12-04T13:57:53.3316146Z adding: test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-0022ebcfefe802db.xml (deflated 85%) 2025-12-04T13:57:53.3317001Z adding: test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-dac6db8ed6c97e37.xml (deflated 85%) 2025-12-04T13:57:53.3317857Z adding: test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-040b4092def69522.xml (deflated 86%) 2025-12-04T13:57:53.3318714Z adding: test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-084397a772e6c7f7.xml (deflated 85%) 2025-12-04T13:57:53.3319560Z adding: test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-2942f663738ee80d.xml (deflated 85%) 2025-12-04T13:57:53.3320413Z adding: test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-ac0d8e76747a1a1b.xml (deflated 86%) 2025-12-04T13:57:53.3321268Z adding: test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-7f452630b39063ce.xml (deflated 85%) 2025-12-04T13:57:53.3322118Z adding: test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-bb668a52e682b2dd.xml (deflated 85%) 2025-12-04T13:57:53.3322967Z adding: test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-60466f6816d813eb.xml (deflated 86%) 2025-12-04T13:57:53.3323831Z adding: test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-3b52d4f5cebad92d.xml (deflated 85%) 2025-12-04T13:57:53.3324728Z adding: test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-2ff4d68a3a0ad43c.xml (deflated 85%) 2025-12-04T13:57:53.3325588Z adding: test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-7240a6bd78b3c1ba.xml (deflated 86%) 2025-12-04T13:57:53.3326445Z adding: test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-7cb1dae7cd5b9c6d.xml (deflated 85%) 2025-12-04T13:57:53.3327291Z adding: test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-1238d86169b8b87a.xml (deflated 85%) 2025-12-04T13:57:53.3328140Z adding: test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-86e4cbaa0e57366b.xml (deflated 86%) 2025-12-04T13:57:53.3328999Z adding: test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-1c1918cf02456db6.xml (deflated 85%) 2025-12-04T13:57:53.3329858Z adding: test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-4d7a6e50e4cb57cd.xml (deflated 85%) 2025-12-04T13:57:53.3330793Z adding: test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-cd0db30d38ee28ec.xml (deflated 86%) 2025-12-04T13:57:53.3331638Z adding: test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-97331d68f5e395ac.xml (deflated 85%) 2025-12-04T13:57:53.3332490Z adding: test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-266b0df1c83533e9.xml (deflated 85%) 2025-12-04T13:57:53.3333341Z adding: test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-0728186d53d0914c.xml (deflated 86%) 2025-12-04T13:57:53.3334240Z adding: test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-ace5a67d4c3c2a01.xml (deflated 85%) 2025-12-04T13:57:53.3335094Z adding: test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-eeeef84645c1bb52.xml (deflated 85%) 2025-12-04T13:57:53.3335954Z adding: test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-ffbbb699e0b6f991.xml (deflated 86%) 2025-12-04T13:57:53.3336812Z adding: test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-90e4a91c9bbb810f.xml (deflated 85%) 2025-12-04T13:57:53.3337670Z adding: test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-2ee8d3487d6282f0.xml (deflated 85%) 2025-12-04T13:57:53.3338532Z adding: test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-2ab4541f9392b10c.xml (deflated 86%) 2025-12-04T13:57:53.3339380Z adding: test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-ce43d3c4fee34ad0.xml (deflated 85%) 2025-12-04T13:57:53.3340239Z adding: test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-d04b19868e6e9971.xml (deflated 85%) 2025-12-04T13:57:53.3341096Z adding: test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-cf6ace4ea5db8dfb.xml (deflated 86%) 2025-12-04T13:57:53.3341960Z adding: test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-8b65a3dabcd39e70.xml (deflated 85%) 2025-12-04T13:57:53.3342817Z adding: test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-c85748e91a458eac.xml (deflated 85%) 2025-12-04T13:57:53.3343716Z adding: test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-85bca917f8457708.xml (deflated 86%) 2025-12-04T13:57:53.3344613Z adding: test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-6a70392d2e5080cc.xml (deflated 85%) 2025-12-04T13:57:53.3345473Z adding: test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-1b506e26b1410ad4.xml (deflated 85%) 2025-12-04T13:57:53.3346330Z adding: test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-f7f9c3afca4dce49.xml (deflated 86%) 2025-12-04T13:57:53.3347177Z adding: test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-c41d68e69339995e.xml (deflated 85%) 2025-12-04T13:57:53.3348034Z adding: test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-dfa47b8dd194f01b.xml (deflated 85%) 2025-12-04T13:57:53.3348899Z adding: test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-6b849be23abc3e95.xml (deflated 86%) 2025-12-04T13:57:53.3349754Z adding: test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-1eeada475e93262f.xml (deflated 85%) 2025-12-04T13:57:53.3350667Z adding: test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-3a596bc2a015c0a5.xml (deflated 85%) 2025-12-04T13:57:53.3351515Z adding: test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-ff629078718dd71f.xml (deflated 87%) 2025-12-04T13:57:53.3352370Z adding: test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-2cc0c50f93d222cd.xml (deflated 85%) 2025-12-04T13:57:53.3353225Z adding: test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-ca536e35d6aacc32.xml (deflated 85%) 2025-12-04T13:57:53.3354110Z adding: test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-732261681906a561.xml (deflated 86%) 2025-12-04T13:57:53.3354958Z adding: test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-dd0f5dd918ba20c2.xml (deflated 85%) 2025-12-04T13:57:53.3355815Z adding: test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-2967af23244bf7ea.xml (deflated 85%) 2025-12-04T13:57:53.3356682Z adding: test/test-reports/python-pytest/inductor.test_cuda_select_algorithm/inductor.test_cuda_select_algorithm-defe8bc47342e70f.xml (deflated 27%) 2025-12-04T13:57:53.3375583Z adding: test/test-reports/python-pytest/inductor.test_compile_subprocess/inductor.test_compile_subprocess-576cab637f4fea56.xml (deflated 96%) 2025-12-04T13:57:53.3376700Z adding: test/test-reports/python-pytest/inductor.test_compile_subprocess/inductor.test_compile_subprocess-c5e45fed58b5f3dc.xml (deflated 73%) 2025-12-04T13:57:53.3387773Z adding: test/test-reports/python-pytest/inductor.test_compile_subprocess/inductor.test_compile_subprocess-825047c883c2cfeb.xml (deflated 96%) 2025-12-04T13:57:53.3388861Z adding: test/test-reports/python-pytest/inductor.test_deterministic/inductor.test_deterministic-3de4e0d14d6571c6.xml (deflated 80%) 2025-12-04T13:57:53.3389896Z adding: test/test-reports/python-pytest/dynamo.test_fx_graph_runnable/dynamo.test_fx_graph_runnable-4db38b1d3487bc19.xml (deflated 87%) 2025-12-04T13:57:53.3390969Z adding: test/test-reports/python-pytest/inductor.test_scatter_optimization/inductor.test_scatter_optimization-7db92b732e333421.xml (deflated 81%) 2025-12-04T13:57:53.3393775Z adding: test/test-reports/python-pytest/inductor.test_padding/inductor.test_padding-2fcd8838081bfd5e.xml (deflated 89%) 2025-12-04T13:57:53.3394674Z adding: test/test-reports/python-pytest/dynamo.test_callback/dynamo.test_callback-dbddf90c7da99eb5.xml (deflated 56%) 2025-12-04T13:57:53.3403402Z adding: test/test-reports/python-pytest/inductor.test_cudagraph_trees_expandable_segments/inductor.test_cudagraph_trees_expandable_segments-e8ff2a368a15c33d.xml (deflated 92%) 2025-12-04T13:57:53.3404735Z adding: test/test-reports/python-pytest/inductor.test_cudagraph_trees_expandable_segments/inductor.test_cudagraph_trees_expandable_segments-99f752af3a8e76fa.xml (deflated 55%) 2025-12-04T13:57:53.3408596Z adding: test/test-reports/python-pytest/inductor.test_cudagraph_trees_expandable_segments/inductor.test_cudagraph_trees_expandable_segments-3b9cbca8ed6c6a9f.xml (deflated 90%) 2025-12-04T13:57:53.3422626Z adding: test/test-reports/python-pytest/test_autograd/test_autograd-428700f7d6c2123c.xml (deflated 89%) 2025-12-04T13:57:53.3424750Z adding: test/test-reports/python-pytest/test_dataloader/test_dataloader-67cc6525e23b4ee5.xml (deflated 83%) 2025-12-04T13:57:53.3432277Z adding: test/test-reports/python-pytest/test_decomp/test_decomp-0baa94e9decc4066.xml (deflated 91%) 2025-12-04T13:57:53.3439608Z adding: test/test-reports/python-pytest/test_decomp/test_decomp-febede57690ded63.xml (deflated 92%) 2025-12-04T13:57:53.3468699Z adding: test/test-reports/python-pytest/test_ops_fwd_gradients/test_ops_fwd_gradients-308b20ea781f4444.xml (deflated 95%) 2025-12-04T13:57:53.3479240Z adding: test/test-reports/python-pytest/test_ops_jit/test_ops_jit-8aecbb695376eca7.xml (deflated 93%) 2025-12-04T13:57:53.3511492Z adding: test/test-reports/python-pytest/test_nestedtensor/test_nestedtensor-3292279b9435c183.xml (deflated 98%) 2025-12-04T13:57:53.3575832Z adding: test/test-reports/python-pytest/test_ops/test_ops-383153bca27cf48b.xml (deflated 94%) 2025-12-04T13:57:53.3599930Z adding: test/test-reports/python-pytest/functorch.test_ops/functorch.test_ops-78f655d60a12ab9a.xml (deflated 93%) 2025-12-04T13:57:53.3646385Z adding: test/test-reports/python-pytest/functorch.test_vmap/functorch.test_vmap-e514136507c8340c.xml (deflated 94%) 2025-12-04T13:57:53.3647417Z adding: test/test-reports/python-pytest/dynamo.test_after_aot/dynamo.test_after_aot-392fab221b48829f.xml (deflated 51%) 2025-12-04T13:57:53.3649747Z adding: test/test-reports/python-pytest/inductor.test_snode_runtime/inductor.test_snode_runtime-a0450ad8c50e7ed1.xml (deflated 91%) 2025-12-04T13:57:53.3805497Z adding: test/test-reports/python-pytest/inductor.test_minifier/inductor.test_minifier-02a4962c79ea03ff.xml (deflated 97%) 2025-12-04T13:57:53.3836782Z adding: test/test-reports/python-pytest/inductor.test_compiled_autograd/inductor.test_compiled_autograd-4b5e4b19f368f980.xml (deflated 92%) 2025-12-04T13:57:53.3846885Z adding: test/test-reports/python-pytest/inductor.test_torchinductor_strided_blocks/inductor.test_torchinductor_strided_blocks-2a8dc82a74796451.xml (deflated 96%) 2025-12-04T13:57:53.3847921Z adding: test/test-reports/python-pytest/inductor.test_pad_mm/inductor.test_pad_mm-e95ecbf722b780e7.xml (deflated 91%) 2025-12-04T13:57:53.3848843Z adding: test/test-reports/python-pytest/inductor.test_triton_syntax/inductor.test_triton_syntax-445bbe43cdf23550.xml (deflated 61%) 2025-12-04T13:57:53.3849903Z adding: test/test-reports/python-pytest/inductor.test_triton_extension_backend/inductor.test_triton_extension_backend-a1753ebb46ebf85e.xml (deflated 28%) 2025-12-04T13:57:53.3852681Z adding: test/test-reports/python-pytest/test_sparse_semi_structured/test_sparse_semi_structured-8ce56cf98dd64834.xml (deflated 94%) 2025-12-04T13:57:53.3858745Z adding: test/test-reports/python-pytest/inductor.test_gpu_cpp_wrapper/inductor.test_gpu_cpp_wrapper-5772573b88e52d5b.xml (deflated 93%) 2025-12-04T13:57:53.3859531Z adding: test/test-reports/python-pytest/inductor.test_control_deps/inductor.test_control_deps-2309f470268f6ec0.xml (deflated 47%) 2025-12-04T13:57:53.3860299Z adding: test/test-reports/python-pytest/inductor.test_benchmarking/inductor.test_benchmarking-2c8c5f4991e26591.xml (deflated 87%) 2025-12-04T13:57:53.3861073Z adding: test/test-reports/python-pytest/inductor.test_best_config/inductor.test_best_config-0c0fda7542001e9d.xml (deflated 51%) 2025-12-04T13:57:53.3861878Z adding: test/test-reports/python-pytest/export.test_tools/export.test_tools-2cb5c16e1bf3618d.xml (deflated 47%) 2025-12-04T13:57:53.3873822Z adding: test/test-reports/python-pytest/inductor.test_compiled_optimizers/inductor.test_compiled_optimizers-c666cafe979acadb.xml (deflated 96%) 2025-12-04T13:57:53.3888488Z adding: test/test-reports/python-pytest/inductor.test_control_flow/inductor.test_control_flow-7c7af97e129282ca.xml (deflated 97%) 2025-12-04T13:57:53.3889411Z adding: test/test-reports/python-pytest/test_per_overload_api/test_per_overload_api-380dd5ceb2a22fbd.xml (deflated 60%) 2025-12-04T13:57:53.3890304Z adding: test/test-reports/python-pytest/test_multiprocessing/test_multiprocessing-1ba78cf82b9f2c08.xml (deflated 81%) 2025-12-04T13:57:53.3904095Z adding: test/test-reports/python-pytest/functorch.test_control_flow/functorch.test_control_flow-f3515033a541aea0.xml (deflated 95%) 2025-12-04T13:57:53.3905040Z adding: test/test-reports/python-pytest/test_tensorboard/test_tensorboard-fca5410f597c8a55.xml (deflated 83%) 2025-12-04T13:57:53.3906864Z adding: test/test-reports/python-pytest/torch_np.numpy_tests.core.test_indexing/torch_np.numpy_tests.core.test_indexing-55ec670873971da8.xml (deflated 89%) 2025-12-04T13:57:53.3907811Z adding: test/test-reports/python-pytest/test_futures/test_futures-365879bca82fb89f.xml (deflated 83%) 2025-12-04T13:57:53.3915223Z adding: test/test-reports/python-pytest/test_tensor_creation_ops/test_tensor_creation_ops-3fdf3b4be21ec825.xml (deflated 94%) 2025-12-04T13:57:53.3918992Z adding: test/test-reports/python-pytest/torch_np.numpy_tests.linalg.test_linalg/torch_np.numpy_tests.linalg.test_linalg-5c045926bb031e0f.xml (deflated 94%) 2025-12-04T13:57:53.3921236Z adding: test/test-reports/python-pytest/torch_np.numpy_tests.core.test_dtype/torch_np.numpy_tests.core.test_dtype-14b751af817e6a8b.xml (deflated 95%) 2025-12-04T13:57:53.3922337Z adding: test/test-reports/python-pytest/lazy.test_debug_util/lazy.test_debug_util-84565a365481d73c.xml (deflated 36%) 2025-12-04T13:57:53.3923207Z adding: test/test-reports/python-pytest/nn.test_load_state_dict/nn.test_load_state_dict-243a96ba7564bbce.xml (deflated 89%) 2025-12-04T13:57:53.3924043Z adding: test/test-reports/python-pytest/test_shape_ops/test_shape_ops-3d3d95d2af19f7b9.xml (deflated 92%) 2025-12-04T13:57:53.3925145Z adding: test/test-reports/python-pytest/profiler.test_memory_profiler/profiler.test_memory_profiler-c5c6304c0eac96b6.xml (deflated 79%) 2025-12-04T13:57:53.3926729Z adding: test/test-reports/python-pytest/test_indexing/test_indexing-05fc579dc970f1f4.xml (deflated 90%) 2025-12-04T13:57:53.3927582Z adding: test/test-reports/python-pytest/torch_np.numpy_tests.core.test_getlimits/torch_np.numpy_tests.core.test_getlimits-9dc4d2d2b9b5672c.xml (deflated 86%) 2025-12-04T13:57:53.3931316Z adding: test/test-reports/python-pytest/torch_np.test_ndarray_methods/torch_np.test_ndarray_methods-30a1cc814f90c4ef.xml (deflated 96%) 2025-12-04T13:57:53.3934725Z adding: test/test-reports/python-pytest/test_view_ops/test_view_ops-6b7df7431405f809.xml (deflated 92%) 2025-12-04T13:57:53.3935464Z adding: test/test-reports/python-pytest/test_type_info/test_type_info-d4f338100cffb528.xml (deflated 67%) 2025-12-04T13:57:53.3949913Z adding: test/test-reports/python-pytest/functorch.test_aotdispatch/functorch.test_aotdispatch-c6f1fb2e499ba29b.xml (deflated 93%) 2025-12-04T13:57:53.3950919Z adding: test/test-reports/python-pytest/test_native_mha/test_native_mha-948cc387c4d4054b.xml (deflated 93%) 2025-12-04T13:57:53.3951954Z adding: test/test-reports/python-pytest/torch_np.numpy_tests.core.test_numerictypes/torch_np.numpy_tests.core.test_numerictypes-39c198af6c0e3f47.xml (deflated 90%) 2025-12-04T13:57:53.3953000Z adding: test/test-reports/python-pytest/test_function_schema/test_function_schema-aaa43be3c6542b34.xml (deflated 82%) 2025-12-04T13:57:53.3953806Z adding: test/test-reports/python-pytest/nn.test_init/nn.test_init-f88b1f9c9665697b.xml (deflated 83%) 2025-12-04T13:57:53.3954780Z adding: test/test-reports/python-pytest/torch_np.test_scalars_0D_arrays/torch_np.test_scalars_0D_arrays-94a518ca987c13a8.xml (deflated 91%) 2025-12-04T13:57:53.3955709Z adding: test/test-reports/python-pytest/test_cuda_primary_ctx/test_cuda_primary_ctx-4703821b78d2c346.xml (deflated 43%) 2025-12-04T13:57:53.3956401Z adding: test/test-reports/python-pytest/test_cuda_primary_ctx/test_cuda_primary_ctx-b8c58bb378a7356d.xml (deflated 43%) 2025-12-04T13:57:53.3957091Z adding: test/test-reports/python-pytest/test_cuda_primary_ctx/test_cuda_primary_ctx-772a569d1e3ca8af.xml (deflated 36%) 2025-12-04T13:57:53.3957767Z adding: test/test-reports/python-pytest/test_cuda_primary_ctx/test_cuda_primary_ctx-6abad4fe89929138.xml (deflated 43%) 2025-12-04T13:57:53.3958501Z adding: test/test-reports/python-pytest/profiler.test_profiler_tree/profiler.test_profiler_tree-96e743ca3bbe44d8.xml (deflated 82%) 2025-12-04T13:57:53.3959190Z adding: test/test-reports/python-pytest/test_dlpack/test_dlpack-f4bd83620146cc39.xml (deflated 94%) 2025-12-04T13:57:53.3959813Z adding: test/test-reports/python-pytest/lazy.test_reuse_ir/lazy.test_reuse_ir-494000dbe63af35c.xml (deflated 62%) 2025-12-04T13:57:53.3960573Z adding: test/test-reports/python-pytest/test_functional_autograd_benchmark/test_functional_autograd_benchmark-632b3fce6e67c954.xml (deflated 54%) 2025-12-04T13:57:53.3962553Z adding: test/test-reports/python-pytest/distributions.test_distributions/distributions.test_distributions-0cc6093cedae9699.xml (deflated 90%) 2025-12-04T13:57:53.3988069Z ##[group]Run # Remove any previous usage logs if they exist 2025-12-04T13:57:53.3988408Z # Remove any previous usage logs if they exist 2025-12-04T13:57:53.3988677Z rm -f logs-*.zip 2025-12-04T13:57:53.3988938Z zip "logs-${FILE_SUFFIX}.zip" 'usage_log.txt' || true 2025-12-04T13:57:53.3989397Z zip -r "logs-${FILE_SUFFIX}.zip" test/test-reports -i '*.log' || true 2025-12-04T13:57:53.3996714Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-12-04T13:57:53.3996991Z env: 2025-12-04T13:57:53.3997168Z GIT_DEFAULT_BRANCH: main 2025-12-04T13:57:53.3997362Z HAS_NVIDIA_GPU: true 2025-12-04T13:57:53.3997592Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2025-12-04T13:57:53.3997983Z DOCKER_CONTAINER_ID: 657e31a8be774492437c4338a4b954650ffa0203d69f0088e2d19ede1e69df31 2025-12-04T13:57:53.3998481Z FILE_SUFFIX: test-default-6-7-linux.g6.4xlarge.experimental.nvidia.gpu_57120265687 2025-12-04T13:57:53.3998830Z ##[endgroup] 2025-12-04T13:57:53.4069956Z adding: usage_log.txt (deflated 58%) 2025-12-04T13:57:53.4134195Z adding: test/test-reports/inductor.test_torchinductor_1.2_e5d5d93df97dd444_.log (deflated 88%) 2025-12-04T13:57:53.4147324Z adding: test/test-reports/inductor.test_torchinductor_dynamic_shapes_1.4_f457e39532c32c3d_.log (deflated 91%) 2025-12-04T13:57:53.4148293Z adding: test/test-reports/inductor.test_kernel_benchmark_1.1_7a99c0fc76c1cb85_.log (deflated 76%) 2025-12-04T13:57:53.4157009Z adding: test/test-reports/inductor.test_torchinductor_opinfo_4.14_d6c80c60df76d1ae_.log (deflated 92%) 2025-12-04T13:57:53.4163968Z adding: test/test-reports/inductor.test_torchinductor_opinfo_11.14_4593ceecb4e38359_.log (deflated 91%) 2025-12-04T13:57:53.4173175Z adding: test/test-reports/dynamo.test_repros_1.1_e85852541ac7eea0_.log (deflated 85%) 2025-12-04T13:57:53.4305004Z adding: test/test-reports/inductor.test_cuda_select_algorithm_1.1_4871a8a3689f9e63_.log (deflated 97%) 2025-12-04T13:57:53.4352453Z adding: test/test-reports/inductor.test_compile_subprocess_1.2_abe6ea6a5cdf0a6c_.log (deflated 95%) 2025-12-04T13:57:53.4353206Z adding: test/test-reports/inductor.test_deterministic_2.3_06e251a75554d673_.log (deflated 73%) 2025-12-04T13:57:53.4354136Z adding: test/test-reports/dynamo.test_fx_graph_runnable_1.1_ccfc333edc67c9c4_.log (deflated 80%) 2025-12-04T13:57:53.4355299Z adding: test/test-reports/inductor.test_scatter_optimization_1.1_74b917c016dc1507_.log (deflated 78%) 2025-12-04T13:57:53.4357179Z adding: test/test-reports/inductor.test_padding_1.1_52cec87b51539027_.log (deflated 86%) 2025-12-04T13:57:53.4357697Z adding: test/test-reports/dynamo.test_callback_1.1_35bc12fe46f76cf6_.log (deflated 58%) 2025-12-04T13:57:53.4365916Z adding: test/test-reports/inductor.test_cudagraph_trees_expandable_segments_1.1_7984e53c82f556c8_.log (deflated 90%) 2025-12-04T13:57:53.4382931Z adding: test/test-reports/test_autograd_1.1_afae20be844dcfb4_.log (deflated 88%) 2025-12-04T13:57:53.4386006Z adding: test/test-reports/test_dataloader_2.2_c5a35c6a3d01e804_.log (deflated 86%) 2025-12-04T13:57:53.4398224Z adding: test/test-reports/test_decomp_5.17_bbc96279acf4332b_.log (deflated 89%) 2025-12-04T13:57:53.4410374Z adding: test/test-reports/test_decomp_12.17_25a251c7912fccac_.log (deflated 89%) 2025-12-04T13:57:53.4445867Z adding: test/test-reports/test_ops_fwd_gradients_2.2_14b4262c9eb464e0_.log (deflated 94%) 2025-12-04T13:57:53.4459098Z adding: test/test-reports/test_ops_jit_2.2_10156177962e62ad_.log (deflated 91%) 2025-12-04T13:57:53.4475239Z adding: test/test-reports/test_nestedtensor_3.3_f869b12f9fd94276_.log (deflated 92%) 2025-12-04T13:57:53.4558136Z adding: test/test-reports/test_ops_7.9_4196a3b6f281d326_.log (deflated 92%) 2025-12-04T13:57:53.4586013Z adding: test/test-reports/functorch.test_ops_2.8_bb188d7ff7219e71_.log (deflated 92%) 2025-12-04T13:57:53.4631903Z adding: test/test-reports/functorch.test_vmap_1.1_e45df0fc68d2af64_.log (deflated 93%) 2025-12-04T13:57:53.4632535Z adding: test/test-reports/inductor.test_smoke_1.1_beac475249fa3092_.log (stored 0%) 2025-12-04T13:57:53.4633115Z adding: test/test-reports/dynamo.test_after_aot_1.1_dfa0ff058fa49cd8_.log (deflated 54%) 2025-12-04T13:57:53.4633736Z adding: test/test-reports/inductor.test_snode_runtime_1.1_25b6d9448d0f8a56_.log (deflated 79%) 2025-12-04T13:57:53.4638918Z adding: test/test-reports/inductor.test_minifier_1.1_e91d133d4d8ff89c_.log (deflated 93%) 2025-12-04T13:57:53.4653833Z adding: test/test-reports/inductor.test_compiled_autograd_1.2_5de33680d1df73db_.log (deflated 90%) 2025-12-04T13:57:53.4662397Z adding: test/test-reports/inductor.test_torchinductor_strided_blocks_1.1_a1d3464a2f8c57ea_.log (deflated 95%) 2025-12-04T13:57:53.4663115Z adding: test/test-reports/inductor.test_pad_mm_1.1_ee372f763bdd660b_.log (deflated 79%) 2025-12-04T13:57:53.4663854Z adding: test/test-reports/inductor.test_triton_syntax_1.1_eefa874937932af8_.log (deflated 51%) 2025-12-04T13:57:53.4664550Z adding: test/test-reports/inductor.test_triton_extension_backend_1.1_e08968c1ce90c8be_.log (deflated 50%) 2025-12-04T13:57:53.4670404Z adding: test/test-reports/test_sparse_semi_structured_1.1_3df7ac6bb8c91876_.log (deflated 94%) 2025-12-04T13:57:53.4675209Z adding: test/test-reports/inductor.test_gpu_cpp_wrapper_2.2_8a9e88db96a4576a_.log (deflated 91%) 2025-12-04T13:57:53.4675885Z adding: test/test-reports/inductor.test_control_deps_1.1_3aef2ceb2e6eb739_.log (deflated 51%) 2025-12-04T13:57:53.4676676Z adding: test/test-reports/inductor.test_benchmarking_1.1_9e4c863097582737_.log (deflated 79%) 2025-12-04T13:57:53.4677389Z adding: test/test-reports/inductor.test_best_config_1.1_f3ac44d45d5a649c_.log (deflated 53%) 2025-12-04T13:57:53.4678267Z adding: test/test-reports/export.test_tools_1.1_b989024045a73d46_.log (deflated 63%) 2025-12-04T13:57:53.4685755Z adding: test/test-reports/inductor.test_compiled_optimizers_1.3_2096799fbf01ca07_.log (deflated 92%) 2025-12-04T13:57:53.5111545Z adding: test/test-reports/inductor.test_control_flow_5.5_8fba914687b14422_.log (deflated 96%) 2025-12-04T13:57:53.5112203Z adding: test/test-reports/inductor.test_pallas_1.1_29615ac8ce961259_.log (deflated 7%) 2025-12-04T13:57:53.5112808Z adding: test/test-reports/test_per_overload_api_1.1_fd510f1512712c78_.log (deflated 59%) 2025-12-04T13:57:53.5113719Z adding: test/test-reports/test_multiprocessing_1.1_889420b6a5edd168_.log (deflated 80%) 2025-12-04T13:57:53.5380939Z adding: test/test-reports/functorch.test_control_flow_2.3_c6fd620740f70764_.log (deflated 96%) 2025-12-04T13:57:53.5381586Z adding: test/test-reports/optim.test_swa_utils_1.1_2ca1b92ac8acd203_.log (deflated 7%) 2025-12-04T13:57:53.5382349Z adding: test/test-reports/test_tensorboard_1.1_041aa7283ea5c0c5_.log (deflated 83%) 2025-12-04T13:57:53.5384829Z adding: test/test-reports/torch_np.numpy_tests.core.test_indexing_1.1_e67fb534e4576208_.log (deflated 86%) 2025-12-04T13:57:53.5385925Z adding: test/test-reports/test_futures_1.1_c5e13566c134c7a8_.log (deflated 86%) 2025-12-04T13:57:53.5397628Z adding: test/test-reports/test_tensor_creation_ops_1.1_21234e58f391600d_.log (deflated 93%) 2025-12-04T13:57:53.5403411Z adding: test/test-reports/torch_np.numpy_tests.linalg.test_linalg_1.1_e7d4ead41bf88610_.log (deflated 92%) 2025-12-04T13:57:53.5405895Z adding: test/test-reports/torch_np.numpy_tests.core.test_dtype_1.1_d6bdce85d660528e_.log (deflated 91%) 2025-12-04T13:57:53.5406421Z adding: test/test-reports/lazy.test_debug_util_1.1_22a1fe9c50887d0a_.log (deflated 50%) 2025-12-04T13:57:53.5407320Z adding: test/test-reports/nn.test_load_state_dict_1.1_7f9d0868685e0061_.log (deflated 85%) 2025-12-04T13:57:53.5409456Z adding: test/test-reports/test_shape_ops_1.1_e2c81c501f7ea980_.log (deflated 87%) 2025-12-04T13:57:53.5410554Z adding: test/test-reports/profiler.test_memory_profiler_1.1_bd4cd8ef1d0be682_.log (deflated 82%) 2025-12-04T13:57:53.5414586Z adding: test/test-reports/test_indexing_1.1_ea804c0f86ef5646_.log (deflated 89%) 2025-12-04T13:57:53.5415350Z adding: test/test-reports/torch_np.numpy_tests.core.test_getlimits_1.1_13078b14d88df97c_.log (deflated 77%) 2025-12-04T13:57:53.5421969Z adding: test/test-reports/torch_np.test_ndarray_methods_1.1_570f98179517d221_.log (deflated 94%) 2025-12-04T13:57:53.5427309Z adding: test/test-reports/test_view_ops_1.1_843de7c7b78fe573_.log (deflated 91%) 2025-12-04T13:57:53.5427889Z adding: test/test-reports/test_type_info_1.1_6aa92fc777d561da_.log (deflated 61%) 2025-12-04T13:57:53.5442361Z adding: test/test-reports/functorch.test_aotdispatch_1.1_a56217decbf84b1c_.log (deflated 91%) 2025-12-04T13:57:53.5444316Z adding: test/test-reports/test_native_mha_1.1_19664d87c5964474_.log (deflated 93%) 2025-12-04T13:57:53.5445287Z adding: test/test-reports/torch_np.numpy_tests.core.test_numerictypes_1.1_cb1f018ca535f77c_.log (deflated 86%) 2025-12-04T13:57:53.5446095Z adding: test/test-reports/test_function_schema_1.1_9409a5d6f9c95b37_.log (deflated 77%) 2025-12-04T13:57:53.5447089Z adding: test/test-reports/nn.test_init_1.1_7a042e75797ecf7c_.log (deflated 78%) 2025-12-04T13:57:53.5448223Z adding: test/test-reports/torch_np.test_scalars_0D_arrays_1.1_a5e82800420456e1_.log (deflated 86%) 2025-12-04T13:57:53.5448978Z adding: test/test-reports/test_cuda_primary_ctx_1.1_989b62dfd14bfc8d_.log (deflated 85%) 2025-12-04T13:57:53.5449708Z adding: test/test-reports/profiler.test_profiler_tree_1.1_afa882d88da46829_.log (deflated 77%) 2025-12-04T13:57:53.5452844Z adding: test/test-reports/test_dlpack_1.1_2aa44fcbfbf54c48_.log (deflated 91%) 2025-12-04T13:57:53.5453408Z adding: test/test-reports/lazy.test_reuse_ir_1.1_6a37ac220f458a3d_.log (deflated 59%) 2025-12-04T13:57:53.5454647Z adding: test/test-reports/test_functional_autograd_benchmark_1.1_62fb1a6c4d9b68b7_.log (deflated 87%) 2025-12-04T13:57:53.5459703Z adding: test/test-reports/distributions.test_distributions_1.1_c4dba1c406e36ca0_.log (deflated 90%) 2025-12-04T13:57:53.5490190Z ##[group]Run # Remove any previous debugging artifacts if they exist 2025-12-04T13:57:53.5490578Z # Remove any previous debugging artifacts if they exist 2025-12-04T13:57:53.5490867Z rm -f debug-*.zip 2025-12-04T13:57:53.5491076Z if [ -d 'test/debug' ]; then 2025-12-04T13:57:53.5491337Z  zip -r "debug-${FILE_SUFFIX}.zip" test/debug 2025-12-04T13:57:53.5491578Z fi 2025-12-04T13:57:53.5499294Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-12-04T13:57:53.5499680Z env: 2025-12-04T13:57:53.5499845Z GIT_DEFAULT_BRANCH: main 2025-12-04T13:57:53.5500041Z HAS_NVIDIA_GPU: true 2025-12-04T13:57:53.5500278Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2025-12-04T13:57:53.5500674Z DOCKER_CONTAINER_ID: 657e31a8be774492437c4338a4b954650ffa0203d69f0088e2d19ede1e69df31 2025-12-04T13:57:53.5501233Z FILE_SUFFIX: test-default-6-7-linux.g6.4xlarge.experimental.nvidia.gpu_57120265687 2025-12-04T13:57:53.5501587Z ##[endgroup] 2025-12-04T13:57:53.5580912Z ##[group]Run seemethere/upload-artifact-s3@v5 2025-12-04T13:57:53.5581155Z with: 2025-12-04T13:57:53.5581313Z s3-bucket: gha-artifacts 2025-12-04T13:57:53.5581557Z s3-prefix: pytorch/pytorch/19922826259/1/artifact 2025-12-04T13:57:53.5581818Z retention-days: 14 2025-12-04T13:57:53.5581996Z if-no-files-found: warn 2025-12-04T13:57:53.5582193Z path: test-jsons-*.zip 2025-12-04T13:57:53.5582375Z name: artifact 2025-12-04T13:57:53.5582547Z region: us-east-1 2025-12-04T13:57:53.5582716Z env: 2025-12-04T13:57:53.5582872Z GIT_DEFAULT_BRANCH: main 2025-12-04T13:57:53.5583054Z HAS_NVIDIA_GPU: true 2025-12-04T13:57:53.5583279Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2025-12-04T13:57:53.5583718Z DOCKER_CONTAINER_ID: 657e31a8be774492437c4338a4b954650ffa0203d69f0088e2d19ede1e69df31 2025-12-04T13:57:53.5584069Z ##[endgroup] 2025-12-04T13:57:53.8644383Z NOTE: s3-prefix specified, ignoring name parameter 2025-12-04T13:57:53.8645374Z With the provided path, there will be 1 file uploaded 2025-12-04T13:57:53.8645864Z Uploading to s3 prefix: pytorch/pytorch/19922826259/1/artifact 2025-12-04T13:57:53.8715410Z Starting upload of test-jsons-test-default-6-7-linux.g6.4xlarge.experimental.nvidia.gpu_57120265687.zip 2025-12-04T13:57:54.0314405Z Finished upload of test-jsons-test-default-6-7-linux.g6.4xlarge.experimental.nvidia.gpu_57120265687.zip 2025-12-04T13:57:54.0547399Z ##[group]Run seemethere/upload-artifact-s3@v5 2025-12-04T13:57:54.0547749Z with: 2025-12-04T13:57:54.0547913Z s3-bucket: gha-artifacts 2025-12-04T13:57:54.0548156Z s3-prefix: pytorch/pytorch/19922826259/1/artifact 2025-12-04T13:57:54.0548408Z retention-days: 14 2025-12-04T13:57:54.0548586Z if-no-files-found: error 2025-12-04T13:57:54.0548779Z path: test-reports-*.zip 2025-12-04T13:57:54.0548967Z name: artifact 2025-12-04T13:57:54.0549128Z region: us-east-1 2025-12-04T13:57:54.0549308Z env: 2025-12-04T13:57:54.0549462Z GIT_DEFAULT_BRANCH: main 2025-12-04T13:57:54.0549659Z HAS_NVIDIA_GPU: true 2025-12-04T13:57:54.0549901Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2025-12-04T13:57:54.0550297Z DOCKER_CONTAINER_ID: 657e31a8be774492437c4338a4b954650ffa0203d69f0088e2d19ede1e69df31 2025-12-04T13:57:54.0550638Z ##[endgroup] 2025-12-04T13:57:54.3746236Z NOTE: s3-prefix specified, ignoring name parameter 2025-12-04T13:57:54.3746686Z With the provided path, there will be 1 file uploaded 2025-12-04T13:57:54.3747098Z Uploading to s3 prefix: pytorch/pytorch/19922826259/1/artifact 2025-12-04T13:57:54.3815271Z Starting upload of test-reports-test-default-6-7-linux.g6.4xlarge.experimental.nvidia.gpu_57120265687.zip 2025-12-04T13:57:54.5101024Z Finished upload of test-reports-test-default-6-7-linux.g6.4xlarge.experimental.nvidia.gpu_57120265687.zip 2025-12-04T13:57:54.5356841Z ##[group]Run seemethere/upload-artifact-s3@v5 2025-12-04T13:57:54.5357259Z with: 2025-12-04T13:57:54.5357535Z s3-bucket: gha-artifacts 2025-12-04T13:57:54.5357964Z s3-prefix: pytorch/pytorch/19922826259/1/artifact 2025-12-04T13:57:54.5358401Z retention-days: 14 2025-12-04T13:57:54.5358725Z if-no-files-found: ignore 2025-12-04T13:57:54.5359065Z path: logs-*.zip 2025-12-04T13:57:54.5359340Z name: artifact 2025-12-04T13:57:54.5359616Z region: us-east-1 2025-12-04T13:57:54.5359894Z env: 2025-12-04T13:57:54.5360156Z GIT_DEFAULT_BRANCH: main 2025-12-04T13:57:54.5360485Z HAS_NVIDIA_GPU: true 2025-12-04T13:57:54.5360876Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2025-12-04T13:57:54.5361562Z DOCKER_CONTAINER_ID: 657e31a8be774492437c4338a4b954650ffa0203d69f0088e2d19ede1e69df31 2025-12-04T13:57:54.5362331Z ##[endgroup] 2025-12-04T13:57:54.8278382Z NOTE: s3-prefix specified, ignoring name parameter 2025-12-04T13:57:54.8278849Z With the provided path, there will be 1 file uploaded 2025-12-04T13:57:54.8279275Z Uploading to s3 prefix: pytorch/pytorch/19922826259/1/artifact 2025-12-04T13:57:54.8346858Z Starting upload of logs-test-default-6-7-linux.g6.4xlarge.experimental.nvidia.gpu_57120265687.zip 2025-12-04T13:57:55.0094340Z Finished upload of logs-test-default-6-7-linux.g6.4xlarge.experimental.nvidia.gpu_57120265687.zip 2025-12-04T13:57:55.0327935Z ##[group]Run seemethere/upload-artifact-s3@v5 2025-12-04T13:57:55.0328183Z with: 2025-12-04T13:57:55.0328354Z s3-bucket: gha-artifacts 2025-12-04T13:57:55.0328594Z s3-prefix: pytorch/pytorch/19922826259/1/artifact 2025-12-04T13:57:55.0328853Z retention-days: 14 2025-12-04T13:57:55.0329036Z if-no-files-found: ignore 2025-12-04T13:57:55.0329247Z path: debug-*.zip 2025-12-04T13:57:55.0329433Z name: artifact 2025-12-04T13:57:55.0329597Z region: us-east-1 2025-12-04T13:57:55.0329764Z env: 2025-12-04T13:57:55.0329931Z GIT_DEFAULT_BRANCH: main 2025-12-04T13:57:55.0330125Z HAS_NVIDIA_GPU: true 2025-12-04T13:57:55.0330358Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2025-12-04T13:57:55.0330750Z DOCKER_CONTAINER_ID: 657e31a8be774492437c4338a4b954650ffa0203d69f0088e2d19ede1e69df31 2025-12-04T13:57:55.0331090Z ##[endgroup] 2025-12-04T13:57:55.3197497Z No files were found with the provided path: debug-*.zip. No artifacts will be uploaded. 2025-12-04T13:57:55.3438520Z ##[group]Run # shellcheck disable=SC2156 2025-12-04T13:57:55.3438832Z # shellcheck disable=SC2156 2025-12-04T13:57:55.3439268Z find . -iname "core.[1-9]*" -exec docker exec "${DOCKER_CONTAINER_ID}" sh -c "gdb python {} -ex 'bt' -ex 'q'" \; 2025-12-04T13:57:55.3447335Z shell: /usr/bin/bash -e {0} 2025-12-04T13:57:55.3447645Z env: 2025-12-04T13:57:55.3447808Z GIT_DEFAULT_BRANCH: main 2025-12-04T13:57:55.3448013Z HAS_NVIDIA_GPU: true 2025-12-04T13:57:55.3448254Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2025-12-04T13:57:55.3448663Z DOCKER_CONTAINER_ID: 657e31a8be774492437c4338a4b954650ffa0203d69f0088e2d19ede1e69df31 2025-12-04T13:57:55.3449013Z ##[endgroup] 2025-12-04T13:57:55.7428935Z ##[group]Run seemethere/upload-artifact-s3@baba72d0712b404f646cebe0730933554ebce96a 2025-12-04T13:57:55.7429322Z with: 2025-12-04T13:57:55.7429608Z name: coredumps-default-6-7-linux.g6.4xlarge.experimental.nvidia.gpu 2025-12-04T13:57:55.7429949Z retention-days: 14 2025-12-04T13:57:55.7430143Z if-no-files-found: ignore 2025-12-04T13:57:55.7430341Z path: ./**/core.[1-9]* 2025-12-04T13:57:55.7430539Z s3-bucket: gha-artifacts 2025-12-04T13:57:55.7430744Z region: us-east-1 2025-12-04T13:57:55.7430910Z env: 2025-12-04T13:57:55.7431065Z GIT_DEFAULT_BRANCH: main 2025-12-04T13:57:55.7431264Z HAS_NVIDIA_GPU: true 2025-12-04T13:57:55.7431506Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2025-12-04T13:57:55.7431907Z DOCKER_CONTAINER_ID: 657e31a8be774492437c4338a4b954650ffa0203d69f0088e2d19ede1e69df31 2025-12-04T13:57:55.7432252Z ##[endgroup] 2025-12-04T13:58:06.3350192Z No files were found with the provided path: ./**/core.[1-9]*. No artifacts will be uploaded. 2025-12-04T13:58:06.3699552Z Prepare all required actions 2025-12-04T13:58:06.3699892Z Getting action download info 2025-12-04T13:58:06.5171291Z Download action repository 'actions/setup-python@v6' (SHA:83679a892e2d95755f2dac6acb0bfd1e9ac5d548) 2025-12-04T13:58:06.8241219Z ##[group]Run ./.github/actions/upload-utilization-stats 2025-12-04T13:58:06.8241490Z with: 2025-12-04T13:58:06.8241654Z job_id: 57120265687 2025-12-04T13:58:06.8242150Z job_name: linux-jammy-cuda12.8-py3.10-gcc11-debug / test (default, 6, 7, linux.g6.4xlarge.experimental.nvidia.gpu, oncall:debug-build, mem_leak_check) 2025-12-04T13:58:06.8242672Z workflow_name: periodic 2025-12-04T13:58:06.8242871Z workflow_run_id: 19922826259 2025-12-04T13:58:06.8243197Z workflow_attempt: 1 2025-12-04T13:58:06.8243369Z env: 2025-12-04T13:58:06.8243523Z GIT_DEFAULT_BRANCH: main 2025-12-04T13:58:06.8243712Z HAS_NVIDIA_GPU: true 2025-12-04T13:58:06.8243936Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2025-12-04T13:58:06.8244360Z DOCKER_CONTAINER_ID: 657e31a8be774492437c4338a4b954650ffa0203d69f0088e2d19ede1e69df31 2025-12-04T13:58:06.8244708Z ##[endgroup] 2025-12-04T13:58:06.8289120Z ##[group]Run actions/setup-python@v6 2025-12-04T13:58:06.8289331Z with: 2025-12-04T13:58:06.8289501Z python-version: 3.10 2025-12-04T13:58:06.8289691Z check-latest: false 2025-12-04T13:58:06.8289960Z token: *** 2025-12-04T13:58:06.8290133Z update-environment: true 2025-12-04T13:58:06.8290339Z allow-prereleases: false 2025-12-04T13:58:06.8290538Z freethreaded: false 2025-12-04T13:58:06.8290708Z env: 2025-12-04T13:58:06.8290858Z GIT_DEFAULT_BRANCH: main 2025-12-04T13:58:06.8291051Z HAS_NVIDIA_GPU: true 2025-12-04T13:58:06.8291270Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2025-12-04T13:58:06.8291663Z DOCKER_CONTAINER_ID: 657e31a8be774492437c4338a4b954650ffa0203d69f0088e2d19ede1e69df31 2025-12-04T13:58:06.8292013Z ##[endgroup] 2025-12-04T13:58:07.2908206Z ##[group]Installed versions 2025-12-04T13:58:07.2916418Z Version 3.10 was not found in the local cache 2025-12-04T13:58:07.3124194Z (node:267888) [DEP0040] DeprecationWarning: The `punycode` module is deprecated. Please use a userland alternative instead. 2025-12-04T13:58:07.3125612Z (Use `node --trace-deprecation ...` to show where the warning was created) 2025-12-04T13:58:07.6565390Z ##[error]The version '3.10' with architecture 'x64' was not found for this operating system. The list of all available versions can be found here: https://raw.githubusercontent.com/actions/python-versions/main/versions-manifest.json 2025-12-04T13:58:07.6762555Z ##[group]Run pytorch/test-infra/.github/actions/teardown-linux@main 2025-12-04T13:58:07.6763020Z with: 2025-12-04T13:58:07.6763166Z env: 2025-12-04T13:58:07.6763331Z GIT_DEFAULT_BRANCH: main 2025-12-04T13:58:07.6763533Z HAS_NVIDIA_GPU: true 2025-12-04T13:58:07.6763767Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2025-12-04T13:58:07.6764173Z DOCKER_CONTAINER_ID: 657e31a8be774492437c4338a4b954650ffa0203d69f0088e2d19ede1e69df31 2025-12-04T13:58:07.6764542Z ##[endgroup] 2025-12-04T13:58:07.6778626Z ##[group]Run set -eou pipefail 2025-12-04T13:58:07.6778865Z set -eou pipefail 2025-12-04T13:58:07.6779064Z  2025-12-04T13:58:07.6779337Z echo "Holding runner for 2 hours until all ssh sessions have logged out" 2025-12-04T13:58:07.6779669Z for _ in $(seq 1440); do 2025-12-04T13:58:07.6779911Z  # Break if no ssh session exists anymore 2025-12-04T13:58:07.6780160Z  if [ "$(who)" = "" ]; then 2025-12-04T13:58:07.6780394Z  break 2025-12-04T13:58:07.6780563Z  fi 2025-12-04T13:58:07.6780724Z  echo "." 2025-12-04T13:58:07.6780907Z  sleep 5 2025-12-04T13:58:07.6781072Z done 2025-12-04T13:58:07.6789959Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-12-04T13:58:07.6790238Z env: 2025-12-04T13:58:07.6790397Z GIT_DEFAULT_BRANCH: main 2025-12-04T13:58:07.6790600Z HAS_NVIDIA_GPU: true 2025-12-04T13:58:07.6790831Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2025-12-04T13:58:07.6791212Z DOCKER_CONTAINER_ID: 657e31a8be774492437c4338a4b954650ffa0203d69f0088e2d19ede1e69df31 2025-12-04T13:58:07.6791567Z ##[endgroup] 2025-12-04T13:58:07.6819446Z Holding runner for 2 hours until all ssh sessions have logged out 2025-12-04T13:58:07.6905869Z ##[group]Run # ignore expansion of "docker ps -q" since it could be empty 2025-12-04T13:58:07.6906268Z # ignore expansion of "docker ps -q" since it could be empty 2025-12-04T13:58:07.6906579Z # shellcheck disable=SC2046 2025-12-04T13:58:07.6906826Z docker stop $(docker ps -q) || true 2025-12-04T13:58:07.6907077Z # Prune all of the docker images 2025-12-04T13:58:07.6907397Z docker system prune -af 2025-12-04T13:58:07.6914261Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-12-04T13:58:07.6914536Z env: 2025-12-04T13:58:07.6914691Z GIT_DEFAULT_BRANCH: main 2025-12-04T13:58:07.6914889Z HAS_NVIDIA_GPU: true 2025-12-04T13:58:07.6915120Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2025-12-04T13:58:07.6915524Z DOCKER_CONTAINER_ID: 657e31a8be774492437c4338a4b954650ffa0203d69f0088e2d19ede1e69df31 2025-12-04T13:58:07.6915878Z ##[endgroup] 2025-12-04T13:58:18.6341763Z 657e31a8be77 2025-12-04T13:58:23.1323804Z Deleted Containers: 2025-12-04T13:58:23.1324378Z 657e31a8be774492437c4338a4b954650ffa0203d69f0088e2d19ede1e69df31 2025-12-04T13:58:23.1324815Z 2025-12-04T13:58:34.6746416Z Deleted Images: 2025-12-04T13:58:34.6747123Z untagged: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/ci-image:pytorch-linux-jammy-cuda12.8-cudnn9-py3-gcc11-f0cd68561080d537ef3d3d6f81b25a6416ad600a 2025-12-04T13:58:34.6748100Z untagged: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/ci-image@sha256:ba21003510dba4bdeed83df81a56fa468e0ee1b612a9445ae1f402a280804f97 2025-12-04T13:58:34.6748893Z deleted: sha256:add7313791033822205cdb3cf32096534b2cfaa4855bd48119b59000bfe00301 2025-12-04T13:58:34.6749508Z deleted: sha256:85a76b7bf29ad34eb76cce6f46af5d49a58b6272f80f983d5c769e82c7749301 2025-12-04T13:58:34.6749981Z deleted: sha256:0882f3ce59ff5ae30195ee4b059fc713e13eda107a3a7814a4616ac9058a30a4 2025-12-04T13:58:34.6750435Z deleted: sha256:64ba5b9344c11a3e4729136076830b90ac4cf1554046edb1bd4f0784b66ebd9b 2025-12-04T13:58:34.6750874Z deleted: sha256:88213c59cf461a65ab9b6cb07b4195dc9d41b5241c152daa002c7b3112e09124 2025-12-04T13:58:34.6751317Z deleted: sha256:4c0f83afa802ffbc05ebaf1aa50e48a2447c7c295549a6dded80ac63437906ca 2025-12-04T13:58:34.6752047Z deleted: sha256:6f7ec74460e8fb070c8209949095ea3be5f4e2fd69c9f750cd39ac4093f5e64b 2025-12-04T13:58:34.6752616Z deleted: sha256:d6928b0d1021b31942fdcb64e5eb4a34682de66e959dd424ed6ed02c29cd706d 2025-12-04T13:58:34.6753068Z deleted: sha256:4e9fbcb1705a6351bb34dd320558752614308636b94fd9ae6f26063e3deadc0a 2025-12-04T13:58:34.6753498Z deleted: sha256:43aabd0201f48712f21758071352dea029b4de37be08b2e2197706856a9ecbf2 2025-12-04T13:58:34.6753925Z deleted: sha256:940a98dec78303f0548beb1033242a45e9097607ef3e55c8b949b69b73d1b95e 2025-12-04T13:58:34.6754356Z deleted: sha256:d2849fa0e0411cf66e4408831d70e38838afb55b11a80c1c4d8aa0ae7dc9ca40 2025-12-04T13:58:34.6754786Z deleted: sha256:14f40d23c20c7e562623f89deb376520296758bc39dd3c77284049b84ebd8a31 2025-12-04T13:58:34.6755236Z deleted: sha256:a8ccba61f90ca097cb391d0f4fbed0d9f821d06b00e28f7332e9e2dcfcbac4ca 2025-12-04T13:58:34.6755681Z deleted: sha256:91b2060d290547d3b517d4a11d994bbe23f4560b5546cb91918ca1828dde6be1 2025-12-04T13:58:34.6756114Z deleted: sha256:b42a184755715dcfead7fad655a127433541d316d9628f5f730ff17ad5f8071c 2025-12-04T13:58:34.6756562Z deleted: sha256:aa5b4f3c9169061dc3c6da0e677e8a86f11ecb0a3f9fb4861ab3d8c04379775c 2025-12-04T13:58:34.6757006Z deleted: sha256:b4dcf450081a48d77fea0a21b8d810a69c03608a595e754fe7d365058d0579b7 2025-12-04T13:58:34.6757446Z deleted: sha256:4f7fe12d3d4f5bf890c7ada4ce16f17a105472aa6509a778f917dcce2f28174b 2025-12-04T13:58:34.6757894Z deleted: sha256:2d1d5a74182594f9a8553df00fdcfc809dba407bcd6700d667f862cbe9d555ce 2025-12-04T13:58:34.6758346Z deleted: sha256:d901e2f5d449aeed16b727bdcc11fc0e0f6c30c8fc5c39ac7eeac8a74d9d176c 2025-12-04T13:58:34.6758787Z deleted: sha256:a04df2603bd12372c6632469a9a81ebc4a8d677452c250672b9692884fa6a452 2025-12-04T13:58:34.6759214Z deleted: sha256:f438a6b52273a552dc3820a55c74c53a62a0eae9f2a7d21b37125add7d71639f 2025-12-04T13:58:34.6759646Z deleted: sha256:d4b09517e9518d709ac98b0ae6f8446ec9ac51688253607b1fca67aa2c87b3f4 2025-12-04T13:58:34.6760079Z deleted: sha256:c1fa38335237f5e7263e39d3d3de98215bcfbbb12b826955c02e149bf68efd13 2025-12-04T13:58:34.6760508Z deleted: sha256:c898d20a30de901fca74d7611663b17ab48e1726a11e031e40548ed16ee81877 2025-12-04T13:58:34.6760943Z deleted: sha256:3baceec7096518fcc10696feba551639d698b3145c2fc09cac927bb60c0fd751 2025-12-04T13:58:34.6761483Z deleted: sha256:5245aaaa3d5c3a19f76b9a6c920bd82d1a0ff5289f87c8c109652089709d9b3b 2025-12-04T13:58:34.6761915Z deleted: sha256:f05cc789b95246938c377f474c41187965b89ceac0250e7d5124bec32153f447 2025-12-04T13:58:34.6762355Z deleted: sha256:07ec4fc008de4e7a2c794ec7094cc72e0d287c04c8b2156163aee0bae147fe2d 2025-12-04T13:58:34.6762793Z deleted: sha256:c6302601ad5fde573c1f8c900250478fca7fdc6907d8fd4fae651b94b4d9264d 2025-12-04T13:58:34.6763225Z deleted: sha256:cc5e955ee1dc54931f02606c5ea87aae14f03b5d764492be611480ab041f2882 2025-12-04T13:58:34.6763656Z deleted: sha256:f21c03518996d98452338f4e80bcfd9b139a1dab155f4830be0d3f623035269f 2025-12-04T13:58:34.6764102Z deleted: sha256:519ca6f1279f7886f25f0005527cfa627deebbc5b7d7cdbfa7ef962bcfc4c26d 2025-12-04T13:58:34.6764560Z deleted: sha256:0ef990495216807d0175b192045be3f617e72331bc373b3434807f41bf69168d 2025-12-04T13:58:34.6764990Z deleted: sha256:7093edf7319e1f0e01654c3224e32c8dede5b948d106e0b9b03cbf0bb1091e33 2025-12-04T13:58:34.6765415Z deleted: sha256:c478161e058e2f4041555c3e880b95ee1ee047938dc58549a3a88135740996ae 2025-12-04T13:58:34.6765844Z deleted: sha256:9bb853b0d938cd7c36a80ce8ee40653f2c0ff92719209b11beb03acc8855ce3e 2025-12-04T13:58:34.6766284Z deleted: sha256:fdf2ace71a78ce6910ef9c4b073c195531da47022443b606bb92dcd6499b6afc 2025-12-04T13:58:34.6766716Z deleted: sha256:576c2b3770d871937d3cfb7014328bcb4bd1aed0c28bc438764b3bfdac4c1ac2 2025-12-04T13:58:34.6767151Z deleted: sha256:878e92b9cb82de09ac14a9d5f3f7bc2411a799b6f54d0d64b78c2bb4d1fdc0fc 2025-12-04T13:58:34.6767595Z deleted: sha256:85c8c3b98b65a6695f988a10cc66c981d73a3ef03eda15b8e14d227b50b56300 2025-12-04T13:58:34.6768045Z deleted: sha256:ce2ab3ba07794f9ee95d6ea7de6dcd3d2aed96561f9a79192dd56ca5bf29313a 2025-12-04T13:58:34.6768595Z deleted: sha256:37a6e12976ca957286977e696e63012ab9821214b0483fe1a48d29dcb280508a 2025-12-04T13:58:34.6769029Z deleted: sha256:cd1d5d3dd7038144ca6fe961c0d4c8e705625ae0c36190ba8b3e9602abedad19 2025-12-04T13:58:34.6769507Z deleted: sha256:0e707276e0be2e0008b86d594fadc0d16444d66c4fb7227c56f144cbb3c2affd 2025-12-04T13:58:34.6769936Z deleted: sha256:22d4aad6a2ada91b341c1225a0f314042b8aeabef7568c5c019709b058bf070b 2025-12-04T13:58:34.6770378Z deleted: sha256:ee4adacf4e0933131d0275eddad406b3c8147e6cf07a292b99f1aff4b5355f33 2025-12-04T13:58:34.6770918Z deleted: sha256:43da0b9e7c0e18403dcb834e53628dc7c970ccb2dbd091878c0d7c0170dbc97f 2025-12-04T13:58:34.6771385Z deleted: sha256:00571684bdcd75beda15eb7d4e79b5458bc914350f9bb4d87fcdc97ad15e0da1 2025-12-04T13:58:34.6771810Z deleted: sha256:41615f09950259f1d75e82ef35b6fc53b18fe71ebff143744cfd51009d04349e 2025-12-04T13:58:34.6772241Z deleted: sha256:75ab34d2eed3c7915467a506ab6dab2711918fbabe94add2fb5c62780221ab0c 2025-12-04T13:58:34.6772688Z deleted: sha256:0a39ef2bebf44c1c3893d1e5fb42dad48b8fac7ca673141267ee967f85455e89 2025-12-04T13:58:34.6773122Z deleted: sha256:9b7d024e48ba1f9824a54597621b1b062cbc4aa41a77d81ca538d6b5c24a612c 2025-12-04T13:58:34.6773552Z deleted: sha256:392257172de6434c271bd93394218a91e9aa86d7c18abc2f2759317b9d5fb6de 2025-12-04T13:58:34.6773969Z deleted: sha256:6c3232860b930866a463a356124fc392c7e5f04895695229257e8c3e8a02711d 2025-12-04T13:58:34.6774400Z deleted: sha256:63dd55b807215e2fa6c715419ac0c5072d02dddc848dbf74bb7e77b906b5eaed 2025-12-04T13:58:34.6774833Z deleted: sha256:07a8738c1b4584db72ed9aa60f5274321eb0ba16263450da3a75df8326ebc25f 2025-12-04T13:58:34.6775259Z deleted: sha256:053fe2965b01281d12040ec1893e0d1aa77362a49ea9a1067402272c69dad9f5 2025-12-04T13:58:34.6775688Z deleted: sha256:7857fb5eb181c4e80262ecab60bdd3c266cf3d1409ceb76c05882609b416a8d3 2025-12-04T13:58:34.6776124Z deleted: sha256:752528477fc99089de3bd2c6da7b30cf34f2e901fe06d8fcfe685b411461e883 2025-12-04T13:58:34.6776552Z deleted: sha256:cce0210e2f4b042601813df03aa294a86b0c668fcfc75f4c63f6fa12b2952e15 2025-12-04T13:58:34.6776992Z deleted: sha256:f2bb405a26705ecd12d21380d26d9355d01db3a2175080fbdb468f2b5a25a76c 2025-12-04T13:58:34.6777447Z deleted: sha256:ad430120d4ffbaf97cd8d6de6ea8eefa4a8f80ec45f0b176c6b26bff0970fd33 2025-12-04T13:58:34.6778209Z deleted: sha256:225a4910baea7cc540ed43eeac75046293800ab0b8e0192b51e991c8cb50bcf3 2025-12-04T13:58:34.6778662Z deleted: sha256:a259945b0c3507f049fbac10fb3d3ffe43d45e83c91b80ae8cd1dafb855ad83c 2025-12-04T13:58:34.6779105Z deleted: sha256:862a98881b1d5adad5c21d01602773b894794097de80964ef8f47bcaadb43255 2025-12-04T13:58:34.6779529Z deleted: sha256:1cf6d3c8b6c2694b79a2d08719594903811c330a36a4c7a8a7153a350b53d292 2025-12-04T13:58:34.6779954Z deleted: sha256:232a1ae8b0fee817ff7838bb5986a2f38377d3b1dbbf5217b576af0f953b0844 2025-12-04T13:58:34.6780394Z deleted: sha256:c72c5705dabd6314423dd7d4fb260a20d5d9886b2ebce60d19e9d78c4a2335c2 2025-12-04T13:58:34.6780828Z deleted: sha256:296734cf81fd92c913884d058908598424ffe072676e38de289bbab83768c7bd 2025-12-04T13:58:34.6781248Z deleted: sha256:7c76040481b889847a1804021aeff07547eaa4ee706d6137db218d497a8fd9c1 2025-12-04T13:58:34.6781686Z deleted: sha256:d5e293f5b354e8cbcc6de893ea72cc632b02d8fdfbb08ec3127c4e9662f3ebff 2025-12-04T13:58:34.6782125Z deleted: sha256:f35a64e429c88e249645090f21fbe7dae108d98e0ab4ea13184f24b3fd66c315 2025-12-04T13:58:34.6782557Z deleted: sha256:ce6ae8d595c8e69115c51b1ce4f9a9158795d7b863b1cb53f21c39a87974d41b 2025-12-04T13:58:34.6782987Z deleted: sha256:8941abaee59400fb9b3a60765fea4a1fc2a6a447467a6d983e84c7f72494a450 2025-12-04T13:58:34.6783423Z deleted: sha256:ef53c29a9a2c2bc80ffdb9bfaf92842436b5755ec1ce828b9d11e5e27d656ea1 2025-12-04T13:58:34.6783989Z deleted: sha256:7a347fb0acb43f1c814f8c8ff21185e8b5cf64d7bc5988cea060f77d906e08b5 2025-12-04T13:58:34.6784427Z deleted: sha256:cc855dc9be79496e15175569dced2d13477e50b077a5fd3945f9bf50018880c1 2025-12-04T13:58:34.6784850Z deleted: sha256:f7a9946ada3d4786658bc0b643808bb32a9a45e4e90e30dc43ee19e2dbe24024 2025-12-04T13:58:34.6786153Z deleted: sha256:c22a9215f62812c1d2e32827f5221ff556c5b6702aadbdab6b87b8293f19635e 2025-12-04T13:58:34.6786608Z deleted: sha256:959a56746620012e37c1def1a83c5afb1e7c0adc59b021a28beb53c24df98032 2025-12-04T13:58:34.6787107Z deleted: sha256:31a0fff0695bf6100c17954be72eab2095b466d559c75c3faf2a17d8c41e6ebe 2025-12-04T13:58:34.6787539Z deleted: sha256:c15e2b5241b9e55af1b2593e544391b4b44d0505e6528e8f12425136e93b424c 2025-12-04T13:58:34.6787962Z deleted: sha256:73974f74b436f39a2fdb6461b1e3f7c3e41c73325776fa71d16b942a5b4a365b 2025-12-04T13:58:34.6788329Z untagged: public.ecr.aws/docker/library/python:3.13 2025-12-04T13:58:34.6788820Z untagged: public.ecr.aws/docker/library/python@sha256:3f986299a7b8b44b0d8cf9bda2b22361ce5c3058ef5d7cb17fb7452506680ab0 2025-12-04T13:58:34.6789392Z deleted: sha256:44438aecfedf7b6086fce506dae0db5ba7fc0027f9b743f1a75a6b5cbc7de70a 2025-12-04T13:58:34.6789837Z deleted: sha256:6f09a1f5d8a107c2532fbd116e75116cb75fa77b1a7d72d3bdf1ac12de152acd 2025-12-04T13:58:34.6790276Z deleted: sha256:fe5f3ac0be086125eb1e3cd10cc33e8e426f4e079381f7ce5a987b626e99fa67 2025-12-04T13:58:34.6790817Z deleted: sha256:79dd2061a22cf919cfc4f1f02704bfda09afadb017265e670ee54441d296c06c 2025-12-04T13:58:34.6791611Z deleted: sha256:9447ad402aafdbee17e999b0ec84ad89c2646dbebf054d469d4f8bee77f66212 2025-12-04T13:58:34.6792375Z deleted: sha256:7a4909f3c1975be52292f53107495ee1b41c17494918767ccedf1cf1688ae318 2025-12-04T13:58:34.6793106Z deleted: sha256:3474923d97f1f498237650a7d51bd4aea37d5e6b9d8a778777920584af5dd560 2025-12-04T13:58:34.6793568Z deleted: sha256:683afd1773444401a9cbd24842ee5d9154a11abb4fab63ddea5c03df788597ee 2025-12-04T13:58:34.6793828Z 2025-12-04T13:58:34.6793921Z Total reclaimed space: 38.01GB 2025-12-04T13:58:34.6824075Z ##[group]Run set +e 2025-12-04T13:58:34.6824366Z set +e 2025-12-04T13:58:34.6824536Z set -x 2025-12-04T13:58:34.6824700Z  2025-12-04T13:58:34.6824855Z nvidia-smi 2025-12-04T13:58:34.6825185Z # NB: Surprisingly, nvidia-smi command returns successfully with return code 0 even in 2025-12-04T13:58:34.6825690Z # the case where the driver has already crashed as it still can get the driver version 2025-12-04T13:58:34.6826170Z # and some basic information like the bus ID. However, the rest of the information 2025-12-04T13:58:34.6826652Z # would be missing (ERR!), for example: 2025-12-04T13:58:34.6826884Z # 2025-12-04T13:58:34.6827115Z # +-----------------------------------------------------------------------------+ 2025-12-04T13:58:34.6827522Z # | NVIDIA-SMI 525.89.02 Driver Version: 525.89.02 CUDA Version: 12.0 | 2025-12-04T13:58:34.6827918Z # |-------------------------------+----------------------+----------------------+ 2025-12-04T13:58:34.6828301Z # | GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC | 2025-12-04T13:58:34.6828707Z # | Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. | 2025-12-04T13:58:34.6829040Z # | | | MIG M. | 2025-12-04T13:58:34.6829286Z # |===============================+======================+======================| 2025-12-04T13:58:34.6829577Z # | 0 ERR! Off | 00000000:00:1E.0 Off | ERR! | 2025-12-04T13:58:34.6829907Z # |ERR! ERR! ERR! ERR! / ERR! | 4184MiB / 23028MiB | ERR! Default | 2025-12-04T13:58:34.6830208Z # | | | ERR! | 2025-12-04T13:58:34.6830492Z # +-------------------------------+----------------------+----------------------+ 2025-12-04T13:58:34.6830756Z # 2025-12-04T13:58:34.6830972Z # +-----------------------------------------------------------------------------+ 2025-12-04T13:58:34.6831290Z # | Processes: | 2025-12-04T13:58:34.6831617Z # | GPU GI CI PID Type Process name GPU Memory | 2025-12-04T13:58:34.6831921Z # | ID ID Usage | 2025-12-04T13:58:34.6832229Z # |=============================================================================| 2025-12-04T13:58:34.6832517Z # +-----------------------------------------------------------------------------+ 2025-12-04T13:58:34.6832773Z # 2025-12-04T13:58:34.6833050Z # This should be reported as a failure instead as it will guarantee to fail when 2025-12-04T13:58:34.6833411Z # Docker tries to run with --gpus all 2025-12-04T13:58:34.6833642Z # 2025-12-04T13:58:34.6833897Z # So, the correct check here is to query one of the missing piece of info like 2025-12-04T13:58:34.6834266Z # GPU name, so that the command can fail accordingly 2025-12-04T13:58:34.6834612Z nvidia-smi --query-gpu=gpu_name --format=csv,noheader --id=0 2025-12-04T13:58:34.6834910Z NVIDIA_SMI_STATUS=$? 2025-12-04T13:58:34.6835102Z  2025-12-04T13:58:34.6835415Z # These are acceptable return code from nvidia-smi as copied from setup-nvidia GitHub action 2025-12-04T13:58:34.6835880Z if [ "$NVIDIA_SMI_STATUS" -ne 0 ] && [ "$NVIDIA_SMI_STATUS" -ne 14 ]; then 2025-12-04T13:58:34.6836284Z  echo "NVIDIA driver installation has failed, shutting down the runner..." 2025-12-04T13:58:34.6836640Z  .github/scripts/stop_runner_service.sh 2025-12-04T13:58:34.6836883Z fi 2025-12-04T13:58:34.6837030Z  2025-12-04T13:58:34.6837423Z # For runner with multiple GPUs, we also want to confirm that the number of GPUs are the 2025-12-04T13:58:34.6837870Z # power of 2, i.e. 1, 2, 4, or 8. This is to avoid flaky test issue when one GPU fails 2025-12-04T13:58:34.6838247Z # https://github.com/pytorch/test-infra/issues/4000 2025-12-04T13:58:34.6838555Z GPU_COUNT=$(nvidia-smi --list-gpus | wc -l) 2025-12-04T13:58:34.6838810Z NVIDIA_SMI_STATUS=$? 2025-12-04T13:58:34.6839004Z  2025-12-04T13:58:34.6839339Z # These are acceptable return code from nvidia-smi as copied from setup-nvidia GitHub action 2025-12-04T13:58:34.6839794Z if [ "$NVIDIA_SMI_STATUS" -ne 0 ] && [ "$NVIDIA_SMI_STATUS" -ne 14 ]; then 2025-12-04T13:58:34.6840196Z  echo "NVIDIA driver installation has failed, shutting down the runner..." 2025-12-04T13:58:34.6840560Z  .github/scripts/stop_runner_service.sh 2025-12-04T13:58:34.6840796Z fi 2025-12-04T13:58:34.6840943Z  2025-12-04T13:58:34.6841117Z # Check the GPU count to be a power of 2 2025-12-04T13:58:34.6841508Z if [ "$GPU_COUNT" -le 8 ] && [ "$GPU_COUNT" -ne 1 ] && [ "$GPU_COUNT" -ne 2 ] && [ "$GPU_COUNT" -ne 4 ] && [ "$GPU_COUNT" -ne 8 ]; then 2025-12-04T13:58:34.6842052Z  echo "NVIDIA driver detects $GPU_COUNT GPUs. The runner has a broken GPU, shutting it down..." 2025-12-04T13:58:34.6842445Z  .github/scripts/stop_runner_service.sh 2025-12-04T13:58:34.6842678Z fi 2025-12-04T13:58:34.6854062Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-12-04T13:58:34.6854352Z env: 2025-12-04T13:58:34.6854514Z GIT_DEFAULT_BRANCH: main 2025-12-04T13:58:34.6854722Z HAS_NVIDIA_GPU: true 2025-12-04T13:58:34.6854959Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2025-12-04T13:58:34.6855356Z DOCKER_CONTAINER_ID: 657e31a8be774492437c4338a4b954650ffa0203d69f0088e2d19ede1e69df31 2025-12-04T13:58:34.6866640Z ##[endgroup] 2025-12-04T13:58:34.6897408Z + nvidia-smi 2025-12-04T13:58:34.7082039Z Thu Dec 4 13:58:34 2025 2025-12-04T13:58:34.7082608Z +-----------------------------------------------------------------------------------------+ 2025-12-04T13:58:34.7083368Z | NVIDIA-SMI 580.82.07 Driver Version: 580.82.07 CUDA Version: 13.0 | 2025-12-04T13:58:34.7084079Z +-----------------------------------------+------------------------+----------------------+ 2025-12-04T13:58:34.7085058Z | GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC | 2025-12-04T13:58:34.7085831Z | Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. | 2025-12-04T13:58:34.7086420Z | | | MIG M. | 2025-12-04T13:58:34.7086882Z |=========================================+========================+======================| 2025-12-04T13:58:34.7218937Z | 0 NVIDIA L4 On | 00000000:35:00.0 Off | 0 | 2025-12-04T13:58:34.7219401Z | N/A 34C P8 16W / 72W | 0MiB / 23034MiB | 0% Default | 2025-12-04T13:58:34.7219802Z | | | N/A | 2025-12-04T13:58:34.7220203Z +-----------------------------------------+------------------------+----------------------+ 2025-12-04T13:58:34.7222629Z 2025-12-04T13:58:34.7222850Z +-----------------------------------------------------------------------------------------+ 2025-12-04T13:58:34.7223280Z | Processes: | 2025-12-04T13:58:34.7223782Z | GPU GI CI PID Type Process name GPU Memory | 2025-12-04T13:58:34.7224162Z | ID ID Usage | 2025-12-04T13:58:34.7224714Z |=========================================================================================| 2025-12-04T13:58:34.7227927Z | No running processes found | 2025-12-04T13:58:34.7228401Z +-----------------------------------------------------------------------------------------+ 2025-12-04T13:58:34.9490494Z + nvidia-smi --query-gpu=gpu_name --format=csv,noheader --id=0 2025-12-04T13:58:34.9646303Z NVIDIA L4 2025-12-04T13:58:34.9680254Z + NVIDIA_SMI_STATUS=0 2025-12-04T13:58:34.9680508Z + '[' 0 -ne 0 ']' 2025-12-04T13:58:34.9686630Z ++ nvidia-smi --list-gpus 2025-12-04T13:58:34.9687928Z ++ wc -l 2025-12-04T13:58:34.9875752Z + GPU_COUNT=1 2025-12-04T13:58:34.9876012Z + NVIDIA_SMI_STATUS=0 2025-12-04T13:58:34.9876248Z + '[' 0 -ne 0 ']' 2025-12-04T13:58:34.9876458Z + '[' 1 -le 8 ']' 2025-12-04T13:58:34.9876657Z + '[' 1 -ne 1 ']' 2025-12-04T13:58:34.9932796Z Post job cleanup. 2025-12-04T13:58:34.9990320Z Post job cleanup. 2025-12-04T13:58:35.0025379Z Post job cleanup. 2025-12-04T13:58:35.0946850Z [command]/usr/bin/git version 2025-12-04T13:58:35.1005120Z git version 2.50.1 2025-12-04T13:58:35.1038503Z Copying '/home/ec2-user/.gitconfig' to '/home/ec2-user/actions-runner/_work/_temp/93a0a531-6784-4033-826a-b4af6eba89e6/.gitconfig' 2025-12-04T13:58:35.1048292Z Temporarily overriding HOME='/home/ec2-user/actions-runner/_work/_temp/93a0a531-6784-4033-826a-b4af6eba89e6' before making global git config changes 2025-12-04T13:58:35.1049277Z Adding repository directory to the temporary git global config as a safe directory 2025-12-04T13:58:35.1053039Z [command]/usr/bin/git config --global --add safe.directory /home/ec2-user/actions-runner/_work/pytorch/pytorch 2025-12-04T13:58:35.1094500Z [command]/usr/bin/git config --local --name-only --get-regexp core\.sshCommand 2025-12-04T13:58:35.1132953Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'core\.sshCommand' && git config --local --unset-all 'core.sshCommand' || :" 2025-12-04T13:58:35.1494687Z Entering 'android/libs/fbjni' 2025-12-04T13:58:35.1563400Z Entering 'third_party/FP16' 2025-12-04T13:58:35.1632503Z Entering 'third_party/FXdiv' 2025-12-04T13:58:35.1702122Z Entering 'third_party/NNPACK' 2025-12-04T13:58:35.1771809Z Entering 'third_party/NVTX' 2025-12-04T13:58:35.1843736Z Entering 'third_party/VulkanMemoryAllocator' 2025-12-04T13:58:35.1911246Z Entering 'third_party/XNNPACK' 2025-12-04T13:58:35.1995037Z Entering 'third_party/aiter' 2025-12-04T13:58:35.2070440Z Entering 'third_party/aiter/3rdparty/composable_kernel' 2025-12-04T13:58:35.2150472Z Entering 'third_party/benchmark' 2025-12-04T13:58:35.2220423Z Entering 'third_party/composable_kernel' 2025-12-04T13:58:35.2298683Z Entering 'third_party/cpp-httplib' 2025-12-04T13:58:35.2376162Z Entering 'third_party/cpuinfo' 2025-12-04T13:58:35.2451140Z Entering 'third_party/cudnn_frontend' 2025-12-04T13:58:35.2521524Z Entering 'third_party/cutlass' 2025-12-04T13:58:35.2605169Z Entering 'third_party/fbgemm' 2025-12-04T13:58:35.2679342Z Entering 'third_party/fbgemm/external/asmjit' 2025-12-04T13:58:35.2750129Z Entering 'third_party/fbgemm/external/composable_kernel' 2025-12-04T13:58:35.2826128Z Entering 'third_party/fbgemm/external/cpuinfo' 2025-12-04T13:58:35.2893341Z Entering 'third_party/fbgemm/external/cutlass' 2025-12-04T13:58:35.2968594Z Entering 'third_party/fbgemm/external/googletest' 2025-12-04T13:58:35.3039831Z Entering 'third_party/fbgemm/external/hipify_torch' 2025-12-04T13:58:35.3109411Z Entering 'third_party/fbgemm/external/json' 2025-12-04T13:58:35.3184208Z Entering 'third_party/flash-attention' 2025-12-04T13:58:35.3252709Z Entering 'third_party/flash-attention/csrc/composable_kernel' 2025-12-04T13:58:35.3326270Z Entering 'third_party/flash-attention/csrc/cutlass' 2025-12-04T13:58:35.3403792Z Entering 'third_party/flatbuffers' 2025-12-04T13:58:35.3475125Z Entering 'third_party/fmt' 2025-12-04T13:58:35.3543034Z Entering 'third_party/gemmlowp/gemmlowp' 2025-12-04T13:58:35.3612059Z Entering 'third_party/gloo' 2025-12-04T13:58:35.3682611Z Entering 'third_party/googletest' 2025-12-04T13:58:35.3751537Z Entering 'third_party/ideep' 2025-12-04T13:58:35.3819461Z Entering 'third_party/ideep/mkl-dnn' 2025-12-04T13:58:35.3897666Z Entering 'third_party/ittapi' 2025-12-04T13:58:35.3971252Z Entering 'third_party/kineto' 2025-12-04T13:58:35.4039516Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2025-12-04T13:58:35.4108836Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2025-12-04T13:58:35.4185606Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2025-12-04T13:58:35.4252900Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2025-12-04T13:58:35.4322023Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2025-12-04T13:58:35.4389241Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2025-12-04T13:58:35.4463941Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2025-12-04T13:58:35.4531593Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2025-12-04T13:58:35.4601650Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2025-12-04T13:58:35.4671617Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2025-12-04T13:58:35.4742385Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/prometheus-cpp' 2025-12-04T13:58:35.4810095Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/prometheus-cpp/3rdparty/civetweb' 2025-12-04T13:58:35.4882669Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/prometheus-cpp/3rdparty/googletest' 2025-12-04T13:58:35.4958538Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2025-12-04T13:58:35.5026568Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2025-12-04T13:58:35.5096173Z Entering 'third_party/kleidiai' 2025-12-04T13:58:35.5169065Z Entering 'third_party/mimalloc' 2025-12-04T13:58:35.5246090Z Entering 'third_party/nlohmann' 2025-12-04T13:58:35.5314044Z Entering 'third_party/onnx' 2025-12-04T13:58:35.5397486Z Entering 'third_party/onnx/third_party/pybind11' 2025-12-04T13:58:35.5471758Z Entering 'third_party/opentelemetry-cpp' 2025-12-04T13:58:35.5543428Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2025-12-04T13:58:35.5611550Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2025-12-04T13:58:35.5680576Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2025-12-04T13:58:35.5749723Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2025-12-04T13:58:35.5821896Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2025-12-04T13:58:35.5890970Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2025-12-04T13:58:35.5960576Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2025-12-04T13:58:35.6031674Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2025-12-04T13:58:35.6102158Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2025-12-04T13:58:35.6174789Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2025-12-04T13:58:35.6262199Z Entering 'third_party/pocketfft' 2025-12-04T13:58:35.6341508Z Entering 'third_party/protobuf' 2025-12-04T13:58:35.6412572Z Entering 'third_party/protobuf/third_party/benchmark' 2025-12-04T13:58:35.6480560Z Entering 'third_party/protobuf/third_party/googletest' 2025-12-04T13:58:35.6553233Z Entering 'third_party/psimd' 2025-12-04T13:58:35.6622464Z Entering 'third_party/pthreadpool' 2025-12-04T13:58:35.6693297Z Entering 'third_party/pybind11' 2025-12-04T13:58:35.6761945Z Entering 'third_party/python-peachpy' 2025-12-04T13:58:35.6831906Z Entering 'third_party/sleef' 2025-12-04T13:58:35.6901143Z Entering 'third_party/tensorpipe' 2025-12-04T13:58:35.6970300Z Entering 'third_party/tensorpipe/third_party/googletest' 2025-12-04T13:58:35.7040656Z Entering 'third_party/tensorpipe/third_party/libnop' 2025-12-04T13:58:35.7110989Z Entering 'third_party/tensorpipe/third_party/libuv' 2025-12-04T13:58:35.7181956Z Entering 'third_party/tensorpipe/third_party/pybind11' 2025-12-04T13:58:35.7250367Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2025-12-04T13:58:35.7347519Z [command]/usr/bin/git config --local --name-only --get-regexp http\.https\:\/\/github\.com\/\.extraheader 2025-12-04T13:58:35.7371365Z http.https://github.com/.extraheader 2025-12-04T13:58:35.7380932Z [command]/usr/bin/git config --local --unset-all http.https://github.com/.extraheader 2025-12-04T13:58:35.7413641Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'http\.https\:\/\/github\.com\/\.extraheader' && git config --local --unset-all 'http.https://github.com/.extraheader' || :" 2025-12-04T13:58:35.7768648Z Entering 'android/libs/fbjni' 2025-12-04T13:58:35.7815073Z http.https://github.com/.extraheader 2025-12-04T13:58:35.7864153Z Entering 'third_party/FP16' 2025-12-04T13:58:35.7909261Z http.https://github.com/.extraheader 2025-12-04T13:58:35.7952373Z Entering 'third_party/FXdiv' 2025-12-04T13:58:35.8001170Z http.https://github.com/.extraheader 2025-12-04T13:58:35.8044235Z Entering 'third_party/NNPACK' 2025-12-04T13:58:35.8091091Z http.https://github.com/.extraheader 2025-12-04T13:58:35.8133793Z Entering 'third_party/NVTX' 2025-12-04T13:58:35.8180034Z http.https://github.com/.extraheader 2025-12-04T13:58:35.8223500Z Entering 'third_party/VulkanMemoryAllocator' 2025-12-04T13:58:35.8270593Z http.https://github.com/.extraheader 2025-12-04T13:58:35.8313507Z Entering 'third_party/XNNPACK' 2025-12-04T13:58:35.8358639Z http.https://github.com/.extraheader 2025-12-04T13:58:35.8414970Z Entering 'third_party/aiter' 2025-12-04T13:58:35.8459190Z http.https://github.com/.extraheader 2025-12-04T13:58:35.8502909Z Entering 'third_party/aiter/3rdparty/composable_kernel' 2025-12-04T13:58:35.8546957Z http.https://github.com/.extraheader 2025-12-04T13:58:35.8600022Z Entering 'third_party/benchmark' 2025-12-04T13:58:35.8646730Z http.https://github.com/.extraheader 2025-12-04T13:58:35.8691449Z Entering 'third_party/composable_kernel' 2025-12-04T13:58:35.8737907Z http.https://github.com/.extraheader 2025-12-04T13:58:35.8789684Z Entering 'third_party/cpp-httplib' 2025-12-04T13:58:35.8835833Z http.https://github.com/.extraheader 2025-12-04T13:58:35.8881007Z Entering 'third_party/cpuinfo' 2025-12-04T13:58:35.8926969Z http.https://github.com/.extraheader 2025-12-04T13:58:35.8971318Z Entering 'third_party/cudnn_frontend' 2025-12-04T13:58:35.9017742Z http.https://github.com/.extraheader 2025-12-04T13:58:35.9062106Z Entering 'third_party/cutlass' 2025-12-04T13:58:35.9108053Z http.https://github.com/.extraheader 2025-12-04T13:58:35.9159921Z Entering 'third_party/fbgemm' 2025-12-04T13:58:35.9206761Z http.https://github.com/.extraheader 2025-12-04T13:58:35.9251889Z Entering 'third_party/fbgemm/external/asmjit' 2025-12-04T13:58:35.9298590Z http.https://github.com/.extraheader 2025-12-04T13:58:35.9341882Z Entering 'third_party/fbgemm/external/composable_kernel' 2025-12-04T13:58:35.9387737Z http.https://github.com/.extraheader 2025-12-04T13:58:35.9439282Z Entering 'third_party/fbgemm/external/cpuinfo' 2025-12-04T13:58:35.9489751Z http.https://github.com/.extraheader 2025-12-04T13:58:35.9532753Z Entering 'third_party/fbgemm/external/cutlass' 2025-12-04T13:58:35.9577752Z http.https://github.com/.extraheader 2025-12-04T13:58:35.9631173Z Entering 'third_party/fbgemm/external/googletest' 2025-12-04T13:58:35.9676853Z http.https://github.com/.extraheader 2025-12-04T13:58:35.9721087Z Entering 'third_party/fbgemm/external/hipify_torch' 2025-12-04T13:58:35.9767447Z http.https://github.com/.extraheader 2025-12-04T13:58:35.9811062Z Entering 'third_party/fbgemm/external/json' 2025-12-04T13:58:35.9855582Z http.https://github.com/.extraheader 2025-12-04T13:58:35.9904136Z Entering 'third_party/flash-attention' 2025-12-04T13:58:35.9949557Z http.https://github.com/.extraheader 2025-12-04T13:58:35.9992909Z Entering 'third_party/flash-attention/csrc/composable_kernel' 2025-12-04T13:58:36.0038513Z http.https://github.com/.extraheader 2025-12-04T13:58:36.0088745Z Entering 'third_party/flash-attention/csrc/cutlass' 2025-12-04T13:58:36.0135378Z http.https://github.com/.extraheader 2025-12-04T13:58:36.0189742Z Entering 'third_party/flatbuffers' 2025-12-04T13:58:36.0236328Z http.https://github.com/.extraheader 2025-12-04T13:58:36.0283432Z Entering 'third_party/fmt' 2025-12-04T13:58:36.0328846Z http.https://github.com/.extraheader 2025-12-04T13:58:36.0372180Z Entering 'third_party/gemmlowp/gemmlowp' 2025-12-04T13:58:36.0418546Z http.https://github.com/.extraheader 2025-12-04T13:58:36.0463632Z Entering 'third_party/gloo' 2025-12-04T13:58:36.0508928Z http.https://github.com/.extraheader 2025-12-04T13:58:36.0552800Z Entering 'third_party/googletest' 2025-12-04T13:58:36.0598760Z http.https://github.com/.extraheader 2025-12-04T13:58:36.0642074Z Entering 'third_party/ideep' 2025-12-04T13:58:36.0687850Z http.https://github.com/.extraheader 2025-12-04T13:58:36.0729367Z Entering 'third_party/ideep/mkl-dnn' 2025-12-04T13:58:36.0773451Z http.https://github.com/.extraheader 2025-12-04T13:58:36.0825744Z Entering 'third_party/ittapi' 2025-12-04T13:58:36.0870569Z http.https://github.com/.extraheader 2025-12-04T13:58:36.0913685Z Entering 'third_party/kineto' 2025-12-04T13:58:36.0958715Z http.https://github.com/.extraheader 2025-12-04T13:58:36.1001257Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2025-12-04T13:58:36.1045767Z http.https://github.com/.extraheader 2025-12-04T13:58:36.1090189Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2025-12-04T13:58:36.1136792Z http.https://github.com/.extraheader 2025-12-04T13:58:36.1182190Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2025-12-04T13:58:36.1228101Z http.https://github.com/.extraheader 2025-12-04T13:58:36.1272491Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2025-12-04T13:58:36.1317410Z http.https://github.com/.extraheader 2025-12-04T13:58:36.1361904Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2025-12-04T13:58:36.1408414Z http.https://github.com/.extraheader 2025-12-04T13:58:36.1450835Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2025-12-04T13:58:36.1498102Z http.https://github.com/.extraheader 2025-12-04T13:58:36.1545961Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2025-12-04T13:58:36.1596149Z http.https://github.com/.extraheader 2025-12-04T13:58:36.1641238Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2025-12-04T13:58:36.1688445Z http.https://github.com/.extraheader 2025-12-04T13:58:36.1732755Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2025-12-04T13:58:36.1779308Z http.https://github.com/.extraheader 2025-12-04T13:58:36.1822723Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2025-12-04T13:58:36.1868471Z http.https://github.com/.extraheader 2025-12-04T13:58:36.1913527Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/prometheus-cpp' 2025-12-04T13:58:36.1960167Z http.https://github.com/.extraheader 2025-12-04T13:58:36.2003023Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/prometheus-cpp/3rdparty/civetweb' 2025-12-04T13:58:36.2048823Z http.https://github.com/.extraheader 2025-12-04T13:58:36.2094829Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/prometheus-cpp/3rdparty/googletest' 2025-12-04T13:58:36.2139423Z http.https://github.com/.extraheader 2025-12-04T13:58:36.2190896Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2025-12-04T13:58:36.2236252Z http.https://github.com/.extraheader 2025-12-04T13:58:36.2278929Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2025-12-04T13:58:36.2323574Z http.https://github.com/.extraheader 2025-12-04T13:58:36.2371491Z Entering 'third_party/kleidiai' 2025-12-04T13:58:36.2416810Z http.https://github.com/.extraheader 2025-12-04T13:58:36.2461128Z Entering 'third_party/mimalloc' 2025-12-04T13:58:36.2506093Z http.https://github.com/.extraheader 2025-12-04T13:58:36.2552296Z Entering 'third_party/nlohmann' 2025-12-04T13:58:36.2597446Z http.https://github.com/.extraheader 2025-12-04T13:58:36.2642235Z Entering 'third_party/onnx' 2025-12-04T13:58:36.2688657Z http.https://github.com/.extraheader 2025-12-04T13:58:36.2746191Z Entering 'third_party/onnx/third_party/pybind11' 2025-12-04T13:58:36.2798342Z http.https://github.com/.extraheader 2025-12-04T13:58:36.2843951Z Entering 'third_party/opentelemetry-cpp' 2025-12-04T13:58:36.2889208Z http.https://github.com/.extraheader 2025-12-04T13:58:36.2931609Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2025-12-04T13:58:36.2978176Z http.https://github.com/.extraheader 2025-12-04T13:58:36.3021569Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2025-12-04T13:58:36.3068527Z http.https://github.com/.extraheader 2025-12-04T13:58:36.3112964Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2025-12-04T13:58:36.3157761Z http.https://github.com/.extraheader 2025-12-04T13:58:36.3201091Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2025-12-04T13:58:36.3246433Z http.https://github.com/.extraheader 2025-12-04T13:58:36.3291913Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2025-12-04T13:58:36.3337326Z http.https://github.com/.extraheader 2025-12-04T13:58:36.3381288Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2025-12-04T13:58:36.3426540Z http.https://github.com/.extraheader 2025-12-04T13:58:36.3469746Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2025-12-04T13:58:36.3514644Z http.https://github.com/.extraheader 2025-12-04T13:58:36.3557216Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2025-12-04T13:58:36.3604720Z http.https://github.com/.extraheader 2025-12-04T13:58:36.3651189Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2025-12-04T13:58:36.3697246Z http.https://github.com/.extraheader 2025-12-04T13:58:36.3744340Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2025-12-04T13:58:36.3790001Z http.https://github.com/.extraheader 2025-12-04T13:58:36.3851906Z Entering 'third_party/pocketfft' 2025-12-04T13:58:36.3898838Z http.https://github.com/.extraheader 2025-12-04T13:58:36.3941746Z Entering 'third_party/protobuf' 2025-12-04T13:58:36.3988076Z http.https://github.com/.extraheader 2025-12-04T13:58:36.4033903Z Entering 'third_party/protobuf/third_party/benchmark' 2025-12-04T13:58:36.4079694Z http.https://github.com/.extraheader 2025-12-04T13:58:36.4122695Z Entering 'third_party/protobuf/third_party/googletest' 2025-12-04T13:58:36.4168462Z http.https://github.com/.extraheader 2025-12-04T13:58:36.4215864Z Entering 'third_party/psimd' 2025-12-04T13:58:36.4265440Z http.https://github.com/.extraheader 2025-12-04T13:58:36.4310825Z Entering 'third_party/pthreadpool' 2025-12-04T13:58:36.4355943Z http.https://github.com/.extraheader 2025-12-04T13:58:36.4400716Z Entering 'third_party/pybind11' 2025-12-04T13:58:36.4447385Z http.https://github.com/.extraheader 2025-12-04T13:58:36.4490790Z Entering 'third_party/python-peachpy' 2025-12-04T13:58:36.4537286Z http.https://github.com/.extraheader 2025-12-04T13:58:36.4582505Z Entering 'third_party/sleef' 2025-12-04T13:58:36.4627970Z http.https://github.com/.extraheader 2025-12-04T13:58:36.4671198Z Entering 'third_party/tensorpipe' 2025-12-04T13:58:36.4718161Z http.https://github.com/.extraheader 2025-12-04T13:58:36.4760642Z Entering 'third_party/tensorpipe/third_party/googletest' 2025-12-04T13:58:36.4806698Z http.https://github.com/.extraheader 2025-12-04T13:58:36.4851356Z Entering 'third_party/tensorpipe/third_party/libnop' 2025-12-04T13:58:36.4897533Z http.https://github.com/.extraheader 2025-12-04T13:58:36.4940883Z Entering 'third_party/tensorpipe/third_party/libuv' 2025-12-04T13:58:36.4987467Z http.https://github.com/.extraheader 2025-12-04T13:58:36.5030990Z Entering 'third_party/tensorpipe/third_party/pybind11' 2025-12-04T13:58:36.5076830Z http.https://github.com/.extraheader 2025-12-04T13:58:36.5119457Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2025-12-04T13:58:36.5166377Z http.https://github.com/.extraheader 2025-12-04T13:58:36.5240985Z [command]/usr/bin/git config --local --name-only --get-regexp ^includeIf\.gitdir: 2025-12-04T13:58:36.5272291Z [command]/usr/bin/git submodule foreach --recursive git config --local --show-origin --name-only --get-regexp remote.origin.url 2025-12-04T13:58:36.5636158Z Entering 'android/libs/fbjni' 2025-12-04T13:58:36.5665615Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/android/libs/fbjni/config remote.origin.url 2025-12-04T13:58:36.5687243Z Entering 'third_party/FP16' 2025-12-04T13:58:36.5717226Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/FP16/config remote.origin.url 2025-12-04T13:58:36.5738926Z Entering 'third_party/FXdiv' 2025-12-04T13:58:36.5775567Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/FXdiv/config remote.origin.url 2025-12-04T13:58:36.5797859Z Entering 'third_party/NNPACK' 2025-12-04T13:58:36.5832727Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK/config remote.origin.url 2025-12-04T13:58:36.5854159Z Entering 'third_party/NVTX' 2025-12-04T13:58:36.5884466Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NVTX/config remote.origin.url 2025-12-04T13:58:36.5909341Z Entering 'third_party/VulkanMemoryAllocator' 2025-12-04T13:58:36.5940510Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/VulkanMemoryAllocator/config remote.origin.url 2025-12-04T13:58:36.5962156Z Entering 'third_party/XNNPACK' 2025-12-04T13:58:36.5992625Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/XNNPACK/config remote.origin.url 2025-12-04T13:58:36.6026987Z Entering 'third_party/aiter' 2025-12-04T13:58:36.6058288Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/aiter/config remote.origin.url 2025-12-04T13:58:36.6081210Z Entering 'third_party/aiter/3rdparty/composable_kernel' 2025-12-04T13:58:36.6111254Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/aiter/modules/3rdparty/composable_kernel/config remote.origin.url 2025-12-04T13:58:36.6141591Z Entering 'third_party/benchmark' 2025-12-04T13:58:36.6177634Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/benchmark/config remote.origin.url 2025-12-04T13:58:36.6200439Z Entering 'third_party/composable_kernel' 2025-12-04T13:58:36.6231267Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/composable_kernel/config remote.origin.url 2025-12-04T13:58:36.6260809Z Entering 'third_party/cpp-httplib' 2025-12-04T13:58:36.6292413Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cpp-httplib/config remote.origin.url 2025-12-04T13:58:36.6313281Z Entering 'third_party/cpuinfo' 2025-12-04T13:58:36.6344122Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cpuinfo/config remote.origin.url 2025-12-04T13:58:36.6365564Z Entering 'third_party/cudnn_frontend' 2025-12-04T13:58:36.6395954Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cudnn_frontend/config remote.origin.url 2025-12-04T13:58:36.6418420Z Entering 'third_party/cutlass' 2025-12-04T13:58:36.6449519Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cutlass/config remote.origin.url 2025-12-04T13:58:36.6480292Z Entering 'third_party/fbgemm' 2025-12-04T13:58:36.6511315Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/config remote.origin.url 2025-12-04T13:58:36.6533313Z Entering 'third_party/fbgemm/external/asmjit' 2025-12-04T13:58:36.6563019Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/external/asmjit/config remote.origin.url 2025-12-04T13:58:36.6584117Z Entering 'third_party/fbgemm/external/composable_kernel' 2025-12-04T13:58:36.6613711Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/external/composable_kernel/config remote.origin.url 2025-12-04T13:58:36.6642534Z Entering 'third_party/fbgemm/external/cpuinfo' 2025-12-04T13:58:36.6672692Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/external/cpuinfo/config remote.origin.url 2025-12-04T13:58:36.6693789Z Entering 'third_party/fbgemm/external/cutlass' 2025-12-04T13:58:36.6723110Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/external/cutlass/config remote.origin.url 2025-12-04T13:58:36.6751986Z Entering 'third_party/fbgemm/external/googletest' 2025-12-04T13:58:36.6781817Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/external/googletest/config remote.origin.url 2025-12-04T13:58:36.6802454Z Entering 'third_party/fbgemm/external/hipify_torch' 2025-12-04T13:58:36.6832316Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/external/hipify_torch/config remote.origin.url 2025-12-04T13:58:36.6852966Z Entering 'third_party/fbgemm/external/json' 2025-12-04T13:58:36.6883939Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/external/json/config remote.origin.url 2025-12-04T13:58:36.6908227Z Entering 'third_party/flash-attention' 2025-12-04T13:58:36.6939725Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/flash-attention/config remote.origin.url 2025-12-04T13:58:36.6961074Z Entering 'third_party/flash-attention/csrc/composable_kernel' 2025-12-04T13:58:36.6992079Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/flash-attention/modules/csrc/composable_kernel/config remote.origin.url 2025-12-04T13:58:36.7018281Z Entering 'third_party/flash-attention/csrc/cutlass' 2025-12-04T13:58:36.7051756Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/flash-attention/modules/csrc/cutlass/config remote.origin.url 2025-12-04T13:58:36.7083086Z Entering 'third_party/flatbuffers' 2025-12-04T13:58:36.7113331Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/flatbuffers/config remote.origin.url 2025-12-04T13:58:36.7138972Z Entering 'third_party/fmt' 2025-12-04T13:58:36.7170504Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fmt/config remote.origin.url 2025-12-04T13:58:36.7191758Z Entering 'third_party/gemmlowp/gemmlowp' 2025-12-04T13:58:36.7222118Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/gemmlowp/gemmlowp/config remote.origin.url 2025-12-04T13:58:36.7243507Z Entering 'third_party/gloo' 2025-12-04T13:58:36.7273748Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/gloo/config remote.origin.url 2025-12-04T13:58:36.7295412Z Entering 'third_party/googletest' 2025-12-04T13:58:36.7331609Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/googletest/config remote.origin.url 2025-12-04T13:58:36.7353327Z Entering 'third_party/ideep' 2025-12-04T13:58:36.7384511Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/ideep/config remote.origin.url 2025-12-04T13:58:36.7403325Z Entering 'third_party/ideep/mkl-dnn' 2025-12-04T13:58:36.7432842Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/ideep/modules/mkl-dnn/config remote.origin.url 2025-12-04T13:58:36.7461873Z Entering 'third_party/ittapi' 2025-12-04T13:58:36.7494301Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/ittapi/config remote.origin.url 2025-12-04T13:58:36.7520927Z Entering 'third_party/kineto' 2025-12-04T13:58:36.7551307Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/config remote.origin.url 2025-12-04T13:58:36.7571134Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2025-12-04T13:58:36.7602458Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/config remote.origin.url 2025-12-04T13:58:36.7622652Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2025-12-04T13:58:36.7652769Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/DCGM/config remote.origin.url 2025-12-04T13:58:36.7674247Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2025-12-04T13:58:36.7704871Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/cpr/config remote.origin.url 2025-12-04T13:58:36.7727336Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2025-12-04T13:58:36.7770464Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/fmt/config remote.origin.url 2025-12-04T13:58:36.7781963Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2025-12-04T13:58:36.7812221Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/gflags/config remote.origin.url 2025-12-04T13:58:36.7831548Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2025-12-04T13:58:36.7862073Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/gflags/modules/doc/config remote.origin.url 2025-12-04T13:58:36.7886071Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2025-12-04T13:58:36.7917900Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/glog/config remote.origin.url 2025-12-04T13:58:36.7939930Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2025-12-04T13:58:36.7971096Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/googletest/config remote.origin.url 2025-12-04T13:58:36.7992904Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2025-12-04T13:58:36.8023227Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/json/config remote.origin.url 2025-12-04T13:58:36.8045155Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2025-12-04T13:58:36.8074671Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/pfs/config remote.origin.url 2025-12-04T13:58:36.8095376Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/prometheus-cpp' 2025-12-04T13:58:36.8124964Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/prometheus-cpp/config remote.origin.url 2025-12-04T13:58:36.8147976Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/prometheus-cpp/3rdparty/civetweb' 2025-12-04T13:58:36.8178401Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/prometheus-cpp/modules/civetweb/config remote.origin.url 2025-12-04T13:58:36.8203276Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/prometheus-cpp/3rdparty/googletest' 2025-12-04T13:58:36.8234154Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/prometheus-cpp/modules/googletest/config remote.origin.url 2025-12-04T13:58:36.8260676Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2025-12-04T13:58:36.8291736Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/fmt/config remote.origin.url 2025-12-04T13:58:36.8312703Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2025-12-04T13:58:36.8342687Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/googletest/config remote.origin.url 2025-12-04T13:58:36.8366646Z Entering 'third_party/kleidiai' 2025-12-04T13:58:36.8399040Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kleidiai/config remote.origin.url 2025-12-04T13:58:36.8421308Z Entering 'third_party/mimalloc' 2025-12-04T13:58:36.8451982Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/mimalloc/config remote.origin.url 2025-12-04T13:58:36.8473349Z Entering 'third_party/nlohmann' 2025-12-04T13:58:36.8504331Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/nlohmann/config remote.origin.url 2025-12-04T13:58:36.8531745Z Entering 'third_party/onnx' 2025-12-04T13:58:36.8562476Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx/config remote.origin.url 2025-12-04T13:58:36.8598799Z Entering 'third_party/onnx/third_party/pybind11' 2025-12-04T13:58:36.8629970Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx/modules/third_party/pybind11/config remote.origin.url 2025-12-04T13:58:36.8655236Z Entering 'third_party/opentelemetry-cpp' 2025-12-04T13:58:36.8688514Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/config remote.origin.url 2025-12-04T13:58:36.8709274Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2025-12-04T13:58:36.8739332Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/benchmark/config remote.origin.url 2025-12-04T13:58:36.8760782Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2025-12-04T13:58:36.8791582Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/googletest/config remote.origin.url 2025-12-04T13:58:36.8812811Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2025-12-04T13:58:36.8842487Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/ms-gsl/config remote.origin.url 2025-12-04T13:58:36.8862704Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2025-12-04T13:58:36.8893140Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/nlohmann-json/config remote.origin.url 2025-12-04T13:58:36.8915082Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2025-12-04T13:58:36.8944578Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/opentelemetry-proto/config remote.origin.url 2025-12-04T13:58:36.8965051Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2025-12-04T13:58:36.8994009Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/opentracing-cpp/config remote.origin.url 2025-12-04T13:58:36.9015455Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2025-12-04T13:58:36.9046837Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/prometheus-cpp/config remote.origin.url 2025-12-04T13:58:36.9070790Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2025-12-04T13:58:36.9100211Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/prometheus-cpp/modules/civetweb/config remote.origin.url 2025-12-04T13:58:36.9123163Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2025-12-04T13:58:36.9153158Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/prometheus-cpp/modules/googletest/config remote.origin.url 2025-12-04T13:58:36.9176232Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2025-12-04T13:58:36.9207417Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/tools/vcpkg/config remote.origin.url 2025-12-04T13:58:36.9248220Z Entering 'third_party/pocketfft' 2025-12-04T13:58:36.9280834Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/pocketfft/config remote.origin.url 2025-12-04T13:58:36.9301915Z Entering 'third_party/protobuf' 2025-12-04T13:58:36.9332482Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/protobuf/config remote.origin.url 2025-12-04T13:58:36.9355263Z Entering 'third_party/protobuf/third_party/benchmark' 2025-12-04T13:58:36.9385227Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/protobuf/modules/third_party/benchmark/config remote.origin.url 2025-12-04T13:58:36.9405271Z Entering 'third_party/protobuf/third_party/googletest' 2025-12-04T13:58:36.9434412Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/protobuf/modules/third_party/googletest/config remote.origin.url 2025-12-04T13:58:36.9462432Z Entering 'third_party/psimd' 2025-12-04T13:58:36.9494098Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/psimd/config remote.origin.url 2025-12-04T13:58:36.9519870Z Entering 'third_party/pthreadpool' 2025-12-04T13:58:36.9550704Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/pthreadpool/config remote.origin.url 2025-12-04T13:58:36.9571877Z Entering 'third_party/pybind11' 2025-12-04T13:58:36.9602782Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/pybind11/config remote.origin.url 2025-12-04T13:58:36.9624056Z Entering 'third_party/python-peachpy' 2025-12-04T13:58:36.9654180Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/python-peachpy/config remote.origin.url 2025-12-04T13:58:36.9680974Z Entering 'third_party/sleef' 2025-12-04T13:58:36.9711919Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/sleef/config remote.origin.url 2025-12-04T13:58:36.9732759Z Entering 'third_party/tensorpipe' 2025-12-04T13:58:36.9762357Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/config remote.origin.url 2025-12-04T13:58:36.9783246Z Entering 'third_party/tensorpipe/third_party/googletest' 2025-12-04T13:58:36.9812974Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/googletest/config remote.origin.url 2025-12-04T13:58:36.9834026Z Entering 'third_party/tensorpipe/third_party/libnop' 2025-12-04T13:58:36.9863915Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/libnop/config remote.origin.url 2025-12-04T13:58:36.9884660Z Entering 'third_party/tensorpipe/third_party/libuv' 2025-12-04T13:58:36.9914169Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/libuv/config remote.origin.url 2025-12-04T13:58:36.9942529Z Entering 'third_party/tensorpipe/third_party/pybind11' 2025-12-04T13:58:36.9972396Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/pybind11/config remote.origin.url 2025-12-04T13:58:36.9991585Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2025-12-04T13:58:37.0021560Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/pybind11/modules/tools/clang/config remote.origin.url 2025-12-04T13:58:37.0072873Z [command]/usr/bin/git config --file /home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/android/libs/fbjni/config --name-only --get-regexp ^includeIf\.gitdir: 2025-12-04T13:58:37.0106555Z [command]/usr/bin/git config --file /home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/FP16/config --name-only --get-regexp ^includeIf\.gitdir: 2025-12-04T13:58:37.0134552Z [command]/usr/bin/git config --file /home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/FXdiv/config --name-only --get-regexp ^includeIf\.gitdir: 2025-12-04T13:58:37.0163068Z [command]/usr/bin/git config --file /home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK/config --name-only --get-regexp ^includeIf\.gitdir: 2025-12-04T13:58:37.0190710Z [command]/usr/bin/git config --file /home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NVTX/config --name-only --get-regexp ^includeIf\.gitdir: 2025-12-04T13:58:37.0218771Z [command]/usr/bin/git config --file /home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/VulkanMemoryAllocator/config --name-only --get-regexp ^includeIf\.gitdir: 2025-12-04T13:58:37.0248138Z [command]/usr/bin/git config --file /home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/XNNPACK/config --name-only --get-regexp ^includeIf\.gitdir: 2025-12-04T13:58:37.0276386Z [command]/usr/bin/git config --file /home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/aiter/config --name-only --get-regexp ^includeIf\.gitdir: 2025-12-04T13:58:37.0306001Z [command]/usr/bin/git config --file /home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/aiter/modules/3rdparty/composable_kernel/config --name-only --get-regexp ^includeIf\.gitdir: 2025-12-04T13:58:37.0332950Z [command]/usr/bin/git config --file /home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/benchmark/config --name-only --get-regexp ^includeIf\.gitdir: 2025-12-04T13:58:37.0363423Z [command]/usr/bin/git config --file /home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/composable_kernel/config --name-only --get-regexp ^includeIf\.gitdir: 2025-12-04T13:58:37.0390755Z [command]/usr/bin/git config --file /home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cpp-httplib/config --name-only --get-regexp ^includeIf\.gitdir: 2025-12-04T13:58:37.0417979Z [command]/usr/bin/git config --file /home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cpuinfo/config --name-only --get-regexp ^includeIf\.gitdir: 2025-12-04T13:58:37.0445300Z [command]/usr/bin/git config --file /home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cudnn_frontend/config --name-only --get-regexp ^includeIf\.gitdir: 2025-12-04T13:58:37.0473179Z [command]/usr/bin/git config --file /home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cutlass/config --name-only --get-regexp ^includeIf\.gitdir: 2025-12-04T13:58:37.0500591Z [command]/usr/bin/git config --file /home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/config --name-only --get-regexp ^includeIf\.gitdir: 2025-12-04T13:58:37.0527646Z [command]/usr/bin/git config --file /home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/external/asmjit/config --name-only --get-regexp ^includeIf\.gitdir: 2025-12-04T13:58:37.0559159Z [command]/usr/bin/git config --file /home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/external/composable_kernel/config --name-only --get-regexp ^includeIf\.gitdir: 2025-12-04T13:58:37.0587122Z [command]/usr/bin/git config --file /home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/external/cpuinfo/config --name-only --get-regexp ^includeIf\.gitdir: 2025-12-04T13:58:37.0613854Z [command]/usr/bin/git config --file /home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/external/cutlass/config --name-only --get-regexp ^includeIf\.gitdir: 2025-12-04T13:58:37.0642881Z [command]/usr/bin/git config --file /home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/external/googletest/config --name-only --get-regexp ^includeIf\.gitdir: 2025-12-04T13:58:37.0670781Z [command]/usr/bin/git config --file /home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/external/hipify_torch/config --name-only --get-regexp ^includeIf\.gitdir: 2025-12-04T13:58:37.0699189Z [command]/usr/bin/git config --file /home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/external/json/config --name-only --get-regexp ^includeIf\.gitdir: 2025-12-04T13:58:37.0726732Z [command]/usr/bin/git config --file /home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/flash-attention/config --name-only --get-regexp ^includeIf\.gitdir: 2025-12-04T13:58:37.0757042Z [command]/usr/bin/git config --file /home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/flash-attention/modules/csrc/composable_kernel/config --name-only --get-regexp ^includeIf\.gitdir: 2025-12-04T13:58:37.0784902Z [command]/usr/bin/git config --file /home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/flash-attention/modules/csrc/cutlass/config --name-only --get-regexp ^includeIf\.gitdir: 2025-12-04T13:58:37.0814023Z [command]/usr/bin/git config --file /home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/flatbuffers/config --name-only --get-regexp ^includeIf\.gitdir: 2025-12-04T13:58:37.0844467Z [command]/usr/bin/git config --file /home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fmt/config --name-only --get-regexp ^includeIf\.gitdir: 2025-12-04T13:58:37.0875562Z [command]/usr/bin/git config --file /home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/gemmlowp/gemmlowp/config --name-only --get-regexp ^includeIf\.gitdir: 2025-12-04T13:58:37.0903314Z [command]/usr/bin/git config --file /home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/gloo/config --name-only --get-regexp ^includeIf\.gitdir: 2025-12-04T13:58:37.0929356Z [command]/usr/bin/git config --file /home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/googletest/config --name-only --get-regexp ^includeIf\.gitdir: 2025-12-04T13:58:37.0956460Z [command]/usr/bin/git config --file /home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/ideep/config --name-only --get-regexp ^includeIf\.gitdir: 2025-12-04T13:58:37.0984882Z [command]/usr/bin/git config --file /home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/ideep/modules/mkl-dnn/config --name-only --get-regexp ^includeIf\.gitdir: 2025-12-04T13:58:37.1014257Z [command]/usr/bin/git config --file /home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/ittapi/config --name-only --get-regexp ^includeIf\.gitdir: 2025-12-04T13:58:37.1042722Z [command]/usr/bin/git config --file /home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/config --name-only --get-regexp ^includeIf\.gitdir: 2025-12-04T13:58:37.1069423Z [command]/usr/bin/git config --file /home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/config --name-only --get-regexp ^includeIf\.gitdir: 2025-12-04T13:58:37.1097640Z [command]/usr/bin/git config --file /home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/DCGM/config --name-only --get-regexp ^includeIf\.gitdir: 2025-12-04T13:58:37.1127021Z [command]/usr/bin/git config --file /home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/cpr/config --name-only --get-regexp ^includeIf\.gitdir: 2025-12-04T13:58:37.1154452Z [command]/usr/bin/git config --file /home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/fmt/config --name-only --get-regexp ^includeIf\.gitdir: 2025-12-04T13:58:37.1181128Z [command]/usr/bin/git config --file /home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/gflags/config --name-only --get-regexp ^includeIf\.gitdir: 2025-12-04T13:58:37.1207737Z [command]/usr/bin/git config --file /home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/gflags/modules/doc/config --name-only --get-regexp ^includeIf\.gitdir: 2025-12-04T13:58:37.1234362Z [command]/usr/bin/git config --file /home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/glog/config --name-only --get-regexp ^includeIf\.gitdir: 2025-12-04T13:58:37.1266799Z [command]/usr/bin/git config --file /home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/googletest/config --name-only --get-regexp ^includeIf\.gitdir: 2025-12-04T13:58:37.1294114Z [command]/usr/bin/git config --file /home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/json/config --name-only --get-regexp ^includeIf\.gitdir: 2025-12-04T13:58:37.1320032Z [command]/usr/bin/git config --file /home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/pfs/config --name-only --get-regexp ^includeIf\.gitdir: 2025-12-04T13:58:37.1346619Z [command]/usr/bin/git config --file /home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/prometheus-cpp/config --name-only --get-regexp ^includeIf\.gitdir: 2025-12-04T13:58:37.1374000Z [command]/usr/bin/git config --file /home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/prometheus-cpp/modules/civetweb/config --name-only --get-regexp ^includeIf\.gitdir: 2025-12-04T13:58:37.1399697Z [command]/usr/bin/git config --file /home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/prometheus-cpp/modules/googletest/config --name-only --get-regexp ^includeIf\.gitdir: 2025-12-04T13:58:37.1426211Z [command]/usr/bin/git config --file /home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/fmt/config --name-only --get-regexp ^includeIf\.gitdir: 2025-12-04T13:58:37.1452972Z [command]/usr/bin/git config --file /home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/googletest/config --name-only --get-regexp ^includeIf\.gitdir: 2025-12-04T13:58:37.1477522Z [command]/usr/bin/git config --file /home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kleidiai/config --name-only --get-regexp ^includeIf\.gitdir: 2025-12-04T13:58:37.1504280Z [command]/usr/bin/git config --file /home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/mimalloc/config --name-only --get-regexp ^includeIf\.gitdir: 2025-12-04T13:58:37.1531708Z [command]/usr/bin/git config --file /home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/nlohmann/config --name-only --get-regexp ^includeIf\.gitdir: 2025-12-04T13:58:37.1557149Z [command]/usr/bin/git config --file /home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx/config --name-only --get-regexp ^includeIf\.gitdir: 2025-12-04T13:58:37.1583838Z [command]/usr/bin/git config --file /home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx/modules/third_party/pybind11/config --name-only --get-regexp ^includeIf\.gitdir: 2025-12-04T13:58:37.1608488Z [command]/usr/bin/git config --file /home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/config --name-only --get-regexp ^includeIf\.gitdir: 2025-12-04T13:58:37.1635204Z [command]/usr/bin/git config --file /home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/benchmark/config --name-only --get-regexp ^includeIf\.gitdir: 2025-12-04T13:58:37.1662976Z [command]/usr/bin/git config --file /home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/googletest/config --name-only --get-regexp ^includeIf\.gitdir: 2025-12-04T13:58:37.1692479Z [command]/usr/bin/git config --file /home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/ms-gsl/config --name-only --get-regexp ^includeIf\.gitdir: 2025-12-04T13:58:37.1718440Z [command]/usr/bin/git config --file /home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/nlohmann-json/config --name-only --get-regexp ^includeIf\.gitdir: 2025-12-04T13:58:37.1744703Z [command]/usr/bin/git config --file /home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/opentelemetry-proto/config --name-only --get-regexp ^includeIf\.gitdir: 2025-12-04T13:58:37.1769882Z [command]/usr/bin/git config --file /home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/opentracing-cpp/config --name-only --get-regexp ^includeIf\.gitdir: 2025-12-04T13:58:37.1798639Z [command]/usr/bin/git config --file /home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/prometheus-cpp/config --name-only --get-regexp ^includeIf\.gitdir: 2025-12-04T13:58:37.1827543Z [command]/usr/bin/git config --file /home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/prometheus-cpp/modules/civetweb/config --name-only --get-regexp ^includeIf\.gitdir: 2025-12-04T13:58:37.1856053Z [command]/usr/bin/git config --file /home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/prometheus-cpp/modules/googletest/config --name-only --get-regexp ^includeIf\.gitdir: 2025-12-04T13:58:37.1890987Z [command]/usr/bin/git config --file /home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/tools/vcpkg/config --name-only --get-regexp ^includeIf\.gitdir: 2025-12-04T13:58:37.1919848Z [command]/usr/bin/git config --file /home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/pocketfft/config --name-only --get-regexp ^includeIf\.gitdir: 2025-12-04T13:58:37.1948792Z [command]/usr/bin/git config --file /home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/protobuf/config --name-only --get-regexp ^includeIf\.gitdir: 2025-12-04T13:58:37.1980338Z [command]/usr/bin/git config --file /home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/protobuf/modules/third_party/benchmark/config --name-only --get-regexp ^includeIf\.gitdir: 2025-12-04T13:58:37.2007992Z [command]/usr/bin/git config --file /home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/protobuf/modules/third_party/googletest/config --name-only --get-regexp ^includeIf\.gitdir: 2025-12-04T13:58:37.2036408Z [command]/usr/bin/git config --file /home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/psimd/config --name-only --get-regexp ^includeIf\.gitdir: 2025-12-04T13:58:37.2064884Z [command]/usr/bin/git config --file /home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/pthreadpool/config --name-only --get-regexp ^includeIf\.gitdir: 2025-12-04T13:58:37.2092773Z [command]/usr/bin/git config --file /home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/pybind11/config --name-only --get-regexp ^includeIf\.gitdir: 2025-12-04T13:58:37.2124558Z [command]/usr/bin/git config --file /home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/python-peachpy/config --name-only --get-regexp ^includeIf\.gitdir: 2025-12-04T13:58:37.2152600Z [command]/usr/bin/git config --file /home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/sleef/config --name-only --get-regexp ^includeIf\.gitdir: 2025-12-04T13:58:37.2181438Z [command]/usr/bin/git config --file /home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/config --name-only --get-regexp ^includeIf\.gitdir: 2025-12-04T13:58:37.2206659Z [command]/usr/bin/git config --file /home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/googletest/config --name-only --get-regexp ^includeIf\.gitdir: 2025-12-04T13:58:37.2234693Z [command]/usr/bin/git config --file /home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/libnop/config --name-only --get-regexp ^includeIf\.gitdir: 2025-12-04T13:58:37.2263022Z [command]/usr/bin/git config --file /home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/libuv/config --name-only --get-regexp ^includeIf\.gitdir: 2025-12-04T13:58:37.2290233Z [command]/usr/bin/git config --file /home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/pybind11/config --name-only --get-regexp ^includeIf\.gitdir: 2025-12-04T13:58:37.2315758Z [command]/usr/bin/git config --file /home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/pybind11/modules/tools/clang/config --name-only --get-regexp ^includeIf\.gitdir: 2025-12-04T13:58:37.2431355Z A job completed hook has been configured by the self-hosted runner administrator 2025-12-04T13:58:37.2445738Z ##[group]Run '/home/ec2-user/runner-scripts/after_job.sh' 2025-12-04T13:58:37.2452532Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-12-04T13:58:37.2452931Z ##[endgroup] 2025-12-04T13:58:37.2587026Z [!ALERT!] Swap in detected! [!ALERT!] 2025-12-04T13:58:47.0392817Z [!ALERT!] Swap out detected [!ALERT!] 2025-12-04T13:59:03.8706949Z Cleaning up orphan processes